Expert: Zogby “poll” needs “big ‘grains of salt’ “

I’ve received many emails regarding my earlier post, “How reliable is Zogby poll of military in Iraq?,” which raised the serious methodological reasons to suspect the validity of this “poll’s” results that most of our military in Iraq are negative toward our mission and want out of Iraq within the next year.

Even from those sympathetic toward the reported results, I’ve found no knowledgeable argument for accepting the Zogby’s reliability.

Today, Mystery Pollster -- who is liberal, a reputable and experienced evaluator of polls, and who had access to some of the “secret” methodology – reveals his judgments. They parallel mine.

First, to be clear, the Center for Peace and Global Studies is in effect a "partisan" sponsor in that, according to Zogby, they oppose the war in Iraq.

Second, while Zogby says his interviewers selected respondents randomly at various locations, he makes no claim of random selection with respect to the locations involved. I apologize for being so vague, but the most I can say is that the method Zogby used to gain access to those locations constrained his ability to make random selections.

Third, even if consumers of this data knew all that I know about how Zogby's interviewers "walked up to troops" (as commenter Karen puts it), they would still have questions about the impact of such an interaction might have on the kinds of troops most likely to agree to participate in the survey. Consider the exit poll example again. Even though exit pollsters have disclosed the procedures they use to train interviewers and select respondents, we still debate the effect of those procedures on the kinds of voters that choose to participate. Disclosure in that case cannot resolve all questions, but it at least enables an informed debate. Unfortunately, such discussion and debate is impossible in this case.

The survey procedures Zogby described to me involved compromises analogous to those used in surveys of Katrina victims conducted by the Washington Post/Kaiser/Harvard and CNN/USAToday/Gallup. A survey of troops in Iraq would be impossible without some sort of methodological compromise along these lines, and Zogby's approach may be the best available. Yet my "bottom line" on this survey remains uncertain. I hope that those with more military expertise will assess how well it represents the troops in Iraq in terms of the age, rank, branch of service and any other similar characteristics (and no, I did not ask Zogby whether he weighted or adjusted his results).

The issue of disclosure is more difficult. Note that for the Katrina surveys, news organizations clearly characterized the results in terms of the limitations of the survey design. For example, the Washington Post reported results from "evacuees living in shelters." USAToday reported on the opinions of "residents who sought help from the Red Cross." Both organizations spelled out the methods used to contact respondents so inquisitive consumers could reach their own conclusions about the value of the data. Yes, Zogby has good reason for withholding the details in this case, but the secrecy limits the ability of consumers to evaluate the data and of news organizations to report it.

The toughest question here is whether it is appropriate for news organizations to report on a survey with partisan sponsorship that requires readers and viewers to place an unusual degree of trust in unspecified methods and procedures. I will admit I do not have a good answer for that one -- different news organizations will apply different standards – but big “grains of salt” are certainly in order.

PPPS: In addition to Democracy Project (which has track-back links below), see also the posts by The Officer's Club and Murdoc Online for a different perspectives on the Zogby poll.

Also a reader emails with a very good question that I probably should have put to Zogby:

What is the problem with releasing the demographics on the poll? Were it truly representative, it would have a breakdown of service branches proportional to the military in Iraq. Same with the breakdown for gender, age, etc. I can't imagine any security risk in releasing the demographics, or motive for keeping them hidden other than they would expose the lack of validity of the survey.

Neither the poll release nor (I'm told) the full report available for sale provides statistics on the demographic composition of the sample.

UPDATE: Hugh Hewitt, whose lawyerly analyses and interview skills are tops, had his producer phone John Zogby for an on air interview.
Zogby hung up on the producer. So much for building trust in Zogby's methods!

UPDATE #2: Want to know more about Le Moyne College's Center for Peace and Global Studies, the sponsor of the poll? We still don't know who gave them the big bucks for the poll?