Criticism of the Lancet study
Stephen E. Moore questions the methodology: (hat tip: Volokh Conspiracy)
However, the key to the validity of cluster sampling is to use enough cluster points. In their 2006 report, “Mortality after the 2003 invasion of Iraq: a cross-sectional sample survey,” the Johns Hopkins team says it used 47 cluster points for their sample of 1,849 interviews. This is astonishing: I wouldn’t survey a junior high school, no less an entire country, using only 47 cluster points.Neither would anyone else. For its 2004 survey of Iraq, the United Nations Development Program (UNDP) used 2,200 cluster points of 10 interviews each for a total sample of 21,688. True, interviews are expensive and not everyone has the U.N.’s bank account. However, even for a similarly sized sample, that is an extraordinarily small number of cluster points. A 2005 survey conducted by ABC News, Time magazine, the BBC, NHK and Der Spiegel used 135 cluster points with a sample size of 1,711–almost three times that of the Johns Hopkins team for 93% of the sample size.
I have seen a few descent arguements that say the Lancet study is in the ball park, the best one probably being that we know about 30,000 Coalition and Iraqi soldiers have died, and it is not impossible to believe that 20 times that number of insurgents and civilians have been killed. Not impossible to believe though is not itself evidence.
One of the things that bothers me with this study is that there doesn’t seem to be anything we can point to that provides evidence that they got it right. For example, 80% of their recorded deaths had death certificates, but any counts that have been cited by issuing authorities are far less than 80% of the Lancet number. Moore points out another area where the Lancet study cannot be authenticated:
Dr. Roberts said that his team’s surveyors did not ask demographic questions. I was so surprised to hear this that I emailed him later in the day to ask a second time if his team asked demographic questions and compared the results to the 1997 Iraqi census. Dr. Roberts replied that he had not even looked at the Iraqi census.
And so, while the gender and the age of the deceased were recorded in the 2006 Johns Hopkins study, nobody, according to Dr. Roberts, recorded demographic information for the living survey respondents. This would be the first survey I have looked at in my 15 years of looking that did not ask demographic questions of its respondents. But don’t take my word for it–try using Google to find a survey that does not ask demographic questions.
Without demographic information to assure a representative sample, there is no way anyone can prove–or disprove–that the Johns Hopkins estimate of Iraqi civilian deaths is accurate.
I know just enough about survey methodology to know that I am far from an expert. However, having worked with survey data collection in a previous job, I do know how important researchers consider their demographic information and that without it, you have no ability to gauge whether you have a ‘good’ survey or not. I have yet to see anything in from the Lancet study that you can point to and say, look the survey says this and we can confirm it from some other source. Without that, I think this study is pretty unreliable.
Iraqi Body Count, hardly considered a sight that under-inflates casualty counts, also has criticisms of the study along these lines.
Of course, to a large extent this issue is simply a sideshow. The numbers of Iraqi dead, and certainly quite a few have died and their deaths are regretable, doesn’t tell us anything about the morality of the war or the best course to take now. My view is that as long as the Iraqi people are willing to continue to fight for a free society, and it appears that they still are, we should be there with them. If they decide that the price is too high, I will regret that but certainly admit that it is time to leave and that Iraq was a failure.
