To verify the accuracy of his model, the CIA set up a kind of forecasting face-off that pit predictions from his model against those of Langley’s more traditional in-house intelligence analysts and area specialists. “We tested Bueno de Mesquita’s model on scores of issues that were conducted in real time—that is, the forecasts were made before the events actually happened,” says Stanley Feder, a former high-level CIA analyst. “We found the model to be accurate 90 percent of the time,” he wrote. Another study evaluating Bueno de Mesquita’s real-time forecasts of 21 policy decisions in the European community concluded that “the probability that the predicted outcome was what indeed occurred was an astounding 97 percent.” What’s more, Bueno de Mesquita’s forecasts were much more detailed than those of the more traditional analysts. “The real issue is the specificity of the accuracy,” says Feder. “We found that DI (Directorate of National Intelligence) analyses, even when they were right, were vague compared to the model’s forecasts. To use an archery metaphor, if you hit the target, that’s great. But if you hit the bull’s eye—that’s amazing.”
This is way cool -- not for the first time, I wish I was better at math. He uses lots of math and game theory for his predictions, and for some reason he absolutely drives many of his critics bonkers.
So am I the only one who is reminded of Asimov's pyschohistorians?
UPDATE: Some skeptical points in the comments to Reihan's post. I agree that in order to really assess this guy, you'd need a database of predictions in advance, which you could then check against other methods, or just random chance. But I still like the coolness of it all.
No comments:
Post a Comment