On the day of the election, Nate Silver told us that the election was a complete coin flip - actually, closer than a coin flip.
This doesn’t mean that Silver believed the election was definitely going to be close. It just means that when he put polling results and fundamentals into his model, the model responded with… a shrug. His model did not even have a suggestion as to which candidate was more likely to win.
But by November 5, there was basically no “real uncertainty” around the election results. We now know that Trump won fairly decisively, and if a model could have accurately looked into the hearts and minds of 300 million Americans it would have known that the probability of a Trump victory was 100%. In each of the seven swing states, the number of Americans who were planning to vote for Trump was higher than the number of Americans who were planning to vote for Kamala. This was, in some abstract sense, knowable; but the accumulated polling efforts of the entire country did not come to know this result. Put another way: there was epistemic uncertainty, but there was no aleatoric uncertainty.
So why isn’t polling more effective? I’ll run through the different types of uncertainty that models and polls are dealing with, the incentives at play for both, and the success of one French Polymarket trader who made out quite handsomely with his bets on Trump.
First, polls of all kinds deal with problems with selection bias. Response rates for political polls are typically somewhere around 4%, which means that of all of the people who are contacted just 4% of them follow through with the poll. This minority group will not be representative of the broader electorate, so pollsters have to try to figure out how to adjust their figures to line up with the electorate. They also have to question whether people are answering them honestly. For nearly a decade now, people have written about the “Shy Trump Voter Effect,” the idea that people wouldn’t want to admit that they’re voting for Trump and will claim something else. I’m skeptical about this effect being real.
Questions of how to weigh polling results bring us to the uncertainty around the changing composition of the electorate. Every four years, different kinds of people show up to vote at different rates, and we don’t know exactly how that will play out until the day of the election. Different pollsters take different approaches. The New York Times/Siena polls, for example, ask people how likely they are to vote and also look at their voting history, as people tend to overestimate their own likelihood of voting. This is more art than science, and models have to review how polls implement this and decide how much to trust each poll and how to weigh polls.
Models also incorporate some amount of time-based uncertainty. Even if they knew exactly how each American would vote two months before the election, votes are subject to change. If a model has some reason to suspect that the results will lean in one particular direction between a given moment and the election, that can be built in. Silver implements this with a “convention bounce”; he expects that after a party’s convention their candidate will rise in the polls by a few points, only to fall back down later. But if there’s no suspected bounce, the time-based uncertainty should just give error bounds that approach zero as the day of the election comes.
Consumers of polls and models alike should also consider the incentives at play for both. Pollsters likely have worse incentives. Of course, they want to be right, but they don’t want to be embarrassed by standing out with a bad poll. As a result, too many of them mess with their results to be in line with the pack in a process called herding. On average, this makes any individual poll more accurate, flattening out the variance that would occur naturally in their results. But if others are doing it then it means that the polling averages end up being more of a collective guess than the result of a rigorous, mathematical process. We know that herding happens because for most pollsters the poll-to-poll variance is significantly lower than we would expect from random sampling. Some pollsters are honest and don’t herd, like the now-infamous Selzer poll or NYT/Siena. Others herd.
When it comes to models, the incentives are more complicated. Election models certainly want to be directionally correct on every state and on the election more broadly; if they think Kamala is definitely more likely to win, they should say that so that when she wins they get credit for being correct. But if they swing too far in one direction, then they can be excoriated for it. In 2016, some forecasters gave Hillary a 99% chance of winning. Nate Silver’s FiveThirtyEight put the probability of a Clinton victory at 70%, lower than any other model, and still took a lot of flak when that ended up being wrong. So models are incentivized to avoid that as best they can by flattening out results towards a 50/50 outcome; the sample size of presidential elections, which occur only four years, is small enough that it’s hard to retroactively determine with any confidence whether presidential election models are well-calibrated or not.
I don’t know how much the incentives end up distorting the methodology behind these models. But it’s something worth considering.
One area in which the incentives neatly align with accuracy is in the world of prediction markets, which have become increasingly popular in recent months.
Although it varied from one prediction market to another, prediction markets did tend to deviate from election models. The models were clustered around 50/50 in the run-up to the election, but just one week before the election Trump got as high as 67% on Polymarket. This was dismissed by many as a poor signal because Polymarket utilizes blockchain technology and as such disproportionately attracts “crypto bros” who would be more likely to support Trump. There was also one individual trader, a French man with a background in financial services who went by the name Théo, who put over $70 million on Trump, shifting the odds as a result.
Théo ended up being right, of course. This begs the question: did he parse the noise of the election and find the signal? His estimate that Trump had a 80% to 90% chance of winning has since been vindicated, but it could be that he was just a dumb rich guy.
There was some method to his madness. Although the “Shy Trump Voter Effect” hasn’t found much empirical backing otherwise, Théo believed in it and thought that polls didn’t do enough to account for it.
Per Théo’s claims to the Wall Street Journal, he commissioned his own series of polls using something called the neighbor method. In this type of poll, people are asked who they expect their neighbors to support rather than being asked directly who they would support. Théo thought this got around that “Shy Trump Voter Effect” and would be more reliable than other forms of polling. He isn’t sharing the results publicly, but Théo claims that this method showed dramatically better results for Trump such that he was confident putting $70 million down on the man.
My own inclination was that Harris was being underestimated by models and prediction markets alike. In both 2016 and 2020, pollsters nationally had a bias in favor of the Democrats by at least 3 points. When I saw the Ann Selzer Iowa poll, which came from a pollster who had a history of coming out with shocking polls that were later proven correct, I figured that the widespread herding from other pollsters was probably biased in favor of Trump in an attempt to overcorrect for the last two election cycles. I put down a lot of Mana in favor of Harris on Manifold Markets, another prediction market that uses play money.
Before election night I had a good run on Manifold. I bet on all sorts of stuff, and profited especially handsomely when Biden was pushed out in favor of Harris. On the morning of Election Day, I had lifetime profits of over 10,000 Mana. But on the night of November 5, I kept doubling down on my preferred candidate. I probably should’ve put my phone away after my third shot of the night. I was wrong, of course. And Théo was right.
Models and prediction markets are more about understanding the incentives and uncertainties behind the information we have than they are about analyzing the information immediately available. The real alpha lies in examining data critically.
The rise of Polymarkets and Kalshis means that understanding these dynamics is now a skill you can be directly rewarded for in the market—and it's crucial for understanding the world around us. Théo's success wasn’t necessarily just luck; he dug deep to find his own information, filling in gaps in public understanding to update his beliefs and make bets accordingly.
Those who understand the difference between epistemic uncertainty and aleatoric uncertainty, between real signals and noise, and who are willing to seek out their own data, will be the ones who thrive in an interconnected world increasingly shaped by the wisdom—and folly—of crowds.