As is often the case, G. Elliott Morris worked for big publications before he went independent. That does give you some credibility before you go out on your own, though, and credibility is extremely important, especially when it comes to polling.
Morris’ newsletter, Strength In Numbers, is a widely recognized and respected source of new polling data. Morris started as a data journalist the better part of a decade ago at The Economist, then moved on to work at ABC News and polling analysis website FiveThirtyEight. Now, he’s doing things his own way with this newsletter.
There are some big advantages to being able to do your own polls and focus on what you think is important, according to Morris. In this edition of Depth Perception, we speak with him about what it’s like competing with the big guys, what he’s learned from the questions he’s asked voters, and how things are changing in the world of polling. —Thor Benson
Learn from top longform journalists and find the best in-depth reporting. Subscribe to Depth Perception:
You’re part of an industry that is filled with big names — Reuters, YouGov, the AP, Gallup. How does it feel to kind of be a lesser known, but influential voice in the polling world? You have experience with the big companies, but you’re doing this independently.
Well, you frame it as a potential downside, but I think it’s actually quite an upside in a couple of ways. I’m just talking about running polls that are financed through a model of independent journalism, rather than whatever mainstream news networks are financed by these days.
The first advantage is I can ask pretty much whatever I want at a pace that works for me in producing journalism. That process is incredibly streamlined in a one- or two- or three-person operation, versus the way that this has traditionally worked, where you come up with some idea of the things that are important to ask right now, and those get filtered through several layers of bureaucracy.
It becomes the 22nd segment on “Good Morning America” or whatever. They’re balancing the other priorities of the organization, which most of the time are really far removed from the journalistic questions. What do people care about from a “small d” democratic point of view? I feel like I’m doing polling that is much closer to what people care about now. I don’t have a bunch of incentives working against me in the broader organization that I had before. It’s just easier to do when you don’t have an institution bringing the boot down on your neck.
For sure. How is it competing with those big name organizations, though? People are used to seeing their polls, and they pretty much trust them. They might not be familiar with your work.
Traditional media organizations still have a lot of purchase when it comes to sharing information, which is a source of frustration. There’s a hesitancy to believe what independent journalists are putting out there. There’s a belief in the public, rightly or wrongly, that institutions still have some layers of fact checking and standards and credibility. From my experience, that is a very dubious assumption on the part of the public. That is a downside to the independent journalism model.
I spend a lot of time doing what is essentially reputation building. That involves me giving interviews to people, like you, but also a lot of the stuff that I would normally do, which is sharing as much information about the process of polling as possible. I think I have an advantage in that I wrote a book about surveys, but it’s still hard to get attention to these surveys sometimes. There are lots of traditional journalists, as opposed to independent journalists, who are really into what I’m doing at Strength In Numbers.
I’ve seen you call out how people have misread data. For instance, people assume Democrats moving to the right benefits them electorally. How do you approach that?
Those are built on assumptions about politics and voter psychology, not data. I guess there’s some data that people will use to make these arguments on occasion, but for the most part, I find it to be a misreading of voter psychology that is actually rooted in ignorance of the data.
The advantage of independent journalism is that I’ve assembled an audience of people that really care about getting questions like this correct. There’s the old 1,000 true fans model of independent media from the guy who founded Wired. Strength In Numbers is more than 1,000 poll sickos who believe that these arguments are important to make and that you can get in the weeds. These are not the type of arguments or articles I’d be able to write in a traditional outlet, because of all those other incentives I mentioned earlier.
So I’m really lucky that the audience supports the work, but it’s really as simple as doing the work the right way and continuing to justify it to the audience. Do your work the right way, and the audience is there.
“There’s the old 1,000 true fans model of independent media from the guy who founded Wired. Strength In Numbers is more than 1,000 poll sickos who believe that these arguments are important to make and that you can get in the weeds.” —G. Elliott Morris
Yeah, you’re able to ask whatever questions you want. I’m sure that’s kind of fun to experiment with. What have been the most interesting findings you’ve had recently, or maybe the past six months?
This was in either the very first or second poll that I ran in Strength In Numbers with our polling partner… I did an experiment on how voters were reacting to information about the kidnapping of Kilmar Abrego Garcia. So people who were reading the news at the time would recall there was some debate over whether or not members of Congress should draw attention to this. Chris Van Hollen, a senator from Maryland, went down to El Salvador to try to interview — and did get some time with — Kilmar Abrego Garcia.
The reaction in the House of Representatives was, “Don’t go down there and draw attention to this.” So my thing was, let’s root this conversation in data. Are voters actually responding to this negatively, in terms of how they were viewing the Democrats? Is going down to El Salvador, in this case, turning voters against the Democrats? Or does this actually increase support for what Democrats are arguing for, which was, at the time, less leeway for the government to deport pretty much whoever they wanted in violation of federal law?
So we did a survey experiment where half of the respondents were asked first, “Do you support the administration’s actions on mass deportations?” And then they were given a question that was more of a priming of information that said, “Here’s what just happened to Kilmar Abrego Garcia — do you think he should be returned home?” So half of the poll saw the questions in that order: First, how do you feel about deportations, and then how do you feel about Garcia? And the other half of the poll got this in reverse.
We saw a large increase in support for the Democratic position for people who were told or heard about Garcia before being asked about mass deportations. So we could write this article and really root the discourse in some data. That was not the type of poll that we could have run at ABC News or FiveThirtyEight. There was no buy-in on running survey experiments and trying to explain that to an audience of mostly boomers who fell asleep on their couch at 5 p.m. watching the local weather.
But it is the type of thing that you can write up for people who have self-selected into spending time with the data and taking discourse about politics seriously, rather than ideologically, first.
The Telly Awards: Long Lead needs you!
What do twins know that the rest of us don’t? Something clearly important since they tend to live longer than the rest of us. While the data behind that fact is clear, the reasons aren’t, so Long Lead creative director Sarah Rogersdecided to ask them in Double Meaning, a portrait and photo essay exploring the people who attend the annual Twins Day festival in Twinsburg, Ohio.
But get this: Double Meaningis actually a double-feature. Tucked inside is Twins Fest, a documentary film short by filmmaker Kate Bennis that was recently named an honoree at the Webby Awards and “Best Documentary Feature” and “Video of the Year” by the Society of Publication Designers.
To top it all off, Twins Fest is a contender for “Best Documentary Film Short” by the Telly Awards! Please vote for Twins Fest in the Telly Awards today. And if you have a twin — or some friends! — tell them to vote, too. After all, two votes is better than one.
Something I’ve been noticing, and I’m sure you have too, is CNN and others are starting to incorporate prediction markets into their reporting. I’ve seen a lot of popular figures in politics say this could replace polling. What do you think about that?
Prediction markets are valuable sources of information to an extent. The theory is that they are valuable when there are enough people willing to put their money where their mouth is and where those people are informed about the thing they are trying to predict. But those are not conditions that are actually satisfied in prediction markets all of the time. So you do get some pretty big errors that you might not get with polls. They’re not foolproof.
On the empirical question of whether or not they are, even while not being foolproof, better than polls on average, the literature, which really predates things like Polymarket and Kalshi, had pretty mixed reviews of prediction markets. It found that they were really good on stuff like who’s going to win the presidential election but really bad on stuff like what vote share these candidates are going to get. And especially once you get down to the House or sometimes the governor level, then the amount of information being aggregated is not enough to make the prediction markets better than the polls.
They are marketing themselves as oracular distillers of all the information in the world about some subject. That just frankly doesn’t pass the bullshit detector. When you look at it, there are also some huge predictive errors on questions that are really, really important, like who is going to be the next pope. I think the prediction markets gave the eventual pope a 2% chance of winning at most, usually closer to 1%. So if you are the oracle at Delphi, that’s a pretty bad track record.
In an election context, they can be biased by what we call “whales,” or people with a lot of money entering the market. And then the aggregation principle is no longer satisfied. So the whole argument for why you would want to trust the market is invalidated when there’s not a whole lot of money in the market and one person, which we can sometimes call dumb money, ruins the market.
So even if they are good at predicting elections, I think they fail at the thing that they’re trying to be, which is like a new kind of news engine, right? Kalshi is like, “You don’t need to go to CNN, all you need to do is follow our markets. They tell you everything you need to know about the world.” Or like on election night, “You don’t need to watch the statistical model at The New York Times, you can just watch the Kalshi odds.” And those odds can end up being wrong.
I'm just a liberal arts major, but it seems that prediction markets, unless carefully constructed with enough information, enough participants who have ingested this information and a circumscribed set of questions, is just word or mouth or popular folk belief. And if you go to the trouble of carefully setting up the betting environment in this way, you sort of get right back to professional polling. Prediction markets are best for horse races.
I'm just a liberal arts major, but it seems that prediction markets, unless carefully constructed with enough information, enough participants who have ingested this information and a circumscribed set of questions, is just word or mouth or popular folk belief. And if you go to the trouble of carefully setting up the betting environment in this way, you sort of get right back to professional polling. Prediction markets are best for horse races.