Some polling firms are starting to survey AI personas instead of people. By the end of this article, you'll know what synthetic polling is, why it can mislead, and which questions to ask the next time a poll or market-research claim cites "AI respondents."
What synthetic polling is
In synthetic polling, a large language model (LLM) is set up to play a voter or customer. It might be told to answer as a 45-year-old suburban homeowner, for example. You ask it survey questions, collect the answers, and treat them as if a real sample had responded.
The appeal is easy to see. According to a report from Phys, firms are exploring these personas to cut costs and scale research, while campaigns, brands and media outlets want faster audience insight. No recruiting, no waiting for responses, no paying participants.
The core problem: people change their minds, personas may not
Phys's reporting frames the issue with a simple contrast. A voter surveyed in October may give a different answer by January. An AI persona might not, depending on how it's built.
That matters because real public opinion moves. People react to news, to their wallets, to a conversation with a neighbor. A persona that gives the same tidy answer every time looks reliable, but that consistency may not reflect reality. It may simply reflect how the model was built.
Three risks to understand
1. Model bias
An AI persona is not a person. It is the model's guess at what such a person would say, shaped by whatever data the model learned from. Jennifer Stromer-Galley, whose concerns Phys highlights, points to model bias as a central worry. If the model's picture of a group is skewed, every "respondent" inherits that skew, and a larger sample doesn't fix it. You just get the same distortion at scale.
2. Missing outliers
Real surveys capture odd, surprising and minority answers. Those outliers are often where the story is: an emerging concern, a group that doesn't fit the pattern. Per the Phys report, Stromer-Galley raises the concern that synthetic personas can miss them. An average-sounding persona tends to produce average-sounding answers.
3. Mistaking consistency for truth
This is the subtle one. Steady, coherent results feel trustworthy. But the Phys piece warns about the risk of mistaking AI consistency for public reality. A tidy chart is not evidence that people actually think that way.
What the Pew findings suggest
Phys notes that recent Pew findings showed major gaps between AI and human survey responses. I can't give you the specific numbers here, so check Pew's own publication if you need them. The takeaway is that AI answers and human answers can diverge in meaningful ways, which is exactly why a reliability question exists at all.
Where synthetic data might still help
This isn't a case for dismissing the technique entirely. My take: low-stakes uses look more defensible than high-stakes ones. Consider:
- Brainstorming. Using personas to draft survey questions or spot confusing wording before you ask real people.
- Stress-testing messages. Treating persona reactions as hypotheses to check, not findings.
- Early exploration. Forming a rough idea you then verify with a real sample.
The danger comes when decision-makers use persona output in place of voters when deciding where to spend money, what to say, or what the public believes.
How to read a poll that mentions AI respondents
You don't need technical skills for this. Ask these five questions:
- Were the respondents real people, AI personas, or a mix? The answer should be stated plainly.
- Was it checked against real survey data? Look for a comparison with a human sample.
- Does it show change over time? Real opinion shifts. Suspiciously flat results deserve a second look.
- Are there surprising minority answers? A complete absence of outliers can be a warning sign.
- Who built the personas, and how? The setup affects the answers, so vague methodology is a red flag.
Treat it as a distorted mirror
A useful way to think about synthetic data is as a mirror of society that may be warped. It reflects the patterns in what the model learned, not the living, changing public. Sometimes a warped mirror still shows you something helpful. You just wouldn't use it to check your appearance before an interview.
The takeaway
Next time a headline says "voters think" or "customers want," find out whether a real person ever answered the question. If none did, treat the result as a starting hypothesis, and confirm it with people before acting on it.
