The State of Synthetic Audiences in Marketing Research

Traditional consumer research is slow and expensive. Industry pricing benchmarks for 2026 put focus groups at $7,000 to $20,000 per group with most studies requiring four to six groups, a 1,000-respondent quantitative survey at $20,000 to $80,000, and a full concept testing programme for a large CPG company at $200,000 to $500,000 annually across 10 to 20 studies.^[33] Typical timelines run 4 to 12 weeks from brief to final report.

The market has moved quickly. Qualtrics added synthetic respondents to its survey platform.^[32] Toluna built over one million synthetic personas from its 79-million-member human panel.^[33] NIQ BASES, a standard vendor inside CPG innovation pipelines, now offers synthetic capability.^[34] Simile raised $100 million from Index Ventures in February 2026.^[33]

Practitioner sentiment has not kept pace with vendor activity. The Rival Group 2026 Market Research Trends Report found 42.75% of market researchers described themselves as not excited about synthetic respondents, even while welcoming other AI applications.^[28] A separate 2026 survey of UX researchers found 47% skeptical and wanting more evidence before trusting them, with 24% cautiously optimistic.^[29] The hesitation is concentrated among the people closest to the work, which is worth taking seriously.

Any honest account of this field has to start with how poorly it performed at launch. The 2023 evidence base is unambiguous.

That is the baseline. Anyone selling synthetic audiences in 2023 was selling something the evidence did not support.

Three things improved between 2023 and 2026. They are worth separating because they carry different implications for how a brand should evaluate a vendor.

The obvious factor. GPT-3.5 gave way to GPT-4, GPT-4o, and successors. Reasoning, instruction following, and consistency all improved. This matters, and it is the driver vendors talk about most.

It is also the least interesting of the three, because it is the one a buyer has no control over and the one that explains the smallest share of the gain.

This is the central finding of the last two years and it deserves emphasis.

Early work asked models to output a number directly. Give this concept a purchase intent score from 1 to 5. Models responded with distributions that were narrow, clustered toward the center, and lacking the variance real human panels produce. This was read as evidence of a deep structural flaw.

Researchers at PyMC Labs, working with Colgate-Palmolive, tested a different approach. Instead of requesting a number, they elicited a natural language response about purchase intent, then mapped that text to a Likert distribution using embedding similarity against reference statements. They called the method Semantic Similarity Rating.^[14]

The validation set was substantial and commercially real: 57 consumer surveys on personal care product concepts, 150 to 400 US participants each, 9,300 human responses total, with demographic attributes attached.

The method recovered both panel-level response distributions and the relative ranking of concepts by mean purchase intent. It generalized to other question types, reaching ρ = 82% and K = 0.81 on concept relevance, and zero-shot elicitation outperformed a LightGBM classifier trained on demographic and product features. Accuracy required prompting the model to consider demographic attributes, and model response behavior with respect to age and income mirrored human response behavior reasonably well.

The model was GPT-4o at temperature 0.5. Nothing exotic. The variance collapse that the 2023 literature treated as a fundamental limitation turned out to be substantially an artifact of asking the question the wrong way.

The second large gain came from what the model is conditioned on.

Park and colleagues at Stanford, Northwestern, and Google DeepMind built generative agents from two-hour interviews with 1,052 real individuals. Those agents replicated their subjects' General Social Survey answers 85% as accurately as the individuals replicated their own answers two weeks later.^[9] Interview-grounded agents outperformed demographic-only agents and reduced accuracy bias across racial and ideological groups.

Toubia and colleagues at Columbia extended this with Twin-2K-500, building digital twins of 2,058 people from 500-plus questions each. Twins reached 72% accuracy on holdout data, an 88% relative accuracy against the test-retest baseline.^[15] Brand, Israeli and Ngwe found that fine-tuning on prior survey data improved alignment for existing and new features within a category.^[3]

The pattern across all three: richer grounding in real data about real people produces better results than demographic prompting. This is the second question to ask a vendor. What real data is this grounded in, and how much of it is mine?

Organized by what is being measured. Concept testing and purchase intent is where the strongest results sit: the Colgate-Palmolive work above, alongside Li and colleagues finding agreement rates above 75% on brand similarity and product attribute ratings.^[8]

On predicting experimental outcomes, Hewitt and colleagues tested 70 pre-registered nationally representative US survey experiments, around 470 treatment effects across 105,000 to 119,000 participants. Correlation with actual treatment effects reached r = 0.85, rising to r = 0.90 for unpublished studies outside the training data. The models systematically overestimated effect magnitude.^[19] Toubia's digital twins replicated only about half of between- and within-subject treatment effects.^[15]

On population-level survey replication the picture is more mixed. Kieslich and colleagues tested six models against the World Values Survey and found 94.4% of 90 model-question comparisons were statistically different from the human benchmark at p ≤ 0.05.^[18] Rafikova and Voronin, simulating attitudes toward immigration, gender stereotypes, and family values, found ChatGPT produced responses aligned with liberal-prosocial norms that likely reflect alignment and safety training rather than stable ideological commitments.^[20]

The largest comparison came from Muthukrishna and Warner: 7,755 UK participants, one synthetic platform against seven human sampling sources, same instrument, pre-registered. The synthetic audience fell within the range of outcomes produced by human samples across most questions.^[21]

Separately, Wang, Zhang and Zhang built a transfer-learning estimator pooling LLM-generated and real choice data, recovering efficiency gains of 25 to 80 percent in conjoint estimation.^[22] This is a bounded, testable application that reduces human panel cost without replacing the panel.

The improvement narrative has real limits, and several of these appear structural rather than temporary.

Individual-level variance structure. Aggregate averages replicate well. The distribution of individual differences does not. Park, Schoenegger and Zhu ran replications of 14 Many Labs 2 studies with GPT-3.5 and replicated only 37.5% of the original results. Six studies could not be analysed at all because of what they named the "correct answer" effect, where different runs returned near-zero variation on questions probing political orientation, economic preference, and moral philosophy. In their Moral Foundations replication, the model identified as politically conservative in 99.6% of cases and as liberal in 99.3% of cases under reversed answer order, while both conditions showed right-leaning moral foundations.^[10]

STRAT7, testing two synthetic providers against real respondent data with Dunnhumby, found a bunching effect with fewer responses at the extremes of the scale than real data produces.^[23] Its 2026 follow-up found purely synthetic respondents breaking logical order in a price ordering exercise 68% of the time, falling to 32.8% when blended with real data.^[24] The Nuremberg Institute for Market Decisions, testing personalized digital twins built across ten demographic variables, found that models tend to overgeneralize and default to socially desirable responses, limiting their ability to reflect the nuance and diversity of real consumer opinion.^[25]

Subgroup and segment fidelity. This is the most persistent failure. Wang, Morgenstern and Dickerson tested four LLMs against 3,200 human participants across 16 demographic identities, and found models both misportray and flatten identity groups.^[16] Kantar found subgroup failure in 2023, and the STRAT7 work with Dunnhumby found divergent segmentation and key driver analysis, ruling synthetic out for both.^[23]

Detectability. Dominguez-Olmedo, Hardt and Mendler-Dünner evaluated 43 models against the US American Community Survey. Responses were governed by ordering and labeling biases, with a systematic pull toward the first option. Correcting for those biases pushed models toward uniformly random answers. A binary classifier could distinguish model-generated data from real census responses almost perfectly.^[12]

Causal estimation. Gui and Toubia showed that varying a treatment such as price in blind simulations systematically affects variables that should remain constant, violating unconfoundedness and producing implausible demand curves that fail to slope downward.^[7] This is a logical problem with the simulation design, and model scale does not obviously fix it.

Novel categories. Synthetic respondents are backward-looking by construction. They are trained on how people responded to things that already exist. Performance on genuinely novel concepts with no analog in the training distribution is unreliable, and this limitation is structural.

Cultural and linguistic coverage. Durmus and colleagues found model responses skew toward the opinions of certain populations, particularly the USA and some European and South American countries.^[6]

Prompt sensitivity and drift. Bisbee and colleagues found ChatGPT persona averages tracked ANES averages while producing too little variance, different regression coefficients, sensitivity to prompt wording, and measurable drift over a three-month period.^[11] Drift means validation is a recurring obligation, not a one-time gate.

Two further critiques sit alongside these. Agnew and colleagues argue that substituting synthetic participants conflicts with the foundational research values of representation, inclusion, and understanding.^[13] Brucks and Toubia identify five systematic distortions that persist in digital twins even when built on validated data.^[17]

One finding from 2026 reframes the entire debate and deserves its own section.

The LSE / NYU study compared eight sources on an identical instrument: two opt-in panels, Prolific, two multi-source aggregators, two river samples recruited through Facebook and Instagram, and one synthetic benchmark. Low-quality responses including bots, duplicates, and failed attention checks were removed before analysis.^[21]

The synthetic audience landing within the human range was one finding. The other was that human panels disagreed with each other substantially.

River samples showed nearly twice the rate of failed attention checks and duplicate accounts seen in Prolific and traditional panels. No single human platform performed best across demographic representativeness, response variability, and data quality considered together. After demographic weighting, cross-platform differences fell by more than 80%, indicating that sample composition rather than response behavior drove most of the divergence.

This matters for how the validity question gets framed. Human panel data is not a fixed ground truth. It is an estimate carrying its own substantial, and often undisclosed, source-dependent error. The honest comparison is between two imperfect instruments with different error profiles, and the choice of human recruitment source alone can materially change the strategic conclusion.

This point should be made carefully and not overplayed. Human panel variance does not license synthetic error. It does mean that a brand demanding perfect correspondence between synthetic and human results is holding synthetic to a standard human panels do not meet against each other. The relevant test is decision consistency, whether both methods lead to the same choice.

A related pressure runs in the same direction. Survey panels carry their own data quality problem. The eight-source comparison above found river samples showing nearly twice the rate of failed attention checks and duplicate accounts seen in Prolific and traditional panels.^[21] The Advertising Research Foundation has opened a dedicated initiative on survey fraud, low-quality respondents, and AI-assisted response behaviour, which indicates the industry treats this as unresolved.^[31] Some proportion of responses in any human panel is already machine-generated, undisclosed and unvalidated. Synthetic data that is labeled, inspectable, and reproducible has a governance argument that unverified human panel data does not.

The boundary below is the practical output of everything above. Both columns are supported by the same evidence base.

Any organization adopting synthetic audiences should be able to show its parent company, its insights function, and its legal team a documented validation process. The following is a defensible standard.

Stage 1. Backtest against your own history. Select 8 to 12 completed studies where the human answer is already known. Span easy questions such as concept ranking and awareness and hard ones such as segmentation, pricing, and emotionally nuanced qualitative. Field the identical instruments to candidate synthetic platforms blind to the known results.

Stage 2. Score on metrics that expose the known failure modes. Correlation of averages alone will hide every documented weakness. Measure rank-order agreement on concept ordering, top-box and top-two-box agreement in percentage points, distributional similarity to catch variance collapse, variance and entropy against the human distribution, subgroup-level error reported separately for every segment that matters commercially, decision consistency, and a discriminator test of whether a simple classifier can separate synthetic from human responses.

Stage 3. Stress-test robustness. Randomize answer option order and re-run. Reverse question order and re-run. Quantify how much results move. Bisbee and colleagues documented meaningful drift over three months, so establish the baseline you will monitor against.^[11]

Stage 4. Set thresholds before seeing results. Committing in advance prevents rationalizing a disappointing result. Reasonable working bars for directional use: rank-order agreement above 0.8, top-box agreement within a few percentage points, decision consistency above 85% on backtested studies, and distributional similarity in the range the published literature achieves, with 0.85 as a useful reference point from the Colgate-Palmolive work. Anything below those bars is triage-only, and subgroup results should be assumed unreliable until specifically demonstrated otherwise.

Stage 5. Govern it as an ongoing obligation. Maintain a human holdout on live work. Re-validate quarterly, since models update and drift is documented. Keep an audit trail. Disclose synthetic methodology in any deliverable that travels beyond the immediate team.

The direction of travel is consistent. Wrong-signed pricing estimates in 2023. Recovery of 90% of human test-retest reliability on real CPG concept data in 2025. Publication in Nature in 2026. Every year of the last three has produced results better than the year before, across independent research groups using different methods.

The gains are coming from method as much as model. This is the more useful observation, because method improvements are faster and more reproducible than waiting for the next frontier model. The variance collapse problem that dominated the 2023 critique was substantially solved by changing the elicitation format. Similar unlock potential likely remains in grounding architecture, calibration against first-party data, and hybrid designs.

Some limitations may not resolve. Causal estimation, novel category performance, and individual-level variance structure have resisted improvement so far and have plausible structural explanations. It would be a mistake to assume all current weaknesses are temporary.

Two developments will materially update this assessment. The Advertising Research Foundation is running a controlled comparison using identical questionnaires across synthetic and traditional online samples, publishing monthly briefs with a synthesis report at the end of 2026.^[30] That will be the first substantial independent industry benchmark. Separately, independent replication of the Colgate-Palmolive elicitation results in other categories would establish whether that finding generalizes beyond personal care.

Organizations that build validation capability during this period will be able to expand usage as the evidence supports it. Organizations that wait for certainty will adopt later and without the internal muscle to evaluate what they are buying.

The evidence supports a specific and bounded position.

Synthetic audiences work well enough today for early-stage screening, message triage, instrument piloting, hypothesis generation, and exploration of hard-to-reach audiences, provided they are grounded in real data, use validated elicitation methods, and are checked against a human holdout. In those applications the speed and cost advantages are large and the risk is contained.

They do not work well enough for final decisions, population estimates, claim substantiation, segmentation, or novel categories. Those boundaries are supported by consistent evidence across multiple independent research groups and should be stated plainly to any stakeholder evaluating adoption.

The field has improved substantially and continues to improve. That improvement is real, documented, and driven by identifiable factors. It does not yet extend to every use case, and the organizations getting the most value are the ones that know precisely where the boundary sits and validate against it continuously.

Each claim in this review, with its supporting figures and how strongly it is supported. Filter it. If you only trust strong evidence, read only that. If you want to know which findings were paid for by the companies selling the product, that is one click.