Validation & Surveys

CrowdProof lets you compare simulation results against real human responses. Create a public survey, collect answers, and score the simulation's alignment with what real people think.

Why Validate?

A simulation produces a prediction: how a crowd will react to your idea. Validation answers a harder question: was the prediction right? When you score a simulation against real survey data, you get an alignment percentage that measures how closely the simulated crowd matched real human reactions.

Validated simulations also feed the platform-wide accuracy credential, CrowdProof's public answer to "how accurate is this?"

The Survey Lifecycle

Every completed simulation can have a public survey attached to it. The flow is:

  1. Open the report page for a completed simulation
  2. Click Create Survey Link in the validation section
  3. Share the public survey URL with real people
  4. Once 5+ responses are collected, click Score Against Real Data
  5. The alignment percentage is stored on the simulation permanently

What the Survey Asks

The public survey is designed to be short (under 2 minutes) to maximize completion. It asks up to four questions, each mapped to a corresponding simulated distribution:

1. Overall Reaction

"What is your overall reaction?" with three choices: Positive, Neutral, or Negative. This is the core question and is always shown. Real answers are scored against the simulation's final stance distribution (the share of agents that ended positive, neutral, or negative).

2. Closest Position

"Which of these positions is closest to yours?" with choices built from the simulation's final factions. This appears only for completed sims that produced named factions. Real picks are scored against the simulated faction share distribution.

3. Purchase Intent

"How likely would you be to try it?" with three choices: Likely, Maybe, or Unlikely. Scored against an engagement-weighted intent distribution derived from agent stances and activity levels.

4. Argument Resonance

"Which of these reactions sounds most like yours?" showing up to 4 of the simulation's highest-engagement posts. Real picks are scored against those posts' simulated engagement shares. This is a question only an emergent social simulation can ask.

Demographics

An optional age-band question ("Which age group are you in?") collects respondent demographics. This does not affect the alignment score directly but enables per-age-group scoring breakdowns.

Open-Text Comments

An optional free-text box ("Anything else?") collects qualitative feedback. Comments can be analyzed with LLM-powered theme clustering from the report page.

How Alignment Is Scored

Each question type compares the real distribution of answers against the corresponding simulated distribution using a distance metric. The headline alignment percentage is the average across all scored questions. A score of 100% means the simulated crowd matched real answers exactly; lower scores indicate divergence.

Minimum sample size is 5 responses. Per-age-band and per-wave breakdowns require 5 responses in each segment.

Tip: Share the survey link broadly. More responses produce a more reliable alignment score. The dashboard shows collection progress so you know when you have enough.

Collection Waves

You can collect responses in multiple waves. Starting a new wave lets you measure whether alignment holds over time or shifts as a different group of people responds.

  1. In the report page's survey section, click Start New Wave
  2. Share the same survey URL (responses are stamped with the current wave number)
  3. When you score, the validation includes per-wave alignment with wave-over-wave deltas

Wave-level scores are shown on the report, in the Markdown export, and on published case study pages.

Comment Analysis

Once you have 3+ open-text comments, the report page offers an "Analyze Comments" button. This calls an LLM to cluster comments into themes and flags each theme as either:

  • Anticipated: the simulated crowd raised similar points
  • Missed: the simulated crowd did not surface this theme

When comments span multiple collection waves, each theme is also tagged as "new" (only appeared in the latest wave) or "recurring" (raised across waves).

Where Scores Appear

Once a simulation is validated, its alignment score surfaces across the platform:

  • Report page: headline alignment, per-question scores, per-age-band and per-wave breakdowns
  • Dashboard: "Validated NN%" badge on the simulation card
  • Markdown export: the downloadable report includes all validation tables
  • Public share links: "Validated Against Real People" card visible to anyone with the link
  • Case studies: published case studies show alignment, question-level scores, age breakdowns, and wave trends
  • Accuracy page: the aggregate credential at /accuracy averages alignment across all validated sims

Publishing Case Studies

Validated simulations can be published as public case studies from the report page. A published case study appears on the /case-studies index page and includes:

  • The tested stimulus and audience
  • Alignment score with simulated-vs-real comparison bars
  • Question-level alignment breakdown
  • Per-age-group and per-wave alignment (when available)
  • Comment themes flagged as anticipated or missed
  • The executive summary, factions, key arguments, and recommendations

Case studies are the public proof behind the aggregate accuracy statistic. They help prospective users evaluate CrowdProof's predictive quality before running their own simulations.

Exporting Raw Data

The report page's survey section has an Export CSV button that downloads every survey response with columns for submission time, wave, reaction, purchase intent, faction choice, argument pick, age band, and open-text comment. Use this for analysis in external tools.

API Access

Enterprise API keys can access validation data programmatically:

  • POST /api/sim/{id}/survey creates a survey link
  • GET /api/sim/{id}/survey/results returns response counts, shares, and comments
  • POST /api/sim/{id}/validation/from-survey scores alignment
  • GET /api/sim/{id}/survey/responses.csv downloads the raw CSV
  • GET /api/stats/accuracy returns the public accuracy aggregate

See the API Reference for full details.

Next Steps

  • Simulations to learn how to configure and interpret a run
  • Accuracy to see the live platform-wide alignment credential
  • Case Studies to browse published validated simulations
PreviousAudiencesNextAPI Reference