Your Conference Leads Are Not Your Best Data

TechCrunch is selling a more measurable Disrupt 2026. In an August 6 announcement, the conference highlighted lead generation through its mobile app, with the goal of turning conversations into follow-up instead of letting them disappear on business cards. Disrupt runs October 13-15 at Moscone West in San Francisco, and the operational message is clear: if an interaction cannot become a trackable next step, it probably did not happen.

That is useful for sales operations. It is incomplete for AI companies.

A contact record tells you who stopped by, what company they work for, and whether they agreed to a follow-up. It does not tell you why they stopped listening, which answer made them skeptical, what edge case exposed a weakness, or which capability they would actually trust in production.

Your conference leads are not your best data. The best data is the evidence produced when a technically credible person puts your system under pressure.

A demo is an unlogged evaluation

Most AI demos are treated as performances. The team prepares a clean workflow, selects friendly prompts, and guides the audience toward the product's strongest path. The audience asks questions. Someone scans a QR code. The CRM receives a lead score.

The interaction contains far more information than the CRM records.

Consider what a technical buyer might do during a 15-minute demo:

• Ask the agent to explain where a cited answer came from.
• Introduce a contradictory document and watch which source wins.
• Request a workflow that requires permissions the product has not discussed.
• Challenge a latency claim with a realistic batch size.
• Reject a generated design because it is technically correct but visually unusable.
• Ask what happens when the model is uncertain, unavailable, or given incomplete context.

These are not objections to be routed to sales. They are observations about system behavior and user expectations. Each one can become product evidence, but only if we capture it with enough structure to preserve its meaning.

The mistake is not that companies collect leads. The mistake is stopping there.

Interest is a weak signal

A lead score usually compresses a conversation into a few fields: company, role, budget, timeline, and next action. Those fields help answer whether a salesperson should follow up. They are poor inputs for deciding what to build.

A technical buyer saying, “Send me the deck,” could mean genuine interest, politeness, or a desire to compare vendors. A buyer interrupting the demo to ask how retrieval handles stale permissions is providing a much stronger product signal, even if they never book a meeting.

We need to separate commercial intent from technical evidence. They are related, but they are not interchangeable.

A useful evidence record might include:

• The exact question or task the buyer posed.
• The product version, model, tools, and data sources involved.
• The system's response, including latency, citations, refusals, and errors.
• The buyer's judgment: accepted, rejected, uncertain, or conditional.
• The reason for that judgment, in the buyer's own words.
• The severity and frequency the buyer expects in their environment.
• The follow-up action, such as reproduce, instrument, prioritize, or discard.

This is not a request to turn every booth conversation into a research interview. It is a request to stop throwing away the most expensive part of the interaction: expert attention.

Capture judgments, not just comments

Raw feedback is not automatically useful. “The output feels off” is a starting point, not a product requirement. The job is to preserve the judgment while adding just enough context to make it actionable.

For example, a weak note says:

Prospect disliked the answer and asked about compliance.

A stronger record says:

Security architect rejected the answer because the citation pointed to a document the requesting role could not access. They considered this a release blocker for internal knowledge search. Reproduce with role-based permissions enabled and test whether citations expose restricted metadata.

The second record contains a testable behavior, a user segment, a deployment context, and a consequence. It can be handed to engineering, design, security, or product without requiring someone to reconstruct the entire conversation from memory.

The same approach works for positive judgments. “Looks good” is weak. “Accepted the generated SQL after checking the query plan, but requested a dry-run mode before allowing write operations” is evidence. It tells you which part of the workflow earned trust and where the adoption boundary sits.

Build a feedback schema before the event

If you wait until after a conference to organize the data, you will get summaries, anecdotes, and a pile of recordings nobody has time to review. Define the schema before the first demo.

At minimum, create fields for:

• Capability tested.
• User role and technical environment.
• Input or scenario.
• Observed output.
• Human verdict.
• Failure or acceptance reason.
• Business impact.
• Confidence in the observation.
• Owner and next action.

Give people controlled vocabularies where consistency matters. A verdict might be accepted, accepted with condition, rejected, or not enough evidence. An action might be reproduce, investigate, add to benchmark, change UX, document limitation, or no action.

Keep free text for the important details. Do not force a security architect's nuanced concern into a dropdown called “feature request.” The schema should make evidence comparable without flattening the judgment that makes it valuable.

You should also record provenance. Which demo build produced the output? Which prompt was used? Was the response edited by a human before presentation? Was the buyer reacting to a live system or a staged workflow? Without that context, later teams may mistake a polished example for production evidence.

Turn the event into a product instrument

The highest-leverage change is to make evidence collection part of the demo workflow itself. After a challenge, the presenter should be able to mark the scenario, capture the output, record the buyer's verdict, and assign an owner in under a minute.

Then review the evidence in a dedicated session, not inside the sales pipeline. Group observations by capability and judgment, not by account size. Look for patterns such as:

• Three buyers independently reject the same citation behavior.
• Several teams accept the core answer but require audit logs.
• Developers ask for API access while executives praise the interface.
• A capability performs well in a scripted demo but fails when buyers introduce their own data shape.
• Buyers describe the same limitation as a trust issue, even when the underlying bug is recoverable.

This is where human feedback becomes technical infrastructure. It can inform roadmap priority, test scenarios, documentation, onboarding, model selection, and instrumentation. It can also tell you when not to build something. If five people ask for a feature but none can explain the workflow it would change, that request may be less urgent than one precise rejection tied to a deployment blocker.

In Your Evals Are Green. Your Users Are Not., we argued that production quality cannot be inferred from automated scores alone. Conference evidence adds a different layer: it shows which behaviors matter to real technical users before they become tickets, churn, or security reviews. The point is not to replace evals with opinions. It is to connect human judgments to the systems and decisions that follow.

The practical standard

Before your next event, ask one uncomfortable question: if we removed every email address from our conference data, would we still have learned anything about the product?

If the answer is no, your capture process is optimized for contact acquisition, not product learning.

Keep the leads. Capture the evidence beside them. Log the question, the behavior, the judgment, and the consequence. Review it with engineering and product while the context is fresh. Feed reproducible scenarios into your existing validation process, and keep the original human rationale attached.

CrowdProof helps teams turn those interactions into structured, traceable evidence instead of leaving them as scattered notes and sales anecdotes. At your next demo, capture what the buyer trusted and what they would not accept.

The contact gets you a follow-up. The judgment tells you what to build.