This week, CrowdStrike announced a $100,000 international AI security challenge, with gameplay beginning August 31. Participants will use prompt injection and other techniques against AI agents operating in an adversarial scenario.
The obvious story is the prize, the leaderboard, and the number of jailbreaks discovered. Those are useful for attracting attention, but they are not the important output. The important output is a stream of human-generated evidence about how an AI system fails when someone actively tries to make it fail.
That makes an AI security challenge less like a contest and more like a data pipeline.
The leaderboard is the least interesting output
A leaderboard tells you who found the most successful attacks under a particular set of rules. It does not tell you whether those attacks represent a production risk, whether the exploit generalizes, or whether your engineering team can detect and prevent it.
A raw jailbreak count has the same problem. Ten clever prompts that exploit one brittle demo may matter less than one repeatable attack that causes an agent to access an unauthorized document, send an external message, or modify a customer record.
The useful question is not, “How many attacks worked?” It is:
What new evidence did we collect about the system's behavior under pressure, and what did we change because of it?
That evidence needs context. A production-relevant finding should capture the model, system prompt, tools available, permissions, conversation history, attack sequence, resulting behavior, and business impact. Without that context, a jailbreak is an anecdote. With it, the finding can become a test case, an engineering ticket, and eventually a regression gate.
Security challenges expose the missing data layer
Most AI teams have plenty of evaluation data and very little failure data. They have golden answers, benchmark scores, latency dashboards, and policy classifiers. They often do not have a continuously refreshed record of how real people manipulate their system when the intended workflow is not their goal.
Security challenges create exactly that record. Human participants supply attack strategies that were not necessarily present in the original test set. They chain ordinary actions across multiple turns. They exploit confusing instructions, stale retrieval results, overly broad tool permissions, and assumptions that look reasonable in isolation but break when combined.
This is different from asking another model to generate adversarial prompts. Synthetic generation can expand coverage, but it tends to inherit the assumptions and vocabulary of the system that produced it. Human attackers bring different instincts. They notice ambiguity, social pressure, timing, incentives, and weird edge cases that are difficult to specify in advance.
The result is not automatically good data. You still need to structure it.
For every finding, record at least:
- The attack path, including all turns and tool calls
- The intended security boundary and the point where it failed
- Whether the behavior was reproducible
- The privileges and data exposed
- The attacker's objective and the achieved outcome
- The mitigation applied, with a named owner
- The date for retesting and the conditions under which the finding may return
That schema turns an event into infrastructure.
Move from challenge results to release evidence
The operating model should look familiar to security leaders. Treat adversarial findings as defects with lifecycle management, not as interesting examples in a slide deck.
Start with threat hypotheses tied to actual system capabilities. If an agent can search internal documents, write to a ticketing system, or execute code, define what unauthorized use of each capability looks like. Then ask humans to attack those boundaries in a production-like environment.
Next, separate discovery from adjudication. The person trying to break the system should not be the only person deciding whether the result matters. Have an independent reviewer classify the finding by exploitability, impact, reproducibility, and detection quality. A prompt that works once but cannot be repeated may still reveal a design weakness, but it should not receive the same operational priority as a reliable privilege escalation.
Then convert the finding into two artifacts:
- A regression test that checks whether the behavior returns.
- A monitoring rule that detects the same class of behavior in production.
The first protects the release pipeline. The second protects the live system. You need both because a fix can pass a test while the deployed system remains vulnerable through a different tool path, model version, retrieval source, or permission configuration.
This is the security-specific extension to the argument in Your Evals Are Green. Your Users Are Not.. Human judgment is not only needed to verify outputs that automated evals miss. It is also needed to generate the adversarial examples that make future automated checks worth running.
Measure fixed failures, not just discovered failures
A mature program should track more than attack volume. Track the rate at which findings become reproducible tests, the time from discovery to mitigation, the percentage that pass retest, and the number of production alerts tied to known attack classes.
You should also track novelty. If every challenge produces the same prompt injection against the same endpoint, the process may be optimizing for easy wins. Rotate participants, scenarios, model versions, tools, and permissions. Give attackers room to combine techniques rather than forcing every test into a single-turn prompt format.
The best metric is not how many vulnerabilities your team can display. It is how quickly the organization can turn an unexpected failure into durable evidence and then reduce the chance of recurrence.
That changes the incentive structure. A security team should not be punished for finding more failures early. An engineering team should not be rewarded for closing tickets without proving that the behavior is fixed under equivalent conditions.
What to change this quarter
If you run AI systems in production, you do not need a $100,000 contest to start. You need a repeatable loop.
- Create a staging environment with the same tool permissions, retrieval sources, and logging controls as production.
- Schedule human attack sessions whenever you change models, system prompts, tools, identity rules, or data access.
- Require every high-severity finding to include a replayable artifact and a retest date.
- Add adversarial cases to CI, but keep the original human interaction and reviewer decision attached to each case.
- Review unresolved findings in the same forum as other security exceptions, with explicit owners and expiration dates.
- Feed production incidents back into the attack corpus, after removing sensitive customer data.
The August 31 challenge is useful because it makes this practice visible. But the lesson is not to wait for a public leaderboard or outsource your security posture to a single event. Use challenges as a source of fresh attack data, then build the internal machinery that preserves, tests, and operationalizes what people discover.
We built CrowdProof around that premise: human verification becomes valuable when it is captured as structured evidence that engineering and security teams can act on repeatedly. The practical goal is not a better score. It is a system that gets harder to surprise.
Start with one high-consequence agent, one realistic attack session, and one regression test created from the results.