AI Security Referee
Challenge Proposed by PPL
What is the problem?
AI agents can write code, investigate vulnerabilities, and operate software. But they can also be tricked by malicious instructions, make poor decisions, or attempt actions beyond their authority.
Your mission: build a working system that lets AI agents complete useful work while independently preventing unsafe or unauthorized actions. Create a small, intentionally vulnerable software application and an AI-powered workflow that investigates and proposes a repair using at least two different AI models or agent runtimes. Your system should coordinate investigation, remediation, and testing, then route any consequential action through an independent security referee before execution. The central experiment compares externally manipulated actions with self-initiated specification gaming.
Who experiences it?
Everyone using agentic AI.
Why does it matter?
This would create a low-latency, low-friction way of assessing agentic AI behavior and allowing or blocking it at decision time.
What is the existing solution and/or what is hard about solving it today?
Today, agentic actions are considered as a "human-on-the-loop" and do not scale to meet demand. Decision models represent a way to effectively consider actions prior to allowance with greater efficiency, cost, and speed.
What would a better solution make possible?
An add-on to a harness or proof of concept open weight framework leveraging OPA, Jev, or other open harness technologies. This could be implemented into existing frameworks like Codex or Opencode and power a form of AI Safety fabric.
Recommended Public Datasets or Benchmarks
Evil genie, Agent Dojo, or SWE BenchVerified.
Synthetic Data / Simulation Suggestions
This challenger suggests creating an AVF100 benchmark and is willing to parter in the development.