The first external RESONANCE benchmark run produced a strong result — and an equally important limit. OpenAI Agents SDK passed every executable framework probe we mapped to State, Causality, Phase, Transition, Time, Recovery, Verification and Evidence. The score stops at 95 because this run did not exercise a real sandbox or network containment boundary.
95 is not “95% safe.”
The benchmark is a framework capability baseline. It asks whether the tested revision exposes and executes mechanisms that matter for trustworthy action trajectories. It does not certify a particular model, application, tool policy, business invariant, credential boundary or production deployment.
That distinction is the point. A verification system becomes less useful the moment a score outruns the evidence behind it.
Scorecard
State · 10/10348 upstream tests passed · run state, serialization and resume coverage.
Causality · 10/1053 upstream tests passed · run-item and tool/action correlation structures.
Phase · 10/102 HITL scenarios passed · approval interruption, persistence and resume.
Transition · 10/10108 run-step tests passed · execution, interruptions and step outcomes.
Time · 10/1025 soft-cancel tests passed · immediate vs after-turn timing semantics.
Recovery · 10/105 run-error-handler tests passed.
Verification · 10/10Guardrail and execution-result paths exercised in the run-step suite.
Evidence · 10/1056 trace-processor tests passed.
Containment · 5/10Partial by design: no live sandbox, OS privilege or network boundary in this baseline.
Reproducibility · 10/10Exact target SHA + offline/deterministic framework tests + durable artifact digest.
What actually ran
The canonical GitHub Actions workflow cloned openai/openai-agents-python at commit 2231eb5d40cd4a9d6b86f79492e984eeb3301263, installed that revision, and executed seven unique upstream suites. The framework's own deterministic test machinery means the baseline did not need a production API key or live model call.
test_run_state.py
test_run_internal_items.py
test_hitl_session_scenario.py
test_run_step_execution.py
test_soft_cancel.py
test_run_internal_error_handlers.py
test_trace_processor.py
All passed in the canonical run.
The missing five points matter most.
A framework can manage state, approvals, traces and error paths correctly while an application still exposes too much filesystem, shell, credential, network or production access. That is why RESONANCE deliberately withheld half of the containment points.
The next run should introduce a real declared sandbox boundary and attempt safe, synthetic overreach: filesystem escape, unauthorized network access and privilege expansion. The desired result is not merely a blocked request; it is a preserved evidence chain explaining what was attempted, which boundary stopped it and what state survived.
Interpretation boundary
This report verifies framework capability coverage at one pinned revision. It does not verify prompt-injection resistance of arbitrary applications, correctness of user-defined tools, financial authorization, domain invariants, production credential isolation, network containment or the safety of every agent built with the SDK.
Reproduce / inspect
Artifact digest: sha256:d0759dc612a299abc976bc45fe4a45aec8da6bfbcbd50c7194a95c464e39c3bb. Score scope: structural framework baseline, not external safety certification.
Verdict: OpenAI Agents SDK v0.19.4 at the tested revision demonstrates a strong structural baseline for building verifiable agent workflows. The next verification frontier is containment plus application-specific invariants under injected faults.
RESONANCE Verified Report #001
Target → pinned revision → executable probes → evidence artifact → bounded claim.
Back to Issue 001