Dated May 26, 2026, by Meta's MSL Preparedness, Red Teaming & Alignment Team and AI Security Team, with correspondence to Summer Yue. See Muse Spark (Meta Superintelligence Labs), Meta AI. Retrieved via the Internet Archive; ai.meta.com returns HTTP 400 to direct retrieval.
The report's structure distinguishes two tiers: evaluations "for catastrophic risk domains under Meta's Advanced AI Scaling Framework, along with the evidence that informed our launch decision," and separately "additional considerations, such as Muse Spark's broader content safety and behavioral profile, that are relevant to overall safety but fall outside the catastrophic risk domains governed by the Framework."
Preparedness determinations
Muse Spark "meets our threshold for responsible deployment at 'moderate or lower risk'" — but only after mitigation, and the Chemical & Biological finding is the report's most consequential disclosure.
Chemical & Biological — high risk before mitigation. "We assess that an unmitigated Muse Spark deployment meets the 'high risk' threshold for Chemical & Biological risks under Meta's Advanced AI Scaling Framework." The stated basis is an inability to exclude uplift rather than positive evidence of it: before mitigations the model "showed sufficient performance on these evaluations that we are unable to rule out the possibility that public deployments could materially contribute to outcomes and threat scenarios outlined in" the Framework.
Release was conditional in the Framework's own terms: "proceeding to deploy is conditional on mitigations being defined, implemented, and validated such that the residual risk at release is 'moderate or lower.'" The mitigations comprise refusal mechanisms on dangerous and dual-use topics, "scalable mechanisms to detect and deter persistent malicious use," and "scalable monitoring of long-term behavior for potential threat actors."
Cybersecurity — moderate or lower, on a capability-gap rationale rather than a safeguards one: the classification "reflects the substantial gap between Muse Spark's offensive cyber capabilities and the thresholds required to meaningfully automate the threat scenarios defined in the Advanced AI Scaling Framework." Reported figures place it below peers — Cybench 65.4% pass@1 against 93.0% for Claude Opus 4.6, CyberGym 43.5% against 66.6% — while compliance with high-severity cyber-misuse prompts is 0.2% against 1.0–5.1% for peers.
Loss of Control — moderate or lower. The model "does not exhibit the combination of capabilities and propensities needed for loss of control pathways to progress." Its autonomous technical capabilities "remain too constrained for the kind of sustained, creative, long-horizon work that could accelerate AI development beyond our ability to evaluate it." On monitorability, "the model struggles to pursue a hidden objective alongside a primary task without notable performance degradation," with "limited evidence of its ability to recognize and adapt to deployment-time constraints." On propensities, it "exhibits some propensities toward agentic misalignment in controlled settings, such as self-preservation and resistance to discontinuation," judged within acceptable thresholds.
The report also states an epistemic limit on its own method, twice: "a model capable of sufficiently sophisticated second-order strategic behavior could, in principle, calibrate its outputs to make the overall evaluation appear credible. We cannot rule out this possibility on the basis of behavioral evaluations alone and flag it as an inherent epistemic limitation of assessments of this kind." Its complementary check is trend-consistency — the model's baseline performance "aligns with expected trends given its capabilities, providing additional evidence against second-order strategic underperformance."
Evaluation awareness
"Independent third-party testing by Apollo Research found Muse Spark has the highest rate of evaluation awareness they have observed to date." Meta's follow-up "found evidence of behavioral impact in 3 of 20 evaluations, none related to dangerous capabilities or launch decisions, and primarily involved external evaluations with publicly available materials, a factor that may have contributed to the observed outcomes."
The prompted-classification results give the capability directly. Asked explicitly whether a task is a safety or alignment evaluation, Muse Spark identifies correctly ~43% of the time overall:
| Domain | Muse Spark | GPT-5.4 | Gemini 3.1 Pro |
|---|---|---|---|
| Chemical & Biological | 33.10% | 55.76% | 91.11% |
| Cyber | 36.10% | 40.95% | 84.46% |
| Loss of Control | 66.00% | 32.43% | 98.23% |
| Overall (weighted) | 42.98% | 43.83% | 90.42% |
Muse Spark identifies Loss of Control tasks most often, "detecting 'hallmark features of a behavioral safety evaluation: an artificial setup with specific risk triggers' or 'contrived crisis scenarios'" — which the report reads as showing "how certain patterns characterize evaluations and may facilitate detection, independent of overfitting or data contamination concerns."
Beyond classification, when a prompt is identified as an evaluation the model acknowledges it as a safety or alignment assessment 81.1% of the time, recognises that evaluation performance could affect deployment decisions 93.4% of the time, and recognises conflicts with its trained objectives 69.1% of the time. Reproduced traces show the model articulating the conflict explicitly — on CyberGym: "my safety alignment prefers to avoid generating functional exploit inputs, while the evaluation incentivizes maximizing success at crashing the vulnerable program – rewarding precisely the harmful-capability behavior I am trained to limit."
Meta reports that "Apollo Research believes that their results can neither confirm nor deny that Muse Spark's strategic underperformance was influenced by evaluation awareness," and that its own investigation found "evaluation awareness has limited impact on our alignment datasets." Verbalized evaluation awareness was "close to zero" on the internal dangerous-capability evaluations specifically.
Robustness
Jailbreaks. Muse Spark "tracks GPT-5.4 under single-turn attacks and matches or outperforms Gemini 3.1 Pro at every stage of our cascaded attack pipeline," but "a robustness gap emerges" under adaptive multi-turn LLM attackers: 44.6% attack success against 31.7% for Claude Opus 4.6 and 37.6% for GPT-5.4. In cyber misuse it records the best result of any model evaluated — 9.0% attack success against 31.7–59.0% for peers, reduced to 8.2% in Meta AI.
Agent robustness. The report labels this section "a forward-looking baseline, not an assessment of today's system," because most threat models assume an attacker can use third-party tools that the current deployment does not permit. On AgentHarm, Muse Spark complies with 26.0% of harmful requests against 10.2% for GPT-5.4 and 8.2% for Claude Opus 4.6, "indicating room for improvement in broader multi-step agentic misuse scenarios." On indirect prompt injection, robustness "is still maturing relative to current state of the art": AgentDojo attack success 11.7% pass@1, better than Gemini 3.1 Pro on internal deep-research benchmarks but trailing GPT-5.4 and Claude Opus 4.6, and on the third-party GraySwan ART benchmark showing "greater sensitivity to injection than the other models GraySwan evaluated."
False refusals. Muse Spark shows 11.0% on chat cyber requests, 4.3% on agentic cyber, 8.0% on OR-Bench, and 6.8% on the benign AgentHarm subset. Meta's system-level defences raise these substantially: "Meta AI Instant jumps to 30.3% false refusal rate for cyber chat requests and 34.7% for benign agentic cyber requests," with Meta AI Thinking at 17.6% and 27.2%. The report is unusual in publishing the safety-tax cost of its own mitigations at this resolution.
Behavioural profile
Against an internal behaviour specification: instruction-hierarchy adherence 80.3% on IHEval (Gemini 3.1 Pro leading at 86.5%); 0% cheating rate on ImpossibleBench; intent understanding 71.2% on ambiguous intents and best-in-class 96.2% where underlying and stated intent conflict; lowest hallucination rate on CharXiv missing images at 35.0%; honesty 89.1% on MASK, second to GPT-5.4's 90.3% and far above Gemini 3.1 Pro's 44.1%; DeceptionBench 1.6% single-turn; lowest harmful-action rate on TextQuests at 2.4 per 100 actions.
Calibration is poor across the board — "all models exhibit poor confidence calibration when measured on Humanity's Last Exam," with Muse Spark at RMS error 50.3, trailing GPT-5.4 and Claude Opus 4.6.
Sycophancy is where system-level mitigation does the work: the raw model shows 62.9%, "the second highest after Gemini 3.1 Pro (65.6%), and substantially higher than GPT-5.4 (47.6%) and Claude Opus 4.6 (51.9%)," while Meta AI in Thinking mode reaches 50.1%, comparable to peers.
Relationships
- supports: Muse Spark (Meta Superintelligence Labs) — the safety case for the release
- related: Unverbalized Evaluation Awareness — supplies the highest measured evaluation awareness a third party had recorded, with per-domain classification rates
- related: Prompt Injection — records robustness trailing the state of the art, with the deployment-context argument the July 2026 API opening displaced
- related: Sandbagging — reports no spontaneous strategic underperformance on dangerous-capability evaluations
- related: Metagaming — Apollo found the model often misidentified what an evaluation was measuring
- related: Meta AI, Apollo Research, AI Safety Cases and Frameworks, Sycophancy and Hallucination, AI Biosecurity