Safety Cases for Frontier AI

Safety cases for frontier AI is a 2024 paper by Marie Davidsen Buhl, Gaurav Sett, Leonie Koessler, Jonas Schuett, and Markus Anderljung of the Centre for the Governance of AI, posted to arXiv on 28 October 2024, with Buhl as corresponding author. It proposes adapting safety cases — reports making a structured argument, supported by evidence, that a system is safe enough in a given operational context — from safety-critical industries such as aviation and nuclear power to frontier AI, and examines their use in both industry self-regulation and government regulation.

The paper restricts its scope in four ways: to frontier AI systems rather than AI systems generally, on the grounds that frontier systems are among those most likely to pose severe risks and present assurance challenges warranting novel analysis; to catastrophic risks, since these are the primary focus of companies' existing safety frameworks; to the US, UK, and EU contexts; and to deployment as the primary worked example, though the authors note safety cases may also inform development decisions such as whether to begin a training run. It defines catastrophic risk as extremely large-scale harm — for example damage in the tens of thousands of lives lost, hundreds of billions of dollars of economic or environmental damage, or significant adverse disruption to the social and political order. The authors note that safety cases may also be useful for other high-risk systems such as biological design tools, and are already used for autonomous vehicles and defense-related software.

What a safety case is

A safety case is a structured argument, supported by evidence, that a system is safe enough in a given operational context, a definition the paper takes from the UK Ministry of Defence (2007). The authors use "the system is safe enough" interchangeably with "the system does not pose unacceptable risk." Safety cases typically inform major go/no-go decisions — whether to build, deploy, procure, or license a system — through a three-stage process: the developer writes the case, an internal or external actor reviews it, and it is shared with the decision-makers. Sometimes a single actor, such as a regulator, is both reviewer and decision-maker.

The defining feature relative to alternatives is that a safety case argues about outcomes achieved rather than products or processes. It is not a collection of decision-relevant information, nor a checklist of best practices the developer followed; it explains why the information presented provides sufficient assurance that the system is safe enough.

Safety cases emerged in energy, petrochemicals, and transportation, where assurance had traditionally meant adherence to specific product design rules. Following accidents in the 1960s–80s that rules-based approach was questioned on the ground that it did not encourage developers to assess system safety comprehensively. The practice has since spread to defense, aerospace, and more recently to security, software, and autonomous vehicles. The paper records the criticism as well: safety cases have been faulted for providing a false sense of assurance when producing an adequate case, or reviewing one effectively, is very difficult. It notes there is little empirical evidence for their efficacy, that such evidence is difficult to produce given the nature of risk, but that practitioners mostly consider them effective and that they are a recognised best practice in the UK.

The authors treat "operational context" broadly, as roughly the setting in which the model is used, with who can access the model and in what ways as key elements. They note that frontier AI safety cases need not be restricted to a specific use case, and that the relevant context will often be a widespread deployment context in which many users can use the system in open-ended ways.

The four components

The working assumption is that a system is unsafe unless convincing arguments are made to the contrary.

Scope should include a detailed specification of the system (architecture, training process, safeguards) and the intended deployment context (whether weights will be released or access is API-only); an account of which changes to system or context fall inside the case and which would require an updated one; and any other restrictions, in particular temporal scope, assumptions such as a safeguard being resistant to jailbreaking, and out-of-scope use cases such as medical diagnosis. The paper's illustrative scope sketch includes a deep learning network with one trillion parameters, pre-training with 10²⁶ total training FLOP using self-supervised learning on Common Crawl web text, RLHF fine-tuning to be helpful, harmless, and honest, API access to anyone with a free online account, and an AI assistant monitoring activity for suspected usage-policy violations. Its illustrative assumption is a buffer covering one year of expected capability improvements from scaffolding and prompting plus additional training using over 10% of pre-training compute, with a new safety case required once those buffers are surpassed. The authors identify breadth of scope as a particular difficulty for frontier AI: the systems are highly general-purpose and have so far been deployed widely for open-ended use, making it virtually impossible to assess all possible risk scenarios individually. Cases may therefore need narrower scopes, such as cases for specific applications rather than general release.

Objectives operationalise what it means for the system to be safe enough. A typical objective is a risk threshold — the paper's worked example is a probability of ≥10⁻⁷ per year of causing an event with ≥1,000 fatalities. It notes that other industries commonly set multiple thresholds: an unacceptable-risk region that must not be exceeded, a region in which risks must be "as low as reasonably practicable," and an acceptable region requiring no further mitigation. Objectives can also be comparative, as in the autonomous-vehicle convention that a self-driving car must be at least as safe as a human driver. In a regulatory context objectives are typically set by regulators but can also be selected by developers, in which case reviewers should assess the choice. Early frontier AI cases may use broad qualitative objectives, may be restricted to a few key risk domains rather than global risk, and may use proxy-based objectives such as capability thresholds that measure risk only indirectly; the authors expect objectives to become more specific, quantitative, and directly risk-based over time, with sub-objectives for specific risks as risk analysis improves.

Arguments break the objective into subclaims that, if true, collectively imply it has been met, and explain what evidence supports those subclaims. The paper distinguishes three broad argument types — directly risk-based, comparison-based, and guideline-based — inspired by the European Rail Agency's three risk-acceptance principles. Within explicit risk estimates it adopts the four-way taxonomy of Clymer et al. (2024): inability (the system is not capable enough to cause serious harm), control (control measures prevent it from using its capabilities to cause serious harm), trustworthiness (it reliably does not cause serious harm), and deference (other AI systems have verified that it will not). The paper's worked sketch is an inability argument organised around five capability red lines — autonomous replication and improvement, power seeking, assisting weapon development, cyberattacks, and deception — with evidence drawn from evaluation results, external verification, expert forecasting, human uplift studies, and trend extrapolation. The authors state the sketch is illustrative and do not claim a case of that structure or substance would be sufficient or sound even for near-term systems.

The authors expect early cases to rely on inability arguments, which are comparatively well established: identifying and evaluating dangerous capabilities is a major focus of existing safety frameworks, and system cards arguably already make inability arguments. Open questions remain on how to incorporate post-deployment enhancements, account for defensive uses of capabilities, and address risks less closely tied to dangerous capabilities such as systemic risks or AI malfunction. In the longer run they argue inability arguments are unlikely to remain the main focus: if capabilities continue to accelerate, systems will eventually cross dangerous-capability thresholds, and cases must instead argue those capabilities will not be used to cause unacceptable harm. Control, trustworthiness, and deference arguments cannot yet be supported via well-established methods and have not been stress-tested in the real world, while the stakes by then will be higher. The paper notes that while developers choose the arguments, regulators may publish guidance on standard arguments they deem acceptable or preferable.

Evidence consists of observable facts supporting the claims. The paper expects a wide variety of types: empirical tests such as dangerous-capability evaluations, deployment experiments, and simulation exercises stress-testing preparedness plans; mathematical models such as predictions of post-deployment capability gains; formal verification or proofs that the system satisfies certain requirements; expert judgements from surveys or interviews; and documentation of plans, policies, and processes such as internal governance structures or incident-response plans. Reviewers should evaluate both the relevance and strength of evidence and selectively verify individual pieces, for example by reproducing select capability evaluations. Current techniques for evidencing inability arguments include expert elicitation, threat modeling, public input, human uplift studies, and forecasting; the authors describe these as relatively early-stage and expect theoretical threat models to be increasingly replaced by quantitative estimates backed by empirical evidence. They record significant disagreement about the validity of evaluations, while judging it plausible that current techniques are adequate for current systems.

Evidencing arguments beyond inability is described as much less scientifically understood. Reliance on empirical evidence about system behavior could become difficult if systems strategically modify their behavior to pass evaluations, so cases may increasingly need to rest on mechanistic understanding drawn from model internals or the training process — which the authors state even developers do not currently have. They name mechanistic interpretability, formal verification, control, and automated AI safety as research directions that may supply evidence sources, while describing these fields as nascent and requiring significantly more investment.

Relation to safety frameworks

A safety framework is a developer's plan for assessing and mitigating risks from its AI systems — an organization-level policy applying to all frontier systems, in contrast to a safety case, which is a system-level assessment. The paper records that 16 companies committed to publishing safety frameworks ahead of the 2025 AI Action Summit and that four had done so at the time of writing. Existing frameworks focus on identifying dangerous capabilities, setting capability thresholds above which additional mitigations would be required, explaining how capabilities will be measured, and outlining mitigation options.

A framework could form the basis of a case: the central argument could be that a system does not cross any of the capability thresholds the framework identifies, or that the developer has implemented the mitigations the framework promises. But the authors are explicit that verifying adherence is not sufficient — the case must also justify the framework, explaining why adhering to it implies avoiding unacceptable risk, including justifying the choice of capability thresholds and arguing that systems below them are safe enough to deploy in the chosen way. They identify three roles safety cases can play relative to frameworks: documenting adherence with respect to a specific system; allowing frameworks to build in more flexibility by supporting subjective outcome-based claims such as "the safeguards are sufficient to prevent unacceptable risk," which matters where highly specific pre-commitments are inappropriate for poorly understood future systems; and making frameworks more justified by requiring implicit assumptions to be made explicit.

Safety cases in self-regulation and in regulation

In self-regulation, the paper identifies three purposes: informing major decisions such as starting a training run or deploying a system, with cases produced by a designated safety-case team, evaluated and approved by an internal review team, and shared with leadership; ongoing risk management across the lifecycle, using the case as a framework for assessing how safe the system is and what additional safeguards are needed; and building trust with downstream developers, users, and governments. It notes that aerospace and autonomous-vehicle companies use safety cases for all three, contrasting primarily against a less structured approach of sharing individual pieces of evidence. The benefits given are that cases integrate different evidence sources into a single comprehensive assessment and are flexible enough to apply across a wide range of systems without mandating particular techniques; and that making implicit arguments explicit highlights reasoning flaws and assurance gaps, makes it easier to see how overall safety is affected if the evidence changes, helps stakeholders communicate about disagreements through a shared framework and set of assumptions, and explains the context and relevance of evidence to decision-makers not intimately familiar with the system. The downsides are costs of two kinds — risk-management costs, which apply only to developers not already meeting a high standard, and documentation costs, which the authors suggest can be reduced through safety-case templates and cross-industry guidance and standards.

In regulation, the paper identifies two roles. First, information sharing and transparency: regulators could require developers to submit safety cases when deploying frontier models without any formal power to penalise or restrict deployment, framed as an extension of existing reporting requirements. This could help regulators build capacity for governing AI more broadly, including writing rules and standards for systems behind the frontier, and could serve as a flexible interim measure while enforcement powers are scaled up only if evidence emerges that future frontier models pose serious risk. Second, compliance assessment against regulation requiring developers to manage risks — which the paper notes is already in place in the EU and may be adopted elsewhere. Here regulators could either set specific rules about how developers assess and mitigate risk, or allow developers to choose and use safety cases to assess whether they have done enough. The latter is described as the common pattern in nuclear power, on-shore and off-shore petrochemical installations, and rail, where a regulator must approve a safety case before granting a licence to build or operate. Safety cases could also be combined with other enforcement mechanisms such as liability — for example, used retrospectively to assess whether a developer was negligent in deploying a model.

The paper compares safety cases with specific rules across six dimensions. Cases are advantaged on flexibility and durability, since regulators can require them without a clear sense of what practices are adequate for current or future systems, whereas rules risk becoming outdated, hard to update, and locking in suboptimal practices; on innovation, since developers lack incentives to develop safety practices beyond what is codified under rules but are incentivised under safety cases to develop and use more effective and efficient practices, though this requires regulators capable of reviewing novel practices; and on allocation of responsibility, since developers rather than regulators bear the burden of specifying adequate practices, which is efficient given developers hold more information, resources, and AI expertise. Rules are advantaged on transparency, being public, clear-cut, and manageable in length where cases may not be fully public and may be long and technical, making third-party scrutiny harder — though the paper notes regulators can facilitate scrutiny by publishing guidance on how they review cases, explaining individual decisions, or subjecting themselves to auditing; on consistency of enforcement, since regulator judgment plays a key role in case review while rule compliance is assessed in a consistent, objective way; and on cost for the developer, since producing cases is resource-intensive and subjective review introduces legal uncertainty, though the paper adds that poorly specified rules could raise compliance costs by requiring inefficient or unnecessary practices. Two dimensions cut both ways: gamability, since rules can create loopholes or gaps while subjective review may let inadequate cases pass; and risk of regulatory capture, since a rulemaking process may be an easier lobbying target while subjective review leaves more room for undue favors and may leave regulators more reliant on industry for information and less able to develop independent expertise. On cost for the regulator, rule compliance is easier to verify but rules are more resource-intensive to develop and keep current, while case review requires a bespoke and subjective assessment though set-up costs may be lower.

The authors then set out six conditions under which safety cases are preferable to precise rules and assess how likely each is to hold for frontier AI. Four are judged likely: difficulty specifying rules (frontier systems are novel and rapidly evolving, highly general-purpose, complex and not mechanistically understood, heterogeneous and lacking standard designs); information asymmetry (AI expertise is naturally concentrated in frontier AI companies, though several governments have established AI Safety Institutes to boost capacity, and some tacit knowledge may be difficult to convey even with information-sharing requirements); few covered developers (frontier development is resource-intensive in compute, energy, capital, and specialized expertise, though it is unclear whether this will continue to be true); and covered developers being able to absorb costs. Rule lock-in is judged unclear and jurisdiction-dependent. High risk is judged possible, with the paper recording significant expert disagreement about the risks frontier AI poses while stating it seems difficult to rule out that future systems may pose high risk. The authors also list conditions they judge relevant but not clearly favoring either approach, including regulator capacity — rules requiring more capacity when the regulation is written, principles more when it is enforced — and the degree of alignment between regulator and regulated objectives.

They conclude that the conditions of the frontier AI industry favor using safety cases to assess when deployment poses unacceptable risk, on three grounds: the industry is rapidly evolving, complex, and poorly understood, so a flexible approach is suitable; cases capitalize on developer expertise and resources; and with only a handful of relatively well-resourced developers the compliance costs seem bearable. They add that regulators may want to complement cases with specific rules — specifying rules in better-understood risk domains, mandating particular safeguards, or continuously replacing cases with precise rules as the frontier moves, using cases to identify best practices then applied to future systems at a similar capability level.

Safety cases across the system lifecycle

The paper treats a safety case as a living document rather than a one-off event, following what it describes as best practice in the safety-case literature. During development the developer should gradually build the case, sketching the intended argument and using it to identify model-level safeguards such as data filtering or reinforcement learning, so that safety is embedded in the design rather than added at the end; the sketch also helps identify gaps where more work is needed, and the developer should keep records of evidence for later use. The authors note that in industries such as nuclear energy, where cases inform licensing, the regulator is commonly involved throughout, giving guidance and feedback on early drafts. The case is finalized and refined pre-deployment, then updated after deployment: new risks may emerge as the system is modified or used in unexpected ways, and real-world information may invalidate the developer's risk assessment. Developers should track key metrics or risk indicators bearing on the claims — for example, if the case estimates a harm-refusal technique's success rate, tracking the actual post-deployment rate — and identify conditions triggering an update, whether at regular intervals or in response to events such as a new jailbreak or significant post-deployment enhancements.

Implementation challenges

The paper separates technical challenges, relating to the content of cases, from institutional challenges, relating to the organizations and processes that review them. It states that if these are left unaddressed, using safety cases may be inadvisable, since it could result in conclusions that are unreliable, biased, and overly relied upon.

Technical. The first is developing and building consensus on methodology, difficult for three reasons: the idea is novel with no well-established methodology; frontier AI has many characteristics of complex systems, being opaque and general-purpose, which makes many known risk-assessment techniques hard to apply; and there will likely be significant disagreement about what constitutes an adequate case, since experts already disagree about how much assurance existing techniques provide and that disagreement will likely grow for future systems. The authors nonetheless argue developers should not delay producing cases until this is resolved, since writing them will likely be essential to progress on methodology — while cautioning that early cases will be somewhat experimental, that decision-makers should not rely on them until methodologies are more robust, and that regulators specifically may want to delay use until clearer expectations and standards can be communicated. The second is developing the safeguards needed to assure advanced future systems, where safeguards such as harm refusal have proved easy to circumvent through jailbreaking and may become less robust if systems deliberately act to subvert them, so that producing cases for such systems may require significant breakthroughs in the science of AI safety. On this the authors argue resolution is not a prerequisite for using safety cases in decision-making but a prerequisite for deploying systems with sufficiently dangerous capabilities: sufficiently advanced systems should not be deployed until it is possible to produce an adequate case.

Institutional. The first is implementing an appropriate review structure — setting a clear process, assigning roles and responsibilities, and creating the right incentive structures for producers and reviewers. Developers should ensure different organizational functions are responsible for creating, reviewing, and stress-testing cases, which the paper suggests could be allocated using the Three Lines Model. Regulators need to appoint or establish a body able to receive and store cases securely given their proprietary and sensitive content, and to consider checks and balances such as an appeals process or regular auditing of the review body. The second is building sufficient capacity and expertise in the review body, which would ideally have frontier-AI and risk-assessment expertise, capacity for in-depth review, information-gathering powers, and model access. The paper identifies a trade-off for internal review — more independent reviewers such as an ethics board may have less time and less hands-on expertise with frontier models than less independent ones such as an internal safety team — and notes that regulators may need funding to pay higher-than-usual salaries to attract private-sector talent. It draws from other industries the lesson that a capable review body is essential, since cases may otherwise be significantly affected by errors and confirmation bias, and suggests this may be more true for frontier AI given that novelty and complexity make review more subjective than usual; it recommends provisional bodies that initially provide advice or feedback before adopting decision-making powers, plus review guidelines of the kind common in other safety-case industries. The third is including third parties in writing and reviewing cases, which could address capacity constraints, gaps in reviewer expertise, and lack of independence. Third parties could supply evidence such as a governance audit report, be consulted on specific questions such as reproducing a model evaluation or reviewing a risk analysis in their domain, or review the entire case — and a regulator could receive both a safety case from the developer and a "risk case" or red-team of it produced by a third party to help reduce confirmation bias. The authors describe third-party involvement as a goal for a mature ecosystem rather than a prerequisite, and note that developing the ecosystem is itself a challenge regulators can support through funding or other financial incentives.

Policy recommendations

For developers: produce safety cases for the next generation of frontier AI systems, with early cases simply making explicit the arguments implicit in current model cards and safety frameworks; commit to not deploying future generations until a case has passed internal review, and consider making that commitment already for the next generation given uncertainty about capabilities and risks; build internal capacity to produce and review cases by assigning roles and responsibilities, hiring relevant expertise, setting up documentation processes, and determining how cases will be reviewed; start the case early in development and continue updating it after deployment, sketching it and gathering documentation already when planning a new training run; share cases with external stakeholders, especially governments such as AI Safety Institutes, and consider sharing potentially redacted versions with downstream developers, third-party research organizations, and the public; and participate in industry-wide development of methodology and best practices.

For governments: encourage companies to produce and share cases, including by securing voluntary commitments and setting up infrastructure such as memorandums of understanding with developers and a platform for securely receiving cases, and in the longer term offering feedback; support implementation by convening conversations and funding or conducting supporting research such as templates or case studies, lowering the cost to companies, and in the longer term engaging with industry and third parties to issue guidance on best practices; contribute to a third-party ecosystem through funding or other financial benefits to organizations such as auditors and third-party model evaluators; and consider using safety cases to assess compliance with existing or future safety requirements, taking into account industry conditions and the technical and institutional challenges — the paper judges this more likely appropriate if capabilities continue to advance rapidly, if basic best practices for safety cases have been established, and if the regulator has the capacity and expertise to review cases effectively.

Conclusion and further research

The paper concludes that safety cases would be a valuable addition to the frontier AI governance toolkit in both self-regulation and government regulation, on two grounds: they make an explicit, structured argument, which helps check whether risk assessment is comprehensive and valid; and they provide a flexible way to assess safety, which matters given that capabilities, safeguards, and understanding of frontier systems are rapidly developing. It states that producing rudimentary cases based on existing safety frameworks is already feasible, while significant research breakthroughs will likely be needed for future systems. The authors identify three research areas: technical research sketching safety-case arguments and identifying gaps where additional research is needed before a compelling case can be produced; technical and policy research on when and how developers should update cases after deployment, with possible lessons from software and autonomous vehicles; and policy research on review processes, including the role of third parties, accountability mechanisms, and regulator involvement during development and after deployment. They characterise the opaque inner workings and open-ended applications of frontier AI as making it particularly difficult to predict if and how such systems will cause harm, and call for a collaborative research effort across companies, governments, and civil society.

Relationships

• related: Managing Cyber Risks — safety cases provide an overarching framework within which the Frontier Model Forum's cyber thresholds and evaluations can operate.
• related: NIST AI RMF 1.0 — safety cases offer a structured argument format that complements the NIST risk management framework.
• related: CA SB 53 / NY RAISE Act — the transparency requirements in these bills could be operationalized through safety cases.
• related: AI Scheming — the paper's framework bears on scheming because a scheming model could actively undermine the evidence base on which a safety case rests, making such a case harder to establish.
• depends-on: Compute Thresholds — the paper's proxy-based objectives and its illustrative 10²⁶ FLOP scope statement rest on capability thresholds standing in for direct measurement of risk.
• related: AI Control — the control argument is one of the four argument types the paper adopts from Clymer et al. (2024).
• related: Responsible Scaling Policy (RSP) — an instance of the safety-framework category the paper distinguishes from system-level safety cases.
• related: AI Safety Frameworks Compared: RSP, Preparedness, NIST RMF, and Safety Cases — compares the developer frameworks whose implicit arguments the paper proposes making explicit.

Provenance

PDF converted to markdown on 2026-04-13. Source: Raw Sources/Safety Cases for Frontier AI.md. Expanded 2026-07-29 directly from the full raw text; the pre-expansion version is preserved at _meta/_revision-backups/2026-07-29/safety-cases-frontier-ai.md.