Apple Intelligence Foundation Language Models (Apple, July 2024)

Apple Intelligence Foundation Language Models is Apple's technical report on the foundation language models that power the first generation of Apple Intelligence, announced at the 2024 Worldwide Developers Conference for iOS 18, iPadOS 18 and macOS Sequoia (Source: https://arxiv.org/abs/2407.21075). Dated July 2024 and posted to arXiv on July 29, 2024, it describes two models: AFM-on-device, a language model of about 3 billion parameters designed to run on Apple devices, and AFM-server, a larger model designed for Private Cloud Compute. The report is the primary technical record for the models described on the Apple Foundation Models page; a separate 2025 report covers the following generation and is not summarised here.

Architecture and training

The models are dense decoder-only transformers with shared input/output embeddings, pre-normalisation with RMSNorm, query/key normalisation, grouped-query attention, SwiGLU activations and RoPE with the base frequency set to 500k for long context. Pre-training data comes from Applebot web crawls that respect robots.txt and exclude profanity and unsafe material, licensed publisher datasets, license-filtered code from GitHub, mathematics content and selected public datasets; the report states that no private Apple user data is used and that personally identifiable information is filtered. AFM-server is pre-trained from scratch on 6.3T tokens; AFM-on-device is initialised from a pruned 6.4B model and distilled. Both then undergo continued pre-training and context lengthening to 32k tokens. Post-training combines supervised fine-tuning on human demonstrations and synthetic mathematics, tool-use and coding data with reinforcement learning from human feedback, using a reward model, an "iterative teaching committee" (iTeC) of models, rejection-sampling distillation, and an online RLHF algorithm combining mirror descent policy optimization with a leave-one-out advantage estimator (MDLOO). Section 5 describes the LoRA adapter architecture that specialises the on-device model per feature, mixed 2-bit/4-bit quantization that the report says compresses AFM-on-device to about 3.5 bits per weight on average, and "accuracy-recovery" adapters that restore quantization losses; a summarization case study describes synthetic training summaries and prompt-injection filtering.

Evaluation approach

Section 6 reports pre-training results on the HELM MMLU 5-shot harness, a subset of the Open LLM Leaderboard v1, and HELM-Lite v1.5.0, with the note that many HELM-Lite settings "differ significantly from commonly used settings". Appendix C gives internal-harness results at each pre-training stage for both models, and Appendix D gives RULER long-context results for AFM-server at context lengths from 4,096 to 32,768 tokens. Post-training evaluation relies on human side-by-side comparison against open models (Phi-3, Gemma-1.1, Llama-3, Mistral, DBRX-Instruct) and commercial models (GPT-3.5 and GPT-4), plus IFEval, the Berkeley Function Calling Leaderboard, internal writing benchmarks, GSM8K and MATH, and a human study of the summarization feature; all of these are presented as charts. Appendix F reports the quantized and accuracy-recovered on-device model on IFEval, AlpacaEval 2.0 and GSM8K as percentages of the 16-bit score.

Findings

The report gives AFM-on-device 61.4 and AFM-server 75.4 on HELM MMLU 5-shot, and AFM-server 72.4 on GSM8K (5-shot), 69.7 on ARC-c, 86.9 on HellaSwag and 79.2 on Winogrande on the Open LLM Leaderboard subset. On the internal harness the final pre-trained AFM-server reaches 74.80 on MMLU (5-shot), 75.51 on GSM8K (8-shot CoT), 35.77 on MATH (4-shot CoT) and 39.55 pass@1 on HumanEval-Py; the on-device model reaches 60.64, 40.00, 15.48 and 30.84 respectively. RULER average accuracy for AFM-server falls from 91.7 at 4,096 tokens to 43.3 at 32,768. In human side-by-side evaluation the report states that AFM-on-device obtains a 47.7% win rate against Phi-3-mini "despite being 25% smaller" and outperforms Gemma-7B and Mistral-7B, and that AFM-server "achieves competitive performance" against closed models, with a win rate "of more than 50%" and a tie rate of 27.4% against GPT-3.5; the remaining human-evaluation, instruction-following, tool-use, writing and mathematics results appear only in figures. With accuracy-recovery adapters, the report says the 3.7-bit on-device model retains about 96% of its 16-bit GSM8K score.

Responsible AI

Section 7 sets out four Responsible AI principles, a safety taxonomy of 12 categories with 51 subcategories, pre-training data exclusions, safety-focused post-training data including adversarial examples, a treatment of malicious-code generation as a safety category, and manual and automatic red teaming, the latter including using language models themselves to identify gaps. Safety evaluation uses a curated set of more than a thousand adversarial prompts covering harmful content, sensitive topics and factuality, graded by humans; the report states that both models achieve violation rates "significantly lower than open-source and commercial models" and are preferred as "safe and helpful" in side-by-side comparison, but the rates are shown only in charts.

Reported results

Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.

Provenance

The page summarises the arXiv HTML (version 2, dated May 27, 2026, of a paper first posted July 29, 2024) as captured on September 21, 2026. All results are reported by Apple; comparison-model results in the figures are Apple's own runs. Confidence is medium: a single developer-authored source; the benchmark figures are more than two years old.

Relationships

• supports: Apple Foundation Models
• instance-of: AI System Cards
• related: Apple, MMLU, GSM8K, HumanEval, Berkeley Function Calling Leaderboard (BFCL)