The Amazon Nova Family of Models: Technical Report and Model Card is the document in which Amazon's Artificial General Intelligence organisation describes the first generation of its Amazon Nova foundation models (Source: https://arxiv.org/abs/2506.12103). The arXiv version (2506.12103v1) is dated March 17, 2025; Amazon first published the report on its science site alongside the December 2024 launch of the models on Amazon Bedrock. The report covers five models: Nova Pro, described as a multimodal model with "the best combination of accuracy, speed, and cost"; Nova Lite, a low-cost multimodal model; Nova Micro, a text-only low-latency model; Nova Canvas, an image-generation model; and Nova Reel, a video-generation model. It does not cover the later Nova Premier or Nova 2 models, nor the March 2025 Nova Act agent system.
Models
Section 1 describes Pro, Lite and Micro as offering "frontier intelligence" in their price tiers, with Micro presented as setting new standards in its tier on MMLU, GPQA, MATH and BIG-Bench Hard, and Pro and Lite accepting text, images, documents and video as input. Context lengths are 128k tokens for Micro and 300k for Lite and Pro. The report lists speed, agentic capability (function calling and multi-step tool use), long-context understanding and customisation (fine-tuning and distillation on Bedrock) as design goals. Canvas is described as generating "professional grade images with rich customization controls", and Reel as offering "high-quality outputs, customization, and motion control". Section 6 covers training infrastructure; the report gives no parameter counts.
Evaluation approach
Section 2 reports benchmarks for Pro, Lite and Micro and for "select publicly-available models", citing existing public results where available and measuring the comparison models itself where not. Results measured by Amazon are marked "M" in the tables and were run "after Amazon Nova models completed training" via the Bedrock API (Claude and Meta models), the OpenAI API or the Gemini API. Where a result is an average of binary scores, the report gives a 95% confidence interval from a Gaussian approximation. Shot settings and prompting differ by model group and are listed beneath each group in the tables; prompts and scoring rules are given in Appendix B.
The benchmarks are grouped as core text capability (MMLU, ARC-C, DROP, GPQA, MATH, GSM8K, IFEval, BIG-Bench Hard, and FLORES-200 translation), core multimodal capability (MMMU, ChartQA, DocVQA, TextVQA, VATEX, EgoSchema), agentic workflows (Berkeley Function Calling Leaderboard v3 as of its November 17, 2024 update; VisualWebBench, MM-Mind2Web and GroundUI-1K), long context (a text needle-in-a-haystack test shown as a heatmap, SQuALITY and LVBench), functional expertise (HumanEval, FinQA and CRAG under Amazon's own scoring), and runtime performance in tokens per second. Section 3 evaluates Canvas with the automated TIFA and ImageReward metrics and with human pairwise preference against DALL·E 3 and Imagen 3; Section 4 evaluates Reel by human pairwise preference against Runway Gen3 Alpha and Luma 1.6.
Findings
On the core text table, Nova Pro is reported at 85.9 on MMLU (0-shot CoT), 94.8 on ARC-C, 46.9 on GPQA, 76.6 on MATH, 94.8 on GSM8K, 92.1 on IFEval and 86.9 on BBH; the report presents these as competitive with Claude 3.5 Sonnet, Gemini 1.5 Pro and GPT-4o at higher speed, and Micro as leading its tier. On multimodal benchmarks Nova Pro is reported at 61.7 on MMMU and 89.2 on ChartQA, below Claude 3.5 Sonnet on MMMU (70.4) and ChartQA (90.8). On BFCL v3, Nova Pro's overall accuracy of 68.4 is reported beside GPT-4o (Aug) at 68.9 and Claude 3.5 Sonnet (Jun) at 61.3. The report states that Nova Pro and Lite "surpass" other models on LVBench and on the agentic multimodal benchmarks, where several comparison values were measured by Amazon. Human raters preferred Nova Canvas to DALL·E 3 for overall image quality in 54.5% of pairs and to Imagen 3 in 48.2%; Nova Reel won 56.4% of video-quality comparisons against Gen3 Alpha and 51.1% against Luma 1.6, and 67.0% and 74.7% for video consistency.
Responsible AI
Section 5 sets out eight Responsible AI dimensions, says Amazon signed the White House voluntary commitments, and describes pre-training data curation, SFT and RLHF alignment, and runtime input and output moderation models. It names the public benchmarks BOLD, RealToxicityPrompts and MM-SafetyBench and proprietary dynamically-updated benchmarks, but reports no scores. Red teaming combined internal exercises (more than 300 attack techniques), external firms including ActiveFence (more than 9,700 adversarial prompts across 20 categories) and other named vendors covering CBRN and cybersecurity, and automated red teaming; findings are described as fed back into development without quantitative results.
Reported results
Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.
| Evaluation | Model | Score | Setting | Section |
|---|---|---|---|---|
| ARC-C | Amazon Nova Pro | 94.8% | 0-shot, accuracy; 95% CI ±1.3; developer-reported | 2.1.1, Table 1 |
| DROP | Amazon Nova Pro | 85.4% | 6-shot CoT, F1-score; 95% CI ±0.7; developer-reported | 2.1.1, Table 1 |
| IFEval | Amazon Nova Pro | 92.1% | 0-shot, instruction-level loose accuracy; 95% CI ±1.8; developer-reported | 2.1.1, Table 1 |
| BBH | Amazon Nova Pro | 86.9% | 3-shot CoT, accuracy; developer-reported | 2.1.1, Table 1 |
| ARC-C | Amazon Nova Lite | 92.4% | 0-shot, accuracy; 95% CI ±1.5; developer-reported | 2.1.1, Table 1 |
| DROP | Amazon Nova Lite | 80.2% | 6-shot CoT, F1-score; 95% CI ±0.8; developer-reported | 2.1.1, Table 1 |
| IFEval | Amazon Nova Lite | 89.7% | 0-shot, instruction-level loose accuracy; 95% CI ±2.1; developer-reported | 2.1.1, Table 1 |
| BBH | Amazon Nova Lite | 82.4% | 3-shot CoT, accuracy; developer-reported | 2.1.1, Table 1 |
| ARC-C | Amazon Nova Micro | 90.2% | 0-shot, accuracy; 95% CI ±1.7; developer-reported | 2.1.1, Table 1 |
| DROP | Amazon Nova Micro | 79.3% | 6-shot CoT, F1-score; 95% CI ±0.8; developer-reported | 2.1.1, Table 1 |
| IFEval | Amazon Nova Micro | 87.2% | 0-shot, instruction-level loose accuracy; 95% CI ±2.3; developer-reported | 2.1.1, Table 1 |
| BBH | Amazon Nova Micro | 79.5% | 3-shot CoT, accuracy; developer-reported | 2.1.1, Table 1 |
| ARC-C | Claude 3.5 Sonnet (Oct 2024) | 96.3% | 25-shot, accuracy; 95% CI ±1.1; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| DROP | Claude 3.5 Sonnet (Oct 2024) | 88.3% | 3-shot, F1-score; 95% CI ±0.6; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| IFEval | Claude 3.5 Sonnet (Oct 2024) | 90.2% | 0-shot, instruction-level loose accuracy; scoring methodology unspecified in Anthropic's report (asterisk); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| BBH | Claude 3.5 Sonnet (Oct 2024) | 93.2% | 3-shot CoT, accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| ARC-C | Claude 3.5 Haiku | 90.9% | 25-shot, accuracy; 95% CI ±1.6; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| DROP | Claude 3.5 Haiku | 83.1% | 3-shot, F1-score; 95% CI ±0.8; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| IFEval | Claude 3.5 Haiku | 85.9% | 0-shot, instruction-level loose accuracy; scoring methodology unspecified in Anthropic's report (asterisk); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| BBH | Claude 3.5 Haiku | 86.6% | 3-shot CoT, accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| ARC-C | Gemini 1.5 Pro (002) | 95.4% | 25-shot, accuracy; 95% CI ±1.2; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| DROP | Gemini 1.5 Pro (002) | 74.9% | 3-shot, F1-score; 95% CI ±0.9; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| IFEval | Gemini 1.5 Pro (002) | 91.7% | 0-shot, instruction-level loose accuracy; 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| BBH | Gemini 1.5 Pro (002) | 89.2% | 3-shot, accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| ARC-C | Gemini 1.5 Flash (002) | 94.3% | 25-shot, accuracy; 95% CI ±1.3; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| DROP | Gemini 1.5 Flash (002) | 78.4% | 3-shot, F1-score; 95% CI ±0.8; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| IFEval | Gemini 1.5 Flash (002) | 91.6% | 0-shot, instruction-level loose accuracy; 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| BBH | Gemini 1.5 Flash (002) | 85.5% | 3-shot, accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| ARC-C | Gemini 1.5 Flash 8B (001) | 88.7% | 25-shot, accuracy; 95% CI ±1.8; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| DROP | Gemini 1.5 Flash 8B (001) | 68.1% | 3-shot, F1-score; 95% CI ±0.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| IFEval | Gemini 1.5 Flash 8B (001) | 86.1% | 0-shot, instruction-level loose accuracy; 95% CI ±2.3; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| BBH | Gemini 1.5 Flash 8B (001) | 69.5% | 3-shot, accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| ARC-C | GPT-4o | 96.2% | 25-shot, accuracy; 95% CI ±1.1; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| DROP | GPT-4o | 83.4% | 3-shot, F1-score; 95% CI ±0.7; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| IFEval | GPT-4o | 89.8% | 0-shot, instruction-level loose accuracy; 95% CI ±2.1; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| BBH | GPT-4o | 83.0% | 3-shot, accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| ARC-C | GPT-4o mini | 92.3% | 25-shot, accuracy; 95% CI ±1.5; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| DROP | GPT-4o mini | 79.7% | 3-shot, F1-score; 95% CI ±0.8; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| IFEval | GPT-4o mini | 87.4% | 0-shot, instruction-level loose accuracy; 95% CI ±2.3; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| BBH | GPT-4o mini | 81.0% | 3-shot, accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| ARC-C | Llama 3.2 90B | 94.8% | 25-shot, accuracy; 95% CI ±1.3; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| IFEval | Llama 3.2 90B | 90.9% | -, instruction-level loose accuracy; 95% CI ±2.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| ARC-C | Llama 3.2 11B | 83.4% | 25-shot, accuracy; 95% CI ±2.1; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| IFEval | Llama 3.2 11B | 85.0% | -, instruction-level loose accuracy; 95% CI ±2.4; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| ARC-C | Llama 3.1 8B | 83.4% | 25-shot, accuracy; 95% CI ±2.1; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.1, Table 1 |
| IFEval | Llama 3.1 8B | 85.0% | -, instruction-level loose accuracy; 95% CI ±2.4; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 1 |
| FLORES-200 — en→Set1 spBleu | Amazon Nova Pro | 43.4 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | Amazon Nova Pro | 89.1 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | Amazon Nova Pro | 44.4 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | Amazon Nova Pro | 89.0 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 spBleu | Amazon Nova Lite | 41.5 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | Amazon Nova Lite | 88.8 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | Amazon Nova Lite | 43.1 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | Amazon Nova Lite | 88.8 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 spBleu | Amazon Nova Micro | 40.2 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | Amazon Nova Micro | 88.5 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | Amazon Nova Micro | 42.6 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | Amazon Nova Micro | 88.7 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 spBleu | Claude 3.5 Sonnet (Oct 2024) | 42.5 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | Claude 3.5 Sonnet (Oct 2024) | 89.4 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | Claude 3.5 Sonnet (Oct 2024) | 43.5 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | Claude 3.5 Sonnet (Oct 2024) | 89.1 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 spBleu | Claude 3.5 Haiku | 40.0 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | Claude 3.5 Haiku | 88.5 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | Claude 3.5 Haiku | 40.2 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | Claude 3.5 Haiku | 88.3 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 spBleu | Gemini 1.5 Pro (002) | 43.0 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | Gemini 1.5 Pro (002) | 89.1 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | Gemini 1.5 Pro (002) | 45.6 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | Gemini 1.5 Pro (002) | 89.1 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 spBleu | Gemini 1.5 Flash (002) | 40.0 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | Gemini 1.5 Flash (002) | 88.5 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | Gemini 1.5 Flash (002) | 42.9 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | Gemini 1.5 Flash (002) | 88.8 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 spBleu | Gemini 1.5 Flash 8B (001) | 38.2 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | Gemini 1.5 Flash 8B (001) | 88.0 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | Gemini 1.5 Flash 8B (001) | 41.4 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | Gemini 1.5 Flash 8B (001) | 88.5 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 spBleu | GPT-4o | 43.1 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | GPT-4o | 89.2 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | GPT-4o | 43.9 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | GPT-4o | 89.0 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 spBleu | GPT-4o mini | 41.1 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | GPT-4o mini | 88.7 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | GPT-4o mini | 41.9 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | GPT-4o mini | 88.7 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 spBleu | Llama 3.2 90B | 39.7 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | Llama 3.2 90B | 88.2 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | Llama 3.2 90B | 43.7 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | Llama 3.2 90B | 88.5 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 spBleu | Llama 3.2 11B | 33.0 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | Llama 3.2 11B | 85.7 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | Llama 3.2 11B | 36.3 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | Llama 3.2 11B | 86.3 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 spBleu | Llama 3.1 8B | 32.7 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — en→Set1 COMET22 | Llama 3.1 8B | 85.5 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en spBleu | Llama 3.1 8B | 36.5 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| FLORES-200 — Set1→en COMET22 | Llama 3.1 8B | 86.5 | 0-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.1, Table 2 |
| ChartQA | Amazon Nova Pro | 89.2% | test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.2; developer-reported | 2.1.2, Table 3 |
| DocVQA | Amazon Nova Pro | 93.5% | test set, ANLS; developer-reported | 2.1.2, Table 3 |
| TextVQA | Amazon Nova Pro | 81.5% | validation set, weighted accuracy; developer-reported | 2.1.2, Table 3 |
| VATEX | Amazon Nova Pro | 77.8 | public test set, CIDEr; developer-reported | 2.1.2, Table 3 |
| EgoSchema | Amazon Nova Pro | 72.1% | test set, accuracy; 95% CI ±5.4; developer-reported | 2.1.2, Table 3 |
| ChartQA | Amazon Nova Lite | 86.8% | test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.3; developer-reported | 2.1.2, Table 3 |
| DocVQA | Amazon Nova Lite | 92.4% | test set, ANLS; developer-reported | 2.1.2, Table 3 |
| TextVQA | Amazon Nova Lite | 80.2% | validation set, weighted accuracy; developer-reported | 2.1.2, Table 3 |
| VATEX | Amazon Nova Lite | 77.8 | public test set, CIDEr; developer-reported | 2.1.2, Table 3 |
| EgoSchema | Amazon Nova Lite | 71.4% | test set, accuracy; 95% CI ±5.4; developer-reported | 2.1.2, Table 3 |
| ChartQA | Claude 3.5 Sonnet (Oct 2024) | 90.8% | test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.1; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| DocVQA | Claude 3.5 Sonnet (Oct 2024) | 94.2% | test set, ANLS; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| TextVQA | Claude 3.5 Sonnet (Oct 2024) | 61.7% | validation set, weighted accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.2, Table 3 |
| ChartQA | Claude 3 Haiku | 82.0% | test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.5; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| DocVQA | Claude 3 Haiku | 88.8% | test set, ANLS; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| ChartQA | Gemini 1.5 Pro (001) | 87.2% | test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.3; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| DocVQA | Gemini 1.5 Pro (001) | 93.1% | test set, ANLS; external OCR used; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| TextVQA | Gemini 1.5 Pro (001) | 78.7% | validation set, weighted accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| VATEX | Gemini 1.5 Pro (001) | 64.6 | public test set, CIDEr; 4-shot; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| EgoSchema | Gemini 1.5 Pro (001) | 72.2% | test set, accuracy; 95% CI ±5.4; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| ChartQA | Gemini 1.5 Flash (001) | 85.4% | test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.4; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| DocVQA | Gemini 1.5 Flash (001) | 89.9% | test set, ANLS; external OCR used; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| TextVQA | Gemini 1.5 Flash (001) | 78.7% | validation set, weighted accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| VATEX | Gemini 1.5 Flash (001) | 57.1 | public test set, CIDEr; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| EgoSchema | Gemini 1.5 Flash (001) | 65.7% | test set, accuracy; 95% CI ±5.7; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| ChartQA | Gemini 1.5 Flash 8B (001) | 78.2% | test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.6; reported in ref. 4; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| DocVQA | Gemini 1.5 Flash 8B (001) | 73.6% | test set, ANLS; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| TextVQA | Gemini 1.5 Flash 8B (001) | 66.7% | validation set, weighted accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| VATEX | Gemini 1.5 Flash 8B (001) | 53.2 | public test set, CIDEr; 4-shot; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| ChartQA | GPT-4o (May 2024) | 85.7% | test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.4; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| DocVQA | GPT-4o (May 2024) | 92.8% | test set, ANLS; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| TextVQA | GPT-4o (May 2024) | 77.2% | validation set, weighted accuracy; GPT-4o (Nov); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| EgoSchema | GPT-4o (May 2024) | 72.2% | test set, accuracy; 95% CI ±5.4; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| ChartQA | GPT-4o mini (Jul 2024) | 79.2% | test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.6; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.2, Table 3 |
| TextVQA | GPT-4o mini (Jul 2024) | 70.3% | validation set, weighted accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.2, Table 3 |
| ChartQA | Llama 3.2 90B | 85.5% | test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.4; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| DocVQA | Llama 3.2 90B | 90.1% | test set, ANLS; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| TextVQA | Llama 3.2 90B | 80.7% | validation set, weighted accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.2, Table 3 |
| ChartQA | Llama 3.2 11B | 83.4% | test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.5; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| DocVQA | Llama 3.2 11B | 88.4% | test set, ANLS; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.1.2, Table 3 |
| TextVQA | Llama 3.2 11B | 71.3% | validation set, weighted accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.1.2, Table 3 |
| VisualWebBench | Amazon Nova Pro | 79.7% | composite (macro average over metrics); developer-reported | 2.2.2, Table 5 |
| MM-Mind2Web | Amazon Nova Pro | 63.7% | step accuracy; developer-reported | 2.2.2, Table 5 |
| GroundUI-1K | Amazon Nova Pro | 81.4% | accuracy; developer-reported | 2.2.2, Table 5 |
| VisualWebBench | Amazon Nova Lite | 77.7% | composite (macro average over metrics); developer-reported | 2.2.2, Table 5 |
| MM-Mind2Web | Amazon Nova Lite | 60.7% | step accuracy; developer-reported | 2.2.2, Table 5 |
| GroundUI-1K | Amazon Nova Lite | 80.2% | accuracy; developer-reported | 2.2.2, Table 5 |
| VisualWebBench | Claude 3.5 Sonnet (Oct 2024) | 76.7% | composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| MM-Mind2Web | Claude 3.5 Sonnet (Oct 2024) | 61.6% | step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| GroundUI-1K | Claude 3.5 Sonnet (Oct 2024) | 16.3% | accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.2.2, Table 5 |
| VisualWebBench | GPT-4o (Nov 2024) | 77.5% | composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| MM-Mind2Web | GPT-4o (Nov 2024) | 55.0% | step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| GroundUI-1K | GPT-4o (Nov 2024) | 13.4% | accuracy; GPT-4o (May); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.2.2, Table 5 |
| VisualWebBench | GPT-4o mini (Jul 2024) | 71.3% | composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| MM-Mind2Web | GPT-4o mini (Jul 2024) | 58.6% | step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| GroundUI-1K | GPT-4o mini (Jul 2024) | 7.2% | accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| VisualWebBench | GPT-4 (Apr 2024) | 64.6% | composite (macro average over metrics); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.2.2, Table 5 |
| MM-Mind2Web | GPT-4 (Apr 2024) | 36.8% | step accuracy; uses in-context learning; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.2.2, Table 5 |
| VisualWebBench | Gemini 1.5 Pro (002) | 76.4% | composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| MM-Mind2Web | Gemini 1.5 Pro (002) | 58.4% | step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| GroundUI-1K | Gemini 1.5 Pro (002) | 35.2% | accuracy; Gemini 1.5 Pro (001); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.2.2, Table 5 |
| VisualWebBench | Gemini 1.5 Flash (002) | 76.1% | composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| MM-Mind2Web | Gemini 1.5 Flash (002) | 46.2% | step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| GroundUI-1K | Gemini 1.5 Flash (002) | 59.9% | accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| VisualWebBench | Gemini 1.0 Pro (001) | 48.0% | composite (macro average over metrics); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.2.2, Table 5 |
| MM-Mind2Web | Gemini 1.0 Pro (001) | 17.9% | step accuracy; uses in-context learning; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report | 2.2.2, Table 5 |
| VisualWebBench | Llama 3.2 90B | 73.2% | composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| MM-Mind2Web | Llama 3.2 90B | 21.6% | step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| GroundUI-1K | Llama 3.2 90B | 8.3% | accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| VisualWebBench | Llama 3.2 11B | 65.1% | composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| MM-Mind2Web | Llama 3.2 11B | 22.1% | step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| GroundUI-1K | Llama 3.2 11B | 3.7% | accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.2.2, Table 5 |
| SQuALITY | Amazon Nova Pro | 19.8 | ROUGE-L, ZeroScrolls; 95% CI ±8.7; developer-reported | 2.3, Table 6 |
| LVBench | Amazon Nova Pro | 41.6% | accuracy; 95% CI ±2.5; developer-reported | 2.3, Table 6 |
| SQuALITY | Amazon Nova Lite | 19.2 | ROUGE-L, ZeroScrolls; 95% CI ±8.6; developer-reported | 2.3, Table 6 |
| LVBench | Amazon Nova Lite | 40.4% | accuracy; 95% CI ±2.4; developer-reported | 2.3, Table 6 |
| SQuALITY | Amazon Nova Micro | 18.8 | ROUGE-L, ZeroScrolls; 95% CI ±8.6; developer-reported | 2.3, Table 6 |
| SQuALITY | Claude 3.5 Sonnet (Jun 2024) | 13.4 | ROUGE-L, ZeroScrolls; 95% CI ±7.5; taken from the Llama 3 report, as reprinted in the Amazon Nova technical report | 2.3, Table 6 |
| LVBench | Gemini 1.5 Pro (001) | 33.1% | accuracy; 95% CI ±2.3; taken from the LVBench leaderboard, as reprinted in the Amazon Nova technical report | 2.3, Table 6 |
| SQuALITY | Gemini 1.5 Pro (002) | 19.1 | ROUGE-L, ZeroScrolls; 95% CI ±8.6; measured by Amazon; reported in the Amazon Nova technical report | 2.3, Table 6 |
| SQuALITY | Gemini 1.5 Flash (002) | 18.1 | ROUGE-L, ZeroScrolls; 95% CI ±8.4; measured by Amazon; reported in the Amazon Nova technical report | 2.3, Table 6 |
| SQuALITY | GPT-4o | 18.8 | ROUGE-L, ZeroScrolls; 95% CI ±8.6; taken from the Llama 3 report, as reprinted in the Amazon Nova technical report | 2.3, Table 6 |
| LVBench | GPT-4o | 30.8% | accuracy; 95% CI ±2.3; taken from the LVBench leaderboard, as reprinted in the Amazon Nova technical report | 2.3, Table 6 |
| SQuALITY | Llama 3 70B | 16.4 | ROUGE-L, ZeroScrolls; 95% CI ±8.1; taken from the Llama 3 report, as reprinted in the Amazon Nova technical report | 2.3, Table 6 |
| SQuALITY | Llama 3 8B | 15.3 | ROUGE-L, ZeroScrolls; 95% CI ±7.9; taken from the Llama 3 report, as reprinted in the Amazon Nova technical report | 2.3, Table 6 |
| FinQA | Amazon Nova Pro | 77.2% | 0-shot accuracy; 95% CI ±0.9; developer-reported | 2.4, Table 7 |
| CRAG | Amazon Nova Pro | 50.3% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; developer-reported | 2.4, Table 7 |
| FinQA | Amazon Nova Lite | 73.6% | 0-shot accuracy; 95% CI ±0.9; developer-reported | 2.4, Table 7 |
| CRAG | Amazon Nova Lite | 43.8% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; developer-reported | 2.4, Table 7 |
| FinQA | Amazon Nova Micro | 65.2% | 0-shot accuracy; 95% CI ±1.0; developer-reported | 2.4, Table 7 |
| CRAG | Amazon Nova Micro | 43.1% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; developer-reported | 2.4, Table 7 |
| FinQA | Claude 3.5 Sonnet (Oct 2024) | 77.3% | 0-shot accuracy; 95% CI ±0.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| CRAG | Claude 3.5 Sonnet (Oct 2024) | 52.6% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.8; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| FinQA | Claude 3.5 Haiku | 73.9% | 0-shot accuracy; 95% CI ±0.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| CRAG | Claude 3.5 Haiku | 31.9% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.8; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| FinQA | Gemini 1.5 Pro (002) | 74.4% | 0-shot accuracy; 95% CI ±0.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| CRAG | Gemini 1.5 Pro (002) | 48.9% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| FinQA | Gemini 1.5 Flash (002) | 73.5% | 0-shot accuracy; 95% CI ±1.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| CRAG | Gemini 1.5 Flash (002) | 42.4% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| FinQA | Gemini 1.5 Flash 8B (001) | 63.7% | 0-shot accuracy; 95% CI ±1.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| CRAG | Gemini 1.5 Flash 8B (001) | 37.7% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.8; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| FinQA | GPT-4o | 71.1% | 0-shot accuracy; 95% CI ±1.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| CRAG | GPT-4o | 52.0% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| FinQA | GPT-4o mini | 70.6% | 0-shot accuracy; 95% CI ±1.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| CRAG | GPT-4o mini | 49.9% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| FinQA | Llama 3.2 90B | 72.8% | 0-shot accuracy; 95% CI ±1.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| CRAG | Llama 3.2 90B | 45.2% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| FinQA | Llama 3.2 11B | 60.8% | 0-shot accuracy; 95% CI ±1.1; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| CRAG | Llama 3.2 11B | 42.2% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| FinQA | Llama 3.1 8B | 61.2% | 0-shot accuracy; 95% CI ±1.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| CRAG | Llama 3.1 8B | 42.2% | RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.8; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report | 2.4, Table 7 |
| TIFA | Amazon Nova Canvas | 0.897 | text-to-image faithfulness; developer-reported | 3.1, Table 8 |
| ImageReward | Amazon Nova Canvas | 1.250 | human-preference reward model score; developer-reported | 3.1, Table 8 |
| TIFA | DALL.E 3 | 0.863 | text-to-image faithfulness; measured by Amazon; reported in the Amazon Nova technical report | 3.1, Table 8 |
| ImageReward | DALL.E 3 | 1.052 | human-preference reward model score; measured by Amazon; reported in the Amazon Nova technical report | 3.1, Table 8 |
| TIFA | Stable Diffusion 3.5 Large | 0.891 | text-to-image faithfulness; measured by Amazon; reported in the Amazon Nova technical report | 3.1, Table 8 |
| ImageReward | Stable Diffusion 3.5 Large | 1.082 | human-preference reward model score; measured by Amazon; reported in the Amazon Nova technical report | 3.1, Table 8 |
| TIFA | Stable Diffusion 3 Medium | 0.881 | text-to-image faithfulness; measured by Amazon; reported in the Amazon Nova technical report | 3.1, Table 8 |
| ImageReward | Stable Diffusion 3 Medium | 0.952 | human-preference reward model score; measured by Amazon; reported in the Amazon Nova technical report | 3.1, Table 8 |
| TIFA | Flux Pro 1.0 | 0.875 | text-to-image faithfulness; measured by Amazon; reported in the Amazon Nova technical report | 3.1, Table 8 |
| ImageReward | Flux Pro 1.0 | 1.075 | human-preference reward model score; measured by Amazon; reported in the Amazon Nova technical report | 3.1, Table 8 |
| TIFA | Flux Schnell | 0.882 | text-to-image faithfulness; measured by Amazon; reported in the Amazon Nova technical report | 3.1, Table 8 |
| ImageReward | Flux Schnell | 0.999 | human-preference reward model score; measured by Amazon; reported in the Amazon Nova technical report | 3.1, Table 8 |
| Human pairwise preference — Overall preference (image quality) — vs. DALL·E 3 | Amazon Nova Canvas | 54.5% | win rate; human raters, pairwise; developer-reported | 3.2, Table 9 |
| Human pairwise preference — Overall preference (image quality) — vs. DALL·E 3 | Amazon Nova Canvas | 6.4% | tie rate; human raters, pairwise; developer-reported | 3.2, Table 9 |
| Human pairwise preference — Overall preference (image quality) — vs. DALL·E 3 | Amazon Nova Canvas | 39.1% | loss rate; human raters, pairwise; developer-reported | 3.2, Table 9 |
| Human pairwise preference — Overall preference (image quality) — vs. Imagen 3 | Amazon Nova Canvas | 48.2% | win rate; human raters, pairwise; developer-reported | 3.2, Table 9 |
| Human pairwise preference — Overall preference (image quality) — vs. Imagen 3 | Amazon Nova Canvas | 5.3% | tie rate; human raters, pairwise; developer-reported | 3.2, Table 9 |
| Human pairwise preference — Overall preference (image quality) — vs. Imagen 3 | Amazon Nova Canvas | 46.5% | loss rate; human raters, pairwise; developer-reported | 3.2, Table 9 |
| Human pairwise preference — Instruction following (text-image alignment) — vs. DALL·E 3 | Amazon Nova Canvas | 39.4% | win rate; human raters, pairwise; developer-reported | 3.2, Table 9 |
| Human pairwise preference — Instruction following (text-image alignment) — vs. DALL·E 3 | Amazon Nova Canvas | 22.5% | tie rate; human raters, pairwise; developer-reported | 3.2, Table 9 |
| Human pairwise preference — Instruction following (text-image alignment) — vs. DALL·E 3 | Amazon Nova Canvas | 38.1% | loss rate; human raters, pairwise; developer-reported | 3.2, Table 9 |
| Human pairwise preference — Instruction following (text-image alignment) — vs. Imagen 3 | Amazon Nova Canvas | 38.4% | win rate; human raters, pairwise; developer-reported | 3.2, Table 9 |
| Human pairwise preference — Instruction following (text-image alignment) — vs. Imagen 3 | Amazon Nova Canvas | 28.1% | tie rate; human raters, pairwise; developer-reported | 3.2, Table 9 |
| Human pairwise preference — Instruction following (text-image alignment) — vs. Imagen 3 | Amazon Nova Canvas | 33.5% | loss rate; human raters, pairwise; developer-reported | 3.2, Table 9 |
| Human pairwise preference — Video Quality — vs. Runway Gen3 Alpha | Amazon Nova Reel | 56.4% | win rate; human raters, pairwise; developer-reported | 4.3, Table 10 |
| Human pairwise preference — Video Quality — vs. Runway Gen3 Alpha | Amazon Nova Reel | 9.9% | tie rate; human raters, pairwise; developer-reported | 4.3, Table 10 |
| Human pairwise preference — Video Quality — vs. Runway Gen3 Alpha | Amazon Nova Reel | 33.7% | loss rate; human raters, pairwise; developer-reported | 4.3, Table 10 |
| Human pairwise preference — Video Quality — vs. Luma 1.6 | Amazon Nova Reel | 51.1% | win rate; human raters, pairwise; developer-reported | 4.3, Table 10 |
| Human pairwise preference — Video Quality — vs. Luma 1.6 | Amazon Nova Reel | 3.4% | tie rate; human raters, pairwise; developer-reported | 4.3, Table 10 |
| Human pairwise preference — Video Quality — vs. Luma 1.6 | Amazon Nova Reel | 45.5% | loss rate; human raters, pairwise; developer-reported | 4.3, Table 10 |
| Human pairwise preference — Video Consistency — vs. Runway Gen3 Alpha | Amazon Nova Reel | 67.0% | win rate; human raters, pairwise; developer-reported | 4.3, Table 10 |
| Human pairwise preference — Video Consistency — vs. Runway Gen3 Alpha | Amazon Nova Reel | 9.1% | tie rate; human raters, pairwise; developer-reported | 4.3, Table 10 |
| Human pairwise preference — Video Consistency — vs. Runway Gen3 Alpha | Amazon Nova Reel | 23.9% | loss rate; human raters, pairwise; developer-reported | 4.3, Table 10 |
| Human pairwise preference — Video Consistency — vs. Luma 1.6 | Amazon Nova Reel | 74.7% | win rate; human raters, pairwise; developer-reported | 4.3, Table 10 |
| Human pairwise preference — Video Consistency — vs. Luma 1.6 | Amazon Nova Reel | 5.1% | tie rate; human raters, pairwise; developer-reported | 4.3, Table 10 |
| Human pairwise preference — Video Consistency — vs. Luma 1.6 | Amazon Nova Reel | 20.2% | loss rate; human raters, pairwise; developer-reported | 4.3, Table 10 |
Provenance
The page summarises the arXiv HTML (v1, March 17, 2025) as captured on September 21, 2026. All Nova figures are Amazon's own; comparison figures are either the other developers' published numbers or Amazon's post-training measurements via public APIs, as marked in each table. Confidence is medium: a single developer-authored source with fast-decay benchmark claims.
Relationships
- supports: Amazon Nova
- instance-of: AI System Cards
- related: Amazon, Berkeley Function Calling Leaderboard (BFCL), MMLU, GPQA, HumanEval, Claude 3.5 Haiku, Llama 3 (Meta), GPT-4 Family (OpenAI)