AI Policy Wiki
Dashboard
⚲

The Amazon Nova Family of Models: Technical Report and Model Card (Amazon, 2024–2025)

medium confidence · updated 2026-09-21

Amazon's technical report and model card for the first Nova generation — Nova Pro, Lite and Micro (text and multimodal understanding) and Nova Canvas and Reel (image and video generation) — first published December 2024 and posted to arXiv as 2506.12103 on March 17, 2025. Reports core, agentic, long-context and functional benchmarks against Claude 3.5, Gemini 1.5, GPT-4o and Llama 3.x, human preference studies for Canvas and Reel, and a Responsible AI section without numeric results.

The Amazon Nova Family of Models: Technical Report and Model Card is the document in which Amazon's Artificial General Intelligence organisation describes the first generation of its Amazon Nova foundation models (Source: https://arxiv.org/abs/2506.12103). The arXiv version (2506.12103v1) is dated March 17, 2025; Amazon first published the report on its science site alongside the December 2024 launch of the models on Amazon Bedrock. The report covers five models: Nova Pro, described as a multimodal model with "the best combination of accuracy, speed, and cost"; Nova Lite, a low-cost multimodal model; Nova Micro, a text-only low-latency model; Nova Canvas, an image-generation model; and Nova Reel, a video-generation model. It does not cover the later Nova Premier or Nova 2 models, nor the March 2025 Nova Act agent system.

Models

Section 1 describes Pro, Lite and Micro as offering "frontier intelligence" in their price tiers, with Micro presented as setting new standards in its tier on MMLU, GPQA, MATH and BIG-Bench Hard, and Pro and Lite accepting text, images, documents and video as input. Context lengths are 128k tokens for Micro and 300k for Lite and Pro. The report lists speed, agentic capability (function calling and multi-step tool use), long-context understanding and customisation (fine-tuning and distillation on Bedrock) as design goals. Canvas is described as generating "professional grade images with rich customization controls", and Reel as offering "high-quality outputs, customization, and motion control". Section 6 covers training infrastructure; the report gives no parameter counts.

Evaluation approach

Section 2 reports benchmarks for Pro, Lite and Micro and for "select publicly-available models", citing existing public results where available and measuring the comparison models itself where not. Results measured by Amazon are marked "M" in the tables and were run "after Amazon Nova models completed training" via the Bedrock API (Claude and Meta models), the OpenAI API or the Gemini API. Where a result is an average of binary scores, the report gives a 95% confidence interval from a Gaussian approximation. Shot settings and prompting differ by model group and are listed beneath each group in the tables; prompts and scoring rules are given in Appendix B.

The benchmarks are grouped as core text capability (MMLU, ARC-C, DROP, GPQA, MATH, GSM8K, IFEval, BIG-Bench Hard, and FLORES-200 translation), core multimodal capability (MMMU, ChartQA, DocVQA, TextVQA, VATEX, EgoSchema), agentic workflows (Berkeley Function Calling Leaderboard v3 as of its November 17, 2024 update; VisualWebBench, MM-Mind2Web and GroundUI-1K), long context (a text needle-in-a-haystack test shown as a heatmap, SQuALITY and LVBench), functional expertise (HumanEval, FinQA and CRAG under Amazon's own scoring), and runtime performance in tokens per second. Section 3 evaluates Canvas with the automated TIFA and ImageReward metrics and with human pairwise preference against DALL·E 3 and Imagen 3; Section 4 evaluates Reel by human pairwise preference against Runway Gen3 Alpha and Luma 1.6.

Findings

On the core text table, Nova Pro is reported at 85.9 on MMLU (0-shot CoT), 94.8 on ARC-C, 46.9 on GPQA, 76.6 on MATH, 94.8 on GSM8K, 92.1 on IFEval and 86.9 on BBH; the report presents these as competitive with Claude 3.5 Sonnet, Gemini 1.5 Pro and GPT-4o at higher speed, and Micro as leading its tier. On multimodal benchmarks Nova Pro is reported at 61.7 on MMMU and 89.2 on ChartQA, below Claude 3.5 Sonnet on MMMU (70.4) and ChartQA (90.8). On BFCL v3, Nova Pro's overall accuracy of 68.4 is reported beside GPT-4o (Aug) at 68.9 and Claude 3.5 Sonnet (Jun) at 61.3. The report states that Nova Pro and Lite "surpass" other models on LVBench and on the agentic multimodal benchmarks, where several comparison values were measured by Amazon. Human raters preferred Nova Canvas to DALL·E 3 for overall image quality in 54.5% of pairs and to Imagen 3 in 48.2%; Nova Reel won 56.4% of video-quality comparisons against Gen3 Alpha and 51.1% against Luma 1.6, and 67.0% and 74.7% for video consistency.

Responsible AI

Section 5 sets out eight Responsible AI dimensions, says Amazon signed the White House voluntary commitments, and describes pre-training data curation, SFT and RLHF alignment, and runtime input and output moderation models. It names the public benchmarks BOLD, RealToxicityPrompts and MM-SafetyBench and proprietary dynamically-updated benchmarks, but reports no scores. Red teaming combined internal exercises (more than 300 attack techniques), external firms including ActiveFence (more than 9,700 adversarial prompts across 20 categories) and other named vendors covering CBRN and cybersecurity, and automated red teaming; findings are described as fed back into development without quantitative results.

Reported results

Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.

EvaluationModelScoreSettingSection
ARC-CAmazon Nova Pro94.8%0-shot, accuracy; 95% CI ±1.3; developer-reported2.1.1, Table 1
DROPAmazon Nova Pro85.4%6-shot CoT, F1-score; 95% CI ±0.7; developer-reported2.1.1, Table 1
IFEvalAmazon Nova Pro92.1%0-shot, instruction-level loose accuracy; 95% CI ±1.8; developer-reported2.1.1, Table 1
BBHAmazon Nova Pro86.9%3-shot CoT, accuracy; developer-reported2.1.1, Table 1
ARC-CAmazon Nova Lite92.4%0-shot, accuracy; 95% CI ±1.5; developer-reported2.1.1, Table 1
DROPAmazon Nova Lite80.2%6-shot CoT, F1-score; 95% CI ±0.8; developer-reported2.1.1, Table 1
IFEvalAmazon Nova Lite89.7%0-shot, instruction-level loose accuracy; 95% CI ±2.1; developer-reported2.1.1, Table 1
BBHAmazon Nova Lite82.4%3-shot CoT, accuracy; developer-reported2.1.1, Table 1
ARC-CAmazon Nova Micro90.2%0-shot, accuracy; 95% CI ±1.7; developer-reported2.1.1, Table 1
DROPAmazon Nova Micro79.3%6-shot CoT, F1-score; 95% CI ±0.8; developer-reported2.1.1, Table 1
IFEvalAmazon Nova Micro87.2%0-shot, instruction-level loose accuracy; 95% CI ±2.3; developer-reported2.1.1, Table 1
BBHAmazon Nova Micro79.5%3-shot CoT, accuracy; developer-reported2.1.1, Table 1
ARC-CClaude 3.5 Sonnet (Oct 2024)96.3%25-shot, accuracy; 95% CI ±1.1; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
DROPClaude 3.5 Sonnet (Oct 2024)88.3%3-shot, F1-score; 95% CI ±0.6; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
IFEvalClaude 3.5 Sonnet (Oct 2024)90.2%0-shot, instruction-level loose accuracy; scoring methodology unspecified in Anthropic's report (asterisk); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
BBHClaude 3.5 Sonnet (Oct 2024)93.2%3-shot CoT, accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
ARC-CClaude 3.5 Haiku90.9%25-shot, accuracy; 95% CI ±1.6; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
DROPClaude 3.5 Haiku83.1%3-shot, F1-score; 95% CI ±0.8; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
IFEvalClaude 3.5 Haiku85.9%0-shot, instruction-level loose accuracy; scoring methodology unspecified in Anthropic's report (asterisk); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
BBHClaude 3.5 Haiku86.6%3-shot CoT, accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
ARC-CGemini 1.5 Pro (002)95.4%25-shot, accuracy; 95% CI ±1.2; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
DROPGemini 1.5 Pro (002)74.9%3-shot, F1-score; 95% CI ±0.9; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
IFEvalGemini 1.5 Pro (002)91.7%0-shot, instruction-level loose accuracy; 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
BBHGemini 1.5 Pro (002)89.2%3-shot, accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
ARC-CGemini 1.5 Flash (002)94.3%25-shot, accuracy; 95% CI ±1.3; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
DROPGemini 1.5 Flash (002)78.4%3-shot, F1-score; 95% CI ±0.8; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
IFEvalGemini 1.5 Flash (002)91.6%0-shot, instruction-level loose accuracy; 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
BBHGemini 1.5 Flash (002)85.5%3-shot, accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
ARC-CGemini 1.5 Flash 8B (001)88.7%25-shot, accuracy; 95% CI ±1.8; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
DROPGemini 1.5 Flash 8B (001)68.1%3-shot, F1-score; 95% CI ±0.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
IFEvalGemini 1.5 Flash 8B (001)86.1%0-shot, instruction-level loose accuracy; 95% CI ±2.3; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
BBHGemini 1.5 Flash 8B (001)69.5%3-shot, accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
ARC-CGPT-4o96.2%25-shot, accuracy; 95% CI ±1.1; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
DROPGPT-4o83.4%3-shot, F1-score; 95% CI ±0.7; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
IFEvalGPT-4o89.8%0-shot, instruction-level loose accuracy; 95% CI ±2.1; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
BBHGPT-4o83.0%3-shot, accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
ARC-CGPT-4o mini92.3%25-shot, accuracy; 95% CI ±1.5; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
DROPGPT-4o mini79.7%3-shot, F1-score; 95% CI ±0.8; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
IFEvalGPT-4o mini87.4%0-shot, instruction-level loose accuracy; 95% CI ±2.3; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
BBHGPT-4o mini81.0%3-shot, accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
ARC-CLlama 3.2 90B94.8%25-shot, accuracy; 95% CI ±1.3; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
IFEvalLlama 3.2 90B90.9%-, instruction-level loose accuracy; 95% CI ±2.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
ARC-CLlama 3.2 11B83.4%25-shot, accuracy; 95% CI ±2.1; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
IFEvalLlama 3.2 11B85.0%-, instruction-level loose accuracy; 95% CI ±2.4; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
ARC-CLlama 3.1 8B83.4%25-shot, accuracy; 95% CI ±2.1; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.1, Table 1
IFEvalLlama 3.1 8B85.0%-, instruction-level loose accuracy; 95% CI ±2.4; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 1
FLORES-200 — en→Set1 spBleuAmazon Nova Pro43.40-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported2.1.1, Table 2
FLORES-200 — en→Set1 COMET22Amazon Nova Pro89.10-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported2.1.1, Table 2
FLORES-200 — Set1→en spBleuAmazon Nova Pro44.40-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported2.1.1, Table 2
FLORES-200 — Set1→en COMET22Amazon Nova Pro89.00-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported2.1.1, Table 2
FLORES-200 — en→Set1 spBleuAmazon Nova Lite41.50-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported2.1.1, Table 2
FLORES-200 — en→Set1 COMET22Amazon Nova Lite88.80-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported2.1.1, Table 2
FLORES-200 — Set1→en spBleuAmazon Nova Lite43.10-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported2.1.1, Table 2
FLORES-200 — Set1→en COMET22Amazon Nova Lite88.80-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported2.1.1, Table 2
FLORES-200 — en→Set1 spBleuAmazon Nova Micro40.20-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported2.1.1, Table 2
FLORES-200 — en→Set1 COMET22Amazon Nova Micro88.50-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported2.1.1, Table 2
FLORES-200 — Set1→en spBleuAmazon Nova Micro42.60-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported2.1.1, Table 2
FLORES-200 — Set1→en COMET22Amazon Nova Micro88.70-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; developer-reported2.1.1, Table 2
FLORES-200 — en→Set1 spBleuClaude 3.5 Sonnet (Oct 2024)42.50-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 COMET22Claude 3.5 Sonnet (Oct 2024)89.40-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en spBleuClaude 3.5 Sonnet (Oct 2024)43.50-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en COMET22Claude 3.5 Sonnet (Oct 2024)89.10-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 spBleuClaude 3.5 Haiku40.00-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 COMET22Claude 3.5 Haiku88.50-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en spBleuClaude 3.5 Haiku40.20-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en COMET22Claude 3.5 Haiku88.30-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 spBleuGemini 1.5 Pro (002)43.00-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 COMET22Gemini 1.5 Pro (002)89.10-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en spBleuGemini 1.5 Pro (002)45.60-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en COMET22Gemini 1.5 Pro (002)89.10-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 spBleuGemini 1.5 Flash (002)40.00-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 COMET22Gemini 1.5 Flash (002)88.50-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en spBleuGemini 1.5 Flash (002)42.90-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en COMET22Gemini 1.5 Flash (002)88.80-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 spBleuGemini 1.5 Flash 8B (001)38.20-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 COMET22Gemini 1.5 Flash 8B (001)88.00-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en spBleuGemini 1.5 Flash 8B (001)41.40-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en COMET22Gemini 1.5 Flash 8B (001)88.50-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 spBleuGPT-4o43.10-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 COMET22GPT-4o89.20-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en spBleuGPT-4o43.90-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en COMET22GPT-4o89.00-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 spBleuGPT-4o mini41.10-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 COMET22GPT-4o mini88.70-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en spBleuGPT-4o mini41.90-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en COMET22GPT-4o mini88.70-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; alternate prompt (Appendix B.1); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 spBleuLlama 3.2 90B39.70-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 COMET22Llama 3.2 90B88.20-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en spBleuLlama 3.2 90B43.70-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en COMET22Llama 3.2 90B88.50-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 spBleuLlama 3.2 11B33.00-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 COMET22Llama 3.2 11B85.70-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en spBleuLlama 3.2 11B36.30-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en COMET22Llama 3.2 11B86.30-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 spBleuLlama 3.1 8B32.70-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — en→Set1 COMET22Llama 3.1 8B85.50-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en spBleuLlama 3.1 8B36.50-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
FLORES-200 — Set1→en COMET22Llama 3.1 8B86.50-shot; Set1 = de, es, fr, it, pt, ja, ar, hi, ru, nl, tr, he, ko, zh; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.1, Table 2
ChartQAAmazon Nova Pro89.2%test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.2; developer-reported2.1.2, Table 3
DocVQAAmazon Nova Pro93.5%test set, ANLS; developer-reported2.1.2, Table 3
TextVQAAmazon Nova Pro81.5%validation set, weighted accuracy; developer-reported2.1.2, Table 3
VATEXAmazon Nova Pro77.8public test set, CIDEr; developer-reported2.1.2, Table 3
EgoSchemaAmazon Nova Pro72.1%test set, accuracy; 95% CI ±5.4; developer-reported2.1.2, Table 3
ChartQAAmazon Nova Lite86.8%test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.3; developer-reported2.1.2, Table 3
DocVQAAmazon Nova Lite92.4%test set, ANLS; developer-reported2.1.2, Table 3
TextVQAAmazon Nova Lite80.2%validation set, weighted accuracy; developer-reported2.1.2, Table 3
VATEXAmazon Nova Lite77.8public test set, CIDEr; developer-reported2.1.2, Table 3
EgoSchemaAmazon Nova Lite71.4%test set, accuracy; 95% CI ±5.4; developer-reported2.1.2, Table 3
ChartQAClaude 3.5 Sonnet (Oct 2024)90.8%test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.1; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
DocVQAClaude 3.5 Sonnet (Oct 2024)94.2%test set, ANLS; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
TextVQAClaude 3.5 Sonnet (Oct 2024)61.7%validation set, weighted accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.2, Table 3
ChartQAClaude 3 Haiku82.0%test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.5; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
DocVQAClaude 3 Haiku88.8%test set, ANLS; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
ChartQAGemini 1.5 Pro (001)87.2%test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.3; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
DocVQAGemini 1.5 Pro (001)93.1%test set, ANLS; external OCR used; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
TextVQAGemini 1.5 Pro (001)78.7%validation set, weighted accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
VATEXGemini 1.5 Pro (001)64.6public test set, CIDEr; 4-shot; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
EgoSchemaGemini 1.5 Pro (001)72.2%test set, accuracy; 95% CI ±5.4; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
ChartQAGemini 1.5 Flash (001)85.4%test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.4; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
DocVQAGemini 1.5 Flash (001)89.9%test set, ANLS; external OCR used; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
TextVQAGemini 1.5 Flash (001)78.7%validation set, weighted accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
VATEXGemini 1.5 Flash (001)57.1public test set, CIDEr; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
EgoSchemaGemini 1.5 Flash (001)65.7%test set, accuracy; 95% CI ±5.7; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
ChartQAGemini 1.5 Flash 8B (001)78.2%test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.6; reported in ref. 4; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
DocVQAGemini 1.5 Flash 8B (001)73.6%test set, ANLS; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
TextVQAGemini 1.5 Flash 8B (001)66.7%validation set, weighted accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
VATEXGemini 1.5 Flash 8B (001)53.2public test set, CIDEr; 4-shot; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
ChartQAGPT-4o (May 2024)85.7%test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.4; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
DocVQAGPT-4o (May 2024)92.8%test set, ANLS; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
TextVQAGPT-4o (May 2024)77.2%validation set, weighted accuracy; GPT-4o (Nov); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
EgoSchemaGPT-4o (May 2024)72.2%test set, accuracy; 95% CI ±5.4; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
ChartQAGPT-4o mini (Jul 2024)79.2%test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.6; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.2, Table 3
TextVQAGPT-4o mini (Jul 2024)70.3%validation set, weighted accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.2, Table 3
ChartQALlama 3.2 90B85.5%test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.4; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
DocVQALlama 3.2 90B90.1%test set, ANLS; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
TextVQALlama 3.2 90B80.7%validation set, weighted accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.2, Table 3
ChartQALlama 3.2 11B83.4%test set, relaxed accuracy; all models except Nova use CoT; 95% CI ±1.5; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
DocVQALlama 3.2 11B88.4%test set, ANLS; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.1.2, Table 3
TextVQALlama 3.2 11B71.3%validation set, weighted accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.1.2, Table 3
VisualWebBenchAmazon Nova Pro79.7%composite (macro average over metrics); developer-reported2.2.2, Table 5
MM-Mind2WebAmazon Nova Pro63.7%step accuracy; developer-reported2.2.2, Table 5
GroundUI-1KAmazon Nova Pro81.4%accuracy; developer-reported2.2.2, Table 5
VisualWebBenchAmazon Nova Lite77.7%composite (macro average over metrics); developer-reported2.2.2, Table 5
MM-Mind2WebAmazon Nova Lite60.7%step accuracy; developer-reported2.2.2, Table 5
GroundUI-1KAmazon Nova Lite80.2%accuracy; developer-reported2.2.2, Table 5
VisualWebBenchClaude 3.5 Sonnet (Oct 2024)76.7%composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
MM-Mind2WebClaude 3.5 Sonnet (Oct 2024)61.6%step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
GroundUI-1KClaude 3.5 Sonnet (Oct 2024)16.3%accuracy; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.2.2, Table 5
VisualWebBenchGPT-4o (Nov 2024)77.5%composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
MM-Mind2WebGPT-4o (Nov 2024)55.0%step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
GroundUI-1KGPT-4o (Nov 2024)13.4%accuracy; GPT-4o (May); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.2.2, Table 5
VisualWebBenchGPT-4o mini (Jul 2024)71.3%composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
MM-Mind2WebGPT-4o mini (Jul 2024)58.6%step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
GroundUI-1KGPT-4o mini (Jul 2024)7.2%accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
VisualWebBenchGPT-4 (Apr 2024)64.6%composite (macro average over metrics); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.2.2, Table 5
MM-Mind2WebGPT-4 (Apr 2024)36.8%step accuracy; uses in-context learning; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.2.2, Table 5
VisualWebBenchGemini 1.5 Pro (002)76.4%composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
MM-Mind2WebGemini 1.5 Pro (002)58.4%step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
GroundUI-1KGemini 1.5 Pro (002)35.2%accuracy; Gemini 1.5 Pro (001); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.2.2, Table 5
VisualWebBenchGemini 1.5 Flash (002)76.1%composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
MM-Mind2WebGemini 1.5 Flash (002)46.2%step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
GroundUI-1KGemini 1.5 Flash (002)59.9%accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
VisualWebBenchGemini 1.0 Pro (001)48.0%composite (macro average over metrics); taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.2.2, Table 5
MM-Mind2WebGemini 1.0 Pro (001)17.9%step accuracy; uses in-context learning; taken from the developer's own technical report or website, as reprinted in the Amazon Nova technical report2.2.2, Table 5
VisualWebBenchLlama 3.2 90B73.2%composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
MM-Mind2WebLlama 3.2 90B21.6%step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
GroundUI-1KLlama 3.2 90B8.3%accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
VisualWebBenchLlama 3.2 11B65.1%composite (macro average over metrics); measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
MM-Mind2WebLlama 3.2 11B22.1%step accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
GroundUI-1KLlama 3.2 11B3.7%accuracy; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.2.2, Table 5
SQuALITYAmazon Nova Pro19.8ROUGE-L, ZeroScrolls; 95% CI ±8.7; developer-reported2.3, Table 6
LVBenchAmazon Nova Pro41.6%accuracy; 95% CI ±2.5; developer-reported2.3, Table 6
SQuALITYAmazon Nova Lite19.2ROUGE-L, ZeroScrolls; 95% CI ±8.6; developer-reported2.3, Table 6
LVBenchAmazon Nova Lite40.4%accuracy; 95% CI ±2.4; developer-reported2.3, Table 6
SQuALITYAmazon Nova Micro18.8ROUGE-L, ZeroScrolls; 95% CI ±8.6; developer-reported2.3, Table 6
SQuALITYClaude 3.5 Sonnet (Jun 2024)13.4ROUGE-L, ZeroScrolls; 95% CI ±7.5; taken from the Llama 3 report, as reprinted in the Amazon Nova technical report2.3, Table 6
LVBenchGemini 1.5 Pro (001)33.1%accuracy; 95% CI ±2.3; taken from the LVBench leaderboard, as reprinted in the Amazon Nova technical report2.3, Table 6
SQuALITYGemini 1.5 Pro (002)19.1ROUGE-L, ZeroScrolls; 95% CI ±8.6; measured by Amazon; reported in the Amazon Nova technical report2.3, Table 6
SQuALITYGemini 1.5 Flash (002)18.1ROUGE-L, ZeroScrolls; 95% CI ±8.4; measured by Amazon; reported in the Amazon Nova technical report2.3, Table 6
SQuALITYGPT-4o18.8ROUGE-L, ZeroScrolls; 95% CI ±8.6; taken from the Llama 3 report, as reprinted in the Amazon Nova technical report2.3, Table 6
LVBenchGPT-4o30.8%accuracy; 95% CI ±2.3; taken from the LVBench leaderboard, as reprinted in the Amazon Nova technical report2.3, Table 6
SQuALITYLlama 3 70B16.4ROUGE-L, ZeroScrolls; 95% CI ±8.1; taken from the Llama 3 report, as reprinted in the Amazon Nova technical report2.3, Table 6
SQuALITYLlama 3 8B15.3ROUGE-L, ZeroScrolls; 95% CI ±7.9; taken from the Llama 3 report, as reprinted in the Amazon Nova technical report2.3, Table 6
FinQAAmazon Nova Pro77.2%0-shot accuracy; 95% CI ±0.9; developer-reported2.4, Table 7
CRAGAmazon Nova Pro50.3%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; developer-reported2.4, Table 7
FinQAAmazon Nova Lite73.6%0-shot accuracy; 95% CI ±0.9; developer-reported2.4, Table 7
CRAGAmazon Nova Lite43.8%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; developer-reported2.4, Table 7
FinQAAmazon Nova Micro65.2%0-shot accuracy; 95% CI ±1.0; developer-reported2.4, Table 7
CRAGAmazon Nova Micro43.1%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; developer-reported2.4, Table 7
FinQAClaude 3.5 Sonnet (Oct 2024)77.3%0-shot accuracy; 95% CI ±0.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
CRAGClaude 3.5 Sonnet (Oct 2024)52.6%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.8; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
FinQAClaude 3.5 Haiku73.9%0-shot accuracy; 95% CI ±0.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
CRAGClaude 3.5 Haiku31.9%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.8; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
FinQAGemini 1.5 Pro (002)74.4%0-shot accuracy; 95% CI ±0.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
CRAGGemini 1.5 Pro (002)48.9%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
FinQAGemini 1.5 Flash (002)73.5%0-shot accuracy; 95% CI ±1.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
CRAGGemini 1.5 Flash (002)42.4%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
FinQAGemini 1.5 Flash 8B (001)63.7%0-shot accuracy; 95% CI ±1.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
CRAGGemini 1.5 Flash 8B (001)37.7%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.8; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
FinQAGPT-4o71.1%0-shot accuracy; 95% CI ±1.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
CRAGGPT-4o52.0%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
FinQAGPT-4o mini70.6%0-shot accuracy; 95% CI ±1.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
CRAGGPT-4o mini49.9%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
FinQALlama 3.2 90B72.8%0-shot accuracy; 95% CI ±1.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
CRAGLlama 3.2 90B45.2%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
FinQALlama 3.2 11B60.8%0-shot accuracy; 95% CI ±1.1; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
CRAGLlama 3.2 11B42.2%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.9; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
FinQALlama 3.1 8B61.2%0-shot accuracy; 95% CI ±1.0; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
CRAGLlama 3.1 8B42.2%RAG; Amazon's own scoring method (2.4.3); 95% CI ±1.8; measured by Amazon after Nova training, via the Bedrock/OpenAI/Gemini API; reported in the Amazon Nova technical report2.4, Table 7
TIFAAmazon Nova Canvas0.897text-to-image faithfulness; developer-reported3.1, Table 8
ImageRewardAmazon Nova Canvas1.250human-preference reward model score; developer-reported3.1, Table 8
TIFADALL.E 30.863text-to-image faithfulness; measured by Amazon; reported in the Amazon Nova technical report3.1, Table 8
ImageRewardDALL.E 31.052human-preference reward model score; measured by Amazon; reported in the Amazon Nova technical report3.1, Table 8
TIFAStable Diffusion 3.5 Large0.891text-to-image faithfulness; measured by Amazon; reported in the Amazon Nova technical report3.1, Table 8
ImageRewardStable Diffusion 3.5 Large1.082human-preference reward model score; measured by Amazon; reported in the Amazon Nova technical report3.1, Table 8
TIFAStable Diffusion 3 Medium0.881text-to-image faithfulness; measured by Amazon; reported in the Amazon Nova technical report3.1, Table 8
ImageRewardStable Diffusion 3 Medium0.952human-preference reward model score; measured by Amazon; reported in the Amazon Nova technical report3.1, Table 8
TIFAFlux Pro 1.00.875text-to-image faithfulness; measured by Amazon; reported in the Amazon Nova technical report3.1, Table 8
ImageRewardFlux Pro 1.01.075human-preference reward model score; measured by Amazon; reported in the Amazon Nova technical report3.1, Table 8
TIFAFlux Schnell0.882text-to-image faithfulness; measured by Amazon; reported in the Amazon Nova technical report3.1, Table 8
ImageRewardFlux Schnell0.999human-preference reward model score; measured by Amazon; reported in the Amazon Nova technical report3.1, Table 8
Human pairwise preference — Overall preference (image quality) — vs. DALL·E 3Amazon Nova Canvas54.5%win rate; human raters, pairwise; developer-reported3.2, Table 9
Human pairwise preference — Overall preference (image quality) — vs. DALL·E 3Amazon Nova Canvas6.4%tie rate; human raters, pairwise; developer-reported3.2, Table 9
Human pairwise preference — Overall preference (image quality) — vs. DALL·E 3Amazon Nova Canvas39.1%loss rate; human raters, pairwise; developer-reported3.2, Table 9
Human pairwise preference — Overall preference (image quality) — vs. Imagen 3Amazon Nova Canvas48.2%win rate; human raters, pairwise; developer-reported3.2, Table 9
Human pairwise preference — Overall preference (image quality) — vs. Imagen 3Amazon Nova Canvas5.3%tie rate; human raters, pairwise; developer-reported3.2, Table 9
Human pairwise preference — Overall preference (image quality) — vs. Imagen 3Amazon Nova Canvas46.5%loss rate; human raters, pairwise; developer-reported3.2, Table 9
Human pairwise preference — Instruction following (text-image alignment) — vs. DALL·E 3Amazon Nova Canvas39.4%win rate; human raters, pairwise; developer-reported3.2, Table 9
Human pairwise preference — Instruction following (text-image alignment) — vs. DALL·E 3Amazon Nova Canvas22.5%tie rate; human raters, pairwise; developer-reported3.2, Table 9
Human pairwise preference — Instruction following (text-image alignment) — vs. DALL·E 3Amazon Nova Canvas38.1%loss rate; human raters, pairwise; developer-reported3.2, Table 9
Human pairwise preference — Instruction following (text-image alignment) — vs. Imagen 3Amazon Nova Canvas38.4%win rate; human raters, pairwise; developer-reported3.2, Table 9
Human pairwise preference — Instruction following (text-image alignment) — vs. Imagen 3Amazon Nova Canvas28.1%tie rate; human raters, pairwise; developer-reported3.2, Table 9
Human pairwise preference — Instruction following (text-image alignment) — vs. Imagen 3Amazon Nova Canvas33.5%loss rate; human raters, pairwise; developer-reported3.2, Table 9
Human pairwise preference — Video Quality — vs. Runway Gen3 AlphaAmazon Nova Reel56.4%win rate; human raters, pairwise; developer-reported4.3, Table 10
Human pairwise preference — Video Quality — vs. Runway Gen3 AlphaAmazon Nova Reel9.9%tie rate; human raters, pairwise; developer-reported4.3, Table 10
Human pairwise preference — Video Quality — vs. Runway Gen3 AlphaAmazon Nova Reel33.7%loss rate; human raters, pairwise; developer-reported4.3, Table 10
Human pairwise preference — Video Quality — vs. Luma 1.6Amazon Nova Reel51.1%win rate; human raters, pairwise; developer-reported4.3, Table 10
Human pairwise preference — Video Quality — vs. Luma 1.6Amazon Nova Reel3.4%tie rate; human raters, pairwise; developer-reported4.3, Table 10
Human pairwise preference — Video Quality — vs. Luma 1.6Amazon Nova Reel45.5%loss rate; human raters, pairwise; developer-reported4.3, Table 10
Human pairwise preference — Video Consistency — vs. Runway Gen3 AlphaAmazon Nova Reel67.0%win rate; human raters, pairwise; developer-reported4.3, Table 10
Human pairwise preference — Video Consistency — vs. Runway Gen3 AlphaAmazon Nova Reel9.1%tie rate; human raters, pairwise; developer-reported4.3, Table 10
Human pairwise preference — Video Consistency — vs. Runway Gen3 AlphaAmazon Nova Reel23.9%loss rate; human raters, pairwise; developer-reported4.3, Table 10
Human pairwise preference — Video Consistency — vs. Luma 1.6Amazon Nova Reel74.7%win rate; human raters, pairwise; developer-reported4.3, Table 10
Human pairwise preference — Video Consistency — vs. Luma 1.6Amazon Nova Reel5.1%tie rate; human raters, pairwise; developer-reported4.3, Table 10
Human pairwise preference — Video Consistency — vs. Luma 1.6Amazon Nova Reel20.2%loss rate; human raters, pairwise; developer-reported4.3, Table 10

Provenance

The page summarises the arXiv HTML (v1, March 17, 2025) as captured on September 21, 2026. All Nova figures are Amazon's own; comparison figures are either the other developers' published numbers or Amazon's post-training measurements via public APIs, as marked in each table. Confidence is medium: a single developer-authored source with fast-decay benchmark claims.

Relationships