Berkeley Function Calling Leaderboard (BFCL)

The Berkeley Function Calling Leaderboard (BFCL) is a benchmark and public leaderboard from the University of California, Berkeley's Gorilla project that measures how accurately language models call functions, also called tools: external APIs or user-defined functions that a model invokes by emitting a function name and arguments. It began in early 2024 with single-turn tests scored by parsing each call's abstract syntax tree, and was extended with user-contributed "live" functions in August 2024 (V2), multi-turn conversations in September 2024 (V3), and agentic web-search, memory and format-sensitivity tests in July 2025 (V4) (Source: github.com; Source: gorilla.cs.berkeley.edu). The benchmark was described in a paper at the International Conference on Machine Learning in 2025, whose authors state that since its preview "BFCL has become the defacto standard for evaluating function-calls" (Source: proceedings.mlr.press). Meta, Z.ai, Salesforce and OpenAI are among the developers whose reports use it (The Llama 3 Herd of Models (Meta, July 2024); GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (Zhipu AI, 2025); Salesforce Koa: An Enterprise Language Model for Agentic Tool Use (technical report); Reasoning models struggle to control their chains of thought, and that's good (OpenAI, March 2026)).

What it measures

The paper defines function calling as "an LLM's ability to invoke external functions, APIs, or user-defined tools in response to user queries" and calls it "an essential capability for agentic LLM applications." Its authors give two reasons that no standard benchmark existed before BFCL: the difficulty of deciding when a function call is valid, and the difficulty of acquiring diverse real-world functions. They restrict the benchmark to a model's native function-calling ability rather than prompt-based or code-generation approaches, and position it against earlier tool-use benchmarks that they say cover a single turn, a narrow domain, or rely on language-model user simulators and judges; they cite τ-bench as supporting 28 functions across two domains and as relying on a simulated user that "remain[s] prone to hallucination and instruction-following errors" (Source: proceedings.mlr.press).

The paper concludes that current models "excel at single-turn calls," while "memory, dynamic decision-making, and long-horizon reasoning remain open challenges." The V4 release describes function calling as "the building blocks of Agents" and adds web search, memory and format sensitivity as a "holistic agentic evaluation" (Source: gorilla.cs.berkeley.edu).

Design

Categories

The V4 leaderboard combines five scored categories and two non-scoring ones. Entry counts are those the V4 release gives (Source: gorilla.cs.berkeley.edu).

The single-turn types are defined by how many functions are offered and how often they must be called: simple offers one function called once; multiple offers several functions of which one is to be called (in the original announcement, "multiple functions input, one function output"); parallel requires several calls to one function; parallel multiple combines the two; and irrelevance offers functions none of which should be called. Multi-turn entries are whole conversations in which each user turn may take several steps. The base case supplies all needed information; the missing-parameter case tests whether the model recognizes that a required value is absent and cannot be inferred; the missing-function case tests whether it recognizes that no available function can fulfil the request; and the long-context case tests accuracy with long conversations or long function outputs (Source: proceedings.mlr.press; Source: gorilla.cs.berkeley.edu).

The web-search category gives the model a DuckDuckGo search function and a page-fetching function and asks 100 human-written multi-hop questions about recent but stable facts. The release's example is "How many floors does the tallest building have in the city that received the most international tourists in 2024?", answered in three searches: Bangkok, then the Magnolias Waterfront Residences at Iconsiam, then 70. Questions were built by using one answer as the basis of the next question, filtered to remove duplicates, common knowledge, ambiguous answers and yes/no questions, and checked by several people independently; each was given a short preceding sentence of context to resemble a real query. Fetch requests fail at random with one of six simulated errors, such as a 503 server error or a timeout, to test recovery (Source: gorilla.cs.berkeley.edu). The memory category spans five domains, including college advising, customer support and a medical assistant: the model is told facts over a series of conversations, a memory snapshot is recorded, and later questions are asked with an empty chat history but access to the snapshot (Source: proceedings.mlr.press).

Construction

Single-turn functions were curated in Python, Java and JavaScript from GitHub repositories, with executable functions for mathematical and physical computation and API wrappers emulating real services such as currency exchange and geocoding; the authors added distractor functions and cases with missing parameters. The crowd-sourced "live" set began as 64,517 real single-turn queries collected between February 26 and April 1, 2024; the authors removed duplicates using ROUGE-L and embedding similarity, excluded queries from public sets to avoid contamination, and had human experts make minimal edits, yielding 2,251 entries from what the introduction describes as more than 67,000 contributions. This set has more multiple-function and fewer parallel-call cases than the expert-written set, and includes multilingual queries and function documentation. For the multi-turn set the authors wrote their own API codebase across domains such as vehicle control, generated tasks with initial API states, and had human annotators label ground-truth trajectories; the introduction describes "eight curated API suites and 1000 queries," and the V4 leaderboard scores 800 multi-turn entries (Source: proceedings.mlr.press; Source: gorilla.cs.berkeley.edu).

Scoring

Single-turn calls are scored by abstract syntax tree (AST) substring matching. The model's output is restricted to Python-callable function calls, Python's ast module extracts the function name and arguments, and a call is correct if the name matches exactly and every argument falls within a predefined set of acceptable values. An executable subset is also run and its responses compared with expected outputs, by exact match, by running the ground-truth call at the same time for time-sensitive functions, or by matching the structure of nested lists and dictionaries. The authors report a strong correlation between AST and execution scores and treat this as validation of AST scoring as a scalable offline method that does not require running every function (Source: proceedings.mlr.press).

Multi-turn entries are checked after every turn in two ways. A state-based check compares the system's state with the ground truth, accepting any sequence of calls that reaches it; a response-based check confirms that the model made the minimal sequence of calls needed for read-only requests, so that it cannot guess a result without calling the function. An entry is correct only if it passes both checks in every turn. Agentic answers are scored by exact match on a dedicated answer field after lower-casing and removing punctuation, so that a reply that mentions the reference answer incidentally is not counted (Source: proceedings.mlr.press).

The V4 overall score is Agentic × 40% + Multi-Turn × 30% + Live × 10% + Non-Live × 10% + Hallucination × 10% (Source: gorilla.cs.berkeley.edu). The V4 changelog entry records the previous weights as one third each for Live, Non-Live and Multi-Turn, with none for irrelevance or agentic tasks, and gives the reason for the change: "As single-turn tasks approach saturation, weighting now favors complex, multi-step agentic tasks" (Source: github.com).

Function-calling and prompt modes

Models with native function calling are given the function definitions in their tools field and are labelled "FC"; others are given the definitions in the system prompt and asked to write calls as text, labelled "Prompt." The paper reports that prompt-mode models had about three times as many decoding failures (412.93 against 182.5 of 4,251 entries on average) but were more flexible in complex multiple and parallel cases, and that some models perform better in prompt mode (Source: proceedings.mlr.press). Many models appear on the leaderboard twice, once in each mode (Source: gorilla.cs.berkeley.edu).

Contamination checks

Because the expert-written single-turn set was public before the crowd-sourced set was released, the authors compared models' perplexity and character-level negative log-likelihood on the two. Most models had equal or lower perplexity on the newer crowd-sourced set than on the older one, which the authors read as evidence against memorization. Salesforce's xLAM-7B was the exception: its perplexity rose from 3.67 on the single-turn set to 5.09 on the crowd-sourced set (character-level negative log-likelihood from 0.340 to 0.427), which the authors describe as a notable performance degradation and as revealing "vulnerabilities that the original static test masked" (Source: proceedings.mlr.press).

No human baseline is reported.

Versions and variants

• V1 (early 2024; the changelog's earliest entry is dated February 29, 2024): expert-curated single-turn simple, multiple and parallel calls, scored by AST matching and by execution, with functions in Python, Java, JavaScript, REST and SQL (Source: gorilla.cs.berkeley.edu; Source: proceedings.mlr.press).
• V2 Live (August 19, 2024): enterprise- and community-contributed functions and queries (Source: gorilla.cs.berkeley.edu).
• V3 (September 19, 2024): multi-turn and multi-step function calling, including going back and forth with the user to ask clarifying questions (Source: gorilla.cs.berkeley.edu; Source: gorilla.cs.berkeley.edu).
• V4 (July 17, 2025): the agentic category (web search and memory), format sensitivity, the revised overall weights, and the exclusion of relevance and irrelevance scores from the Live and Non-Live averages (Source: github.com).

The ICML 2025 paper's results table uses a composition that includes an agentic SQL category not shown on the current leaderboard (Source: proceedings.mlr.press). Scores reported under different versions are not directly comparable, because both the task set and the overall weighting changed. Some developer reports name the version they used, as Z.ai did with "BFCL v3" for GLM-4.5 (GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (Zhipu AI, 2025)); others do not.

Development and maintainers

BFCL is maintained at UC Berkeley by the team behind Gorilla, an earlier project that trained a model to call APIs. The ICML paper's authors are Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica and Joseph E. Gonzalez (Source: proceedings.mlr.press). The code and data are public in the Gorilla GitHub repository, on Hugging Face and as the bfcl-eval package on PyPI. The maintainers run the models themselves and publish the raw responses; the leaderboard, last updated April 12, 2026 when read in September 2026, lists 109 entries evaluated at a stated commit (f7cf735, package version 2025.12.17), with each entry's estimated cost and latency, and asks those who want a model added to contact the team (Source: gorilla.cs.berkeley.edu). The changelog records ground-truth corrections to individual entries, for example a wrong date in one live-simple entry in September 2025 and a multi-turn base entry in October 2025 (Source: github.com).

Results

In the paper's evaluation of models from late 2024 and early 2025, GPT-4o (November 2024) in prompt mode had the highest overall accuracy at 66.4, and the best memory score was 12%, by o1 (Source: proceedings.mlr.press). On the V4 leaderboard as last updated in April 2026, Claude Opus 4.5 in function-calling mode leads at 77.47%, followed by Claude Sonnet 4.5 at 73.24% and Gemini 3 Pro Preview in prompt mode at 72.51% (Source: gorilla.cs.berkeley.edu). Salesforce's Koa report of September 2026 gives 78.18% for Claude Opus 4.8 and 67.63% for GPT-5.5 in its own runs; it breaks down results by category only for the base model and its own checkpoints (Salesforce Koa: An Enterprise Language Model for Agentic Tool Use (technical report)).

BFCL V4 Overall

Overall accuracy under the V4 composition. Leaderboard rows are measured by the BFCL maintainers and dated to the leaderboard's last update (April 12, 2026), as read on September 21, 2026. The Koa report does not name a BFCL version; its category breakdown (web search, memory, multi-turn, live and non-live AST, relevance and irrelevance) matches V4.

BFCL V4 Multi-Turn

Unweighted average of the four multi-turn subcategories, from the Koa report's full BFCL table.

BFCL V4 Web Search

Average of the with-snippet and without-snippet variants, from the Koa report's full BFCL table.

BFCL V4 Memory

Average of the key-value, vector and recursive-summarization memory backends, from the Koa report's full BFCL table.

BFCL V4 Live AST

Average over the live (user-contributed) single-turn subcategories, from the Koa report's full BFCL table.

BFCL V4 Non-Live AST

Average over the expert-written single-turn subcategories, from the Koa report's full BFCL table.

BFCL V4 Irrelevance detection

Share of cases in which the model correctly declines to call any of the offered functions, from the Koa report's full BFCL table.

BFCL V4 Relevance detection

The live relevance cases, which are not scored in the V4 overall, from the Koa report's full BFCL table.

BFCL paper (ICML 2025)

Overall accuracy in the paper's results table, whose composition includes an agentic SQL category and differs from the V4 leaderboard. Selected rows; the table lists about 70 model and mode combinations.

BFCL v3 — Overall accuracy

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

BFCL v3 — Latency (seconds)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

BFCL v3 — Non-Live execution

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

BFCL v3 — Live overall

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

BFCL v3 — Multi-Turn overall

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

BFCL v3 — Hallucination relevance

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

BFCL v3 — Hallucination irrelevance

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

BFCL v3

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Interpreting scores

An overall score is a composite. Under V4, 70% of it comes from the agentic and multi-turn categories and 30% from single-turn tests and irrelevance detection, so two models with the same overall score can differ widely by category (Source: gorilla.cs.berkeley.edu). The leaderboard page's own footnote describes Overall Accuracy as "the unweighted average of all the sub-categories" and links to the V3 composition, while the V4 release and changelog give the weighted formula (Source: gorilla.cs.berkeley.edu; Source: github.com).

The mode in which a model is run changes its score substantially. On the leaderboard as last updated, Claude Opus 4.5 scores 77.47% in function-calling mode and 33.47% in prompt mode, and GPT-4.1 53.96% and 39.38% (Source: gorilla.cs.berkeley.edu). The paper reports the opposite direction for some models, such as Llama 3.1 70B (50.4 in prompt mode, 20.8 in function-calling mode) (Source: proceedings.mlr.press). A score quoted without its mode and version is therefore hard to place. Salesforce's Koa report, for example, does not state either for the models it compares, though its GPT-4.1 figure matches the leaderboard's function-calling entry (Salesforce Koa: An Enterprise Language Model for Agentic Tool Use (technical report)).

The web-search category depends on a live search engine and live web pages; the maintainers chose questions about recent but stable facts so that answers would not change, and chose DuckDuckGo in part so that results are not personalized (Source: gorilla.cs.berkeley.edu).

Limitations and criticism

The authors note that AST matching is a proxy for execution and support it with a correlation study rather than by executing every function. Their own contamination analysis rests on the premise that the older, public single-turn set may be in training data, and they introduced the crowd-sourced set partly as a check on it (Source: proceedings.mlr.press). The maintainers' changelog records corrections to individual ground-truth answers after release (Source: github.com). The single-turn categories were, in the maintainers' description at the V4 release, approaching saturation, which was their stated reason for moving weight to agentic and multi-turn tasks (Source: github.com).

Use in policy and safety frameworks

OpenAI's open-source CoT-Control suite of more than 13,000 tasks, used to measure how well a model can control its own chain of thought, pairs problems drawn from BFCL and four other benchmarks with instructions about the chain of thought (Reasoning models struggle to control their chains of thought, and that's good (OpenAI, March 2026)); the GPT-6 Astra system card reports CoT-controllability results on that suite. Meta's Llama 3 report lists BFCL among its tool-use benchmarks (The Llama 3 Herd of Models (Meta, July 2024)). The paper states that BFCL is used "by all leading labs developing large language models," citing Meta, Qwen and Cohere (Source: proceedings.mlr.press).

Relationships

• instance-of: AI Benchmarks and Evaluation
• related: Agentic AI, Agentic harnesses and capability elicitation — function-calling versus prompt mode changes scores by tens of points
• related: τ-bench — a multi-turn tool-use benchmark with a simulated user, which the BFCL paper contrasts with its own design; MCP Atlas — tool use over live MCP servers
• supports: Salesforce Koa — Salesforce reports Koa's largest gain over its base on BFCL's multi-turn category (Salesforce Koa: An Enterprise Language Model for Agentic Tool Use (technical report))

See also

• BrowseComp, GAIA — other evaluations of web search by agents
• Terminal-Bench, OSWorld — agentic evaluations in other environments