"Salesforce Koa: An Enterprise Language Model for Agentic Tool Use" is a 16-page technical report by 25 authors from Salesforce Agentforce & AI Research, led by co-first authors Zixiang Chen, Sufeng Niu, Yingchi Liu, Wenting Zhao, Akshara Prabhakar and Shubham Mehrotra, with Phil Mui and Huan Wang as co-corresponding authors and Silvio Savarese, Shelby Heinecke and Sitaram Asur among the others. It was posted to arXiv as 2609.15066 on September 14, 2026; the title page is dated September 15, 2026, the day Salesforce and NVIDIA announced the model. The report describes Koa, built by post-training NVIDIA's open-weight Nemotron-3-Super-120B with reinforcement learning using Group Relative Policy Optimization (GRPO) on public and synthetically generated data, "with no customer data." Its stated contribution is a pipeline in which the same declarative specifications that configure a Salesforce Agentforce agent also generate the training tasks and reward criteria for the model. The acknowledgements state that the model "was trained in partnership with NVIDIA" (Salesforce Koa: An Enterprise Language Model for Agentic Tool Use (technical report)).
Method
Specification-driven task construction
For enterprise domains the training tasks start from specifications written in Agent Script, which the report describes as Salesforce's declarative language for Agentforce agents. A specification defines a router, specialized subagents, typed actions and natural-language instructions. A static extraction pass compiles it into a typed workflow graph of per-tool argument schemas, declared state effects, routing conditions and termination settings, and that graph instantiates an executable environment with simulated state that the tools read and change. For public tool-use domains, where no Agent Script specification exists, the authors synthesize the same kind of workflow graph directly, so both sources feed one simulation and reward system.
A simulation pipeline expands each specification into sessions conditioned on a scenario, which fixes the workflows and user needs exercised, and on a persona, which conditions the simulated customer's behaviour. Sessions become reinforcement-learning examples that begin either at the opening request or partway through a generated dialogue, so that the policy is trained on decisions at different depths of a conversation. NVIDIA's NeMo Gym runs each example as an online rollout. The report presents this as a link between agent authoring and model training: "the same specifications that configure an agent also structure its rollout tasks and resolution criteria."
Environments and reward
In simulated-dialogue environments a single frozen helper model plays three roles in every rollout: customer simulator, tool emulator where no production backend exists, and coverage judge. The helper is never updated. Three environment types are described:
| Environment type | Interaction | Verification |
|---|---|---|
| Flat customer support | One agent; all advertised tools callable; persona-conditioned simulated customer | Per-sub-question coverage judged by a language model, with a duplicate-call gate |
| Routed customer service | A router plus subagents reached through go_to_* calls; out-of-scope calls return an error; a focus message is inserted at each switch | The same coverage reward and gate, with domain-specific scope and grounding rules |
| Deterministic tool sandbox | One agent over stateful tools; no simulated customer and no helper | Reward of one only if the predicted actions reproduce the reference end state |
The task intent is written as numbered sub-questions, and the judge returns a resolved or not-resolved verdict for each over the full transcript. The reward is the share of sub-questions resolved, set to zero if the agent made two identical tool calls in a row. The judge prompt allows a sub-question that needs customer- or system-specific data to count as resolved only when the answer rests on a successful, relevant tool call. Each conversation exposes only the tools of its source session, and in routed environments only those of the active subagent.
Optimization
The policy samples a small group of trajectories per prompt. Groups in which every trajectory earns the same reward are discarded and refilled. Advantages are computed against a leave-one-out group baseline with no value network, and a token-level truncated importance-sampling weight corrects the mismatch between training-time and generation-time probabilities. The report performs one update per rollout batch, so the clipped-ratio term of PPO-style objectives is inactive, and applies no KL penalty against a reference model. Trajectories with malformed tool-call or thinking syntax receive a fixed negative advantage on the offending tokens; over-length trajectories are masked.
A second training mode targets what the authors call constrained decision turns, in which the correct action is to tell the user something rather than call a tool: a needed API is unavailable, a required parameter is missing, or an action needs permissions the agent lacks. The authors write that the base policy "largely lacks this skill," most often issuing state-changing calls with similar names and occasionally "fabricating confirmations for actions that never occurred." Building on the PivotRL method, the environment withholds the tool a request needs at a chosen turn; any state-changing call at that turn zeroes the reward, and the score otherwise depends on whether the agent issues the correct call once a follow-up message restores the tool. The report's example is a home-buying service in which the user asks to list inspection requests while that function is withheld and a similarly named read-only function that cannot answer the query is offered. Generating constrained turns only in mid-conversation improved that slice but regressed turn-0 cases until turn-0 examples were added.
Experimental setup
The policy is "Nemotron-3-Super v3 (∼120B)," run in thinking mode during training. Rollouts use NeMo Gym environments for Salesforce support, healthcare administration and real estate, plus a helper-free workplace-assistant sandbox; each simulated-dialogue environment uses a Nemotron-3-Nano (about 30 billion parameters) model as its helper. Training ran on a Slurm cluster of five NVIDIA B200 nodes, one of them serving the helper. Offline evaluation served the trained checkpoints under greedy decoding with a separate judge model for argument equivalence, and all checkpoints were served through vLLM in BF16.
The reward and GRPO recipe were developed on the smaller Nemotron-3-Nano proxy and then transferred. Of the reward designs tried there, a penalized binary reward trended downward, additive and floored tool-bonus variants were flat to declining and noisy, and a dense partial-credit reward scored highest only because, in the authors' words, "partial credit inflates its own scalar (reward hacking)." With the final gated-coverage reward, the report found stability depended on scale: the Super policy's training task-resolution score rose to about 0.65 over about 200 steps, while the Nano proxy peaked near 0.47 around step 30 and fell to about 0.33. The authors conclude that "even the winning reward is thus not sufficient on its own at small scale."
The comparison set is the base model, GPT-4.1, GPT-5.5 and Claude Opus 4.8. The benchmarks are Tau2Bench (airline, retail and telecom customer service, with GPT-4.1 as the user simulator, four trials and pass^1 averaged), the Berkeley Function Calling Leaderboard (BFCL) and CRM Bench, which the report describes as "one released enterprise benchmark" covering "single-turn Salesforce and Agentforce workflows scored along topic, function-call, and free-text accuracy." The report does not state reasoning or sampling settings for the proprietary models, name the BFCL version, or give a location for CRM Bench.
Findings
The report summarises its results as showing that Koa "improves over its open-weight base, with the clearest gains on multi-turn tool use, and surpasses a strong proprietary baseline (GPT-4.1) while remaining below the strongest frontier models." On Tau2Bench Koa's task-weighted average is 69.41 against 68.64 for the base and 54.48 for GPT-4.1; Claude Opus 4.8 scores 74.00 and GPT-5.5 83.99. On BFCL Koa scores 66.63% against 64.73% for the base, 53.96% for GPT-4.1, 67.63% for GPT-5.5 and 78.18% for Claude Opus 4.8. On CRM Bench its average is 0.86, against 0.84 for the base, 0.81 for GPT-4.1, 0.87 for Claude Opus 4.8 and 0.90 for GPT-5.5. The per-domain and per-category figures are recorded on τ-bench and Berkeley Function Calling Leaderboard (BFCL), and the CRM Bench figures below.
Within Tau2Bench, Koa's gain over its base is in airline (62.0 against 61.5) and retail (81.6 against 79.9); telecom is unchanged at 60.5. The larger BFCL gain comes from the multi-turn category, where the Koa checkpoint scores 59.50 against the base's 54.12.
Supervised fine-tuning versus reinforcement learning
The report states that the RL stage was applied "directly to the base model rather than to an SFT checkpoint," and that "we build Salesforce Koa on the RL-trained checkpoint." A preliminary supervised fine-tuning (SFT) study, run before RL, trained the full model on synthetic tool-use trajectories generated with Salesforce's APIGen and APIGen-MT pipelines over BFCL and Tau2Bench airline tool collections. It ran on 32 H200 GPUs at a peak learning rate of 5 × 10⁻⁸, which the authors chose to avoid catastrophic forgetting in a base model that had "already undergone elaborate post-training."
The resulting SFT-only checkpoint scored slightly higher than the RL checkpoint on Tau2Bench (70.04 against 69.41) and on CRM Bench (0.88 against 0.86), but slightly lower on BFCL multi-turn (53.25 against the base's 54.12, where RL reached 59.50). SFT's BFCL gains were in single-turn and static categories such as live parallel calls, memory and relevance detection. The authors conclude that RL is stronger for multi-turn tool use and SFT effective for narrow single-turn CRM adaptation, and they limit the finding: because the base model was itself already RL post-trained, "starting from a pre-RL checkpoint could alter the conclusion." They list co-designing SFT and RL stages and starting from a pre-RL model as future work.
Reported results
CRM Bench has no evaluations page; its results are recorded here. Scores are fractions between 0 and 1 as the report gives them. Tau2Bench and BFCL results from the report are on τ-bench and Berkeley Function Calling Leaderboard (BFCL).
| Evaluation | Model | Score | Setting | Section |
|---|---|---|---|---|
| CRM Bench (weighted average) | Salesforce Koa | 0.86 | RL-only checkpoint; BF16 via vLLM; Salesforce's evaluation; also reported as the RL-only row of Table 5 (average 0.86) | §3, Table 1; App. J, Table 5 |
| CRM Bench (topic accuracy) | Salesforce Koa | 0.97 | as above; Table 5 gives 0.970 | §3, Table 1; App. J, Table 5 |
| CRM Bench (function-call accuracy) | Salesforce Koa | 0.77 | as above; Table 5 gives 0.770 | §3, Table 1; App. J, Table 5 |
| CRM Bench (text accuracy) | Salesforce Koa | 0.85 | as above; Table 5 gives 0.850 | §3, Table 1; App. J, Table 5 |
| CRM Bench (weighted average) | Nemotron-3-Super-120B | 0.84 | open-weight base, before Salesforce post-training; BF16 via vLLM; also the base row of Table 5 | §3, Table 1; App. J, Table 5 |
| CRM Bench (topic accuracy) | Nemotron-3-Super-120B | 0.97 | as above; Table 5 gives 0.970 | §3, Table 1; App. J, Table 5 |
| CRM Bench (function-call accuracy) | Nemotron-3-Super-120B | 0.71 | as above; Table 5 gives 0.710 | §3, Table 1; App. J, Table 5 |
| CRM Bench (text accuracy) | Nemotron-3-Super-120B | 0.85 | as above; Table 5 gives 0.850 | §3, Table 1; App. J, Table 5 |
| CRM Bench (weighted average) | Claude Opus 4.8 | 0.87 | Salesforce's evaluation; model settings not stated | §3, Table 1 |
| CRM Bench (topic accuracy) | Claude Opus 4.8 | 0.99 | as above | §3, Table 1 |
| CRM Bench (function-call accuracy) | Claude Opus 4.8 | 0.83 | as above | §3, Table 1 |
| CRM Bench (text accuracy) | Claude Opus 4.8 | 0.79 | as above | §3, Table 1 |
| CRM Bench (weighted average) | GPT-5.5 | 0.90 | Salesforce's evaluation; model settings not stated | §3, Table 1 |
| CRM Bench (topic accuracy) | GPT-5.5 | 0.99 | as above | §3, Table 1 |
| CRM Bench (function-call accuracy) | GPT-5.5 | 0.82 | as above | §3, Table 1 |
| CRM Bench (text accuracy) | GPT-5.5 | 0.89 | as above | §3, Table 1 |
| CRM Bench (weighted average) | GPT-4.1 | 0.81 | Salesforce's evaluation; model settings not stated | §3, Table 1 |
| CRM Bench (topic accuracy) | GPT-4.1 | 0.98 | as above | §3, Table 1 |
| CRM Bench (function-call accuracy) | GPT-4.1 | 0.85 | as above | §3, Table 1 |
| CRM Bench (text accuracy) | GPT-4.1 | 0.60 | as above | §3, Table 1 |
| CRM Bench (average) | SFT-only checkpoint of Nemotron-3-Super-120B (Salesforce ablation) | 0.88 | supervised fine-tuning only, not the released model; BF16 | App. J, Table 5 |
| CRM Bench (topic accuracy) | SFT-only checkpoint of Nemotron-3-Super-120B (Salesforce ablation) | 0.980 | as above | App. J, Table 5 |
| CRM Bench (function-call accuracy) | SFT-only checkpoint of Nemotron-3-Super-120B (Salesforce ablation) | 0.770 | as above | App. J, Table 5 |
| CRM Bench (text accuracy) | SFT-only checkpoint of Nemotron-3-Super-120B (Salesforce ablation) | 0.890 | as above | App. J, Table 5 |
Table 1 labels the CRM Bench summary column "Weighted Avg." and Table 5 labels it "Average"; neither states the weights.
Relation to the product announcement
The report and the joint Salesforce–NVIDIA press release of September 15, 2026 agree on the base model, the use of GRPO and the absence of customer data, and differ on several points (Source: salesforce.com):
- Training sequence. The press release states that Salesforce "applied Supervised Fine-Tuning (SFT) and reinforcement learning with Group Relative Policy Optimization (GRPO)." The report states that RL was applied directly to the base model, that SFT was a preliminary study, and that Koa is built on the RL-only checkpoint.
- CRM benchmark claim. The press release states that on "Salesforce's CRM benchmark" Koa "already matches or exceeds leading model performance on CRM actions with three times fewer errors," and links to Salesforce AI Research's CRM benchmark page. That page describes scoring on accuracy (factuality, instruction following, conciseness and completeness on a four-point scale), cost, speed, trust and safety, and sustainability (Source: salesforceairesearch.com). The report's CRM Bench is scored on topic, function-call and text accuracy, and no error-rate comparison appears in it. In the report, Koa's CRM Bench average of 0.86 is below Claude Opus 4.8 (0.87) and GPT-5.5 (0.90), and its function-call accuracy of 0.77 is below all three proprietary models.
- Availability. The report does not discuss availability. The press release gives Koa as available to select pilot customers in Agentforce, with general availability expected in winter 2026 in U.S. regions.
- Scale of the base model. The report names the base "Nemotron-3-Super-120B" and "Nemotron-3-Super v3 (∼120B)"; NVIDIA's December 2025 announcement of the Nemotron 3 family described Super as approximately 100 billion parameters with up to 10 billion active (NVIDIA Nemotron).
The report's own framing is narrower than the press release's in one further respect: it states that Koa remains "below the strongest frontier models."
Provenance
The raw file is a full transcription of the arXiv v1 PDF, fetched from arxiv.org on September 21, 2026. Tables were taken from the PDF, whose rule lines assign Table 1's columns to their benchmarks; an earlier extraction attempt from the arXiv HTML rendering had collapsed that table's spanning header. All results are Salesforce's own and have not been independently reproduced. The report's GPT-4.1 BFCL figure (53.96%) equals the BFCL leaderboard's entry for GPT-4.1-2025-04-14 in function-calling mode (Berkeley Function Calling Leaderboard (BFCL)).
Relationships
- supports: Salesforce Koa — the technical documentation for the model
- depends-on: NVIDIA Nemotron — the base model; also the source of the training-time helper (Nemotron-3-Nano)
- related: Salesforce, Inc. — author institution and developer; Nvidia & TSMC — AI Compute Infrastructure — training partner and co-announcer
- instance-of: Post-Training — specialization of an open-weight model by reinforcement learning
- related: Synthetic Data / Model-Generated Training Data — training tasks, personas and SFT trajectories are model-generated
- related: Reward Hacking — the partial-credit reward design the authors report was inflated by the policy
- related: Agentic AI, Open-Weight Frontier Models
- related: τ-bench, Berkeley Function Calling Leaderboard (BFCL) — benchmarks on which the report's public results are recorded