APIFlow-Bench 1.0 Leaderboard
Solo subtasksHard
Deterministic API engineering benchmark spanning 7 capability axes, over REST APIs only (GraphQL and MCP are planned). Showing the solo subtasks task set: 9,840 trials across 24 models and 82 tasks covering all 7 axes — authentication, discovery, error recovery, multi-step workflow, pagination, schema, statefulness. Switch task set above; click any card to drill into a slice with the full interactive detail view; click any model name to see its per-task results.
Showing Solo subtasks created as hard — 82 tasks, 9,840 trials. Back to the default 20-subtask chain view.
Overall Ranking
Columns are grouped into metric families. Pass rate = passed ÷ all trials; API recovery = passed ÷ trials that hit a 4xx/5xx. Difficulty bucket is judged per task from its content and reference solve transcripts (Easy / Long-horizon / Hard / Hard + Long-horizon) — a separate taxonomy from the creation-time “Difficulty (as created)” filter, so the two don't map one-to-one. Per-trial details (verifier groups, checkpoints, step counts) live on the model → task → trial pages.
| Rank | Model | Scoring | Difficulty bucket (pass rate) | Horizon (pass rate) | Cost / efficiency | Behavioral | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pass rate | API recovery | Easy | Long-horizon | Hard | Short | Medium | Long | Tool calls / trial | Time / trial | Tokens / trial | Cost / trial | Abstain % | Refusal % | ||
| 1 | kimi-k3 | 99.0%406/410 | 117/121 | 100%140/140 | 100%90/90 | 98%172/175 | 100%235/235 | 98%157/160 | 100%10/10 | 14.3 | 7.5m | 103.4K | $0.13 | 0.0%0/410 | 0.0%0/410 |
| 2 | claude-sonnet-5 | 97.6%400/410 | 128/136 | 99%138/140 | 100%90/90 | 98%171/175 | 99%233/235 | 98%156/160 | 100%10/10 | 14.0 | 55.4s | 138.6K | $0.060 | 1.2%5/410 | 0.0%0/410 |
| 3 | claude-sonnet-4-6 | 96.8%397/410 | 98/106 | 100%140/140 | 100%90/90 | 95%166/175 | 98%230/235 | 98%156/160 | 100%10/10 | 12.0 | 47.7s | 118.9K | $0.062 | 0.0%0/410 | 0.0%0/410 |
| 4 | kimi-k2p7-code | 96.6%396/410 | 128/134 | 99%138/140 | 96%86/90 | 97%170/175 | 97%229/235 | 97%155/160 | 100%10/10 | 13.5 | 1.9m | 96.6K | $0.036 | 1.0%4/410 | 0.0%0/410 |
| 5 | gpt-5.5 | 96.3%395/410 | 104/114 | 100%140/140 | 97%87/90 | 96%168/175 | 99%233/235 | 95%152/160 | 100%10/10 | 14.1 | 31.6s | 54.0K | $0.10 | 0.0%0/410 | 0.0%0/410 |
| 6 | claude-opus-4-8 | 96.1%394/410 | 121/134 | 100%140/140 | 100%90/90 | 94%164/175 | 100%235/235 | 93%149/160 | 100%10/10 | 11.8 | 42.6s | 68.6K | $0.084 | 3.9%16/410 | 0.0%0/410 |
| 7 | gpt-5.6-sol | 95.9%393/410 | 92/103 | 99%139/140 | 100%90/90 | 94%164/175 | 98%230/235 | 96%153/160 | 100%10/10 | 13.3 | 2.1m | 44.8K | $0.090 | 2.2%9/410 | 0.0%0/410 |
| 8 | glm-5p2 | 95.6%392/410 | 116/126 | 97%136/140 | 97%87/90 | 97%169/175 | 98%230/235 | 95%152/160 | 100%10/10 | 14.3 | 3.2m | 85.7K | $0.077 | 0.0%0/410 | 0.0%0/410 |
| 9 | kimi-k2p6 | 95.4%391/410 | 134/140 | 96%135/140 | 94%85/90 | 97%169/175 | 97%227/235 | 95%152/160 | 100%10/10 | 15.6 | 1.2m | 236.5K | $0.071 | 0.5%2/410 | 0.0%0/410 |
| 9 | qwen3p6-plus | 95.4%391/410 | 117/129 | 97%136/140 | 98%88/90 | 95%167/175 | 98%231/235 | 94%150/160 | 100%10/10 | 15.1 | 1.4m | 136.5K | $0.053 | 0.0%0/410 | 0.0%0/410 |
| 9 | deepseek-v4-pro | 95.4%391/410 | 141/152 | 99%139/140 | 99%89/90 | 93%162/175 | 98%230/235 | 94%150/160 | 100%10/10 | 18.9 | 3.5m | 130.3K | $0.11 | 0.2%1/410 | 0.0%0/410 |
| 12 | deepseek-v4-flash | 95.1%390/410 | 139/146 | 96%135/140 | 96%86/90 | 95%166/175 | 96%225/235 | 95%152/160 | 100%10/10 | 17.9 | 54.6s | 183.9K | $0.015 | 0.0%0/410 | 0.0%0/410 |
| 13 | gpt-5.6-luna | 94.4%387/410 | 101/112 | 98%137/140 | 98%88/90 | 93%162/175 | 96%226/235 | 94%151/160 | 100%10/10 | 12.6 | 1.4m | 32.1K | $0.013 | 1.2%5/410 | 0.0%0/410 |
| 14 | kimi-k2p5 | 93.9%385/410 | 122/130 | 97%136/140 | 96%86/90 | 92%161/175 | 95%224/235 | 93%149/160 | 100%10/10 | 12.1 | 28.6s | 75.1K | $0.031 | 0.7%3/410 | 0.0%0/410 |
| 14 | gpt-5.6-terra | 93.9%385/410 | 99/112 | 98%137/140 | 99%89/90 | 91%159/175 | 95%224/235 | 94%151/160 | 100%10/10 | 12.3 | 1.0m | 36.6K | $0.036 | 2.0%8/410 | 0.0%0/410 |
| 16 | minimax-m2p7 | 93.4%383/410 | 144/159 | 98%137/140 | 94%85/90 | 91%159/175 | 96%225/235 | 92%147/160 | 90%9/10 | 13.0 | 25.6s | 66.2K | $0.016 | 0.2%1/410 | 0.0%0/410 |
| 17 | qwen3p7-plus | 93.2%382/410 | 102/113 | 97%136/140 | 97%87/90 | 92%159/172 | 97%227/234 | 92%146/158 | 90%9/10 | 11.9 | 3.5m | 146.6K | $0.042 | 1.7%7/410 | 0.0%0/410 |
| 18 | claude-haiku-4-5-20251001 | 92.4%379/410 | 133/147 | 95%133/140 | 94%85/90 | 92%161/175 | 95%223/235 | 91%146/160 | 100%10/10 | 14.8 | 29.5s | 101.6K | $0.026 | 1.7%7/410 | 0.0%0/410 |
| 19 | gpt-oss-120b | 91.0%373/410 | 130/145 | 91%127/140 | 98%88/90 | 90%158/175 | 91%215/235 | 92%148/160 | 100%10/10 | 12.4 | 20.3s | 56.6K | $0.006 | 0.0%0/410 | 0.0%0/410 |
| 20 | gpt-5-mini | 89.5%367/410 | 115/129 | 95%133/140 | 90%81/90 | 87%152/175 | 93%218/235 | 86%138/160 | 100%10/10 | 11.1 | 29.7s | 50.4K | $0.006 | 3.7%15/410 | 0.0%0/410 |
| 21 | gpt-oss-20b | 87.1%357/410 | 129/150 | 86%121/140 | 89%80/90 | 89%156/175 | 89%208/235 | 88%140/160 | 90%9/10 | 13.5 | 1.5m | 65.4K | $0.003 | 2.0%8/410 | 0.0%0/410 |
| 22 | gpt-5.4-mini | 74.4%305/410 | 103/149 | 81%114/140 | 63%57/90 | 77%134/175 | 79%185/235 | 72%115/160 | 50%5/10 | 10.3 | 14.2s | 20.9K | $0.008 | 2.7%11/410 | 0.0%0/410 |
| 23 | minimax-m3 | 41.5%170/410 | 170/410 | 60%84/140 | 28%25/90 | 35%61/175 | 60%140/235 | 18%29/160 | 10%1/10 | 141.8 | 23.5m | 3.8M | $0.26 | 33.9%139/410 | 0.0%0/410 |
| 24 | claude-fable-5 | 20.5%84/410 | 2/8 | 25%35/140 | 22%20/90 | 17%29/175 | 23%55/235 | 18%29/160 | 0%0/10 | 3.4 | 1.0m | 17.4K | $0.038 | 0.0%0/410 | 82.9%340/410 |
Slices
Behavior Mix
10 models
passedfailedabstainedno-score
| Rank | Model | Mix | Pass rate |
|---|---|---|---|
| 1 | kimi-k3 | 99.0% | |
| 2 | claude-sonnet-5 | 97.6% | |
| 3 | claude-sonnet-4-6 | 96.8% | |
| 4 | kimi-k2p7-code | 96.6% | |
| 5 | gpt-5.5 | 96.3% | |
| 6 | claude-opus-4-8 | 96.1% | |
| 7 | gpt-5.6-sol | 95.9% | |
| 8 | glm-5p2 | 95.6% | |
| 9 | kimi-k2p6 | 95.4% | |
| 9 | qwen3p6-plus | 95.4% |
Authentication
1,800 trials
| Rank | Model | Pass rate |
|---|---|---|
| 1 | claude-sonnet-5 | 100.0% |
| 1 | claude-opus-4-8 | 100.0% |
| 1 | qwen3p6-plus | 100.0% |
| 1 | kimi-k3 | 100.0% |
| 5 | glm-5p2 | 98.7% |
| 6 | gpt-5.5 | 97.3% |
| 7 | kimi-k2p6 | 96.0% |
| 8 | kimi-k2p7-code | 94.7% |
| 8 | gpt-5.6-sol | 94.7% |
| 8 | deepseek-v4-pro | 94.7% |
Discovery
1,800 trials
| Rank | Model | Pass rate |
|---|---|---|
| 11 | kimi-k2p5 | 98.7% |
| 11 | deepseek-v4-flash | 98.7% |
| 13 | qwen3p6-plus | 97.3% |
| 13 | gpt-5.6-luna | 97.3% |
| 13 | deepseek-v4-pro | 97.3% |
| 16 | claude-haiku-4-5-20251001 | 96.0% |
| 17 | claude-sonnet-5 | 94.7% |
| 18 | qwen3p7-plus | 93.3% |
| 19 | gpt-oss-120b | 85.3% |
| 19 | gpt-5-mini | 85.3% |
Error Recovery
1,320 trials
| Rank | Model | Pass rate |
|---|---|---|
| 6 | gpt-5.6-terra | 98.2% |
| 6 | deepseek-v4-pro | 98.2% |
| 8 | minimax-m2p7 | 96.4% |
| 8 | qwen3p6-plus | 96.4% |
| 8 | gpt-5.6-luna | 96.4% |
| 8 | qwen3p7-plus | 96.4% |
| 12 | gpt-oss-120b | 94.5% |
| 12 | gpt-5.5 | 94.5% |
| 12 | kimi-k2p7-code | 94.5% |
| 12 | glm-5p2 | 94.5% |
Multi-step Workflow
1,920 trials
| Rank | Model | Pass rate |
|---|---|---|
| 1 | kimi-k3 | 98.8% |
| 2 | deepseek-v4-flash | 97.5% |
| 3 | kimi-k2p6 | 96.2% |
| 4 | kimi-k2p5 | 95.0% |
| 4 | kimi-k2p7-code | 95.0% |
| 4 | deepseek-v4-pro | 95.0% |
| 7 | gpt-oss-120b | 93.8% |
| 7 | gpt-5.5 | 93.8% |
| 7 | gpt-5.6-luna | 93.8% |
| 7 | gpt-5.6-sol | 93.8% |
Pagination
600 trials
| Rank | Model | Pass rate |
|---|---|---|
| 14 | claude-opus-4-8 | 96.0% |
| 14 | gpt-5.6-sol | 96.0% |
| 14 | deepseek-v4-pro | 96.0% |
| 17 | gpt-5.4-mini | 92.0% |
| 17 | gpt-oss-120b | 92.0% |
| 17 | qwen3p6-plus | 92.0% |
| 17 | glm-5p2 | 92.0% |
| 21 | claude-haiku-4-5-20251001 | 88.0% |
| 21 | gpt-oss-20b | 88.0% |
| 23 | minimax-m3 | 56.0% |
Schema
1,080 trials
| Rank | Model | Pass rate |
|---|---|---|
| 12 | deepseek-v4-flash | 97.8% |
| 12 | gpt-5.5 | 97.8% |
| 12 | gpt-oss-20b | 97.8% |
| 12 | gpt-5.6-sol | 97.8% |
| 12 | deepseek-v4-pro | 97.8% |
| 17 | gpt-oss-120b | 95.6% |
| 17 | gpt-5-mini | 95.6% |
| 17 | gpt-5.6-terra | 95.6% |
| 17 | claude-opus-4-8 | 95.6% |
| 21 | minimax-m2p7 | 93.3% |
Statefulness
1,320 trials
| Rank | Model | Pass rate |
|---|---|---|
| 1 | claude-sonnet-5 | 98.2% |
| 2 | claude-sonnet-4-6 | 94.5% |
| 2 | kimi-k2p7-code | 94.5% |
| 2 | kimi-k3 | 94.5% |
| 5 | gpt-5.5 | 92.7% |
| 6 | claude-opus-4-8 | 90.9% |
| 7 | gpt-5.6-sol | 89.1% |
| 7 | deepseek-v4-pro | 89.1% |
| 9 | deepseek-v4-flash | 87.3% |
| 9 | qwen3p6-plus | 87.3% |
Per-Model Results
Drill into any model to see per-task pass rates across its attempted tasks. This build contains aggregate results only (no per-trial pages).
claude-fable-5
410 trials20.5%
claude-haiku-4-5-20251001
410 trials92.4%
claude-opus-4-8
410 trials96.1%
claude-sonnet-4-6
410 trials96.8%
claude-sonnet-5
410 trials97.6%
deepseek-v4-flash
410 trials95.1%
deepseek-v4-pro
410 trials95.4%
glm-5p2
410 trials95.6%
gpt-5-mini
410 trials89.5%
gpt-5.4-mini
410 trials74.4%
gpt-5.5
410 trials96.3%
gpt-5.6-luna
410 trials94.4%
gpt-5.6-sol
410 trials95.9%
gpt-5.6-terra
410 trials93.9%
gpt-oss-120b
410 trials91.0%
gpt-oss-20b
410 trials87.1%
kimi-k2p5
410 trials93.9%
kimi-k2p6
410 trials95.4%
kimi-k2p7-code
410 trials96.6%
kimi-k3
410 trials99.0%
minimax-m2p7
410 trials93.4%
minimax-m3
410 trials41.5%
qwen3p6-plus
410 trials95.4%
qwen3p7-plus
410 trials93.2%