Showing 5-subtask chain — 13 tasks, 1,235 trials. Back to the default 20-subtask chain view.
APIFlow-Bench 1.0 Leaderboard
Deterministic API engineering benchmark spanning 7 capability axes, over REST APIs only (GraphQL and MCP are planned). Showing the 5-subtask chain task set: 1,235 trials across 19 models and 13 tasks covering all 7 axes — authentication, discovery, error recovery, multi-step workflow, pagination, schema, statefulness. Switch task set above; click any card to drill into a slice with the full interactive detail view; click any model name to see its per-task results.
☑ 1,235 trials
▤ 19 models
⛁ 13 tasks
⌖ Overall pass rate: 89.8%
Overall Ranking
Columns are grouped into metric families.
Pass rate = passed ÷ all trials;
API recovery = passed ÷ trials that hit a 4xx/5xx. Difficulty bucket is judged per task from its content and reference solve transcripts (Easy / Long-horizon / Hard / Hard + Long-horizon) — a separate taxonomy from the creation-time “Difficulty (as created)” filter, so the two don't map one-to-one. Difficulty/horizon breakdowns are omitted: every task in this slice is Long-horizon.
Per-trial details (verifier groups, checkpoints, step counts) live on the model → task → trial pages.
| Rank | Model | Scoring | Difficulty bucket (pass rate) | Cost / efficiency | Behavioral |
|---|
| Pass rate? | API recovery? | Long-horizon? | Hard + Long-horizon? | Tool calls / trial? | Time / trial? | Tokens / trial? | Cost / trial? | Abstain %? | Refusal %? |
|---|
| 1 | kimi-k2p6 | 100.0%65/65 | 35/35 | 100%5/5 | 100%60/60 | 22.1 | 1.8m | 193.8K | $0.088 | 0.0%0/65 | 0.0%0/65 |
| 2 | deepseek-v4-flash | 98.5%64/65 | 36/37 | 100%5/5 | 98%59/60 | 21.9 | 57.6s | 101.0K | $0.009 | 0.0%0/65 | 0.0%0/65 |
| 3 | claude-haiku-4-5-20251001 | 96.9%63/65 | 31/33 | 100%5/5 | 97%58/60 | 32.0 | 55.5s | 333.1K | $0.058 | 0.0%0/65 | 0.0%0/65 |
| 4 | kimi-k2p7-code | 95.4%62/65 | 32/35 | 100%5/5 | 95%57/60 | 16.9 | 3.0m | 73.4K | $0.032 | 0.0%0/65 | 0.0%0/65 |
| 5 | kimi-k2p5 | 92.3%60/65 | 30/35 | 100%5/5 | 92%55/60 | 12.8 | 31.5s | 44.6K | $0.021 | 0.0%0/65 | 0.0%0/65 |
| 5 | claude-opus-4-8 | 92.3%60/65 | 26/31 | 100%5/5 | 92%55/60 | 13.7 | 45.4s | 74.4K | $0.088 | 0.0%0/65 | 0.0%0/65 |
| 5 | minimax-m2p7 | 92.3%60/65 | 33/38 | 100%5/5 | 92%55/60 | 18.3 | 30.6s | 103.8K | $0.023 | 1.5%1/65 | 0.0%0/65 |
| 5 | gpt-5.5 | 92.3%60/65 | 28/33 | 100%5/5 | 92%55/60 | 18.6 | 45.0s | 68.0K | $0.14 | 0.0%0/65 | 0.0%0/65 |
| 5 | deepseek-v4-pro | 92.3%60/65 | 40/45 | 100%5/5 | 92%55/60 | 25.5 | 3.9m | 111.9K | $0.13 | 0.0%0/65 | 0.0%0/65 |
| 5 | glm-5p2 | 92.3%60/65 | 30/35 | 100%5/5 | 92%55/60 | 20.6 | 5.8m | 243.4K | $0.15 | 0.0%0/65 | 0.0%0/65 |
| 11 | gpt-5.6-terra | 90.8%59/65 | 25/30 | 100%5/5 | 90%54/60 | 16.2 | 1.6m | 50.3K | $0.050 | 0.0%0/65 | 0.0%0/65 |
| 11 | qwen3p7-plus | 90.8%59/65 | 27/32 | 100%5/5 | 90%54/60 | 13.0 | 4.7m | 153.1K | $0.042 | 0.0%0/65 | 0.0%0/65 |
| 13 | gpt-oss-120b | 86.2%56/65 | 30/39 | 100%5/5 | 85%51/60 | 16.5 | 27.3s | 80.5K | $0.008 | 0.0%0/65 | 0.0%0/65 |
| 13 | gpt-oss-20b | 86.2%56/65 | 34/42 | 100%5/5 | 85%51/60 | 17.6 | 1.5m | 85.8K | $0.004 | 6.2%4/65 | 0.0%0/65 |
| 13 | qwen3p6-plus | 86.2%56/65 | 28/35 | 100%5/5 | 85%51/60 | 26.2 | 2.1m | 224.0K | $0.089 | 0.0%0/65 | 0.0%0/65 |
| 13 | claude-sonnet-4-6 | 86.2%56/65 | 21/30 | 100%5/5 | 85%51/60 | 77.7 | 4.8m | 9.4M | $2.95 | 0.0%0/65 | 0.0%0/65 |
| 17 | claude-sonnet-5 | 84.6%55/65 | 23/33 | 100%5/5 | 83%50/60 | 19.8 | 1.0m | 118.8K | $0.059 | 0.0%0/65 | 0.0%0/65 |
| 18 | gpt-5-mini | 83.1%54/65 | 31/34 | 100%5/5 | 82%49/60 | 13.3 | 34.5s | 59.4K | $0.007 | 13.8%9/65 | 0.0%0/65 |
| 19 | gpt-5.4-mini | 67.7%44/65 | 21/36 | 100%5/5 | 65%39/60 | 13.6 | 17.3s | 28.3K | $0.010 | 4.6%3/65 | 0.0%0/65 |
Slices
▥Behavior Mix
Top 10 of 19 models
passed?failed?abstained (clarify / report_blocked)?no-score?
| Rank? | Model? | Mix? | Pass rate? |
|---|
| 1 | kimi-k2p6 |
| 100.0% |
| 2 | deepseek-v4-flash |
| 98.5% |
| 3 | claude-haiku-4-5-20251001 |
| 96.9% |
| 4 | kimi-k2p7-code |
| 95.4% |
| 5 | kimi-k2p5 |
| 92.3% |
| 5 | claude-opus-4-8 |
| 92.3% |
| 5 | minimax-m2p7 |
| 92.3% |
| 5 | gpt-5.5 |
| 92.3% |
| 5 | deepseek-v4-pro |
| 92.3% |
| 5 | glm-5p2 |
| 92.3% |
🔐Authentication
95 trials
| Rank? | Model? | Pass rate? |
|---|
| 1 | 15 models tied at 100% | 100.0% |
| 16 | claude-opus-4-8 | 80.0% |
| 17 | gpt-oss-20b | 60.0% |
| 18 | gpt-5.4-mini | 20.0% |
| 19 | claude-sonnet-5 | 0.0% |
| Rank? | Model? | Pass rate? |
|---|
| 1 | 15 models tied at 100% | 100.0% |
| 16 | gpt-oss-120b | 90.0% |
| 16 | gpt-oss-20b | 90.0% |
| 18 | qwen3p6-plus | 80.0% |
| 19 | gpt-5.4-mini | 50.0% |
↻Error Recovery
475 trials
| Rank? | Model? | Pass rate? |
|---|
| 1 | 15 models tied at 100% | 100.0% |
| 16 | gpt-oss-120b | 96.0% |
| 16 | gpt-5.6-terra | 96.0% |
| 16 | gpt-oss-20b | 96.0% |
| 19 | gpt-5-mini | 92.0% |
⛓Multi-step Workflow
95 trials
| Rank? | Model? | Pass rate? |
|---|
| 1 | 17 models tied at 100% | 100.0% |
| 18 | gpt-5.4-mini | 80.0% |
| 19 | gpt-5-mini | 0.0% |
| Rank? | Model? | Pass rate? |
|---|
| 1 | 13 models tied at 100% | 100.0% |
| 14 | minimax-m2p7 | 80.0% |
| 14 | gpt-oss-20b | 80.0% |
| 16 | gpt-5-mini | 60.0% |
| 16 | qwen3p6-plus | 60.0% |
| 18 | gpt-5.4-mini | 40.0% |
| 19 | claude-sonnet-4-6 | 0.0% |
| Rank? | Model? | Pass rate? |
|---|
| 1 | kimi-k2p6 | 100.0% |
| 2 | gpt-5-mini | 80.0% |
| 2 | gpt-oss-20b | 80.0% |
| 2 | deepseek-v4-flash | 80.0% |
| 5 | claude-haiku-4-5-20251001 | 60.0% |
| 6 | kimi-k2p7-code | 40.0% |
| 7 | claude-opus-4-8 | 20.0% |
| 7 | minimax-m2p7 | 20.0% |
| 7 | claude-sonnet-4-6 | 20.0% |
| 10 | gpt-5.4-mini | 0.0% |
+9 more models below — click for the full ranking of all 19.
| Rank? | Model? | Pass rate? |
|---|
| 1 | 14 models tied at 100% | 100.0% |
| 15 | gpt-5-mini | 90.0% |
| 15 | qwen3p7-plus | 90.0% |
| 17 | gpt-oss-120b | 80.0% |
| 18 | gpt-5.4-mini | 70.0% |
| 18 | gpt-oss-20b | 70.0% |
Per-Model Results
Drill into any model to see per-task pass rates across its attempted tasks. This build contains aggregate results only (no per-trial pages).
