Overview›Statefulness

APIFlow-Bench 1.0 Leaderboard | Statefulness

Whether a model keeps track of server-side state it created or changed earlier in the episode and stays consistent with it in later calls. 76 tasks × 24 models in this slice.

Ranking◴ Generated 2026-07-31☑ 9,118 trials▤ 24 models

Pass rate

83.3%

Error Rate

0.0%

Trials

9,118

Tasks

Models

p95 Time

11.3m

p95 Tokens

594,304

Tool Calls Avg

Rank	Rank Spread	Model	Pass rate ± 90% CI	Pass (no abstain)	Abstain%	p95 Time	p95 Tokens	Avg Tools
1	1 ↔ 5	kimi-k3	93.9% 89.7%–97.6%	94.2% 357/379	0.3% 1/380	25.0m	475,547	15
2	1 ↔ 10	claude-sonnet-4-6	92.4% 87.6%–96.6%	92.4% 351/380	0.0% 0/380	1.9m	122,343	14.4
3	1 ↔ 11	claude-sonnet-5	92.1% 86.8%–96.6%	92.6% 350/378	0.5% 2/380	2.8m	269,878	15.2
4	1 ↔ 12	claude-opus-4-8	91.8% 86.6%–96.8%	94.3% 349/370	2.6% 10/380	1.6m	129,368	11.7
5	2 ↔ 15	qwen3p6-plus	91.3% 86.0%–95.8%	91.3% 346/379	0.0% 0/379	8.0m	362,107	15.7
6	2 ↔ 16	deepseek-v4-flash	90.8% 85.3%–95.5%	90.8% 345/380	0.0% 0/380	2.3m	252,798	19.5
7	3 ↔ 17	kimi-k2p6	90.5% 85.3%–95.3%	90.8% 344/379	0.3% 1/380	5.8m	741,226	17.1
8	3 ↔ 17	minimax-m2p7	90.3% 84.7%–95.3%	90.3% 343/380	0.0% 0/380	1.2m	154,096	12.8
8	3 ↔ 18	kimi-k2p7-code	90.3% 84.7%–95.5%	90.5% 343/379	0.3% 1/380	6.8m	209,244	14.4
10	3 ↔ 18	claude-haiku-4-5-20251001	90.0% 84.7%–95.0%	91.7% 342/373	1.8% 7/380	1.4m	156,362	15.9
11	4 ↔ 18	kimi-k2p5	89.7% 84.5%–94.7%	90.0% 341/379	0.3% 1/380	2.0m	116,759	12.7
11	4 ↔ 18	gpt-5.6-luna	89.7% 84.2%–94.5%	90.5% 341/377	0.8% 3/380	2.9m	73,961	13.8
11	4 ↔ 18	gpt-5.6-sol	89.7% 83.9%–94.7%	91.4% 341/373	1.8% 7/380	6.7m	135,028	14
14	5 ↔ 19	gpt-5.5	89.5% 83.9%–94.5%	89.5% 340/380	0.0% 0/380	1.7m	143,098	13.8
15	6 ↔ 19	deepseek-v4-pro	88.9% 83.2%–94.2%	88.9% 338/380	0.0% 0/380	9.2m	211,006	21.5
15	6 ↔ 19	glm-5p2	88.9% 83.4%–93.9%	88.9% 338/380	0.0% 0/380	19.4m	372,573	14.9
17	8 ↔ 20	qwen3p7-plus	88.2% 82.6%–92.9%	89.6% 335/374	1.6% 6/380	11.9m	584,855	13.2
18	9 ↔ 20	gpt-5.6-terra	87.9% 81.6%–93.4%	88.1% 334/379	0.3% 1/380	2.5m	88,613	13.2
19	11 ↔ 20	gpt-oss-20b	87.3% 81.4%–92.3%	89.9% 331/368	2.9% 11/379	4.0m	146,363	15
20	13 ↔ 21	gpt-oss-120b	86.6% 80.5%–92.1%	86.6% 329/380	0.0% 0/380	53.4s	136,404	13.2
21	20 ↔ 21	gpt-5-mini	82.9% 76.8%–88.9%	88.7% 315/355	6.6% 25/380	1.2m	108,759	12.1
22	22 ↔ 22	gpt-5.4-mini	69.2% 62.9%–75.5%	71.1% 263/370	2.6% 10/380	23.6s	39,312	11.7
23	23 ↔ 23	minimax-m3	32.9% 27.1%–38.4%	56.8% 125/220	42.1% 160/380	62.0m	14,473,879	155.1
24	24 ↔ 24	claude-fable-5	14.5% 9.2%–20.3%	14.5% 55/380	0.0% 0/380	2.4m	44,227	3

Diagnostics

Loading the full trial dataset for this view…