Who leads the mmlu-pro benchmark?

Darwin 180B RSI currently leads mmlu-pro with a score of 88.12 on Accuracy.

What is the state-of-the-art score on mmlu-pro?

The state-of-the-art result on mmlu-pro is 88.12 (Accuracy), achieved by Darwin 180B RSI as of 2024.

How many models are tracked on mmlu-pro?

Codesota tracks 146 models on mmlu-pro.

When was the mmlu-pro leaderboard last updated?

The mmlu-pro leaderboard on Codesota includes results through 2024.

Codesota · Benchmark · mmlu-proHome/Leaderboards/mmlu-pro

Unknown

mmlu-pro.

Name: mmlu-pro Benchmark Results
Creator: Unknown
Published: 2024-01-01
License: https://creativecommons.org/licenses/by/4.0/

mmlu-pro is a state-of-the-art machine learning benchmark indexed on Codesota. This page tracks published model results, top scores per metric, and the SOTA timeline for mmlu-pro.

Paper ↗Leaderboard ↓Lineage

§ 01 · Leaderboard

Results by metric.

Found a wrong score or missing run?

Use row edits to send a sourced correction into moderation.

Add / edit result ↗Report issue ↗

Accuracy

Accuracy is the reported evaluation metric for mmlu-pro. Codesota tracks published model scores on this metric so readers can compare state-of-the-art results across sources and model families.

Higher is better

Trust tiers for Accuracyverifiedpapervendorcommunityunverified

Muted rows were not state of the art when published — an earlier or same-year result already scored better.

Rank	Model	Trust	Score	Year	Links	Fix
01	Darwin 180B RSI FINAL-Bench/Darwin-180B-RSI	vendor	88.12	N/A	Code ↗	Looks wrong?
02	MiniMax M2.1 MiniMaxAI/MiniMax-M2.1	vendor	88	N/A	Code ↗Source ↗	Looks wrong?
03	Intern S2 Preview internlm/Intern-S2-Preview	vendor	88	N/A	Code ↗	Looks wrong?
04	Qwen3.5 397B A17B Qwen/Qwen3.5-397B-A17B	vendor	87.8	N/A	Code ↗	Looks wrong?
05	DeepSeek V4 Pro deepseek-ai/DeepSeek-V4-Pro	vendor	87.5	N/A	Code ↗	Looks wrong?
06	Kimi K2.5 moonshotai/Kimi-K2.5	vendor	87.1	N/A	Code ↗	Looks wrong?
07	NVIDIA Nemotron 3 Ultra 550B A55B BF16 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16	vendor	86.8	N/A	Code ↗	Looks wrong?
08	NVIDIA Nemotron 3 Ultra 550B A55B NVFP4 nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4	vendor	86.8	N/A	Code ↗	Looks wrong?
09	Qwen3.5 122B A10B Qwen/Qwen3.5-122B-A10B	vendor	86.7	N/A	Code ↗Source ↗	Looks wrong?
10	DeepSeek V4 Flash deepseek-ai/DeepSeek-V4-Flash	vendor	86.4	N/A	Code ↗	Looks wrong?
11	Qwen3.6 27B Qwen/Qwen3.6-27B	vendor	86.2	N/A	Code ↗	Looks wrong?
12	Solar Open2 250B upstage/Solar-Open2-250B	vendor	86.2	N/A	Code ↗	Looks wrong?
13	Qwen3.5 27B Qwen/Qwen3.5-27B	vendor	86.1	N/A	Code ↗Source ↗	Looks wrong?
14	GLM 5 zai-org/GLM-5	vendor	86	N/A	Code ↗Source ↗	Looks wrong?
15	Qwen3.6 35B A3B Qwen/Qwen3.6-35B-A3B	vendor	85.2	N/A	Code ↗	Looks wrong?
16	DeepSeek R1 0528 deepseek-ai/DeepSeek-R1-0528	vendor	85	N/A	Code ↗	Looks wrong?
17	GLM 4.5 zai-org/GLM-4.5	vendor	84.6	N/A	Code ↗Source ↗	Looks wrong?
18	Step 3.5 Flash stepfun-ai/Step-3.5-Flash	vendor	84.4	N/A	Code ↗Source ↗	Looks wrong?
19	DeepSeek R1 deepseek-ai/DeepSeek-R1	vendor	84	N/A	Code ↗	Looks wrong?
20	K EXAONE 236B A23B LGAI-EXAONE/K-EXAONE-236B-A23B	vendor	83.8	N/A	Code ↗	Looks wrong?
21	NVIDIA Nemotron 3 Super 120B A12B BF16 nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16	vendor	83.73	N/A	Code ↗	Looks wrong?
22	Intern S1 internlm/Intern-S1	vendor	83.5	N/A	Code ↗Source ↗	Looks wrong?
23	EXAONE 4.5 33B LGAI-EXAONE/EXAONE-4.5-33B	vendor	83.3	N/A	Code ↗	Looks wrong?
24	Qwen3 235B A22B Instruct 2507 Qwen/Qwen3-235B-A22B-Instruct-2507	vendor	83	N/A	Code ↗Source ↗	Looks wrong?
25	Seed OSS 36B Instruct ByteDance-Seed/Seed-OSS-36B-Instruct	vendor	82.7	N/A	Code ↗Source ↗	Looks wrong?
26	LongCat Flash Chat meituan-longcat/LongCat-Flash-Chat	vendor	82.7	N/A	Code ↗Source ↗	Looks wrong?
27	MiniMax M2 MiniMaxAI/MiniMax-M2	vendor	82	N/A	Code ↗	Looks wrong?
28	NVIDIA Nemotron 3.5 Lightning 30B A3B BF16 nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16	vendor	81.94	N/A	Code ↗	Looks wrong?
29	NVIDIA Nemotron 3.5 Lightning 30B A3B NVFP4 nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4	vendor	81.62	N/A	Code ↗	Looks wrong?
30	A.X K1 skt/A.X-K1	vendor	81.5	N/A	Code ↗	Looks wrong?
31	GLM 4.5 Air zai-org/GLM-4.5-Air	vendor	81.4	N/A	Code ↗Source ↗	Looks wrong?
32	DeepSeek V3 0324 deepseek-ai/DeepSeek-V3-0324	vendor	81.2	N/A	Code ↗	Looks wrong?
33	MiniMax M1 40k MiniMaxAI/MiniMax-M1-40k	vendor	81.1	N/A	Code ↗Source ↗	Looks wrong?
34	JoyAI LLM Flash jdopensource/JoyAI-LLM-Flash	vendor	81.02	N/A	Code ↗	Looks wrong?
35	Kimi K2 Instruct moonshotai/Kimi-K2-Instruct	vendor	81	N/A	Code ↗Source ↗	Looks wrong?
36	Qwen3 30B A3B Thinking 2507 Qwen/Qwen3-30B-A3B-Thinking-2507	vendor	80.9	N/A	Code ↗Source ↗	Looks wrong?
37	gpt oss 120b openai/gpt-oss-120b	vendor	80.8	N/A	Code ↗Source ↗	Looks wrong?
38	MiniMax M2.5 MiniMaxAI/MiniMax-M2.5	vendor	80.1	N/A	Code ↗Source ↗	Looks wrong?
39	ERNIE 4.5 300B A47B PT baidu/ERNIE-4.5-300B-A47B-PT	vendor	78.4	N/A	Code ↗Source ↗	Looks wrong?
40	LongCat Flash Lite meituan-longcat/LongCat-Flash-Lite	vendor	78.29	N/A	Code ↗	Looks wrong?
41	MiniMax Text 01 MiniMaxAI/MiniMax-Text-01	vendor	75.7	N/A	Code ↗Source ↗	Looks wrong?
42	gpt oss 20b openai/gpt-oss-20b	vendor	73.6	N/A	Code ↗Source ↗	Looks wrong?
43	GPT-4o Original MMLU-Pro paper, 5-shot CoT	paper	72.6	2024	Paper ↗	Looks wrong?
44	Qwen2.5 72B Qwen/Qwen2.5-72B	vendor	71.59	N/A	Code ↗Source ↗	Looks wrong?
45	phi 4 microsoft/phi-4	vendor	70.4	N/A	Code ↗Source ↗	Looks wrong?
46	Qwen3 4B Instruct 2507 Qwen/Qwen3-4B-Instruct-2507	vendor	69.6	N/A	Code ↗	Looks wrong?
47	ERNIE 4.5 300B A47B Base PT baidu/ERNIE-4.5-300B-A47B-Base-PT	vendor	69.5	N/A	Code ↗Source ↗	Looks wrong?
48	Qwen2.5 32B Qwen/Qwen2.5-32B	vendor	69.23	N/A	Code ↗Source ↗	Looks wrong?
49	Gemini 1.5 Pro Original MMLU-Pro paper, 5-shot CoT	paper	69	2024	Paper ↗	Looks wrong?
50	MiMo V2.5 Pro XiaomiMiMo/MiMo-V2.5-Pro	vendor	68.5	N/A	Code ↗	Looks wrong?
51	Claude 3 Opus Original MMLU-Pro paper, 5-shot CoT	paper	68.5	2024	Paper ↗	Looks wrong?
52	Qwen3 235B A22B Qwen/Qwen3-235B-A22B	vendor	68.18	N/A	Code ↗Source ↗	Looks wrong?
53	Mistral Large Instruct 2411 mistralai/Mistral-Large-Instruct-2411	vendor	67.94	N/A	Code ↗Source ↗	Looks wrong?
54	Hunyuan A13B Instruct tencent/Hunyuan-A13B-Instruct	vendor	67.3	N/A	Code ↗Source ↗	Looks wrong?
55	Mistral Large Instruct 2407 mistralai/Mistral-Large-Instruct-2407	vendor	65.91	N/A	Code ↗Source ↗	Looks wrong?
56	DeepSeek V2.5 deepseek-ai/DeepSeek-V2.5	vendor	65.83	N/A	Code ↗Source ↗	Looks wrong?
57	Seed OSS 36B Base ByteDance-Seed/Seed-OSS-36B-Base	vendor	65.1	N/A	Code ↗Source ↗	Looks wrong?
58	NVIDIA Nemotron 3 Nano 30B A3B Base BF16 nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16	vendor	65.1	N/A	Code ↗Source ↗	Looks wrong?
59	DeepSeek V3 deepseek-ai/DeepSeek-V3	vendor	64.4	N/A	Code ↗	Looks wrong?
60	granite 4.1 30b ibm-granite/granite-4.1-30b	vendor	64.09	N/A	Code ↗	Looks wrong?
61	GPT-4-Turbo Original MMLU-Pro paper, 5-shot CoT	paper	63.7	2024	Paper ↗	Looks wrong?
62	Qwen2.5 14B Qwen/Qwen2.5-14B	vendor	63.69	N/A	Code ↗Source ↗	Looks wrong?
63	Qwen3 30B A3B Base Qwen/Qwen3-30B-A3B-Base	vendor	61.7	N/A	Code ↗Source ↗	Looks wrong?
64	Llama 3.1 405B meta-llama/Llama-3.1-405B	vendor	61.6	N/A	Code ↗Source ↗	Looks wrong?
65	Nemotron H 56B Base 8K nvidia/Nemotron-H-56B-Base-8K	vendor	60.5	N/A	Code ↗Source ↗	Looks wrong?
66	Seed OSS 36B Base woSyn ByteDance-Seed/Seed-OSS-36B-Base-woSyn	vendor	60.4	N/A	Code ↗Source ↗	Looks wrong?
67	Tencent Hunyuan Large tencent/Tencent-Hunyuan-Large	vendor	60.2	N/A	Code ↗Source ↗	Looks wrong?
68	Mellum2 12B A2.5B Base Pretrain JetBrains/Mellum2-12B-A2.5B-Base-Pretrain	vendor	59.31	N/A	Code ↗	Looks wrong?
69	Mellum2 12B A2.5B Base JetBrains/Mellum2-12B-A2.5B-Base	vendor	59.31	N/A	Code ↗	Looks wrong?
70	Gemini 1.5 Flash Original MMLU-Pro paper, 5-shot CoT	paper	59.1	2024	Paper ↗	Looks wrong?
71	EXAONE 3.5 32B Instruct LGAI-EXAONE/EXAONE-3.5-32B-Instruct	vendor	58.91	N/A	Code ↗Source ↗	Looks wrong?
72	MiMo 7B RL XiaomiMiMo/MiMo-7B-RL	vendor	58.6	N/A	Code ↗Source ↗	Looks wrong?
73	internlm3 8b instruct internlm/internlm3-8b-instruct	vendor	57.6	N/A	Code ↗Source ↗	Looks wrong?
74	ERNIE 4.5 21B A3B Base PT baidu/ERNIE-4.5-21B-A3B-Base-PT	vendor	56.7	N/A	Code ↗Source ↗	Looks wrong?
75	Llama 3 70B Instruct Original MMLU-Pro paper, 5-shot CoT	paper	56.2	2024	Paper ↗	Looks wrong?
76	granite 4.1 8b ibm-granite/granite-4.1-8b	vendor	55.99	N/A	Code ↗	Looks wrong?
77	Phi 3 medium 4k instruct microsoft/Phi-3-medium-4k-instruct	vendor	55.7	N/A	Code ↗Source ↗	Looks wrong?
78	DeepSeek V2 Chat deepseek-ai/DeepSeek-V2-Chat	vendor	54.81	N/A	Code ↗Source ↗	Looks wrong?
79	Mistral Small 24B Base 2501 mistralai/Mistral-Small-24B-Base-2501	vendor	54.4	N/A	Code ↗Source ↗	Looks wrong?
80	Phi 4 mini instruct microsoft/Phi-4-mini-instruct	vendor	52.8	N/A	Code ↗Source ↗	Looks wrong?
81	Meta Llama 3 70B meta-llama/Meta-Llama-3-70B	vendor	52.78	N/A	Code ↗Source ↗	Looks wrong?
82	Llama 3.1 70B meta-llama/Llama-3.1-70B	vendor	52.47	N/A	Code ↗Source ↗	Looks wrong?
83	Yi 1.5 34B Chat 01-ai/Yi-1.5-34B-Chat	vendor	52.29	N/A	Code ↗Source ↗	Looks wrong?
84	Phi 3 medium 128k instruct microsoft/Phi-3-medium-128k-instruct	vendor	51.91	N/A	Code ↗Source ↗	Looks wrong?
85	MAmmoTH2 8x7B Plus TIGER-Lab/MAmmoTH2-8x7B-Plus	vendor	50.4	N/A	Code ↗Source ↗	Looks wrong?
86	Qwen1.5 110B Qwen/Qwen1.5-110B	vendor	49.93	N/A	Code ↗Source ↗	Looks wrong?
87	granite 4.1 3b ibm-granite/granite-4.1-3b	vendor	49.83	N/A	Code ↗	Looks wrong?
88	AI21 Jamba Large 1.5 ai21labs/AI21-Jamba-Large-1.5	vendor	49.46	N/A	Code ↗Source ↗	Looks wrong?
89	Mistral Small Instruct 2409 mistralai/Mistral-Small-Instruct-2409	vendor	48.4	N/A	Code ↗Source ↗	Looks wrong?
90	glm 4 9b zai-org/glm-4-9b	vendor	47.92	N/A	Code ↗Source ↗	Looks wrong?
91	Phi 3.5 mini instruct microsoft/Phi-3.5-mini-instruct	vendor	47.87	N/A	Code ↗Source ↗	Looks wrong?
92	EXAONE 3.5 7.8B Instruct LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct	vendor	46.24	N/A	Code ↗Source ↗	Looks wrong?
93	Yi 1.5 9B Chat 01-ai/Yi-1.5-9B-Chat	vendor	45.95	N/A	Code ↗Source ↗	Looks wrong?
94	Phi 3 mini 4k instruct microsoft/Phi-3-mini-4k-instruct	vendor	45.66	N/A	Code ↗Source ↗	Looks wrong?
95	aya expanse 32b CohereLabs/aya-expanse-32b	vendor	45.41	N/A	Code ↗Source ↗	Looks wrong?
96	gemma 2 9b google/gemma-2-9b	vendor	45.1	N/A	Code ↗Source ↗	Looks wrong?
97	Qwen2.5 7B Qwen/Qwen2.5-7B	vendor	45	N/A	Code ↗Source ↗	Looks wrong?
98	Phi 3 mini 128k instruct microsoft/Phi-3-mini-128k-instruct	vendor	43.86	N/A	Code ↗Source ↗	Looks wrong?
99	Qwen2.5 3B Qwen/Qwen2.5-3B	vendor	43.73	N/A	Code ↗Source ↗	Looks wrong?
100	MAmmoTH2 8B Plus TIGER-Lab/MAmmoTH2-8B-Plus	vendor	43.35	N/A	Code ↗Source ↗	Looks wrong?
101	Yi 34B 01-ai/Yi-34B	vendor	43.03	N/A	Code ↗Source ↗	Looks wrong?
102	Mathstral 7B v0.1 mistralai/Mathstral-7B-v0.1	vendor	42	N/A	Code ↗Source ↗	Looks wrong?
103	MiMo 7B Base XiaomiMiMo/MiMo-7B-Base	vendor	41.9	N/A	Code ↗Source ↗	Looks wrong?
104	DeepSeek Coder V2 Lite Instruct deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct	vendor	41.57	N/A	Code ↗Source ↗	Looks wrong?
105	Mixtral 8x7B v0.1 mistralai/Mixtral-8x7B-v0.1	vendor	41.03	N/A	Code ↗Source ↗	Looks wrong?
106	Meta Llama 3 8B Instruct meta-llama/Meta-Llama-3-8B-Instruct	vendor	40.98	N/A	Code ↗Source ↗	Looks wrong?
107	MAmmoTH2 7B Plus TIGER-Lab/MAmmoTH2-7B-Plus	vendor	40.85	N/A	Code ↗Source ↗	Looks wrong?
108	Qwen2 7B Qwen/Qwen2-7B	vendor	40.73	N/A	Code ↗Source ↗	Looks wrong?
109	Mistral Nemo Base 2407 mistralai/Mistral-Nemo-Base-2407	vendor	39.77	N/A	Code ↗Source ↗	Looks wrong?
110	EXAONE 3.5 2.4B Instruct LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct	vendor	39.1	N/A	Code ↗Source ↗	Looks wrong?
111	Yi 1.5 6B Chat 01-ai/Yi-1.5-6B-Chat	vendor	38.23	N/A	Code ↗Source ↗	Looks wrong?
112	Qwen1.5 14B Chat Qwen/Qwen1.5-14B-Chat	vendor	38.02	N/A	Code ↗Source ↗	Looks wrong?
113	Ministral 8B Instruct 2410 mistralai/Ministral-8B-Instruct-2410	vendor	37.93	N/A	Code ↗Source ↗	Looks wrong?
114	c4ai command r v01 CohereLabs/c4ai-command-r-v01	vendor	37.9	N/A	Code ↗Source ↗	Looks wrong?
115	internlm2 math plus 20b internlm/internlm2-math-plus-20b	vendor	37.1	N/A	Code ↗Source ↗	Looks wrong?
116	LLaDA 8B Instruct GSAI-ML/LLaDA-8B-Instruct	vendor	37	N/A	Code ↗Source ↗	Looks wrong?
117	Llama 3 Smaug 8B abacusai/Llama-3-Smaug-8B	vendor	36.93	N/A	Code ↗Source ↗	Looks wrong?
118	Llama 3.1 8B meta-llama/Llama-3.1-8B	vendor	36.6	N/A	Code ↗Source ↗	Looks wrong?
119	Meta Llama 3 8B meta-llama/Meta-Llama-3-8B	vendor	35.36	N/A	Code ↗Source ↗	Looks wrong?
120	deepseek math 7b instruct deepseek-ai/deepseek-math-7b-instruct	vendor	35.3	N/A	Code ↗Source ↗	Looks wrong?
121	DeepSeek Coder V2 Lite Base deepseek-ai/DeepSeek-Coder-V2-Lite-Base	vendor	34.37	N/A	Code ↗Source ↗	Looks wrong?
122	aya expanse 8b CohereLabs/aya-expanse-8b	vendor	33.74	N/A	Code ↗Source ↗	Looks wrong?
123	internlm2 math plus 7b internlm/internlm2-math-plus-7b	vendor	33.5	N/A	Code ↗Source ↗	Looks wrong?
124	granite 3.1 8b base ibm-granite/granite-3.1-8b-base	vendor	33.08	N/A	Code ↗Source ↗	Looks wrong?
125	Qwen2.5 1.5B Qwen/Qwen2.5-1.5B	vendor	32.1	N/A	Code ↗Source ↗	Looks wrong?
126	granite 3.0 8b base ibm-granite/granite-3.0-8b-base	vendor	31.03	N/A	Code ↗Source ↗	Looks wrong?
127	Mistral 7B Instruct v0.2 mistralai/Mistral-7B-Instruct-v0.2	vendor	30.84	N/A	Code ↗Source ↗	Looks wrong?
128	Mistral 7B v0.2 mistral-community/Mistral-7B-v0.2	vendor	30.43	N/A	Code ↗Source ↗	Looks wrong?
129	Qwen1.5 7B Chat Qwen/Qwen1.5-7B-Chat	vendor	29.06	N/A	Code ↗Source ↗	Looks wrong?
130	Yi 6B Chat 01-ai/Yi-6B-Chat	vendor	28.84	N/A	Code ↗Source ↗	Looks wrong?
131	Yi 6B 01-ai/Yi-6B	vendor	26.51	N/A	Code ↗Source ↗	Looks wrong?
132	granite 3.1 2b base ibm-granite/granite-3.1-2b-base	vendor	23.89	N/A	Code ↗Source ↗	Looks wrong?
133	llemma 7b EleutherAI/llemma_7b	vendor	23.45	N/A	Code ↗Source ↗	Looks wrong?
134	Qwen2 1.5B Instruct Qwen/Qwen2-1.5B-Instruct	vendor	22.62	N/A	Code ↗Source ↗	Looks wrong?
135	Qwen2 1.5B Qwen/Qwen2-1.5B	vendor	22.56	N/A	Code ↗Source ↗	Looks wrong?
136	Llama 3.2 3B meta-llama/Llama-3.2-3B	vendor	22.17	N/A	Code ↗Source ↗	Looks wrong?
137	granite 3.0 2b base ibm-granite/granite-3.0-2b-base	vendor	21.72	N/A	Code ↗Source ↗	Looks wrong?
138	granite 3.1 3b a800m base ibm-granite/granite-3.1-3b-a800m-base	vendor	20.39	N/A	Code ↗Source ↗	Looks wrong?
139	SmolLM2 1.7B HuggingFaceTB/SmolLM2-1.7B	vendor	18.31	N/A	Code ↗Source ↗	Looks wrong?
140	Qwen2 0.5B Qwen/Qwen2-0.5B	vendor	14.97	N/A	Code ↗Source ↗	Looks wrong?
141	Qwen2.5 0.5B Qwen/Qwen2.5-0.5B	vendor	14.92	N/A	Code ↗Source ↗	Looks wrong?
142	granite 3.1 1b a400m base ibm-granite/granite-3.1-1b-a400m-base	vendor	12.34	N/A	Code ↗Source ↗	Looks wrong?
143	Llama 3.2 1B meta-llama/Llama-3.2-1B	vendor	11.95	N/A	Code ↗Source ↗	Looks wrong?
144	SmolLM2 360M HuggingFaceTB/SmolLM2-360M	vendor	11.38	N/A	Code ↗Source ↗	Looks wrong?
145	SmolLM2 135M HuggingFaceTB/SmolLM2-135M	vendor	10.85	N/A	Code ↗Source ↗	Looks wrong?
146	Qwen2.5 VL 72B Instruct Qwen/Qwen2.5-VL-72B-Instruct	vendor	0.65	N/A	Code ↗Source ↗	Looks wrong?