Robots Atlas>ROBOTS ATLAS
Evaluation

JEVal

2026ActivePublished: 17 November 2026Updated: 17 November 2026Published
Key innovation
The first bilingual (English-Chinese) benchmark that unifies the evaluation of general decision models, measuring not only accuracy but also confidence calibration (ECE, Brier) for structured decisions across 36 datasets and 10 domains.
Category
Evaluation
Abstraction level
Benchmark
Operation level
ModelSystem
Use cases
Benchmarking decision models and LLMs on structured decisionsMeasuring confidence calibration (ECE, Brier), not just accuracyEvaluating decisions across domains: medicine, law, finance, social scienceDiagnosing model overconfidence in decision tasks

How it works

Each JEVal instance consists of a question and a set of permitted labels/options; the model must return the chosen label together with an assigned probability. Instances come in one of three types: option selection (choice), judgment (noul), and numeric scoring (score). Classification responses (choice and noul, 10,938 items in total) are scored with accuracy and calibration metrics: accuracy (ACC) counts invalid responses as incorrect, while ECE and the raw Brier score are computed over formally valid responses only. The 319 score items are evaluated separately. The benchmark is distributed as a single Parquet file with a test split, retaining a source column so that results can be analysed per subtask and per domain.

Problem solved

Evaluations of language models on decision tasks were scattered across many inconsistent datasets and focused almost solely on accuracy, ignoring whether a model realistically estimates its own confidence. JEVal unifies these tasks into a single bilingual benchmark and adds calibration metrics (ECE, Brier), revealing that models can often identify the most likely outcome while substantially overstating its probability.

Evolution

Original paper · 2026 · arXiv preprint (2026) · Feiyu Duan
General Decision Models: Benchmarking and Insights Beyond Jev
Feiyu Duan, Jiayu Lin, Jia Wang, Jun Xiang, Jialiang Wu, Xinnong Zhang, Hanqi Yan, Siyuan Wang, Zhongyu Wei
2026
Release of JEVal and the SII-Decision models
Inflection point

The SII team and academic partners release JEVal (11,257 instances, 36 datasets, 10 domains) together with the SII-Decision-4B/27B models trained with Reasoning-to-Readout Self-Distillation.