JEVal
How it works
Each JEVal instance consists of a question and a set of permitted labels/options; the model must return the chosen label together with an assigned probability. Instances come in one of three types: option selection (choice), judgment (noul), and numeric scoring (score). Classification responses (choice and noul, 10,938 items in total) are scored with accuracy and calibration metrics: accuracy (ACC) counts invalid responses as incorrect, while ECE and the raw Brier score are computed over formally valid responses only. The 319 score items are evaluated separately. The benchmark is distributed as a single Parquet file with a test split, retaining a source column so that results can be analysed per subtask and per domain.
Problem solved
Evaluations of language models on decision tasks were scattered across many inconsistent datasets and focused almost solely on accuracy, ignoring whether a model realistically estimates its own confidence. JEVal unifies these tasks into a single bilingual benchmark and adds calibration metrics (ECE, Brier), revealing that models can often identify the most likely outcome while substantially overstating its probability.
Evolution
The SII team and academic partners release JEVal (11,257 instances, 36 datasets, 10 domains) together with the SII-Decision-4B/27B models trained with Reasoning-to-Readout Self-Distillation.