A benchmark defines a set of tasks with known answers and a metric (e.g. accuracy, pass@k, Elo). A model is run on the same set and the results are posted on a leaderboard. Good benchmarks include a held-out test set to reduce data leakage and overfitting.
Without common, standardized tests it is impossible to objectively compare which model is better at a task or to track the progress of the field.
