Robots Atlas>ROBOTS ATLAS
Evaluation

Online-Mind2Web

2025ActivePublished: 6 August 2026Updated: 6 August 2026Published
Key innovation
Evaluating web agents on live, dynamic websites with human evaluation (rather than cached pages), which revealed a significant gap between older-benchmark scores and real-world success (an 'illusion of progress').
Category
Evaluation
Abstraction level
Pattern
Operation level
Evaluation (runtime)
Use cases
Evaluating web agents and CUAs on live sitesComparing models/agents with human evaluationDetecting the 'illusion of progress' vs synthetic benchmarksTracking progress in web task automationResearch on agent reliability

How it works

An agent is given a realistic task (one of 300) and performs it on a live site (one of 136 websites), navigating and interacting like a user. Afterwards, humans judge whether the task was actually completed (human rather than automatic evaluation), giving a reliable success metric. Because the sites are dynamic, the test reflects real conditions (UI, content and availability changes). Results for different agents/models go onto a shared leaderboard.

Problem solved

Older web-agent benchmarks relied on cached pages and automatic scoring, overstating agent success. A measure of real performance on live, changing sites with reliable evaluation was missing.

Components

Live-site tasks (300/136)Defines what is measured.

300 realistic tasks across 136 live websites, reflecting real usage.

Human evaluationA reliable success signal.

Humans judge whether the task was actually completed - instead of automatic, inflating scoring.

Live environment (no cache)Realistic conditions.

Real-time testing against dynamic, evolving interfaces.

Official

LeaderboardAggregation and comparison.

A public ranking of agent and model results.

Official

Implementation

Implementation pitfalls
Cost and scalability of human evaluationMedium

Human evaluation is reliable but costly and slower than automatic scoring.

Fix:Clear grading rubrics, sampling, partial automation with verification.
Instability of live sitesMedium

Changes on live sites can affect reproducibility and comparability of results over time.

Fix:Task versioning, site-availability monitoring and reporting the test date.

Evolution

Original paper · 2025 · arXiv:2504.01382 (OSU-NLP)
An Illusion of Progress? Assessing the Current State of Web Agents
2023
Mind2Web (original)

OSU-NLP introduced Mind2Web - a benchmark for generalist web agents, based on collected/cached data.

2025
Online-Mind2Web and the 'illusion of progress'
Inflection point

The live, human-evaluated version revealed that agents' real-world success is far below older benchmarks.