Development ยท Developer Tools
v1ยท
DrivingBench Harness is research software that lets a large language model (LLM) drive a real car at parking-lot speed while a licensed driver sits in the driver's seat with a foot over the brake. The harness exposes the car to any chat app that speaks the MCP protocol (Codex, Claude Code, Cursor) as one server with three tools: observe, set_motion and stop_now.
The system runs on a comma four device with openpilot (v0.11.2) in a supported Toyota (TSS2 models; developed and tested on a Toyota Corolla). A laptop (macOS, or Linux in foreground mode) acts as a gateway that reaches the device over an SSH tunnel and serves the operator UI. In DRIVINGBENCH_MODE=experimental, openpilot's 100 Hz control loop hands each tick to the harness controller, which turns the requested steering percent into a wheel-angle target and curvature and runs a small speed feedback loop.
Every tool call, telemetry (every 100 ms) and image the model saw is recorded on the device as a reproducible trace that can be replayed in the browser. The harness powers the DrivingBench v1 benchmark and a public leaderboard comparing models on a fixed cone course. The authors stress that this is research software for closed courses only and is not affiliated with comma.ai, openpilot, Toyota or any model maker.
A benchmark / evaluation harness is software that provides a standardized test environment: it defines tasks, collects measurements, records traces, and enables reproducible comparison of models or systems. It typically includes a control layer (running attempts), a results recorder, and metrics with a leaderboard.
A Developer Tool is software designed to support the development workflow, including configuration, debugging, testing, monitoring, validation, and integration of robotic and embedded systems. Examples include IDE plugins, visual debuggers, log analysers, hardware-in-the-loop (HIL) test harnesses, and code-generation utilities specific to robotics platforms.
Powers the DrivingBench v1 benchmark and its public leaderboard comparing frontier models on a fixed cone course.
A family of open-source operating systems based on the Linux kernel, common on servers, in the cloud and in robotics.
comma four running openpilot v0.11.2; a supported Toyota (TSS2 models) with the matching comma car harness; a laptop running macOS (or Linux in foreground mode) with Python 3.12 and the uv package manager; a private, empty lot and traffic cones.
An open protocol that connects large language models (LLMs) with external tools and data sources. An application exposes an MCP server with a set of tools the model can call and receive structured results from. Used by clients such as Claude Code, Codex and Cursor.
License family: Permissive
First public release of the DrivingBench harness: three MCP tools (observe, set_motion, stop_now), the native openpilot control hook, a laptop gateway, and trace recording and replay.
