A team at Tsinghua University's Department of Automation, led by Professor Tao Jianhua, has introduced SEED — a self-evolving on-policy distillation method that brings hindsight supervision to the training of AI agents. The method learns from completed trajectories: extracting strategies from successes and correction rules from failures. The work was published as a preprint on arXiv (2607.14777).
Key takeaways
- On the ALFWorld benchmark SEED improves success rate by 45.9 percentage points over the GRPO baseline.
- The method reaches GRPO-level performance using only 60% of the training data.
- On unseen tasks the macro-average success rate rises from 70.9% to 86.2%.
- On vision tasks (Sokoban and EZPoints) average success rises from 77.0% to 91.0%.
- SEED beats Skill-Prompt, OPSD, Skill-SD and GRPO across ALFWorld, Search-based QA and WebShop.
The problem: an outcome reward does not say what to fix
In long, multi-step agentic tasks, classic outcome-based reinforcement learning has a gap. The model gets a signal only about whether the task succeeded or not — but it does not know which specific observations, actions or tool calls in the trajectory?trajectory: One full agent run: the sequence of observations, actions and the final outcome in a single task. deserved reinforcement.
Over dozens of steps this is like grading a whole exam with one letter: pass or fail, with no indication of which answers were good. The Tsinghua authors attack this gap with an idea they call hindsight supervision (事后监督) — learning after the fact, once a trajectory is complete.
Instead of judging only the final outcome, SEED analyzes the entire path and extracts "hindsight skills" from it: reusable strategies from successful runs and correction and avoidance rules from failures. Those insights are then distilled back into the policy model.
How SEED works
The framework runs in two stages. In the first — Hindsight Skill SFT?SFT: Supervised fine-tuning — tuning a model on pairs of input and desired output. — the system collects base-policy trajectories, and an external analyzer summarizes complete runs into "hindsight skills." Trajectory-skill pairs are then used to finetune the model.
The second stage is where the real "self-evolution" begins. The same policy simultaneously samples new task trajectories and acts as the analyzer, extracting reusable skills. The system compares the model's decisions made with and without access to those skills, and turns the difference into an on-policy distillation (OPD) training signal, optimized alongside GRPO. In practice the model teaches itself — using its own hindsight as the teacher.
Results and comparison
The strongest result comes from the ALFWorld benchmark: SEED improves success by 45.9 percentage points over GRPO alone. Equally important is another result — the method reaches GRPO-level performance using just 60% of the training data. Hindsight supervision not only raises the ceiling but also improves data efficiency.
| Test | GRPO | SEED |
|---|---|---|
| ALFWorld (success) | baseline | +45.9 pts |
| Unseen tasks | 70.9% | 86.2% |
| Vision tasks (Sokoban, EZPoints) | 77.0% | 91.0% |
| Data to reach GRPO level | 100% | 60% |
The gains hold beyond the training data. In head-to-head comparison SEED outperforms skill-prompting methods (Skill-Prompt), distillation variants (OPSD, Skill-SD) and plain GRPO — across three different task families: ALFWorld, search (Search-based QA) and WebShop.
Why it matters
SEED targets one of the main bottlenecks for AI agents: the sparse reward signal in long tasks. Training based purely on outcome wastes most of the information in a trajectory, because it cannot assign credit to specific decisions. Turning completed runs into concrete, reusable lessons is a way to get a denser learning signal without hand-labeling every step.
The data result is especially notable: GRPO-level performance at 60% of the data means the method can lower the cost of training agents, not just improve their success rate. The caveat: these are results on specific benchmarks, not an open, production environment — transfer to real, long tasks remains to be shown.
What's next?
- The work is a preprint on arXiv (2607.14777) — peer review and conference publication still lie ahead.
- The method was validated on ALFWorld, Search-based QA, WebShop and vision tasks — the next step is testing on harder, longer agentic tasks.
- An open question is how SEED behaves with larger models and real tools, where the analyzer-policy can make its own mistakes.
Sources
- 36Kr — 清华陶建华团队提出「自进化同策略蒸馏」SEED
- arXiv — SEED (2607.14777)
