Researchers from Peking University published "Taming I2V models for Image HOI Editing: A Cognitive Benchmark and Agentic Self-Correcting Framework" at ICML 2026, introducing three complementary tools for evaluating and improving image editing models in complex human-object interaction scenarios. Code and data are already publicly available.
Key takeaways
- HOI-Edit is a hierarchical dataset of 705 examples split across three difficulty levels: basic interaction editing, spatial understanding, and causal-physical reasoning
- HOI-Eval — a new grounding-based evaluation metric — correlates with human judgment at 0.60 (Interaction), while CLIP shows no measurable correlation
- SCPE improves the Wan 2.2 I2V base model by 22% on L1 tasks, 26% on L2, and 22% on L3 compared to the baseline
- The SCPE framework consists of four agents (Generator, Analyzer, Reflector, Curator) and uses I2V model-generated video as a feedback source
- Results generalize across configurations: replacing the evaluator with Qwen or adding the faster TurboDiffusion diffuser keeps performance trends stable
What HOI-Edit is and why existing benchmarks fell short
Editing images featuring humans and objects is one of the harder tests for generative vision models. An instruction like "hand the mug to the woman" demands not just pixel changes, but understanding character identity, the physical path of movement, and the logic of the event. Existing benchmarks assessed whether the output "looks similar" to the original — ignoring whether the action actually happened.
HOI-Edit introduces three levels of cognitive evaluation. L1 (357 examples) covers basic interactions: creating, removing, and modifying interaction relations. L2 (202 examples) tests whether the model understands spatial context — for instance, whether it places a moved object in the correct location. L3 (146 examples) goes furthest, requiring causal reasoning: the model must complete necessary prerequisite actions (open a door before someone walks through it) before generating the target result.
The dataset was built through a multi-step pipeline: context-aware instruction design, systematic region annotation using SAM 2, multi-faceted question construction, and finally identity verification for subjects.
HOI-Eval: evaluation through questions, not pixels
Standard metrics like CLIPScore or DINOv2 measure global similarity between images — and that is precisely their weakness in HOI tasks. Pixel-level similarity does not guarantee the interaction actually took place.
HOI-Eval shifts to a grounding-based approach: after editing, the system identifies target regions in the modified image, verifies that the identity of the person and object is preserved, and then poses specific questions about the interaction itself (did the motion occur, is the position correct, is the sequence physically plausible).
The comparison with alternative methods speaks clearly. On Pearson correlation with human judgment (Pr): DINOv2 reaches 0.035 for subjects and 0.172 for objects — with no measurable correlation for interactions. CLIP: 0.253 for subjects, –0.027 for objects. HOI-Eval: 0.43, 0.47, and 0.60 respectively — all statistically significant (p < 0.01). The Interaction score of 0.60 is especially meaningful, as that dimension was previously the hardest to measure automatically.
SCPE: an agent that learns from video, not just the final frame
The central innovation in SCPE (Self-Correcting Process Editing) is changing the feedback source. I2V models like Wan 2.2 generate video showing the entire process — from approaching the target to the final placement — not just the last frame. SCPE treats this video as a diagnostic log.
The framework runs four agents in a loop: the Generator builds a detailed prompt and produces the video, the Analyzer extracts the root cause of failure from the trajectory, the Reflector distills individual cases into general rules, and the Curator writes them to a "playbook" — a reusable repository of accumulated experience. On the next attempt, the Generator consults the playbook and avoids repeating known failure patterns.
The ablation study shows each component's contribution. Simply swapping the standard prompt enhancer for the Official Prompt Enhancer (OPE) raises interaction score from 0.68 to 0.70 — but drops IDS (identity consistency) from 0.85 to 0.74. Removing the playbook pulls I back to 0.76, while IDS recovers to 0.88. The full SCPE achieves I=0.82, IDS=0.90.
Results on the HOI-Edit benchmark confirm the gains: relative to the Wan 2.2 I2V baseline, improvements are +22% on L1 (basic interactions), +26% on L2 (spatial relations), and +22% on L3 (causal chains). The model surpasses all available open-source baselines across every category.
Why this matters
Measuring "how well a model changed an image" is technically straightforward. Measuring "whether a model understands what it is doing" — whether the scene is physically coherent, whether characters retain their identity, whether the action sequence makes sense — is a fundamentally different challenge. HOI-Edit, HOI-Eval, and SCPE address it as a unified problem.
Particularly valuable is the shift of diagnostics from the final frame to the video process. When a model should grasp an object located behind another item but instead reaches through a wall, that information is visible in the video trajectory, not in the final frame. SCPE captures these signals and converts them into reusable rules — an approach that mirrors how humans learn: by correcting errors during action, not by polishing finished results.
The backend-independence of the results also matters. Swapping the evaluator model for Qwen or the diffuser for TurboDiffusion does not break performance trends, suggesting the framework can be applied broadly.
What's next?
Code, HOI-Edit data, and weights are already publicly available (GitHub: oceanflowlab/HOI-Edit), so independent replications and benchmark extensions can appear quickly
The paper is accepted to ICML 2026 — post-conference discussion is expected to focus on extending HOI-Eval to video-to-video tasks, not only image-to-video
An identified limitation: the pool of 705 examples is relatively small; the authors point to the need for expansion with more complex multi-person scenarios and longer action chains
Sources
arXiv — Taming I2V models for Image HOI Editing: A Cognitive Benchmark and Agentic Self-Correcting Framework
GitHub oceanflowlab — HOI-Edit repository
