JIT Retrieval
How it works
1) The agent stores lightweight references (file paths, stored queries, links) instead of full data. 2) At runtime it determines what information is needed for the current decision. 3) It uses tools (e.g. grep, glob, Bash head/tail, page fetching) to dynamically retrieve data without loading full objects into context. 4) It leverages environment metadata (file sizes suggest complexity, names hint at purpose, timestamps proxy for relevance) as navigation signals. 5) Through progressive disclosure, each interaction yields context that informs the next step, keeping only what is necessary in working memory. This requires opinionated, thoughtful engineering โ the right tools and heuristics โ because without it the agent wastes context by misusing tools or chasing dead-ends.
Problem solved
Pre-loading all context fills the model's window, raises costs and degrades answer quality due to "context rot" โ as the number of tokens grows, the model's ability to accurately recall relevant information decreases. Classic embedding-based pre-retrieval also suffers from stale indexes and rigid pipelines. Just-in-Time Retrieval mitigates these issues by keeping only the data actually needed at a given step in context.
Components
File paths, stored queries and web links kept instead of full data; they point to where content can be fetched on demand.
Official
Runtime-executed tools that dynamically retrieve data โ e.g. grep, glob, Bash head/tail, page fetching โ without loading whole objects.
Official
Folder hierarchy, naming conventions, file sizes and timestamps as cues about complexity, purpose and relevance of data.
An iterative mechanism where each interaction yields context that informs the next decision; the agent builds understanding layer by layer.
Implementation
Without the right heuristics the agent chases dead-ends, misuses tools, or fails to identify key information.
On-demand exploration is slower than retrieving pre-computed data due to multiple round-trips.
Inconsistent naming and folder structure deprive the agent of navigation signals and degrade retrieval accuracy.
Evolution
Retrieval-Augmented Generation popularizes pre-fetching and injecting data into context โ the baseline against which the just-in-time strategy is positioned.
The article "Effective context engineering for AI agents" (Sep 29, 2025) defines the just-in-time strategy and describes its hybrid use in Claude Code.
Hyperparameters (configurable axes)
Cap on tool round-trips per task; balances thoroughness against latency and cost.
How much data is loaded per step (e.g. whole file vs. a fragment via head/tail/grep).
What share of context is loaded upfront (e.g. CLAUDE.md) versus fetched on demand.
How well naming, folder structure and timestamps guide the agent to the right data.
Computational complexity
Compute bottleneck
Multiple runtime tool round-trips and I/O operations introduce latency; this is the main cost traded for lower context usage.
Execution paradigm
Data loading happens conditionally โ only when the agent deems it necessary.
At runtime the agent decides which data to fetch based on the current need and metadata signals.
Parallelism
Progressive disclosure is inherently sequential โ each step depends on the previous result. Independent fetches can be partially parallelized.
Hardware requirements
It is a software/agent-level orchestration pattern, independent of any specific accelerator.