Robots Atlas>ROBOTS ATLAS
Training

Transformer²

2025ResearchUpdated: 5 May 2026Published
Key innovation
Adapting LLMs to unseen tasks at inference time by selectively adjusting only the singular components of weight matrices, without classical fine-tuning.
Category
Training
Abstraction level
Pattern
Operation level
InferenceTraining
Use cases
Self-adaptive LLMsInference-time task specializationEfficient alternative to LoRA / PEFTMulti-task LLM servingVision-language adaptation

How it works

1) Offline phase: the base LLM's weight matrices are decomposed via SVD; lightweight expert vectors 'Z' are trained via RL — each vector specialises in a task category (e.g., math, coding, reasoning). 2) Inference phase, pass 1 (dispatch): the system analyses the prompt and identifies the task type. 3) Inference phase, pass 2 (execute): expert vectors matching the task are dynamically mixed and applied to the singular values of the weights, yielding a model tailored to the specific prompt — without updating the original weights.

Problem solved

Classical fine-tuning and PEFT (LoRA) produce static adapters that cannot handle arbitrary unseen tasks at runtime. Transformer² solves this by dynamically composing expert vectors at inference time.

Components

SVD-decomposed weight matricesStatic knowledge backbone of the model

Base LLM weight matrices decomposed via SVD into U·Σ·Vᵀ. Singular values (Σ) are the application point for expert vectors.

Expert vectors (Z)Dynamic behavioural adaptation

Lightweight task-specialised vectors trained via Reinforcement Learning. They modulate the singular values Σ during inference.

Official

Dispatch systemTask routing

Lightweight classifier that analyses the prompt in the first pass and selects the appropriate set of expert vectors.

Official

Implementation

Reference implementations
SakanaAI/self-adaptive-llms (official GitHub)
Python · Sakana AI
Official
Implementation pitfalls
Inaccurate task classification in the dispatch phaseHigh

If the dispatcher misidentifies the task type, it will select the wrong expert vectors and quality degrades significantly.

Fix:Train the dispatcher on diverse prompts and fall back to full mode (all experts mixed) when classification confidence is low.
Instability of RL-based expert vector trainingMedium

RL training of Z vectors can be unstable in the presence of sparse or noisy rewards.

Fix:Use reward shaping, KL regularisation against the base policy, and variance-reducing baselines.

Evolution

Original paper · 2025 · arXiv preprint 2501.06252 (Sakana AI, 2025) · Qi Sun
Transformer-Squared: Self-adaptive LLMs
Qi Sun, Edoardo Cetin, Yujin Tang
2014
SVD as a tool for analysing neural network weight matrices — theoretical foundation
2021
LoRA (Hu et al.) — low-rank adaptation as an efficient fine-tuning alternative
Inflection point
2025
Transformer² (Sakana AI) — first self-adaptive LLM method based on SVD and RL-trained expert vectors
Inflection point

Hyperparameters (configurable axes)

Number of expert vectorsHigh

Number of trained Z vectors covering different task categories.

SVD decomposition rankHigh

Number of singular values retained during weight decomposition — trades adaptation capacity vs. cost.

RL objective for expert vectorsCritical

Reward function used to train Z vectors (typically task-specific reward).

Execution paradigm

Primary mode
Conditional

Conceptually similar to MoE, but routing operates in SVD space rather than across FFN blocks.

Activation pattern
Input dependent
Additional modes
Sparse
Routing mechanism

The first pass classifies the task; the second applies a mix of expert vectors to the singular values of the weights.

Parallelism

Parallelism level
Partially parallel

The second inference pass depends on the first pass (sequential dispatch → execute), but expert execution itself is fully parallel.

Scope
InferenceTraining

Hardware requirements

Primary

Both SVD decomposition and LLM inference rely on dense matrix operations well supported by tensor cores.