Conversational Post-Training
How it works
1) Starting point: a pretrained base model. 2) SFT (supervised fine-tuning): training on curated dialogue data — instruction–response pairs and multi-turn conversations formatted with a chat template with roles (system/user/assistant) and special tokens. 3) Preference tuning: RLHF (train a reward model on response rankings, then optimize the policy e.g. with PPO under a KL penalty) or the simpler DPO/APO that learn directly from preference pairs without a separate reward model. 4) Optionally: rejection sampling, checkpoint merging, reasoning-mode switching. The result is a model that interprets the chat template, retains multi-turn context, and produces helpful, preference-aligned responses.
Problem solved
A pretrained language model predicts next tokens but cannot naturally respond to commands or hold a coherent, multi-turn conversation aligned with user intent. Conversational post-training closes this behavioral gap: it teaches the model the dialogue format, instruction following, multi-turn context retention, and preferred helpful, safe responses.
Components
Supervised fine-tuning of the base model on curated conversations and instruction–response pairs, teaching the dialogue format and basic instruction following.
Tuning the model to human preferences: RLHF (reward model + PPO with a KL penalty) or DPO/APO learning directly from preference pairs, improving helpfulness and safety.
Datasets of multi-turn conversations and preference pairs (human or synthetic) that fuel the post-training stage.
The role format (system/user/assistant) and special tokens into which conversations are embedded, defining the input/output interface of the conversational model.
Evolution
OpenAI shows that SFT on demonstrations + RLHF on rankings make a model a useful instruction-following assistant.
A consumer chat assistant built on conversational post-training popularizes the paradigm.
Meta publishes a detailed open conversational post-training recipe with iterative RLHF.
Rafailov et al. simplify preference alignment by removing the separate reward model and RL loop.
AI2 releases a fully open post-training pipeline with data and code.
Hugging Face combines reasoning mid-training, SFT, Anchored Preference Optimization, and checkpoint merging.
Hyperparameters (configurable axes)
Proportion and selection of dialogue datasets (instructions, multi-turn chats, specialized data) in SFT.
Choice of preference alignment method: RLHF (PPO), DPO, APO, or variants.
Strength of the KL regularization toward the reference model in RLHF/DPO, controlling drift from the base policy.
Number of passes over the SFT data; too many risks overfitting and forgetting pretraining knowledge.
Proportion of multi-turn versus single-turn examples, affecting conversational context retention.