Fine-Tuning & Adaptation

Prompting changes what you ask the model, entirely in-context with no weight change — the cheapest and fastest to iterate on. RAG changes what context the model sees by retrieving external information at query time, adding knowledge without touching weights. Fine-tuning changes the model's weights via further training, needed when prompting or RAG can't reliably produce a desired behavior or format.

Catastrophic forgetting is the model losing previously learned general capabilities while specializing on new fine-tuning data. It's caused by gradient updates optimized for the new narrow task overwriting weight patterns responsible for broader pretrained behavior — more likely with full fine-tuning, an aggressive learning rate, or narrow/repetitive fine-tuning data.

LoRA freezes the original weight matrices and instead trains two much smaller low-rank matrices whose product approximates the weight update. Since gradients and optimizer state only need to be tracked for these small added matrices — not the full-size original weights — memory and compute needed drop dramatically compared to updating every original parameter.

QLoRA additionally quantizes the frozen base model's weights, typically to 4-bit, before applying LoRA adapters — further shrinking the memory footprint of the base model itself, not just the trainable adapter. This enables fine-tuning of much larger models on consumer or limited GPU memory that standard LoRA alone couldn't fit.

Fine-tuning bakes knowledge into weights that go stale as soon as underlying facts change, requiring a new training run to update. RAG's knowledge lives in an external, easily re-indexed source, so updating knowledge is just re-indexing documents — far cheaper and faster to keep current than retraining a model.

First, collect comparisons between candidate model responses ranked by human preference. Second, train a reward model to predict human preference scores from those comparisons. Third, use reinforcement learning (e.g. PPO) to fine-tune the base model to maximize the reward model's score — shaping tone, helpfulness, and refusal behavior beyond raw instruction-following.

DPO removes the need for a separate reward model and RL loop by reformulating preference optimization as a direct classification-style loss over preferred vs. rejected response pairs — simpler and more stable to train while targeting the same underlying preference-alignment goal as RLHF.

When maximum task performance is critical and the task requires deep, broad changes to model behavior that a small set of adapter parameters can't capture well, or when the deployment target requires a fully self-contained model without needing to merge or serve separate adapters — though in practice LoRA is competitive for most tasks and is the more common default.

Instruction tuning teaches the model to follow instructions using supervised (instruction, ideal response) examples. RLHF/preference tuning further refines output quality, tone, and safety using relative preference comparisons rather than single correct answers. They're often sequenced because instruction tuning establishes basic instruction-following first, and preference tuning then polishes quality and alignment on top of that base.

Noisy, inconsistent, or low-quality examples teach the model contradictory or undesirable patterns that a small amount of gradient signal can't average out, while a smaller set of carefully curated examples gives a cleaner, more consistent training signal. Quantity without quality can actively hurt performance on the target behavior.

A pretrained model already encodes substantial useful structure. A learning rate that's too high can overwrite that structure quickly (catastrophic forgetting), especially since fine-tuning typically runs far fewer steps than pretraining, leaving less opportunity to recover from an aggressive early update — fine-tuning generally uses a much smaller learning rate than pretraining from scratch.

Distillation trains a smaller "student" model to mimic a larger "teacher" model's outputs, e.g. matching its predicted probabilities or generated responses. After fine-tuning a large model to the desired behavior, you could distill that behavior into a smaller, cheaper-to-serve model for production, trading some quality for lower latency and cost.

Evaluate on a held-out set representative of real usage, not seen during training, to check generalization; specifically test whether the target behavior actually improved; and separately check for regressions on general capabilities or tasks the model previously handled well. Training loss alone doesn't reveal overfitting or forgetting.

Prompt tuning learns a small set of continuous "soft prompt" embeddings prepended to the input, while all model weights, including LoRA-style adapters, remain untouched. It's even more parameter-efficient than LoRA but generally has less capacity to shift model behavior, making it suited to narrower adaptation needs.

Since LoRA freezes the original weights and only trains a small added set of parameters, the base model's original capabilities are largely preserved by construction, making catastrophic forgetting less likely than with full fine-tuning, which directly modifies all the original weights and can more easily drift away from previously learned general behavior.

Few-shot prompting requires no training infrastructure, no labeled dataset curation, and can be iterated on in minutes. It's a fast, low-cost way to validate whether the desired behavior is achievable at all before committing engineering time and compute to collecting fine-tuning data and running training experiments.

Tasks primarily needing up-to-date or rapidly changing factual knowledge are a poor fit, since that's better served by RAG, as are tasks with very little available labeled data, since there's insufficient signal to fine-tune reliably. Fine-tuning is best suited to stable, well-defined behavior or format changes with enough quality examples to learn from.

Collect or generate a diverse set of natural-language-request-to-correctly-formatted-JSON-response pairs covering the range of expected inputs and edge cases, ensuring the JSON responses are consistently well-formed and validated — the model will learn to mimic whatever patterns, including inconsistencies, are present in the training examples.

It can look like the fine-tune succeeded, matching in-distribution examples well, while actually failing to generalize to realistic variation in production inputs — because the evaluation never tests behavior outside what the model has effectively memorized from training.

For open-ended generation, there's often no single objectively correct response, but humans can more reliably and consistently judge which of two candidate responses is better. Pairwise preference judgments provide a more tractable and consistent training signal than trying to define one canonical "correct" answer for subjective quality dimensions like helpfulness or tone.

Since LoRA adapters are small and separate from the frozen base model, multiple task-specific adapters can be trained independently and swapped in and out, or even served simultaneously via multi-adapter serving, against the same base model — avoiding the cost of hosting a full separate fine-tuned model copy per use case.

For example, fine-tuning a model on a company's product documentation to "teach it the facts," then needing to retrain from scratch every time the documentation changes — an expensive, slow cycle compared to RAG, where updating the documentation just means re-indexing it, with no retraining needed at all.

Behavior — format, tone, consistent structured output, task-specific style — is what fine-tuning is well suited to shift, since it's a persistent pattern in how the model responds. Knowledge, specific facts, is better handled by RAG since it can be updated without retraining. Conflating the two leads to reaching for the wrong tool, such as fine-tuning to add facts that then go stale.

Run both the fine-tuned model and the base model on a held-out target-task evaluation set, checking for improvement, and on a separate general-capability benchmark or set of prior task types the model handled, checking for no meaningful regression. Comparing both dimensions, rather than only the target metric, catches forgetting that a single-metric evaluation would miss.

It may not show up as an obvious, uniform capability loss — it can appear as subtle degradation on specific unrelated tasks or edge cases not covered by whatever quick sanity checks were run after fine-tuning. Teams that only spot-check the target task's improvement can ship a model that's quietly worse at things users still relied on.