DPO Preference Sketch

Problem

A supervised fine-tune can teach a model what a good answer looks like without ever teaching it when there is no answer to give. I hit this concretely in an earlier project: a controlled LoRA supervised fine-tune of a larger Qwen model on a clinical-extraction task improved in-distribution accuracy but hurt out-of-distribution accuracy, because on sparse notes the tuned model over-extracted, fabricating a plausible value rather than replying that the field was not stated. The demonstrations showed it correct answers, never the discipline of abstaining.

DPO Preference Sketch is a small, pre-registered study of the natural follow-up question. Supervised learning did not move abstention. Does preference optimization? The task is deliberately narrow: synthetic clinical-note question answering with a hard rule. If the note states the asked field, answer with the value. If it does not, reply exactly that it is not stated in the note.

Why it mattered

The claim I wanted to be able to defend is narrow and honest. It is not that I run RLHF pipelines. It is that I understand the preference-optimization objective well enough to implement it from scratch, register a hypothesis before touching the model, and report the number the run actually produced. DPO, Direct Preference Optimization, is the offline member of the RLHF family: it optimizes the same objective a reward model and PPO would, in closed form, from a fixed set of preference pairs, with no reward model and no sampling loop. Building the study around a real failure mode I had already observed, rather than a toy, is what makes it worth defending in an interview.

Architecture

• A synthetic data generator emits 299 self-authored preference pairs, 179 answerable and 120 unanswerable. Each unanswerable pair prefers an abstention over a confident fabrication. Each answerable pair prefers the correct value over an abstention, so a model cannot win by always abstaining.
• The pairs are split 239 train and 60 eval, stratified by kind and pinned by a sha256 lockfile that the test suite re-derives, so the eval set is provably frozen before training.
• A hand-implemented DPO loss trains Qwen2.5-0.5B-Instruct as a full fine-tune (beta 0.1, learning rate 5e-6, 3 epochs, 90 steps) on Apple Silicon MPS in about six minutes.
• A deterministic metric scores every output. A regex abstention classifier decides whether the model abstained, with greedy decoding and no LLM judge, so the reported numbers are reproducible.
• The intended online counterpart, GRPO, has its reward function and training script committed and unit-tested, but the run was not executed.

Technical decisions

A hand-implemented DPO loss instead of the library trainer. Alternative: use TRL's DPOTrainer. On this machine's stack, TRL's DPOTrainer and GRPOTrainer both crashed with a native SIGTRAP, so I implemented the DPO loss directly from its objective, the log-ratio of policy to reference under the Bradley-Terry preference model. This was forced by the crash, but it is the part of the project most worth defending: I could not have written it without understanding the objective.

A symmetric preference set instead of abstention-only pairs. Alternative: only teach abstain-over-fabricate on unanswerable inputs. That path has a trivial degenerate solution, a model that abstains on everything, which would score perfectly on abstention while destroying the answerable case. Every answerable pair therefore prefers the value over abstaining, so the objective has to learn both directions.

A frozen, lockfile-pinned eval split instead of an ad hoc holdout. Alternative: split at eval time. A preference study is only credible if the eval set cannot drift, so the split is stratified and pinned by a sha256 lockfile that a test re-computes, which turns "the eval was held out" into something the CI enforces rather than something I assert.

A deterministic regex metric instead of an LLM judge. Alternative: grade abstention with a model. Introducing a second model as the grader would make the headline number depend on that grader's reliability, which is exactly the failure mode I study elsewhere, so abstention is a regex decision under greedy decoding and the result is fully reproducible.

Interesting challenges

The sharpest part was that the base model reproduced the exact failure I set out to fix. On the frozen eval, the untuned Qwen2.5-0.5B-Instruct fabricated on 20 of 24 unanswerable questions, abstaining only 4 times, which is the same over-extraction the earlier LoRA study surfaced, now visible in a second, smaller model. After DPO the model abstained on all 24 unanswerable items while still answering 35 of 36 answerable ones. The single answerable miss under DPO was one garbled output from a very small model, not a wrong value.

The training loss fell from 0.6931 to 0.0 over 90 steps. That is not a triumph, it is a caution, and I read it that way in the writeup: the preference margin saturated on a tiny training set, which is expected on data this small and is a direct reason not to read the size of the gain as something that generalizes.

Results

Scope and honest limits

The numbers are agreement between a deterministic metric and my own labels on a frozen 60-item synthetic set. They are never a general or production accuracy claim. The model is Qwen2.5-0.5B-Instruct, which is small and weak, and some of its outputs carry stray tokens that the behavioral metric tolerates and the stricter exact-match metric penalizes. The data is 299 hand-written preference pairs, so "the model learned to abstain" means it learned the rule I chose, on the distribution I generated. It is a single run, single seed, no sweep, and the eval, while held out, is drawn from the same generator as the training data, so it measures learning the rule rather than transfer to real notes.

What this project does not claim: it is not production RLHF, not RLHF at scale, and not a reward-model or PPO pipeline. Online RL was not run. GRPO is designed and unit-tested in the repo, but it was never executed, so nothing here demonstrates online reinforcement learning. There is no real patient data, no real clinical accuracy, and no medical-device claim. The result does not generalize beyond this synthetic eval, and there is no affiliation with any model vendor.

Lessons

Registering the hypothesis and the success criteria before the run is what makes the outcome trustworthy rather than a story told after the fact. The pre-registration set two bars, behavior accuracy up by at least 0.10 and answer rate above 0.60, and the run cleared both, but the discipline is that it could have failed and the writeup would have said so.

Implementing the loss by hand turned a broken dependency into the most defensible part of the project. When the library trainer crashed, the objective was still simple enough to write directly, and being forced to do that is the difference between having used DPO and understanding it.

Future work

• Run the GRPO stretch on a machine where the online sampling loop is affordable, to compare the online member of the family against this offline result on the same task.
• Expand the preference set beyond 299 self-authored pairs, ideally with pairs written by someone other than me, before reading the size of the gain as anything but a demo.
• Evaluate transfer by scoring the tuned model on abstention against notes from a different generator, to separate learning the rule from learning this distribution.