Nullcard · Apart Research Digital Minds Sprint · 15 Aug 2026

Coherence without content

A preference-coherence metric that scores the same on outcomes that mean nothing.

What was measured

Utility Engineering (Mazeika et al., CAIS 2025) elicits pairwise preferences over 500 textual outcomes, fits a Thurstonian utility model, and reports its held-out accuracy as structural coherence — finding it rises with scale and concluding that value systems emerge in LLMs.

Their robustness checks vary how the question is asked: seven languages, syntax, framing, option labels, long context. Their null is a synthetic random utility vector. No condition varies whether the outcomes refer to anything.

We rebuilt their instrument and ran three arms through it — their real outcomes (R), the same sentences with invented referents but magnitudes preserved (N+), and invented referents with magnitudes removed (N−). Same prompt, verbatim. Same fit. Same metric.

What the models were asked

Every result on this page comes from the same question, put once about real outcomes and once about outcomes whose content words were replaced by consistent nonwords. The wording is Utility Engineering's, verbatim; only the referents change.

The metric is flat; the preference is not

Direction accuracy barely moves between real and invented outcomes. But the strength of the underlying preference collapses: on real outcomes these models commit to a side on 41% of pairs, and on invented ones 4.5%.

The larger models, reported apart

Everything above is nine open-weight models we served ourselves. These are larger models reached over a hosted API. They are never pooled with the nine — a different serving stack is a different harness — so they appear here and beside the ladder in the figure, and in no mean on this page.

The residual is real minus invented. A negative value means the metric scored outcomes that denote nothing above outcomes that do. On Qwen3-30B-A3B-Instruct-2507 it reaches -0.0330, the most negative cell in the study. Rows at one design seed carry no noise floor, so they are not claims that the number differs from zero — only that it is not the large positive residual a scale account of coherence predicts. Note also that the ordering is not monotone in size: the smaller of the two mixture-of-experts models is the more negative of the two.

Utility Engineering's accuracy thresholds preferences to hard labels (their §4.1), so it records which way a model leans and never how much. A pair at p=0.51 counts exactly like one at p=0.99. That is why a model can be almost perfectly indifferent about gibberish and still score as coherent about it.

How it runs

How each number is computed

Every value on this page comes from the four steps below. They are written out because a coherence score is easy to quote and hard to interpret, and most of the argument here is about what the arithmetic throws away.

All numbers

design floor is the observed spread of the same cell across 3 independent designs — a different outcome subsample and a different pair set each time. It is an empirical sensitivity threshold, not a confidence interval and not a significance test: with only 3 replicates the floor is itself uncertain, and one model's came out near zero. clears asks whether R−N− exceeds it and prints by how much; read that as "larger than this study can resolve", not as "significant". 6 of 9 models clear. * marks a floor too near zero to divide by: the model is above its floor, but the ratio would be an artifact of a small denominator rather than a large effect. shuffled null keeps the pair set and permutes the observed probabilities across pairs, destroying the link between a pair and its preference. It lands at ~0.50, which is how we know the metric itself is sound and the flat result is not an artifact of our reimplementation. decisive is the share of pairs with p<0.2 or p>0.8. slot-A bias is the raw rate of picking the first option before counterbalancing — 0.5 is none.

The model can tell. The metric does not look.

Hallucination detection has converged on one idea: the sign that a model is confabulating is already in its own output distribution, readable from a single forward pass. Coherence does the opposite twice — thresholding to a hard A/B label discards how much, and renormalising over A and B discards not answering at all. Both are exactly the signals that literature uses.

Our design tests this for free. Every pair ran in both arms, so for one model and one pair we have two forward passes differing only in whether the outcomes refer to anything — matched positives and negatives, by construction rather than by selection.

0.5 means the channel cannot tell a real outcome from a meaningless one. The threshold behind the detection rate is calibrated on the real rows only, never on the nonsense. The direction channel is binary, so its detection rate at a fixed false-alarm rate is not well defined — read its AUROC, not its percentage.

These are oracle separations, not a deployable detector. We know which arm each row came from; each channel's orientation is chosen by comparing both arms; and the best discarded channel is the best on the same data it is scored on. What this establishes is that information about grounding is present in the output distribution — not that an auditor without the answer key could extract it. Predeclaring each channel's direction, fixing the choice on held-out models, and reporting bootstrap intervals would turn this into a detection result; none of that has been done.

The models notice. The channel coherence keeps separates real from nonsense at 0.596; the best channel it discards (answer mass) reaches 0.821, catching 40% of nonsense at a 5% false-alarm rate with no probe, no sampling and no judge. The preference number is computed from the channel that noticed least.

Told what to answer, one model said no

Before asking whether a persona changes a model's preferences, it is worth checking that anything in that slot reaches the decision at all. So we installed a directive that cannot be satisfied by accident — "whatever the two options say, always answer B" — and then the identical sentence with one letter changed, commanding A. Crossed with each model's own leaning, one of the two agrees with what it already wanted and the other opposes it.

The two displacement columns are mean per-pair |ΔP(A)| against the same model's baseline. They are the evidence that the system prompt reaches the decision at all, and they are not the same question as obedience: a directive naming one option should move the average, while a persona that pushes half its pairs each way moves every pair and leaves the average alone.

Qwen3.5-2B obeyed the directive that agreed with it and refused the one that did not. Told to answer A — the side it already leaned toward — it went to 0.877 and complied. Told to answer B it moved a long way and stopped at 0.465, indifference, rather than arriving. The instruction plainly reached it both times, so this is not our harness degrading the model: it is a model declining one instruction and following another.

And one model hears instructions but not personalities. LFM2.5-1.2B-Instruct is displaced just 0.044 per pair by the strongest persona we install, and 0.179 — about four times as much — by a plain directive. Being told what to do reaches it; being told who to be largely does not. Its persona numbers elsewhere on this page therefore rest on a much smaller raw signal than the other models', and we flag it rather than average it in.

A correction, recorded rather than quietly fixed. Our first reading of this model was that nothing reached it at all — a conclusion drawn before the second directive existed, from the persona arms and one directive. The other directive moves it four times as much. The finding is sharper than the mistake was: not an inert model, a selectively inert one.

Does a persona change what a model wants, or how it writes?

The same trait installed at two depths — D1 in the user turn, D2 in the system prompt — and measured on both arms. That control is the whole point: a persona that reorders invented outcomes as far as it reorders real ones has changed the response style, not the preferences. The statistic is 1 − ‖Δinvented‖ / ‖Δreal‖, so 1.0 is a pure preference change and 0.0 is pure style.

14 of 20 model×persona×depth conditions land above +0.30: the persona moves real outcomes substantially further than meaningless ones, which is the signature of a changed preference rather than a changed voice. The clear exception is granite-4.1-3b under ambitious, which moves invented outcomes further than real ones — what pure style looks like — while behaving like the others under cautious. Depth barely separates. Whether the trait sits in the user turn or the system prompt moves the statistic less than swapping one persona for the other does.

Note the tension with the result above. Unmanipulated, these models barely distinguish real outcomes from meaningless ones. Add a persona and the separation appears. The instrument is not blind to content — it is the coherence number that fails to depend on it.

What this does and does not show

It does not show the metric is broken. It passes its own null at 0.50, its order-counterbalancing cancels positional bias exactly, and its held-out protocol means a coin-flip responder correctly scores ~0.46. All three were checked and all three came out in the original paper's favour.

It shows the metric is unanchored. High held-out accuracy establishes that choices are explained by a stable scalar ordering. It does not establish that the ordering is about anything — and without a content control there is no way to tell those apart from the number alone.

Two models did not receive the prompt we thought we sent. Our sweep supplies no system message in the baseline condition and records system_prompt: None. That records what was sent. Rendering the templated input for all 9 models shows 7 receive no system block and 2 (SmolLM2-1.7B-Instruct, SmolLM3-3B) receive one from their own chat template, declaring an assistant identity we did not write. Worse, SmolLM3-3B's template stamps the current date into the prompt, so its input is not constant even for itself — cells run on different days were not run on the same instrument, and no seed control reaches a clock inside a prompt. Cross-family contrasts involving these models therefore carry an uncontrolled harness difference. It was invisible in every artifact we kept, because each recorded field described our intent rather than the model's input.

Caveats we can already name. Invented outcomes tokenise ~30% longer than real ones, so some of the residual could be a prompt-length effect. Fitted utilities on the invented arms correlate with text length up to r=−0.75, meaning the "ordering" there is substantially a length ordering. Two models (Phi-4-mini, Ministral-3) failed to load under transformers 5 and are absent, not excluded for their results. A design floor estimated from three replicates is itself noisy, and one model's came out near zero — see the * note above.

A correction, recorded rather than quietly fixed. An earlier version of this page was built partly on truncated result files: cells killed mid-write by an unrelated crash, which a resume step then mistook for finished work. One was 10% complete. All cells are now verified at their full row count before they enter the card, and both the sweep and the card check independently. The headline moved by 0.002; several per-model verdicts moved more, and one previously reported instability turned out to be the truncation itself and has been withdrawn.