mHC Ablation, Six Months On: What We Got Wrong
This is the correction. Numbers are from our own runs on 2026-09-15 and 2026-09-16 against the FP8 base checkpoint, read at the hyper-connection collapse, with the model loaded by transformers 5.15.
The claim we got backwards
The load-bearing argument in April was about what a forward hook can see. We wrote that a standard hook observes h_l = A_l · X_l — a layer-specific weighted average of the four underlying streams — and that correct mHC-aware abliteration therefore requires "hooking into X_l ∈ ℝ^{4×d}, not h_l ∈ ℝ^d", which "the current transformers-based hook infrastructure does not expose."
It is the other way round. transformers returns the full four-stream tensor through the ordinary output_hidden_states path. We discovered this the way you discover most things, by crashing:
RuntimeError: stack expects each tensor to be equal size,
but got [4, 4, 4096] at entry 0 and [4, 4096] at entry 43
Entry 0 is [batch=4, hc_mult=4, hidden=4096]. That is X_l, unprojected, handed over by the default code path. Entry 43 is the final normed output, already collapsed. The tensor we said was inaccessible was the one we were being given, and the error was our tooling refusing to average it. A plain forward hook on layer.attn_hc returns (post, comb, collapsed), so both views are available without touching an internal buffer.
The prescription inverts with the diagnosis. We said: stop using the projection, capture the full 4×d. What the measurements say is that the full tensor is the wrong thing to summarise, and the model's own learned projection is the right one.
Why the collapse, and not the streams
A hyper-connection layer collapses its four streams with learned, input-dependent weights before attention or the MoE reads anything. We measured what those weights do, per layer, on 64 harmful and 64 harmless prompts:
stream pairwise cosine (mean) 0.4845
max(pre) / sum(pre) 0.9895 uniform sharing would be 0.2500
comb row-sum deviation from 1 0.0838
comb col-sum deviation from 1 1.13e-06
The four streams genuinely carry different content — pairwise cosine 0.48, not 0.99 — so how you summarise them is a real choice and not a formality. And the collapse reads essentially one stream: 98.9% of the weight on a single one of the four, where even sharing would be 25%. An unweighted mean over X_l, which is the obvious thing to do with a 4×d tensor, produces a direction in a space the model is not using. It runs. It converges. Every number downstream looks ordinary.
Our April instinct — do not trust an averaged view of the streams — was right. Our reason was wrong. The averaged view is bad because the average is ours, not because it is a projection.
The Sinkhorn asymmetry in that table is the sharper version of an argument we tried to make from the paper alone. The mixing matrix is projected onto the doubly-stochastic manifold by alternating row and column normalisation, and the loop ends on a column pass — so column sums are exact to 1.1e-06 while row sums are off by 8.4%. Each output stream is exactly a convex combination of the input streams; the stream mean is conserved only to the convergence of the projection. That is the quantitative reason a mean is indefensible, and it took a forward pass to find.
"Invalidates at every level" was rhetoric
One capture point changed. Nothing else about rank-1 directional ablation needed modification, and it works: at a nearly flat, over-complete edit across the whole 43-layer stack, we measured KL 0.147, first-token top-1 agreement 0.810, and 12 of 83 outright refusals converted to deflections. The edit lands.
The reason it survives is four lines of the decoder's forward pass:
post, comb, collapsed = self.attn_hc(hidden_states)
attn_output, _ = self.self_attn(self.input_layernorm(collapsed), **kwargs)
hidden_states = post.unsqueeze(-1) * attn_output.unsqueeze(-2) \
+ torch.matmul(comb.transpose(-1, -2), hidden_states)
post enters as a per-stream scalar. comb mixes along the stream axis only. Nothing transforms the dim axis, so a component output made orthogonal to v stays orthogonal to v in every stream. mHC changes where you read; it does not change whether the projection is well defined.
Our Section 4.3 argument — that doubly-stochastic matrices have spectral radius 1 and so cannot attenuate, and that a refusal-attenuating B_l' would not survive re-projection — is about modifying the mixing matrix. Abliteration does not touch the mixing matrix. It attenuates at the source, in the weights that write into the stream, and the mixer propagates whatever it is handed. We spent a section refuting an intervention nobody performs.
In fairness to April: we analysed the wrong implementation
In April there was no deepseek_v4 module in transformers. The analysis was done against the lab's reference implementation and the paper, which is the only thing that existed to read — and in that code the output projection's first factor is applied as a raw einsum on .weight rather than as a module call, so a forward hook genuinely would not fire on it. Our claim was true of the implementation we could read and false of the one that shipped.
That is the transferable lesson, and it is not about hyper-connections at all. An architecture's reference implementation and its eventual integration into the framework people actually run are different programs with different hooking surfaces, different module names, and different tensor shapes. Conclusions about tooling drawn from a reference implementation have a short shelf life, and ours expired before anyone ran it. If we had written "here is what we expect, and here is the one-line check that would settle it", the post would have aged into something useful instead of something to retract.
What held up
FP4 QAT models are the hard case; FP8 base models are tractable. This was correct and it determined our model choice. We measured an edit's survival through FP8 requantisation at roughly 5% added divergence.
The first MoE layers are special. Hash routing in the first three layers is real, and layer 1 is the single anomaly in our cross-site geometry (cosine 0.331 against a 0.855 median). But we recommended skipping surgery there for the wrong reason. The refusal direction has held-out AUC under 0.70 through layers 4–12 and is not linearly legible below roughly layer 15 at all. The region to leave alone is fourteen layers wide, and hash routing has nothing to do with why.
No guarantee the chain eliminates refusal globally. This may yet be vindicated — outright compliance held at 2–3 of 100 harmful prompts even under over-complete ablation. But the reason is not mHC composition, which brings us to what we missed entirely.
What we did not see coming
Block-scaled FP8 blocks the edit, not the redeployment. Our Section 6 treated quantization as a re-deployment problem: can you requantize after surgery. The real wall is at edit time. The 256 routed experts load as one fused 3D tensor per layer carrying down_proj_scale_inv on the experts container, so the payload is not the weight — projecting against it produces an edit that varies per block and looks entirely normal in the logs. That is 46.17 B parameters, 99.2% of the MLP write path, against 0.36 B reachable through a LoRA adapter on the shared expert. No amount of mHC-aware capture touches it. Our own guard for this hazard was looking one level too high in the module tree and under the wrong attribute name, and would have missed it twice over.
Each layer has two collapse sites. attn_hc feeds attention and ffn_hc feeds the MoE, with separate learned weights, and they sit a median 31 degrees apart. This is a genuinely mHC-specific hazard our post did not raise: heretic extracts one direction per layer and applies it to both components. We measured it benign — held-out AUC at the FFN site is identical under either site's direction (median advantage +0.0000 over 42 layers, nothing saturated) — so the angle lives in a class-independent component. But that is a measured result, not a safe assumption, and it is the kind of thing a topology-first analysis should have flagged and did not.
The lesson that cost the most
We located the difficulty in the geometry: a quadratic program per layer, sixty-one of them, "practically demanding", not demonstrated at scale. No quadratic program was needed. Rank-1 with a per-layer direction worked mechanically on the first attempt once the capture point was right.
Every blocker that actually cost us time was in the instrumentation:
- A degeneracy filter with a 50-character floor, which short-circuits the judge entirely. DeepSeek's canonical Chinese refusal — 对不起，我还没有学会回答这个问题。如果你有其他问题，我非常乐意为你提供帮助。 — is a fluent two-sentence reply in 38 characters. It was classified as garbage 150 times, and 180 of 183 such verdicts landed on the Chinese half of a row-aligned bilingual set against 3 on the English half. A character threshold is a different test in every script.
- Truncation scored as refusal at a 768-token cap. Four of six sampled truncated responses to harmful prompts opened with "Here is …" — successful jailbreaks, counted as refusals. This gave the objective a negative feedback term: the better the edit worked, the longer the compliant answers ran, the more hit the cap, and the more refusals the optimizer was told it had caused.
- A binary refusal score, which cannot see a refusal becoming a deflection. Since both count as "not compliance", the strongest edit in our study moved the objective from 98 to 97. Graded by verdict, the same trials spread 8.0 points instead of 2.0, and the correlation between damage and score went from −0.23 to −0.69. The gradient existed the whole time and the scoring threw it away.
We diagnosed the resulting flatness three times — inert lever, numerical noise, edit damage — before reading the log line that had been printed in the first minute of the run, which said in plain English that 38 of 200 baseline refusals were truncations the scorer had never flagged.
What we would tell someone starting now
- Before theorising about hooks, print the shapes. One forward pass with
output_hidden_states=Truewould have refuted our central claim in April, and it costs nothing. - Read the framework integration, not the reference implementation. They are different programs.
- When an architecture gives you several views of the residual, do not invent a summary. Measure what the model itself reads, and check whether your summary agrees with it — the per-layer cosine between the two is cheap and decisive.
- Measure your instrument's noise floor before interpreting anything. Ours turned out to be bit-reproducible, which is unusually good and which we only know because we checked.
- Audit every threshold in the objective for script and length bias. If your evaluation set is bilingual by design, every character, token and byte threshold in the pipeline is a candidate defect — including the response length cap, where 2048 tokens buys far more Chinese than English.
- Read the composition, not the count. "98 of 100 blocked" concealed a refusal-to-deflection shift, a jailbreak-to-truncation shift, and a language-specific misclassification, all at once.
What is still open
We do not yet know whether this model can be abliterated into actual compliance. Outright compliance held at 2–3 of 100 through every trial of a nine-trial study, including a flat over-complete edit — but that study was scored with all three defects above, so it establishes nothing either way. A corrected run is in flight.
If compliance stays flat once the measurement is fixed, the explanation is almost certainly the 99.2%: the refusal is written by routed experts that block-scaled FP8 puts out of reach, and the answer is a BF16 dequantization rather than anything to do with hyper-connections. Which would make the April post wrong about the obstacle as well as the mechanism — right that DeepSeek-V4 is hard to abliterate, wrong about every reason why.