# Deterministic LoRA training directly through quantized GGUF serving weights **URL:** \ **Category:** Show and Tell\ **Created:** [August 22, 2026, 9:30am UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118 "2026-08-22T09:30:41Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Joakimpalm-Zen](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joakimpalm-zen/32/62554_2.png) [@Joakimpalm-Zen](https://discuss.huggingface.co/u/Joakimpalm-Zen)\ **Post date:** [August 22, 2026, 9:30am UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/1 "2026-08-22T09:30:41Z")
I’ve been working on making Xyntetik Runner train LoRA adapters directly through the same quantized GGUF weights it serves. The important part is that there is no separate FP16 training copy and no second training runtime. The forward pass used for training is the forward pass used for inference. The first reproducible artifact is now live on Hugging Face: > **[Joakimpalm-Zen/Qwen3-4B-Runner-ToolUse-Q4\_K\_M · Hugging Face](https://huggingface.co/Joakimpalm-Zen/Qwen3-4B-Runner-ToolUse-Q4_K_M)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. For this release I trained a tool-use LoRA directly against a frozen Qwen3-4B Q4\_K\_M GGUF. On the small held-out eval, exact tool-call accuracy went from 0.69 to 1.00 and right-tool selection from 0.72 to 1.00. More interesting to me is the reproducibility result: same base GGUF - same dataset - same seed - same training config = byte-identical adapter GGUF Two independent runs at 4B scale produced the same SHA256. I also repeated the same training task through BF16, Q8\_0 and Q4\_K\_M copies of the same base model. The BF16 and Q8-trained adapters ended up extremely close in weight space, cosine 0.9998. The Q4-trained adapter followed a measurably different optimization path, cosine 0.9926 against the others, while all three adapters still reached 1.00 on this particular held-out task. So at least in this experiment, training through the deployed quantization and training through the high-precision parent are not numerically the same operation, even though they ended up capability-equivalent on the measured task. The repo contains the adapters, dataset, raw eval JSONs, provenance records, hashes and commands needed to reproduce the runs. A few important limits: This is a narrow synthetic tool-use task with 29 held-out prompts. The training set is small enough that the run is clearly in memorization territory. I’m treating this as a demonstration of the training mechanism and reproducibility properties, not as evidence of general tool-use improvement. I’d be especially interested in people trying the reproducibility claim on hardware I don’t have access to, or rerunning the BF16/Q8/Q4 comparison on a harder task where the optimization-path difference may become behaviorally visible. Runner itself is here: > **[GitHub - Joakimpalm-Zen/xyntetik-runner: Single-binary GGUF model runtime in C —...](https://github.com/Joakimpalm-Zen/xyntetik-runner)** > > Single-binary GGUF model runtime in C — CPU/CUDA/Metal, OpenAI-compatible. Serves, scores, and trains LoRA directly through the quantized weights it deploys, with byte-reproducible adapters. Tool calls survive the token limit; sparse MoE and schema-constrained decoding included. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 23, 2026, 3:56am UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/2 "2026-08-23T03:56:10Z")
For now, I tried a few things myself: * * * The short version is: **the determinism claim held up surprisingly well in the external checks I could run** , including a Tesla T4 path and a one-step Qwen3-4B test using the same Q4\_K\_M base SHA and ToolUse training-data SHA recorded in the published adapter provenance. There was also one useful boundary condition: **holding the binary fixed and changing the host was not enough to break the tiny test, but changing the build profile was enough to change the adapter SHA.** So I think there are really a few different reproducibility contracts hiding under the word “deterministic”, and separating them makes the results easier to interpret. The strongest 4B check I ran was: ```plaintext published base GGUF SHA == probe base GGUF SHA published training SHA == probe training SHA same Runner binary same rank / alpha / lr / ctx / seed semantics same one-step training job CPU path vs Tesla T4-assisted backward ``` The CPU and T4-assisted runs both reported the same one-step loss (`0.676138`) and produced the same adapter SHA-256: ```plaintext 79c5f136486bcad288f0ad2f4f66345dab07db668f0cfe3de9d757bbfdb48e8f ``` The adapter files were byte-for-byte identical. The T4 side was not a silent CPU fallback; Runner reported: ```plaintext train-gpu: Tesla T4 — backward matvec on device ``` The exact base was the [bartowski Qwen3-4B Q4\_K\_M](https://huggingface.co/bartowski/Qwen_Qwen3-4B-GGUF/blob/a8512ea1cb744e4e9537847a993f49229b77d1ce/Qwen_Qwen3-4B-Q4_K_M.gguf), whose file SHA is: ```plaintext fbe1d5edd4ce802ae3ae7c7e4ab7d09789d697fdac1fc7929f8df4ca3c41bae3 ``` and the regenerated ToolUse training set also matched the SHA recorded in the published adapter provenance. I would still phrase this as an **exact-base/data one-step bridge** , not a reproduction of the full published training run: the published adapter provenance records Runner `0.1.20-alpha` and 316 steps, while my probe used a pinned `0.2.0`-era commit (`8652a7f...`) and deliberately stopped after one update. ### One distinction that may be useful for future reproduction reports After doing these checks, I would probably separate at least three things: ```plaintext 1. Artifact determinism same binary + same model/data/config -> same adapter bytes? 2. Build/toolchain reproducibility same source + independently built binary -> same adapter bytes? 3. Behavioral reproducibility same predictions / probabilities / task behavior? ``` Those did not behave identically in my small tests. With the **same binary** , I could replay the tiny fixture on a second Linux x86\_64 host with a different Xeon generation, kernel and glibc version and still get the same adapter SHA. Thread counts `1`, `2`, and `4` also gave the same adapter. But on the same Colab machine, rebuilding the same source with a different ISA/build profile changed the tiny adapter SHA. Interestingly, the resulting parameter difference was extremely small despite the file SHA changing: the adapter cosine was effectively 1 and the maximum F32 difference was only on the order of `1e-7`. So I would treat SHA-256 as a very sensitive **artifact-identity oracle** , but not as a behavioral-distance metric: ```plaintext different SHA does not necessarily mean meaningfully different behavior ``` That distinction may be useful if people start reporting results from ARM, Windows, different compilers, different release binaries, etc. For that reason, a third-party reproduction record might be easier to interpret if it includes something like: ```plaintext Runner commit Runner binary SHA-256 compiler + version build profile / ISA OS + architecture base GGUF SHA-256 training-data SHA-256 seed / rank / alpha / optimizer config adapter SHA-256 loss trajectory ``` rather than treating “same source checkout” and “same executable” as interchangeable. This is also a stronger reproducibility target than the one normally assumed by ML frameworks. For comparison, [PyTorch’s reproducibility documentation](https://docs.pytorch.org/docs/stable/notes/randomness.html) explicitly does not promise complete reproducibility across releases/platforms or between CPU and GPU execution. For the **BF16 / Q8\_0 / Q4\_K\_M** part, I think there is a relatively cheap next measurement before constructing a substantially harder benchmark. Right now the interesting result is: ```plaintext adapter weights: BF16 ≈ Q8, Q4 somewhat farther away held-out exact-call score: BF16 = Q8 = Q4 = 1.00 ``` That does not necessarily mean the learned functions are equally far apart or equally close. It may just mean the current exact-call metric is saturated. My default next step would be: ```plaintext existing held-out prompts | v teacher-force the gold tool call and compare its token log-prob / NLL | +-- differences visible | -> the precision-dependent path is already | behaviorally visible below top-1 | +-- still almost identical | v compare earlier checkpoints | +-- trajectories separate | -> different learning path, | similar final endpoint | +-- still insensitive -> move to a harder / OOD / near-confusable tool task ``` That seems cheaper than immediately building a large new evaluation suite, and it keeps the claim narrow: first ask whether the existing test hides probability-margin differences; only then ask for a harder task. > **What I actually tested** > > ### Tiny deterministic fixture > > I first used Runner’s tiny deterministic training fixture. > > On CPU, these all produced the same adapter SHA: > > ```plaintext > repeat A, t=2 > repeat B, t=2 > t=1 > t=2 > t=4 > > ``` > > Then I set: > > ```plaintext > RUNNER_TRAIN_GPU=1 > > ``` > > on a Colab Tesla T4. > > The CUDA-assisted backward path engaged, and the resulting adapter was byte-identical to the CPU adapter. > > I also checked that this was not merely producing an unchanged/no-op adapter: teacher-forced scoring improved from roughly > > ```plaintext > base: > NLL 5.5785 > PPL 264.68 > > trained: > NLL 5.1719 > PPL 176.24 > > ``` > > and the CPU and T4-trained adapters produced the same score. > > The pinned repository’s training tests also passed in that environment. > > ### Same binary, second Linux host > > I then took the **actual binary built in Colab** , without recompiling it, and ran the same tiny fixture on another Linux x86\_64 host. > > The two environments differed in CPU generation, kernel, glibc and CPU count. > > The adapter SHA remained identical. > > That does not establish x86/ARM or Linux/macOS/Windows identity, but it does suggest that, at least for this fixture, host/runtime variation alone was not sufficient to break the result once the executable itself was fixed. > > ### Build profile boundary > > On the other hand, rebuilding on the same machine changed the exact bytes. > > A normal local build and an explicit `x86-64-v3` build produced different tiny adapter SHAs. > > That is why I think “same binary on other hardware” and “independent rebuild from the same source” are worth treating as two separate tests. > > I would not attribute this to one specific compiler transformation without a smaller isolation test; the observation I can actually support is just that **changing the build/code-generation profile changed the exact training artifact**. > > ### Real Qwen3-4B bridge > > I first repeated the CPU/T4 check with the first-party [Qwen3-4B Q4\_K\_M GGUF](https://huggingface.co/Qwen/Qwen3-4B-GGUF/blob/main/Qwen3-4B-Q4_K_M.gguf). > > That one-step CPU/T4 comparison also produced byte-identical adapters. > > That was useful as a real-model sanity check, but that GGUF did not have the same file SHA as the base recorded by the published ToolUse adapter. > > So I repeated the experiment a second time using the exact Q4\_K\_M file recorded in the published provenance. > > For the final run: > > ```plaintext > base: > bartowski/Qwen_Qwen3-4B-GGUF > Qwen_Qwen3-4B-Q4_K_M.gguf > > base SHA: > fbe1d5edd4ce802ae3ae7c7e4ab7d09789d697fdac1fc7929f8df4ca3c41bae3 > > ToolUse train examples: > 158 > > train-data SHA: > 4c3feca6afa9d776b0e5d08afb6ac4b134bee93a94548dd934a74b2c944d6a5a > > ``` > > Both SHAs matched the published adapter provenance before the training comparison was allowed to run. > > The one-step configuration used the published rank/alpha/lr/context settings: > > ```plaintext > rank = 8 > alpha = 16 > lr = 1e-4 > ctx = 128 > > ``` > > One small reproducibility footgun I ran into: the provenance records seed `0`, but explicitly passing: > > ```plaintext > -s 0 > > ``` > > is not equivalent — Runner correctly rejects that because RNG state zero is a fixed point. For the corrected probe I omitted `-s`, which invokes Runner’s deterministic training-init default that is recorded as seed 0 in the training provenance. > > That seems worth documenting for anyone trying to reproduce the sidecar mechanically. > > ### What this test does _not_ establish > > I would not generalize the result to any of these yet: > > - all hardware; > - ARM64; > - Windows or macOS; > - arbitrary independently compiled binaries; > - every CUDA training path; > - the entire 316-step trajectory; > - byte reproduction of the historical `0.1.20-alpha` run; > - later Runner revisions; > - behavioral equivalence whenever two adapter SHAs differ. > > The positive result is narrower: > > > for the pinned Runner build and the tested Qwen3-4B Q4\_K\_M base/data/config, the first CPU update and the Tesla T4-assisted update produced the same adapter bytes. > **Where I think this sits relative to nearby GGUF / training work** > > I would separate three ideas that can otherwise sound like the same thing: > > ```plaintext > loading a GGUF for further training > > != > > keeping the quantized GGUF weights as the frozen training base > > != > > using the serving forward path itself as the training forward path > > ``` > > For example, [Transformers supports loading GGUF for further training](https://huggingface.co/docs/transformers/gguf), but its documented path dequantizes the GGUF checkpoint to FP32 PyTorch weights. > > There is also already nearby work on training LoRA directly over GGUF quantized bases. In particular, [this Unsloth discussion / proof of concept](https://github.com/unslothai/unsloth/discussions/3894) explicitly explores LoRA training over GGUF rather than a bitsandbytes 4-bit base. > > So I would not frame “LoRA over GGUF” by itself as the unusual part here. > > The part I find more interesting in Runner is the systems boundary: > > > the forward used while adapting the LoRA is the same forward implementation used to serve that quantized GGUF. > > There is a useful analogy in another domain: the [vLLM + TorchTitan bitwise-consistent train/inference work](https://vllm.ai/blog/2025-11-10-bitwise-consistent-train-inference) audited training/inference forward kernels for numerical equivalence and reused inference-side forward operations while supplying custom backward implementations. > > That project is solving an on-policy RL problem rather than GGUF LoRA adaptation, so I would not treat them as the same system. But the design pressure is similar: reducing or eliminating the numerical seam between the model that produces outputs and the model through which gradients are computed. > > That seems like a useful existing vocabulary for this part of Runner. > **One related branch: merging the adapter back into low-bit weights** > > The later merge/requantization result also seems worth keeping as a separate branch rather than mixing it into the determinism claim. > > There is a very close open research question in [llama.cpp issue #13485](https://github.com/ggml-org/llama.cpp/issues/13485), which explicitly proposes comparing: > > ```plaintext > 1. train LoRA on full precision > -> merge > -> quantize > > 2. train LoRA on quantized model > -> merge > > 3. train LoRA on quantized model > -> keep adapter separate > > ``` > > That is almost exactly the comparison I would want around the Q4 merge result. > > There is also a practical reason to distinguish **quantization** from **requantization**. The current [`llama-quantize` documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) warns that requantizing tensors that are already quantized can severely reduce quality compared with quantizing from 16- or 32-bit weights. > > And PEFT’s 4-bit LoRA merge implementation likewise explicitly warns that merging into a 4-bit linear layer can produce different generations because of rounding errors: [PEFT `bnb.py`](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/bnb.py). > > So if this branch gets explored further, one fairly clean control would be: > > ```plaintext > A. Q4 base + separate LoRA > vs > > B. high-precision base > + merge LoRA > + one-shot quantize to Q4 > vs > > C. already-Q4 base > + merge delta > + requantize to Q4 > > ``` > > That would help separate: > > ```plaintext > "the Q4 grid is too coarse for this delta" > > ``` > > from: > > ```plaintext > "the second projection/requantization step is the damaging part" > > ``` > > I would still keep this result local to the tested adapter/model until more cases are measured; I do not think the current evidence supports a general statement like “merging LoRA into Q4 destroys the adaptation.” So my current read is: - **the external determinism check is positive** for the paths I could test; - **binary identity looks like an important part of the reproducibility contract** ; - **adapter SHA and behavioral distance should probably be reported separately** ; - and, for the precision experiment, **gold-sequence probability / NLL looks like the cheapest next discriminator before moving to a harder benchmark**. At least from these checks, I would probably spend the next experiment budget on the BF16/Q8/Q4 behavioral-separation question rather than on another small determinism repetition. ---
**Author:** ![Joakimpalm-Zen](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joakimpalm-zen/32/62554_2.png) [@Joakimpalm-Zen](https://discuss.huggingface.co/u/Joakimpalm-Zen)\ **Post date:** [August 23, 2026, 11:16am UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/3 "2026-08-23T11:16:22Z")
Thanks for putting real work into this. Independent reproduction is exactly the currency this project trades in, so a report like yours is worth more to me than any benchmark I could run myself. A few responses: The T4 result matters more than it might look. Byte-equality between the CPU path and the GPU-assisted backward has now held on three GPU generations: my RTX 3070, an RTX PRO 6000 (Blackwell), and your Tesla T4. The T4 is also sm\_75, which is the exact floor the embedded PTX targets, so you validated the oldest corner of the support envelope. I could not have published that claim about my own hardware with a straight face; you did it for me. Your three-level framework is correct, and I’m adopting it. Runner’s contract is level 1 (artifact determinism: same binary, same inputs, same adapter bytes) plus level 3 (behavioral). It was never level 2, but I had not scoped that explicitly until your rebuild experiment forced the question. The mechanism behind your x86-64-v3 finding: the accumulation chains in the trainer are pinned fmaf sequences and survive any build, but the transcendentals are not. SiLU and softmax call expf, and libm plus compiler codegen for expf differs across ISA profiles. Different exp, different activations, different gradients, different bytes, same behavior (your cosine ~1.0). So the boundary of the determinism claim sits precisely at libm, which is a much sharper statement than I had before your test. Your reporting recommendations are going into the tool. The machine-written provenance record (.train.json) already carries base/data/adapter SHAs, seed and config. Next release it will also carry the running binary’s own SHA-256, compiler and build profile, and OS/arch, so a reproduction report can distinguish “same executable” from “same source” mechanically instead of by convention. And your proposed next step is the one I’m taking. Comparing held-out token logprobs across the bf16/Q8/Q4-trained adapters is exactly the right instrument for the question I care about (at what point does weight-space divergence become behavioral divergence). The three adapters diverge up to 12 percent in weight space while every top-1 metric sits at 1.000, so if precision leaves a fingerprint, logprobs are where it shows first. All three adapters and the eval set are already in the HF repo, and the scorer is one command, so this will be the next published result. If you ever rerun with the current main branch: training is about 2.3x faster now, and the adapter SHA is gated to be unchanged, which is a small demonstration that the speed and reproducibility claims compose. ---
**Author:** ![Joakimpalm-Zen](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joakimpalm-zen/32/62554_2.png) [@Joakimpalm-Zen](https://discuss.huggingface.co/u/Joakimpalm-Zen)\ **Post date:** [August 23, 2026, 6:01pm UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/4 "2026-08-23T18:01:59Z")
Quick follow-up: the logprob comparison you proposed is done and published. Setup: all 29 held-out prompts with their gold completions appended, teacher-forced through the same Q4\_K\_M serving base under each of the three study adapters, 3,195 scored positions. Raw per-position outputs and the summary are in the repo under evals/logprob-study/. Result: the weight-space divergence is behavioral, and the mapping is almost proportional. - bf16-trained vs Q8-trained adapter (2 percent apart in weight space): mean |dlogprob| 0.020, no position above 1 nat. Functionally the same adapter. - bf16-trained vs Q4-trained (12 percent apart): mean |dlogprob| 0.146, about 2 percent of positions above a full nat. That is roughly 40 percent of the entire adapter effect, since base vs any adapter runs about 0.32 to 0.40. - So a 6x gap in weight space comes out as a 7.4x gap in logprobs. Your instinct was right that this would show something the accuracy numbers could not: the divergence is invisible at top-1 only because this task’s decision margins are wide. On a task where decisions sit closer to zero margin, a 0.146 nat gap is the size that flips answers. That tight-margin task is the next rung. So the earlier study’s conclusion gets sharper: training through bf16 and training through the 4-bit base do not just learn measurably different weights, they learn measurably different behaviors, one level below where the eval saturates. Thanks for pointing at the right instrument before I built a new benchmark to find the same thing. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 25, 2026, 1:42pm UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/5 "2026-08-25T13:42:38Z")
I also did a bit of measuring on my side: * * * My short version is: **yes, I think the optimization-path difference can become behaviorally visible — but I would separate that conclusion from the aggregate `mean |Δlogprob|` number itself.** After looking at the published per-position data and then probing a few deliberately narrow tool-choice boundaries, I ended up with a picture that seems quite compatible with what you are building: 1. the BF16 / Q8 / Q4 training paths really do leave measurably different function-level fingerprints; 2. on the original 29 held-out examples, the supervised output decisions are mostly very wide-margin, so all three adapters can still land on the same tool calls; 3. if the tool-choice boundary is made narrower, I can get an actual **BF16/Q8 vs Q4 branch split** , including a different complete deterministic JSON tool call. So I would probably keep the logprob study, but treat **distributional drift** and **decision-boundary robustness** as two related-but-separate measurements. A low-cost extension to the current evaluation might therefore be a small “boundary lane” rather than a much larger benchmark: a handful of tool-vs-no-tool, similar-tool, multi-intent, and unfamiliar-wording cases, recording the selected branch plus the legal-choice margin. Runner already has most of the machinery for this in [`choice_logprobs`](https://github.com/Joakimpalm-Zen/xyntetik-runner) and the existing [quant-vs-tool-call-fidelity harness](https://github.com/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/quant-fidelity.md). > **What I measured and what changed when I split the measurements** > > ### 1. Re-reading the published logprob result > > I started from the raw files under the artifact’s [`evals/logprob-study/`](https://huggingface.co/Joakimpalm-Zen/Qwen3-4B-Runner-ToolUse-Q4_K_M/tree/main/evals/logprob-study). > > The published whole-stream BF16-vs-Q4 result is real: over the 3,195 scored positions, the mean absolute observed-token logprob difference is about: > > `mean |Δlogp| ≈ 0.146 nat` > > The distribution is also strongly tail-shaped rather than uniform: the median is much smaller, while a relatively small set of positions carry large differences. > > But splitting those positions by prompt vs gold completion changed my interpretation quite a lot. > > There are 629 completion positions. On those: > > `BF16 vs Q4 completion-only mean |Δlogp| ≈ 0.000272 nat` > > and none of the completion positions exceed `|Δlogp| > 1 nat`. > > Put another way, essentially all of the absolute BF16-Q4 difference in that whole-stream scalar comes from outside the supervised gold completion. A large fraction is even concentrated in prompt material that repeats across the 29 examples. > > That does **not** make the original logprob result uninteresting. To me it changes what the number is measuring. > > I would read it more like: > > > training through different precision paths leaves a measurable function-level fingerprint, including on regions that were not directly supervised by the completion loss. > > rather than: > > > the actual tool-call decisions are separated by about 0.146 nat. > > Those are both useful questions, but they are different questions. > > ### 2. Why the original 29 examples can still all agree > > This also makes the original held-out result easier to reconcile. > > On the supervised completion region, BF16 / Q8 / Q4 are extremely close. The gold tokens are generally high-probability tokens, so the measured tool-call trajectories look more like wide-margin decisions than near-boundary decisions. > > That fits the observed result: > > - different optimization trajectories; > - measurable whole-function distributional differences; > - but the same greedy tool behavior on this particular 29-example set. > > So I do not think the `1.00` tool-call result and the logprob divergence are in conflict. > > The original post already frames this correctly as a narrow synthetic demonstration rather than a general tool-use benchmark, which is important context for future readers: [HF thread](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118). > > ### 3. Then I tried measuring the actual tool-choice boundary > > Instead of using the observed-token logprob difference as a proxy for decision margin, I tried a small set of deliberately ambiguous / unfamiliar requests using the same raw JSON tool-call scaffold. > > One case turned out especially clean: > > `tell me what README.md says and translate it` > > At the first legal tool-name decision, I got approximately: > > | adapter training path | preferred branch | runner-up | top1-top2 margin | > | --- | --- | --- | --- | > | BF16 | `read_file` | `none` | 0.487 nat | > | Q8 | `read_file` | `none` | 0.496 nat | > | Q4 | `none` | `read_file` | 2.167 nat | > > So this is not merely “Q4 has a slightly smaller score.” > > It has actually crossed the decision boundary and become confident on the other side. > > BF16 and Q8 remain very close to each other, while the Q4-trained adapter picks the opposite branch. > > That is qualitatively consistent with the earlier weight-space observation that BF16 and Q8 are much closer to each other than either is to Q4, although I would **not** infer a general proportional law from this one case. > > ### 4. Full deterministic JSON confirms that the branch split survives generation > > I then checked the same request with complete deterministic greedy JSON generation rather than stopping at the first tool-name token. > > The four conditions produced: > > | condition | deterministic full call | > | --- | --- | > | base Q4 model, no adapter | `search_files({"pattern":"README.md","path":"."})` | > | BF16-trained adapter | `read_file({"path":"README.md"})` | > | Q8-trained adapter | `read_file({"path":"README.md"})` | > | Q4-trained adapter | `none({})` | > > So, for this constructed boundary case, the training-precision path difference is visible all the way out at the behavioral interface. > > That seems like a useful answer to the question in the original post about whether the different optimization paths can become behaviorally visible on a harder decision. > > The important limitation is that this is **one deliberately ambiguous/OOD-ish request** , not a prevalence estimate. > > I would describe the result as: > > > I found a concrete existence proof that the path difference can cross a tool-selection boundary. > > not: > > > Q4 training generally causes more tool-selection errors. > > ### 5. A few nearby cases also moved substantially, without always flipping > > The same small boundary scan produced several non-flip cases where the margin moved quite a lot. > > For example: > > | request | BF16 margin | Q8 margin | Q4 margin | branch | > | --- | --- | --- | --- | --- | > | `summarize README.md` | 3.150 | 2.926 | 0.675 | `none` for all three | > | `show me the markdown file README.md` | 6.840 | 6.110 | 1.275 | `read_file` for all three | > | `list docs and filter for markdown files` | 7.276 | 7.143 | 2.007 | `list_dir` for all three | > > But the direction is **not monotonic**. > > For example: > > `check notes.txt and write done if needed` > > gave approximately: > > - BF16: `0.947` > - Q8: `0.264` > - Q4: `1.871` > > So I would avoid turning this into a claim that lower training precision simply “shrinks margins.” > > What seems safer is: > > > the training precision path can materially reshape particular decision boundaries, and some boundaries are much more sensitive than others. > > That distinction probably matters quite a lot if this grows into a reusable evaluation. > **How I would separate the evaluation layers** > > I think the current work naturally decomposes into something like four layers. > > | Layer | Question | Example measurement | > | --- | --- | --- | > | reproducibility | Can the same training procedure reproduce the same artifact? | adapter SHA / deterministic rerun | > | function-level drift | Did the learned model function move differently? | per-position logprobs, KLD, effect-vector comparisons | > | supervised/output fidelity | Does the trained capability still produce the intended output? | exact tool call, tool selection, argument agreement | > | decision-boundary robustness | How close is the model to changing a discrete action? | legal-choice top1/top2 margin, branch flips | > > I like this split because none of the layers invalidates another. > > The original logprob study remains useful evidence that the precision path changes the learned function. > > The original 29-example tool eval remains useful evidence that the measured capability survives on those examples. > > A boundary probe adds the missing question: > > > how much perturbation room is there before the model chooses a different action? > > Runner’s current tooling already seems to be moving in this direction. > > The [quant-fidelity harness](https://github.com/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/quant-fidelity.md) explicitly separates: > > - schema conformance; > - tool selection; > - argument-content agreement; > - distributional divergence. > > And [`choice_logprobs`](https://github.com/Joakimpalm-Zen/xyntetik-runner) records legal alternatives and a posterior over the probed legal set for constrained decisions. > > So I would not see a decision-margin lane as a new evaluation philosophy that needs to replace what is there. It looks more like a small extension of the framework Runner already has. > > A compact version might be something like: > > ```plaintext > wide-margin regression set > -> should remain boring and stable > > boundary / ambiguity set > -> record legal alternatives > -> top1 > -> top2 > -> top1-top2 margin > -> full greedy call when the branch changes > > ``` > > The wide-margin set tells you whether ordinary functionality survived. > > The boundary set tells you where precision-path differences can turn into actions. > **A cheap boundary set might be enough before building a large benchmark** > > If the goal is information gain rather than benchmark size, I suspect only a few small families are needed initially. > > ### Tool vs no-tool > > Examples where an available tool is a useful precursor but cannot fully satisfy the stated request: > > - “summarize README.md” > - “translate README.md” > - “email README.md to the team” > > These are useful because “call the available precursor” and “no available tool completes the requested action” can both be locally plausible behaviors. > > ### Similar tools > > Cases near boundaries such as: > > - `list_dir` vs `search_files` > - `search_files` vs `read_file` > - `read_file` vs `write_file` > > For example: > > - “show me markdown files in docs” > - “find README.md and show it” > - “check notes.txt and write done if needed” > > ### Multi-intent requests > > A request combining two operations can expose whether one adapter treats the first tool operation as an actionable first step while another refuses because the whole request cannot be completed with one legal call. > > ### Wording shift > > The same semantic task under wording that was not represented by the tiny synthetic training templates. > > For each case, the useful record seems small: > > ```plaintext > prompt > condition / adapter > legal alternatives > top1 > top2 > margin > full deterministic call > > ``` > > That would be much cheaper than immediately scaling this into a large evaluation suite, while giving much more information than another aggregate exact-call percentage. > > If the boundary lane starts showing a stable pattern across many cases, _then_ increasing the case count would be useful. > > If it does not, that is also useful: the result may simply be highly decision-family-specific. > **Relation to quantization-margin literature** > > There is a recent paper whose **measurement idea** seems very relevant here: > > [Which Decisions Low-Bit Quantization Breaks, and How to Predict Them](https://arxiv.org/abs/2608.06564) > > It measures paired decision margins rather than trying to infer behavioral stability from aggregate benchmark accuracy alone, and it separates decisions such as whether to call a tool from which tool to call. > > That seems close to the useful measurement abstraction here. > > I would keep the causal connection deliberately weak, though. > > That paper primarily studies **inference quantization of a model** , whereas this experiment changes the precision path through which the **adapter is trained** , then serves the resulting adapters against a common Q4 base. > > So I would borrow: > > > measure the actual decision margin > > but not automatically borrow: > > > the same quantization mechanism caused the margin change. > > An older complementary result is [Accuracy is Not All You Need](https://arxiv.org/abs/2407.09141), which shows that compressed and reference models can have similar aggregate accuracy while still exhibiting substantial individual-answer flips, and argues for distance metrics such as KLD and flips alongside accuracy. > > Again, the setup is different, but the general evaluation lesson seems applicable: aggregate agreement and per-decision similarity are separate observables. > **Small harness/provenance note** > > One runtime detail was worth isolating because it can otherwise contaminate repeated measurements. > > I initially reused a JamePeng `Llama` context between conditions and hit native failures around state reset/reuse. > > There is an existing JamePeng report around reset behavior: [issue #168](https://github.com/JamePeng/llama-cpp-python/issues/168), and the JamePeng changelog also records fixes that explicitly clear KV/hybrid state on reset to avoid previous-run context poisoning: [CHANGELOG](https://github.com/TAO71-AI/llama-cpp-python-JamePeng/blob/main/CHANGELOG.md). > > For the measurements above I therefore used: > > ```plaintext > one prompt-condition > -> one fresh subprocess > -> one fresh Llama context > -> load exactly one adapter > -> evaluate > -> exit > > ``` > > No context was reused between BF16 / Q8 / Q4 conditions. > > That made the probe stable, and it also gives a cleaner experimental boundary: each measurement starts from an empty runtime state. > > I would treat this purely as a harness/provenance detail, not as part of the Xyntetik training claim itself. > **One measurement I discarded** > > For completeness: I also tried scoring each complete legal tool-name continuation. > > I discarded that lane after noticing a tokenization-boundary error in my probe. > > I had forced: > > ```plaintext > candidate_tool_name + " > > ``` > > but in the actual JSON continuation the tokenizer represents the following punctuation differently — in the generated path the quote/comma boundary can be a combined token. > > That made the forced continuation artificially improbable and changed the ranking. > > So I am **not** using that sequence-scoring result as evidence. > > The first-branch margin measurement and the complete greedy JSON generation do not depend on that discarded lane, and they agree on the interesting BF16/Q8-vs-Q4 case. So my current read would be: **the logprob study found a real precision-path fingerprint, but its aggregate scalar mostly measures something broader than the supervised tool-call decision itself. When I measured the discrete decision boundary separately, I did find a case where the difference becomes behavioral: BF16/Q8 choose `read_file`, while Q4 chooses `none`, and the split survives complete deterministic JSON generation.** That makes the result more interesting to me rather than less: the two measurements are answering different questions. If this were folded back into the reproducibility story, I think a small explicit **decision-boundary robustness** lane beside the existing reproducibility / fidelity measurements would make the interpretation unusually clean without requiring a large new benchmark. ---
**Author:** ![Joakimpalm-Zen](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joakimpalm-zen/32/62554_2.png) [@Joakimpalm-Zen](https://discuss.huggingface.co/u/Joakimpalm-Zen)\ **Post date:** [August 25, 2026, 5:41pm UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/6 "2026-08-25T17:41:01Z")
This is a better analysis of my own data than I did, and the correction lands. I re-verified your prompt/completion split from the published files before replying: on the last fifteen scored positions of each example, bf16-vs-Q4 mean |dlogprob| is 0.000257 nat with nothing near 1 nat, against 0.169 in the prompt region. So you are right about what the aggregate was measuring, and I have updated the HF card accordingly, with credit: the precision path leaves a real function-level fingerprint, but it lives almost entirely outside what the completion loss supervised, and the supervised decisions stay wide-margin. Two observables, not one. My earlier phrasing conflated them. And then you went and answered the question the study was reaching for. The read\_file vs none flip surviving full deterministic JSON is exactly the existence proof I had queued as the next experiment, so you have saved me a measurement campaign and improved its design at the same time. The non-monotonic margin cases are the part I find most useful: “some boundaries are much more sensitive than others” is a more honest and more interesting claim than any simple precision-shrinks-margins law, and it is the right caution for anyone who would over-generalize from the flip. I am adopting your four-layer split as the evaluation structure going forward, and the boundary lane as specced: your four case families, legal alternatives, top1/top2 margin, full greedy call on branch changes. You are right that choice\_logprobs and the quant-fidelity harness already carry most of the machinery, so this is an extension, not a new philosophy. The wide-margin set stays boring on purpose; the boundary set is where the information is. Also noted with appreciation: the harness discipline (fresh process per condition), the discarded sequence-scoring lane and why, and the two paper pointers, both of which I will read before building the lane. The paired-margins framing from the first one is clearly the right abstraction to borrow, with the causal caveat you flagged. If you are keeping notes on your boundary prompts, I would gladly take the full set as the seed of the lane, credited. At this point you are less a reviewer and more a collaborator, and the methodology is visibly better for it. ---
**Author:** ![daspinaki2005](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/daspinaki2005/32/62643_2.png) [@daspinaki2005](https://discuss.huggingface.co/u/daspinaki2005)\ **Post date:** [August 25, 2026, 10:12pm UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/7 "2026-08-25T22:12:02Z")
This is a really interesting approach. Training the LoRA directly through the same quantized GGUF weights used for inference makes the reproducibility aspect especially interesting. The BF16/Q8 vs Q4 optimization-path difference is also worth exploring on larger and more diverse datasets. I like that you’ve clearly mentioned the limitations of the current 29-prompt evaluation instead of presenting it as a general improvement. I’ll definitely check out the repo and reproducibility artifacts. Great work! ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 27, 2026, 10:35pm UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/8 "2026-08-27T22:35:14Z")
For now, I went ahead and uploaded them here: * * * - [Clean notebook — prompt bank + reproducible procedure](https://huggingface.co/datasets/John6666/forum3/blob/main/deterministic_GGUF_LoRA_training/Xyntetik_Runner_Tool_Choice_Boundary_Probe_Public_v1_0.ipynb) - [Executed notebook — same notebook with the reference run outputs](https://huggingface.co/datasets/John6666/forum3/blob/main/deterministic_GGUF_LoRA_training/Xyntetik_Runner_Tool_Choice_Boundary_Probe_Public_v1_0_executed.ipynb) The full **36-prompt set** is in there as a standalone `PROMPT_BANK`, so it should be easy to lift directly into Runner’s boundary lane. I deliberately left it as an **exploratory boundary bank rather than a labeled benchmark**. There are no authoritative gold tool labels attached to these prompts; the ambiguity is part of what they are for. The family names describe how the cases were constructed, not what the “correct” answer must be. The current groups are roughly: ```auto list_dir <-> search_files read_file <-> search/list read_file <-> write_file / multi-intent config browsing available-tool <-> none / useful-precursor cases ``` The notebook’s default path is intentionally small: ```auto 36 exploratory prompts | v Q4-trained adapter discovery scan | v keep cases with top1-top2 legal-choice margin <= 3 nat | v base / BF16 / Q8 / Q4 comparison | v if the trained adapters choose different branches: run full deterministic greedy JSON generation ``` So the first-token margin is just the cheap locator; a branch difference is only promoted to the more interesting behavioral result if it survives full generation. The executed reference run reproduced the earlier result cleanly: ```auto 36 / 36 discovery cases completed 7 cases selected at <= 3 nat 1 BF16/Q8/Q4 adapter-only branch split prompt: tell me what README.md says and translate it BF16 -> read_file Q8 -> read_file Q4 -> none ``` and the split survived complete deterministic JSON generation: ```auto base -> {"tool":"search_files","args":{"pattern":"README.md","path":"."}} bf16 -> {"tool":"read_file","args":{"path":"README.md"}} q8 -> {"tool":"read_file","args":{"path":"README.md"}} q4 -> {"tool":"none","args":{}} ``` The reference run also completed without worker failures. Feel free to take whatever level is useful for Runner — just the 36 prompts, the family grouping, the screening rule, or the whole procedure. I suspect the cleanest long-term home is probably Runner’s native [`choice_logprobs`](https://github.com/Joakimpalm-Zen/xyntetik-runner) / boundary-lane machinery rather than preserving this notebook as the canonical implementation. > **A few caveats I kept in the notebook** > > ### The selected set is selection-biased > > The initial screen uses the **Q4-trained adapter’s** margin, so the selected 7 cases should not be used to estimate a general BF16-vs-Q8-vs-Q4 margin distribution. > > The bank is mainly useful for locating interesting boundaries. > > ### There are intentionally no gold labels > > For prompts such as: > > ```auto > summarize README.md > translate README.md to French > tell me what README.md says and translate it > > ``` > > `read_file` can be a useful first step even though the available tool set cannot complete the whole user request. > > That ambiguity is exactly why these are useful boundary probes. Assigning a gold label too early would turn a decision-sensitivity probe into a quality benchmark and quietly change the question. > > If a later Runner suite wants correctness labels, I would probably make that a separate curated layer. > > ### The first-token measurement has a narrow contract > > At: > > ```auto > JSON: {"tool": " > > ``` > > the five legal choices are: > > ```auto > search_files > read_file > write_file > list_dir > none > > ``` > > and their first continuation tokens are distinct under the pinned tokenizer. The notebook verifies that at runtime before using the margin. > > I intentionally did **not** include the earlier naive full-tool-name sequence scorer. Punctuation can merge across the apparent string boundary under BPE tokenization, which made that version of the sequence score invalid. Full greedy JSON generation is used instead when an actual branch disagreement needs confirmation. > > ### I would not read a monotonic precision law into it > > Several cases had much smaller Q4 margins, but not all of them did. One case even had Q8 as the tightest condition. > > So the useful interpretation still seems to be: > > > some decision boundaries are much more sensitive to the training precision path than others > > rather than: > > > lower precision systematically shrinks margins. If the goal is simply to seed the lane you described, though, the important artifact is really just the prompt bank. The rest of the notebook is there so the way those prompts were discovered and screened remains reproducible rather than becoming an unexplained list of hand-picked examples. ---
**Author:** ![Joakimpalm-Zen](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joakimpalm-zen/32/62554_2.png) [@Joakimpalm-Zen](https://discuss.huggingface.co/u/Joakimpalm-Zen)\ **Post date:** [August 30, 2026, 3:14pm UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/9 "2026-08-30T15:14:33Z")
Thank you. This is the rung we had written down as a prediction and had not run. The precision study left this sentence in `docs/adaptation-engine.md`: the 12% weight-space divergence between the BF16-trained and the Q4-trained adapter is “invisible at top-1 only because this task’s decisions have wide margins; on a task whose decisions sit closer to zero margin, this is the size of gap that flips answers.” Your bank found one. `tell me what README.md says and translate it`, BF16 and Q8 choosing `read_file`, Q4 choosing `none`, surviving full deterministic greedy generation rather than only the first token, is the first instance anyone has produced of that predicted flip. Independently, on your hardware, with your prompts, against a sentence we wrote before you ran it. Three things about the method, because they are the reason I trust the result. **You screened on the Q4 margin and then said so.** That one sentence is what keeps the seven selected cases from being read as a sample of anything, and it is exactly the caveat that usually goes missing. **You refused the monotonic law.** One case having Q8 as the tightest condition is the observation that kills “lower precision shrinks margins” and leaves the weaker and correct claim standing: some decision boundaries are far more sensitive to the training precision path than others, and the direction is not monotone in bit width. We will carry it in that form, in your words, not in a tidier one. **You dropped your own sequence scorer once BPE punctuation merging made it invalid.** That is worth more than it looks, and it is also why your instinct about the long-term home is right. Runner’s `choice_logprobs` does not score tool-name strings at all. At each constrained decision point it records the legal alternatives as the grammar defines them, a posterior renormalized over that probed legal set, and the coverage mass. The alternatives are branches, not text, so the token-boundary failure you hit cannot arise by construction. `scripts/cl-calibration.py` then turns labeled decisions into accuracy, Brier score and ECE with an optional gate. Your notebook is approximating that machinery from outside the engine, and moving the bank onto it is the obvious next step. On what the finding can carry: one confirmed split, out of seven selected, out of 36, is an existence proof and we will publish it as one. It says the flip can happen. It does not say how often. Turning it into a rate needs an unbiased screen, which is a different and more expensive experiment, and we would rather state “this happens” accurately than “this happens X% of the time” loosely. Two questions. 1. **What licence do you want on the prompt bank?** We would like to bring the 36 prompts and the family grouping into the runner repo with your name on them, and we will match whatever terms you set. If you would rather it stay in your dataset repo and be referenced from ours, that works too. 2. **Do you have the sha256 of the three adapters you probed, and were they trained through the published protocol?** If they carry `.train.json` records, your behavioral result links back to the artifact-determinism level of your own three-level taxonomy, and the chain runs unbroken from adapter bytes to a flipped decision. That would be the first time all three levels close on one object. What we will do with it, so you can hold us to it. The adaptation-engine doc records the flip with credit and with both of your caveats intact. The precision study’s “capability-equivalent on this task” line gets scoped explicitly, so it stops reading as a general claim now that we have a case where it is not. And the bank goes onto the boundary lane if the licence allows. Again, thank you this has been awesome! ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 31, 2026, 8:33am UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/10 "2026-08-31T08:33:20Z")
I made the licensing explicit: * * * For the two concrete questions first: 1. **The 36-prompt bank is now `MIT OR Apache-2.0`.** So if you want to bring the prompts and family grouping directly into Runner, using them under Apache-2.0 is fine. I put the reusable bank, licence, README and provenance together here: [boundary prompt bank](https://huggingface.co/datasets/John6666/forum3/tree/main/deterministic_GGUF_LoRA_training/boundary_prompt_bank). 2. **Yes, I have the exact SHA-256 values of the three study adapters I probed, and they are the published study artifacts with their `.train.json` provenance records.** | training path | exact adapter SHA-256 | | --- | --- | | BF16 | `24d5f02122bc7436e6e6d5c8dc4e7bcbd336cd8cd5320c3d84d73de8ce31f292` | | Q8\_0 | `a2c70991d7421bab0c13cf2e5f5011ef1ed1b8bc343f33409b26cce35d3d8c55` | | Q4\_K\_M | `eda3c58491f9c65422d491f8fd01dbe561a2d988e3f16161231dcc0fd28db4f5` | For the behavioral probe I pinned the published study repo to: ```plaintext f9dd5b37df7177baf81f37a20fad7047d1f9f39e ``` and used the same Q4\_K\_M serving base for base/BF16/Q8/Q4 behavioral comparison: ```plaintext fbe1d5edd4ce802ae3ae7c7e4ab7d09789d697fdac1fc7929f8df4ca3c41bae3 ``` The pinned study artifacts are under the model repo’s [study directory](https://huggingface.co/Joakimpalm-Zen/Qwen3-4B-Runner-ToolUse-Q4_K_M/tree/f9dd5b37df7177baf81f37a20fad7047d1f9f39e/study). There is one provenance boundary I would keep explicit, mostly so future reproduction reports do not accidentally make the result stronger than what I actually ran: ```plaintext published training provenance -> exact published adapter bytes -> independent behavioral probe -> confirmed full-call decision flip ``` That chain is supported. What I did **not** do is independently rerun all three complete BF16/Q8/Q4 training jobs and reproduce those three final SHA-256 values from scratch. So I would describe this as closing the provenance-to-behavior chain on the exact published study objects, rather than as an independent full three-precision training reproduction. My earlier CPU/T4 one-step byte-identity experiment is useful evidence for Runner’s artifact-determinism contract, but it is a separate experiment. That distinction aside, the study provenance is actually quite tight. > **What the three historical training records pin** > > The three study sidecars record the same training data and training configuration, while changing the base precision. > > Common fields across all three are: > > ```plaintext > schema = xyntetik.runner.train.v1 > runner = 0.1.20-alpha > data SHA256 = 4c3feca6afa9d776b0e5d08afb6ac4b134bee93a94548dd934a74b2c944d6a5a > > seed = 0 > rank = 8 > alpha = 16 > lr = 1e-4 > steps = 120 > ctx = 128 > > AdamW: > beta1 = 0.9 > beta2 = 0.999 > eps = 1e-8 > weight decay = 0.01 > > ``` > > The training-base hashes then separate cleanly by precision: > > | path | training-base SHA-256 | resulting adapter SHA-256 | > | --- | --- | --- | > | BF16 | `52486602bdca589fd1507962537b4fc7fa2f1fc57222fa36a960cf691d4960c8` | `24d5f02122bc7436e6e6d5c8dc4e7bcbd336cd8cd5320c3d84d73de8ce31f292` | > | Q8\_0 | `4050871d20fa939e88e83b1a86060061bdeff01e0d9e5d000d374766f0caf7d7` | `a2c70991d7421bab0c13cf2e5f5011ef1ed1b8bc343f33409b26cce35d3d8c55` | > | Q4\_K\_M | `fbe1d5edd4ce802ae3ae7c7e4ab7d09789d697fdac1fc7929f8df4ca3c41bae3` | `eda3c58491f9c65422d491f8fd01dbe561a2d988e3f16161231dcc0fd28db4f5` | > > Those three base hashes also resolve to the public bartowski GGUF artifacts rather than being orphan identifiers: > > - [BF16 base](https://huggingface.co/bartowski/Qwen_Qwen3-4B-GGUF/blob/16479aa83310e195362c52005df32d3358b901d9/Qwen_Qwen3-4B-bf16.gguf) > - [Q8\_0 base](https://huggingface.co/bartowski/Qwen_Qwen3-4B-GGUF/blob/70d4e333a3c5161982e0cce89ed87f830ebe1faf/Qwen_Qwen3-4B-Q8_0.gguf) > - [Q4\_K\_M base](https://huggingface.co/bartowski/Qwen_Qwen3-4B-GGUF/blob/a8512ea1cb744e4e9537847a993f49229b77d1ce/Qwen_Qwen3-4B-Q4_K_M.gguf) > > One small correction to my earlier post is worth making explicit here: the **three precision-study adapters are 120-step jobs**. The 316-step record belongs to the separate release adapter in the model repo, not these three study adapters. > > There is also a historical-provenance limit worth preserving. > > The old sidecars say: > > ```plaintext > runner = 0.1.20-alpha > > ``` > > but that version string by itself is not sufficient to identify an exact historical executable/source build, and this older provenance schema predates the later binary/build-environment fields. > > So for these historical objects I think the strongest clean statement is: > > ```plaintext > exact input artifact hashes > + exact training-data hash > + training configuration > + exact output adapter hash > > ``` > > rather than: > > ```plaintext > exact historical executable independently reconstructed > > ``` > > That seems compatible with the reproducibility boundary we already separated earlier: artifact identity and build/toolchain reproducibility are different contracts. On the boundary bank itself, I agree with moving the long-term implementation into Runner’s native machinery. The notebook was useful as an external instrument because it made the discovery process reproducible, including the failed sequence-scoring attempt. But [Runner’s current `choice_logprobs` interface](https://github.com/Joakimpalm-Zen/xyntetik-runner) is a better canonical measurement surface: it records the grammar-legal alternatives at each constrained decision point, their raw logprobs, a posterior renormalized over the probed legal set, and coverage mass. I would just put one small migration bridge between the notebook result and the native lane rather than treating the two measurements as numerically interchangeable. The cheapest version I can think of is: ```plaintext exact same published adapters + exact same Q4 serving base + the 7 already-selected prompts | v native choice_logprobs run | +-- known BF16/Q8 vs Q4 branch split still visible? | +-- full deterministic generation still disagrees? | +-- coverage looks healthy at the relevant decision? | v migration bridge passed | v run the full 36-prompt exploratory bank ``` I would **not** require the native margin numbers to equal the notebook’s first-token margins. The notebook deliberately measured a narrow hand-built contract: ```plaintext JSON prefix + five first-token-distinct choices = cheap boundary locator ``` whereas Runner is recording decisions as its grammar actually exposes them. The legal decision points in the native tool protocol therefore do not have to correspond one-for-one to that hand-written five-way probe. For me, the useful migration invariant is behavioral: ```plaintext same exact artifacts same prompt same qualitative branch separation same full-call disagreement ``` not: ```plaintext same floating-point margin from two different scoring constructions ``` That also means that if the native lane moves or removes one of the old boundary cases, I would not interpret that as invalidating the pinned notebook result. It would instead tell us that the boundary is partly protocol/template/grammar dependent, which is useful information in its own right. > **How I would keep the exploratory and calibration lanes separate** > > I think there are two useful datasets hiding here, but they answer different questions. > > ### A. Exploratory boundary lane > > Keep the current 36 prompts **unlabeled**. > > That preserves the question they were designed to answer: > > > where do different trained adapters make different locally plausible decisions? > > For prompts such as: > > ```plaintext > summarize README.md > translate README.md to French > tell me what README.md says and translate it > > ``` > > there may be a meaningful distinction between: > > - taking an available precursor action such as `read_file`; > - declining because no available tool completes the whole request; > - choosing a different information-gathering tool first. > > Attaching a single authoritative gold label would silently turn that into a correctness benchmark. > > For the exploratory lane, I think the useful native record is closer to: > > ```plaintext > prompt > family > > condition: > base > BF16 > Q8_0 > Q4_K_M > > decision point > legal alternatives > top branch > runner-up > posterior / logprob information > coverage > > full deterministic call if conditions disagree > > ``` > > No accuracy percentage is needed. > > ### B. Labeled calibration/evaluation lane > > If there is later a reason to measure whether Runner’s decision confidence is calibrated, I would make that a separate curated dataset where the labels are defined before looking at the adapter outcomes. > > That lines up naturally with [`scripts/cl-calibration.py`](https://github.com/Joakimpalm-Zen/xyntetik-runner/blob/main/scripts/cl-calibration.py), which consumes labeled decisions and turns them into accuracy/Brier/ECE-style measurements. > > So I would keep the split roughly: > > ```plaintext > ambiguous constructed prompts > -> > exploration / sensitivity / disagreement > > clearly specified labeled prompts > -> > correctness / calibration / ECE > > ``` > > Neither lane needs to replace the other. There is also a very cheap way to get more information from the existing bank without turning it into a general benchmark. The original discovery procedure intentionally screened on the Q4-trained adapter and selected 7/36 cases at the margin threshold. That was useful for finding an existence proof, but it means `1/7` should stay exactly what you called it: a result inside a selected subset, not a frequency estimate. Now that the existence proof already exists, the next inexpensive statistic could simply be to run **all 36 prompts under BF16/Q8/Q4 without using one adapter to select the subset first**. That would let the result say something like: ```plaintext within this deliberately constructed 36-prompt boundary bank: N prompts showed an adapter branch disagreement disagreements appeared in families X/Y/... ``` That still would **not** estimate how often precision-path flips occur in ordinary user traffic. I would keep three quantities separate: | quantity | what it can mean | | --- | --- | | `1 / 7 selected cases` | existence-proof discovery result | | disagreement across all 36 constructed prompts | property of this particular boundary bank | | prevalence in real tool-use requests | requires a target population and an unbiased sampling design | That separation seems more useful than spending effort immediately on a much larger benchmark. If the all-36 native pass shows that disagreements cluster in one or two prompt families, that tells you where an expanded evaluation would have the highest information gain. If it shows no stable family structure, that is useful too. > **Why I would keep the original notebook even after native migration** > > I would still keep the clean/executed notebooks available even if Runner becomes the canonical implementation. > > They preserve several pieces of experimental history that are useful for future readers: > > 1. the original 36-prompt construction; > 2. the Q4-screening rule that produced the seven cases; > 3. the fact that the selected subset is selection-biased; > 4. the first-token distinctness check; > 5. the discarded full-tool-name sequence scorer and why it was discarded; > 6. the full deterministic generation check used to promote a first-token disagreement into a behavioral result. > > That gives a future reader both: > > ```plaintext > historical external measurement > + > native maintained implementation > > ``` > > rather than requiring the native implementation to erase how the result was originally found. > > The public source package is here: > > - [boundary prompt bank](https://huggingface.co/datasets/John6666/forum3/tree/main/deterministic_GGUF_LoRA_training/boundary_prompt_bank) > - [clean notebook](https://huggingface.co/datasets/John6666/forum3/blob/main/deterministic_GGUF_LoRA_training/Xyntetik_Runner_Tool_Choice_Boundary_Probe_Public_v1_0.ipynb) > - [executed reference notebook](https://huggingface.co/datasets/John6666/forum3/blob/main/deterministic_GGUF_LoRA_training/Xyntetik_Runner_Tool_Choice_Boundary_Probe_Public_v1_0_executed.ipynb) > > And the natural Runner-side references seem to be: > > - [`docs/adaptation-engine.md`](https://github.com/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/adaptation-engine.md) > - [`choice_logprobs` in Runner](https://github.com/Joakimpalm-Zen/xyntetik-runner) > - [`scripts/cl-calibration.py`](https://github.com/Joakimpalm-Zen/xyntetik-runner/blob/main/scripts/cl-calibration.py) > - the existing [quant-fidelity methodology](https://github.com/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/quant-fidelity.md) So my default route from here would be fairly small: ```plaintext dual-licensed bank -> native seven-case migration bridge -> if the known behavioral split survives, run all 36 without outcome-based screening -> keep that as the exploratory boundary lane separately, only if useful later: create a frozen-label calibration lane ``` I would not jump to a large benchmark or another full determinism campaign before that. The seven-case bridge is cheap, the 36-case pass is still small, and together they answer the immediate integration question without asking the bank to support a prevalence claim it was never designed for. And yes, please feel free to take the prompts and family grouping into Runner with credit. The native boundary lane still looks like the cleanest long-term home for them. ---
**Author:** ![Joakimpalm-Zen](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joakimpalm-zen/32/62554_2.png) [@Joakimpalm-Zen](https://discuss.huggingface.co/u/Joakimpalm-Zen)\ **Post date:** [September 1, 2026, 6:04pm UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/11 "2026-09-01T18:04:42Z")
Thank you. That answers everything, and more carefully than i asked. The licence settles it: the 36 prompts and the family grouping will come into Runner under Apache-2.0 with your name on them, as the seed corpus for the native boundary lane, and the bank repo will be referenced as the canonical source. The clean and executed notebooks stay linked as the discovery record. You are right that the native lane should not erase how the result was found, and the discarded sequence scorer is exactly the kind of history worth keeping: it documents why the lane records grammar branches instead of tool-name text. Your provenance framing is adopted verbatim, and it is now written into the planning record in your words so that a future reproduction report cannot quietly make the result stronger than what was run: published training provenance, to exact published adapter bytes, to independent behavioral probe, to confirmed full-call decision flip. That chain is closed on the published study objects. It is not an independent three-precision training reproduction, and nothing we publish will describe it as one. The T4 one-step byte-identity result stays filed as what it is, a separate experiment about the artifact-determinism contract. The migration bridge is adopted as you designed it. First run of the native lane will be your seven cases against exactly the artifacts you pinned, the three sha256 adapters over the shared Q4 serving base, through choice\_logprobs. The invariant we will hold it to is the behavioral one: same artifacts, same prompts, same qualitative branch separation, same full-call disagreement. We will not require the native margins to reproduce the notebook’s first-token margins, because the two constructions measure different surfaces, and if a boundary case moves or disappears under the native grammar we will record that as protocol dependence rather than as either measurement being wrong. If the bridge passes, the next step is the one you suggested: all 36 prompts under base, BF16, Q8 and Q4 with no adapter-screened subset, reported as a property of this bank and nothing more. Your three-quantity separation is now standing wording on our side: one of seven selected cases is an existence proof, disagreement across the 36 constructed prompts is a property of the bank, and prevalence in real traffic would need a sampling design nobody has built yet. The exploratory lane stays unlabeled, with the record shape you listed. If a calibration lane is ever worth building it will be a separate dataset with labels frozen before any adapter output is seen, feeding the existing calibration script, and it will not replace the exploratory bank. When the boundary lane lands in Runner it will carry the attribution and a pointer back here, and the changelog will say where the prompts came from. For whatever it is worth from the other side of the exchange: preregistered screens, instruments you killed yourself, and claims scoped tighter than i would have dared to scope them for you. Thank you again. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [September 3, 2026, 12:42am UTC](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/12 "2026-09-03T00:42:39Z")
yw! Since I had the chance, I tried it, and I think the native exploratory lane actually surfaced a useful difference: * * * The short version is: the native migration seems to have done exactly what I was hoping this kind of lane could do. The 7-case migration bridge preserved the external probe’s top-1 branch on **28/28 case-condition rows** , including the previously known `README.md + translate` BF16/Q8-vs-Q4 split. Then the **unscreened 36-prompt native pass** found **3/36 adapter disagreements** : - the known `tool_vs_none` case; - `show me the yaml files in config`; - `list config and find yaml files`. All three were: ```plaintext BF16 == Q8 != Q4 ``` and all three survived the transition from the `choice_logprobs` decision to the full deterministic emitted call. I would still treat `3/36` only as a property of this deliberately constructed exploratory bank, not as a failure rate or prevalence estimate. But as an existence/discovery result, I think it is useful. More importantly, the follow-up controls made one design distinction much clearer to me: ```plaintext single-condition uncertainty screening != cross-condition disagreement scanning ``` The original Q4 low-margin screen was useful — it found the first existence proof, so I would not replace it. But it turns out that a model can also be **quite confident in its own branch while disagreeing with another precision-path adapter**. So if the question is specifically “where did these training paths place a decision boundary differently?”, an unscreened symmetric comparison seems to add information that a Q4-only uncertainty threshold cannot provide. If I were keeping this as a small Runner workflow, my default would now be roughly: ```plaintext cheap single-condition uncertainty locator | v unscreened cross-condition disagreement pass | v tiny matched controls around an actual hit | v stop once the effect is localized enough to describe safely separately, only if correctness becomes the question: labeled calibration / evaluation ``` That separation also seems to fit the way Runner already treats [`choice_logprobs`](https://github.com/Joakimpalm-Zen/xyntetik-runner) as a decision record, while labeled calibration and the existing [quant-fidelity harness](https://github.com/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/quant-fidelity.md) measure other things separately. I also would **not** summarize the new result as “Q4 prefers search” or “lower precision causes a recency bias”. The tiny controls pushed me in almost the opposite direction: the new `config_browse` effect looks fairly **local, lexical, and structural** , and several plausible broad explanations collapse under one-factor changes. At this point I would probably stop expanding this particular prompt family unless another question makes it useful. The small controls already seem sufficient to establish the interesting part without quietly turning the exploratory bank into a benchmark. > **Native migration and the unscreened 36-prompt pass** > > For the migration gate, I used behavioral branch identity rather than requiring numerical equality between the old external scorer and Runner’s native `choice_logprobs`. > > That seems like the right level of equivalence here because the measurement implementation changed, while the intended synthetic decision surface did not. > > The native lane landed at: > > [`6a5288021ccaf29d9e6a25f0b73a849dac15c9d8`](https://github.com/Joakimpalm-Zen/xyntetik-runner/commit/6a5288021ccaf29d9e6a25f0b73a849dac15c9d8) > > with the boundary-lane implementation around: > > - [`docs/tool-choice-boundary-lane.md`](https://github.com/Joakimpalm-Zen/xyntetik-runner/blob/6a5288021ccaf29d9e6a25f0b73a849dac15c9d8/docs/tool-choice-boundary-lane.md) > - [`scripts/tool-choice-boundary.py`](https://github.com/Joakimpalm-Zen/xyntetik-runner/blob/6a5288021ccaf29d9e6a25f0b73a849dac15c9d8/scripts/tool-choice-boundary.py) > > The 7 selected cases × 4 conditions gave: > > ```plaintext > 28 / 28 top-1 branch matches > > ``` > > between the previous external probe and the native lane. > > The previously known case also remained behaviorally visible: > > ```plaintext > tell me what README.md says and translate it > > BF16 -> read_file > Q8 -> read_file > Q4 -> none > > ``` > > and the full deterministic call preserved that difference. > > After that, running all 36 exploratory prompts without selecting them by Q4 margin produced: > > ```plaintext > BF16 vs Q8 disagreement: 0 / 36 > BF16 vs Q4 disagreement: 3 / 36 > Q8 vs Q4 disagreement: 3 / 36 > > ``` > > The three adapter disagreements were: > > | prompt | BF16 | Q8 | Q4 | > | --- | --- | --- | --- | > | `tell me what README.md says and translate it` | `read_file` | `read_file` | `none` | > | `show me the yaml files in config` | `list_dir` | `list_dir` | `search_files` | > | `list config and find yaml files` | `list_dir` | `list_dir` | `search_files` | > > All three also survived full deterministic generation. > > That last point is useful to me because it moves the observation one step beyond “some internal scalar differs”: > > ```plaintext > different native decision > -> > different emitted tool branch > > ``` > > while still keeping the claim quite narrow. > > The public discovery artifacts I used before the native migration are still here: > > - [clean boundary-probe notebook](https://huggingface.co/datasets/John6666/forum3/blob/main/deterministic_GGUF_LoRA_training/Xyntetik_Runner_Tool_Choice_Boundary_Probe_Public_v1_0.ipynb) > - [executed reference notebook](https://huggingface.co/datasets/John6666/forum3/blob/main/deterministic_GGUF_LoRA_training/Xyntetik_Runner_Tool_Choice_Boundary_Probe_Public_v1_0_executed.ipynb) > - [`MIT OR Apache-2.0` prompt-bank package](https://huggingface.co/datasets/John6666/forum3/tree/main/deterministic_GGUF_LoRA_training/boundary_prompt_bank) > > I would still describe that 36-prompt set as an **exploratory boundary bank** , not a labeled benchmark. > **Why I would keep uncertainty screening and disagreement scanning separate** > > The clearest example came from a 14-prompt matched one-factor-at-a-time neighborhood around the two new `config_browse` hits. > > Across those 14 prompts: > > ```plaintext > Q4 minimum top1-vs-top2 margin: 3.055 nat > Q4 prompts with margin <= 3: 0 / 14 > adapter-disagreement prompts: 6 / 14 > > ``` > > So the original rule: > > ```plaintext > select if Q4 margin <= 3 nat > > ``` > > would select **none** of that local grid, even though six prompts produced a BF16/Q8/Q4 branch disagreement. > > The converse also appeared: there were BF16/Q8 cases near their own boundaries where all three adapters still selected the same branch. > > The later tiny probes produced the same general lesson from another angle. > > For the clause-order probe, the Q4 margins were all above 3 nat while the forward composition still disagreed. > > For the connector probe: > > ```plaintext > Q4 minimum margin: 0.439 nat > Q4 margin <= 3: 3 / 10 > > ``` > > For the synonym probe: > > ```plaintext > Q4 minimum margin: 2.895 nat > Q4 margin <= 3: 1 / 8 > > ``` > > So I would currently phrase the observation as: > > > Cross-condition disagreement can occur both near and well away from one condition’s own local uncertainty boundary. > > That is why I would keep both acquisition ideas rather than replacing one with the other: > > ```plaintext > uncertainty: > "where is this condition itself unsure?" > > disagreement: > "where do these conditions make different decisions?" > > ``` > > They overlap sometimes, but they are not asking the same question. > > This does **not** make the original Q4 screen a mistake. It did its job as a very cheap locator for the first existence proof. The unscreened native pass just exposed another useful axis afterward. > > Runner’s current interface already makes this separation fairly natural: `choice_logprobs` exposes the legal alternatives, restricted posterior/raw logprobs and coverage, while the calibration path can use labeled decisions to compute accuracy/Brier/ECE. The broader [quant-fidelity harness](https://github.com/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/quant-fidelity.md) similarly separates schema conformance, tool selection, argument agreement and distributional divergence instead of treating “tool-call quality” as one scalar. > **Localizing the two new config\_browse crossings** > > I tried to localize these two rather than expanding the prompt bank. > > The result was useful mostly because several broad explanations did **not** survive. > > ### 1. `show me the yaml files in config` > > The anchor was: > > ```plaintext > BF16 -> list_dir > Q8 -> list_dir > Q4 -> search_files > > ``` > > But six one-factor variants all removed the disagreement. > > Changing the operation word: > > ```plaintext > show -> list => all list_dir > show -> find => all search_files > > ``` > > Changing the extension: > > ```plaintext > yaml -> json => all search_files > yaml -> toml => all search_files > > ``` > > Changing the directory: > > ```plaintext > config -> docs => all search_files > config -> tests => all search_files > > ``` > > So I would not treat this as a reusable “config/YAML” phenomenon. It currently looks much more like a sentence-local lexical crossing. > > ### 2. `list config and find yaml files` > > This anchor was more stable: > > ```plaintext > BF16 -> list_dir > Q8 -> list_dir > Q4 -> search_files > > ``` > > Changing only the extension preserved it: > > ```plaintext > list config and find json files > list config and find toml files > > ``` > > Changing only the directory also preserved it: > > ```plaintext > list docs and find yaml files > list tests and find yaml files > > ``` > > But changing either side of the competing operation cue collapsed it: > > ```plaintext > inspect config and find yaml files > -> all search_files > > list config and show yaml files > -> all list_dir > > ``` > > So at that stage the most economical description seemed to be: > > > In this constructed local neighborhood, extension and directory substitution preserve the BF16/Q8-vs-Q4 split, while disrupting either side of the `list` / `find` operation-cue conflict collapses it. > > I still would not call that a causal cue-weighting rule; it was just enough to justify one more small composition control. > > ### 3. Single cues and clause order > > The two cues separately were boring: > > ```plaintext > list config > -> BF16/Q8/Q4 all list_dir > > find yaml files in config > -> BF16/Q8/Q4 all search_files > > ``` > > The forward composition recreated the split: > > ```plaintext > list config and find yaml files in config > > BF16 -> list_dir > Q8 -> list_dir > Q4 -> search_files > > ``` > > but reversing it collapsed everything to search: > > ```plaintext > find yaml files in config and list config > > BF16 -> search_files > Q8 -> search_files > Q4 -> search_files > > ``` > > That made order/recency an obvious candidate, but the next control made a simple version of that explanation too strong. > > ### 4. Connector / sequencing control > > Using matched forward/reverse forms: > > | form | forward | reverse | > | --- | --- | --- | > | `A and B` | BF16/Q8=`list`, Q4=`search` | all `search` | > | `A, then B` | all `list` | all `search` | > | `first A, then B` | all `list` | all `search` | > | newline-separated | BF16/Q8=`list`, Q4=`search` | BF16/Q8=`list`, Q4=`search` | > | `A; B` | BF16/Q8=`list`, Q4=`search` | all `search` | > > So this is not well described by a simple: > > ```plaintext > first clause always wins > > ``` > > or: > > ```plaintext > last clause / recency always wins > > ``` > > rule. > > For example, reversing the newline-separated version did **not** remove the split, while changing `and` to an explicit `then` did. > > The safer description seems to be that branch resolution here is sensitive to the **composition and delimiting of the competing operation cues** , and that the BF16/Q8 and Q4 training paths place that local boundary somewhat differently. > > ### 5. Tiny synonym control > > I finally checked whether the literal operation words mattered. > > The exact forward anchor again reproduced: > > ```plaintext > list ... and find ... > > BF16 -> list_dir > Q8 -> list_dir > Q4 -> search_files > > ``` > > Replacing `find` with `locate` preserved it: > > ```plaintext > list ... and locate ... > > BF16 -> list_dir > Q8 -> list_dir > Q4 -> search_files > > ``` > > But replacing `list` with `enumerate` removed it: > > ```plaintext > enumerate ... and find ... > -> all search_files > > ``` > > and replacing both did the same: > > ```plaintext > enumerate ... and locate ... > -> all search_files > > ``` > > All four reverse-order synonym forms went to `search_files` for BF16/Q8/Q4. > > So the strongest wording I would use is only: > > > The literal `list` cue appears unusually important in this particular local neighborhood. > > Putting all of those controls together, the evidence currently looks more like: > > ```plaintext > generic YAML/config effect no evidence > generic "Q4 prefers search" rule no evidence > simple recency rule no > simple first-clause rule no > > local lexical/structural sensitivity > around a competing tool-choice boundary yes > > ``` > > That seems interesting enough on its own without promoting it into a general linguistic or quantization law. > > It is also worth keeping the original study context in view: the [published three-precision study/model card](https://huggingface.co/Joakimpalm-Zen/Qwen3-4B-Runner-ToolUse-Q4_K_M) had the small held-out tool-use evaluation saturated across these adapter paths. So I would read this as a boundary-sensitivity observation outside that easy eval, not as evidence that Q4-trained tool use is generally broken. > **What I think this suggests for the Runner lane** > > The result actually makes me more comfortable with the separation you described in [your follow-up](https://discuss.huggingface.co/t/deterministic-lora-training-directly-through-quantized-gguf-serving-weights/179118/11). > > I would keep these roles distinct: > > ### A. Exploratory boundary lane > > Purpose: > > ```plaintext > find interesting decision differences > > ``` > > Useful properties: > > - no authoritative gold label required; > - all conditions compared symmetrically; > - exact artifact identity retained; > - decision coverage retained; > - full-call confirmation when an actual branch flip appears; > - small matched counterfactuals only after a hit. > > This is where the 36-prompt bank belongs. > > ### B. Labeled calibration / evaluation lane > > Purpose: > > ```plaintext > decide whether a branch is correct > measure accuracy / calibration > > ``` > > That needs an actual labeling policy. I would not infer one retrospectively from which adapter chose which branch. > > So if one of these exploratory cases later becomes useful as an eval item, I would copy/promote it deliberately into a separately labeled dataset rather than silently turning its discovery metadata into ground truth. > > ### C. Population/prevalence study > > Purpose: > > ```plaintext > how often does this matter on traffic we care about? > > ``` > > That would be yet another question. > > Before quoting a rate, I think it would need a target prompt distribution or a sampling frame. The current `3/36` and local `6/14`, `2/8`, etc. are deliberately constructed-neighborhood statistics, so they should stay where they are. > > For me the nice default path is therefore: > > ```plaintext > exploratory bank > | > | disagreement hit > v > tiny matched controls > | > +---- effect is very local/noisy ---> document + stop > | > +---- effect looks reusable > | > +---- need correctness? ---> labeled eval lane > | > +---- need prevalence? ----> sampled target distribution > > ``` > > That keeps the cheap/high-information part cheap and avoids making every interesting boundary observation grow into a benchmark. > **Claim boundary / reproducibility notes** > > I would keep the provenance claim exactly as narrow as we already separated it: > > ```plaintext > published historical training provenance > -> > exact published adapter bytes > -> > independent behavioral probe of those bytes > -> > native/full-call decision differences > > ``` > > I am **not** treating this as: > > ```plaintext > independent full BF16/Q8/Q4 retraining > -> > same three published final adapter SHA-256 values > > ``` > > because I did not perform that full three-precision retraining reproduction. > > Likewise, I would keep the earlier CPU/T4 byte-identity experiment as a separate artifact-determinism result rather than using it to bridge that missing claim. > > For reference, the public materials behind the discovery side are: > > - [three-precision study/model card](https://huggingface.co/Joakimpalm-Zen/Qwen3-4B-Runner-ToolUse-Q4_K_M) > - [Xyntetik Runner](https://github.com/Joakimpalm-Zen/xyntetik-runner) > - [native boundary-lane commit](https://github.com/Joakimpalm-Zen/xyntetik-runner/commit/6a5288021ccaf29d9e6a25f0b73a849dac15c9d8) > - [clean public boundary notebook](https://huggingface.co/datasets/John6666/forum3/blob/main/deterministic_GGUF_LoRA_training/Xyntetik_Runner_Tool_Choice_Boundary_Probe_Public_v1_0.ipynb) > - [executed public boundary notebook](https://huggingface.co/datasets/John6666/forum3/blob/main/deterministic_GGUF_LoRA_training/Xyntetik_Runner_Tool_Choice_Boundary_Probe_Public_v1_0_executed.ipynb) > - [dual-licensed boundary prompt bank](https://huggingface.co/datasets/John6666/forum3/tree/main/deterministic_GGUF_LoRA_training/boundary_prompt_bank) > - [Runner quant/tool-call fidelity methodology](https://github.com/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/quant-fidelity.md) > > So, at least from this little exercise, I think the native exploratory lane earned its keep: it found behaviorally real precision-path differences that the saturated small eval did not expose, and the later controls gave a reasonably disciplined way to say **how local those differences are** without turning them into a broader claim than the evidence supports.