v2.2: open-taxonomy choice; v2.1's counterfactual and injection training

README.md

@@ -33,10 +33,13 @@ datasets:
 - dgslibisey/MuSiQue
 - xanhho/2WikiMultihopQA
 - google-research-datasets/go_emotions
+- databricks/databricks-dolly-15k
 ---
 
 # circuit-1.7b
 
+**v2.2** (2026-09-24). Choice questions whose options are a taxonomy the model has never seen: a user's own tags, each option a label with a description, "other" right only when nothing listed fits. v2.0 answered "other" too readily on unfamiliar option lists; v2.2 picks a listed option when one fits. Held-out label sets (no_robots task categories, MMLU subjects, DBpedia classes never trained on) rise from .58 to .70 top-1, and 30 tags written as a practitioner would, never seen in training, score 24 of 30. Everything v2.0 did holds: hard tier .851 (v2.0 .842), unseen .742 (.738), grounded tool calls .890 (.872), water calls .930 (.940), order flips 0.1% (0.3%, both with the repo's scorer), multi F1 .72 (.70), locate top-3 .90 (.89). It also carries v2.1's counterfactual and injection training, never released on its own: minimal edits that should and should not change an answer (.872), and instructions planted in the state (.864). See [Open taxonomies](#open-taxonomies-v22). `revision="v2.0"` gets the previous weights.
+
 **v2.0** (2026-09-23). Two new kinds of question, on text states. `multi`: every option that applies, each with its own probability (none may apply, or all). `locate`: which part of the state answers the question (a field, a list element, or a sentence), or none. Ask-next, which follow-up question would settle a case, is an ordinary `choice` it now answers well. Served: multi F1 .70 with per-option ECE .006, locate 71% top-1 and 90% top-3, ask-next 97%. Everything v1.3 did holds or improves: hard tier .842 (v1.3 .822), unseen .737 with ECE .105 (v1.3 .733 / .120), water calls .940 (.900), order flips 0.9%. See [Question types](#question-types-v20). `revision="v1.3"` gets the previous weights.
 
 **v1.3** (2026-09-22). Trained on long documents and multi-step reading as well: merger agreements, contract clauses, whole government policy pages, and multi-hop questions over 10 to 20 paragraphs, at 4,096 tokens instead of 1,024. On held-out rows of those five sources it answers 82.2% (v1.2: 59.1%; Jev, which never trained on them: 74.9%). Unseen families rise from .722 to .733 with better calibration (ECE .147 to .120). The cost: a few small in-house production families drop, see [Versions](#versions). `revision="v1.2"` gets the previous weights.
@@ -79,6 +82,20 @@ Served through the hosted endpoint, 879 held-out rows of the v2 training sources
 
 Locate is the least calibrated type: its confidence runs ahead of its top-1 accuracy. Treat a locate answer as the top three candidates, which hold the answer nine times in ten.
 
+## Open taxonomies (v2.2)
+
+A tagger asks choice questions like "which cost center should this be charged to?" with options the model was never trained on. v2.2 trained on 13,645 such rows built from human or code labels (Dolly-15k task categories, DBpedia classes, MASSIVE scenarios and intents, Banking77 and CLINC intents in random subsets, with the gold label removed a sixth of the time so that "other" is right), plus a 12,000-row replay of the v2 mix. Held-out label sets, top-1 / share answered "other":
+
+| | v2.0 | v2.2 |
+|---|---|---|
+| all held-out (2,303 rows) | .58 / .24 | **.70 / .21** |
+| no_robots task categories (eval only, CC BY-NC) | .58 | **.72** |
+| MMLU subjects | .63 | **.66** |
+| DBpedia classes held out of training | .66 | **.73** |
+| a tagging application's own questions on real chat logs, against a reference tagger | .54 | **.62** |
+
+Where it falls short: a tag about what a conversation *is* can be answered by what it is *about* (a question on how to store API keys read as containing one), and on a business-function tag ("which department") it still answers "other" often. Its confidence tracks this: when it is 0.7 sure or more it agrees with the reference tagger 80 to 96% of the time, so a floor on confidence, which decision-circuits' `argmax(..., min_confidence=)` sets, keeps the tags it gives trustworthy.
+
 ## Results
 
 Same items for every model, human labels, accuracy / ECE (15 bins).
@@ -172,6 +189,7 @@ Reproduce: `uv run python scripts/eval_permutations.py lora:<this repo> --out ou
 
 | tag | date | what changed |
 |---|---|---|
+| `v2.2` | 2026-09-24 | Open-taxonomy choice: +13,645 items with random option lists and a sometimes-right "other" (Dolly-15k, DBpedia, MASSIVE, Banking77, CLINC; human or code labels; no training row repeats an eval question), with v2.1's counterfactual and injection rows and a 12,000-row replay of the v2 mix. Continued from v2.0 at half the learning rate, 1 epoch (kept step 5,600 of about 6,100, 2.4 h on one RTX 3090). Held-out label sets .70 (v2.0 .58); hard .851; unseen .742 / ECE .133; grounded tools .890; water .930; order flips 0.1%. Measured with the repo's scorer on the training machine, not through the hosted endpoint. Dolly-15k and DBpedia text are CC BY-SA 3.0. |
 | `v2.0` | 2026-09-23 | `multi` and `locate` questions, and ask-next as a choice: +8,739 items (MuSiQue, 2Wiki, ShARC, CUAD and SQuAD 2.0 for locate; CUAD and GoEmotions, with the share of raters as the reference, for multi; ShARC follow-ups for ask-next), on top of the v1.3 mix, 4,096 tokens, 1 epoch (kept step 6,000 of 7,830, 5.6 h on one A40). Hard tier .842 / ECE .051; unseen .737 / ECE .105 / Brier .303; water calls .940; order flips 0.9% with accuracy .740 (v1.3 .706). All numbers measured through the hosted endpoint. |
 | `v1.3` | 2026-09-22 | +6,300 long-document and multi-hop items, 4,096 tokens, 1 epoch (kept step 5,200 of 5,864, 3 h on one A40). Hard-tier eval (650 held-out rows of the five new sources) .822 / ECE .040 against v1.2's .591 and Jev's .749. Unseen sets .733 / ECE .120 / Brier .303; water calls .900; order flips 0.9%. On the 981-item order set, accuracy .706 against v1.2's .729: the drop is in small in-house production families (30 items or fewer each), while every public held-out family held or improved. All numbers measured through the hosted endpoint. |
 | `v1.2` | 2026-09-22 | Options encoded side by side: order-stable answers (0.5% flips). Unseen sets .722 / ECE .147 / Brier .327 against v1.1's .738 / .133 / .306; water calls .850 against .930; grid .961 against .970. Per-type temperatures (choice 1.3) in the config. |

adapter/adapter_config.json

@@ -35,13 +35,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "q_proj",
-    "k_proj",
-    "gate_proj",
     "down_proj",
     "v_proj",
+    "gate_proj",
     "o_proj",
-    "up_proj"
+    "up_proj",
+    "q_proj",
+    "k_proj"
   ],
   "target_parameters": null,
   "task_type": null,

adapter/adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:393f2f02221bd0844ed41dd573ca7cc16c5a89f626aa9929dea5680b3ee54832
+oid sha256:8e64e6c6f7a5fd060d7b011b1f4c295ca51debfe104ea3b28951fb1ca3a2dae6
 size 69782384

config.json

@@ -13,6 +13,7 @@
   "<|fim_middle|>"
  ],
  "parallel_options": true,
+ "masked": false,
  "question_types": [
   "noul",
   "choice",
@@ -21,524 +22,388 @@
   "locate"
  ],
  "temperatures": {
-  "noul": 2.0,
-  "choice": 2.5,
+  "noul": 1.75,
+  "choice": 2.0,
   "score": 1.0
  },
  "best": {
-  "ece": 0.05545587268843555,
-  "accuracy": 0.8856610800744878,
-  "kl": 0.3611638357698157,
+  "ece": 0.05495522255122487,
+  "accuracy": 0.8948283355063016,
+  "kl": 0.3489468937737143,
   "ece_by_type": {
-   "noul": 0.04219939842704649,
-   "choice": 0.08952805843863466,
-   "score": 2.1930535634373527e-05,
-   "multi": 0.005115650817968373,
-   "locate": 0.16475388922789636
+   "noul": 0.03301019309669409,
+   "choice": 0.06774300191745326,
+   "multi": 0.0088773566603521,
+   "locate": 0.1752706635885566
   },
   "multi": {
-   "n": 299,
-   "exact_set": 0.43812709030100333,
-   "f1": 0.6994047619047619
+   "n": 82,
+   "exact_set": 0.4634146341463415,
+   "f1": 0.7106598984771574
   },
   "locate": {
-   "n": 473,
-   "top1": 0.7167019027484144,
-   "top3": 0.8752642706131079
+   "n": 131,
+   "top1": 0.7251908396946565,
+   "top3": 0.8854961832061069
   },
-  "ece_worst": 0.16475388922789636,
-  "step": 6000
+  "ece_worst": 0.1752706635885566,
+  "step": 5600
  },
  "history": [
   {
-   "ece": 0.06883794354965124,
-   "accuracy": 0.6636871508379888,
-   "kl": 0.7090902082948101,
+   "ece": 0.08584639207273398,
+   "accuracy": 0.8365927857453281,
+   "kl": 0.5910362811236936,
    "ece_by_type": {
-    "noul": 0.07981667523236236,
-    "choice": 0.07539085847298796,
-    "score": 0.20667400674687494,
-    "multi": 0.04436400984039529,
-    "locate": 0.2054201467691214
+    "noul": 0.023615736474272973,
+    "choice": 0.10823203112135801,
+    "multi": 0.013472037295757265,
+    "locate": 0.22944260974421757
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.22742474916387959,
-    "f1": 0.0
+    "n": 82,
+    "exact_set": 0.4634146341463415,
+    "f1": 0.7
    },
    "locate": {
-    "n": 473,
-    "top1": 0.49682875264270615,
-    "top3": 0.7315010570824524
+    "n": 131,
+    "top1": 0.7480916030534351,
+    "top3": 0.8702290076335878
    },
-   "ece_worst": 0.20667400674687494,
+   "ece_worst": 0.22944260974421757,
    "step": 400
   },
   {
-   "ece": 0.0548305833722626,
-   "accuracy": 0.7426443202979516,
-   "kl": 0.573386100766419,
+   "ece": 0.055616019563227356,
+   "accuracy": 0.8670143415906127,
+   "kl": 0.4500976491261067,
    "ece_by_type": {
-    "noul": 0.06620688738749006,
-    "choice": 0.06049149208721312,
-    "score": 0.10243611716561847,
-    "multi": 0.010564134443887839,
-    "locate": 0.2184526875898652
+    "noul": 0.02860243448647127,
+    "choice": 0.071587355912699,
+    "multi": 0.008698512188276051,
+    "locate": 0.24214290850262607
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.27424749163879597,
-    "f1": 0.42398286937901497
+    "n": 82,
+    "exact_set": 0.4268292682926829,
+    "f1": 0.6629213483146067
    },
    "locate": {
-    "n": 473,
-    "top1": 0.5623678646934461,
-    "top3": 0.8160676532769556
+    "n": 131,
+    "top1": 0.7862595419847328,
+    "top3": 0.8931297709923665
    },
-   "ece_worst": 0.2184526875898652,
+   "ece_worst": 0.24214290850262607,
    "step": 800
   },
   {
-   "ece": 0.02867637172313596,
-   "accuracy": 0.8115456238361266,
-   "kl": 0.4371429148751139,
+   "ece": 0.06092668626812012,
+   "accuracy": 0.8683181225554107,
+   "kl": 0.43823268247499647,
    "ece_by_type": {
-    "noul": 0.0426548361316208,
-    "choice": 0.03077342929663485,
-    "score": 0.030355580978923408,
-    "multi": 0.010345635710125953,
-    "locate": 0.15709535626351964
+    "noul": 0.020365813855201913,
+    "choice": 0.08031533988714888,
+    "multi": 0.015809024437894285,
+    "locate": 0.18877925382543154
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.3612040133779264,
-    "f1": 0.6326194398682042
+    "n": 82,
+    "exact_set": 0.3780487804878049,
+    "f1": 0.6470588235294118
    },
    "locate": {
-    "n": 473,
-    "top1": 0.638477801268499,
-    "top3": 0.8414376321353065
+    "n": 131,
+    "top1": 0.7251908396946565,
+    "top3": 0.8778625954198473
    },
-   "ece_worst": 0.15709535626351964,
+   "ece_worst": 0.18877925382543154,
    "step": 1200
   },
   {
-   "ece": 0.06664805306291664,
-   "accuracy": 0.8126629422718809,
-   "kl": 0.44059206739287016,
+   "ece": 0.0723707175601933,
+   "accuracy": 0.8744024337244676,
+   "kl": 0.47455943871286976,
    "ece_by_type": {
-    "noul": 0.06549387772878008,
-    "choice": 0.07826259436676046,
-    "score": 0.015112782518068971,
-    "multi": 0.011307363245307869,
-    "locate": 0.1515529298442111
+    "noul": 0.024952563162772824,
+    "choice": 0.09539862066323884,
+    "multi": 0.01554111122524691,
+    "locate": 0.19624842748842167
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.3110367892976589,
-    "f1": 0.44025157232704404
+    "n": 82,
+    "exact_set": 0.45121951219512196,
+    "f1": 0.6731707317073171
    },
    "locate": {
-    "n": 473,
-    "top1": 0.6173361522198731,
-    "top3": 0.8054968287526427
+    "n": 131,
+    "top1": 0.7786259541984732,
+    "top3": 0.9083969465648855
    },
-   "ece_worst": 0.1515529298442111,
+   "ece_worst": 0.19624842748842167,
    "step": 1600
   },
   {
-   "ece": 0.10378278813548594,
-   "accuracy": 0.7895716945996276,
-   "kl": 0.5127488619592051,
+   "ece": 0.04666243936511963,
+   "accuracy": 0.8770099956540635,
+   "kl": 0.3581923115652219,
    "ece_by_type": {
-    "noul": 0.08858446728351507,
-    "choice": 0.13592801827462125,
-    "score": 0.021776424679491292,
-    "multi": 0.04203863114282169,
-    "locate": 0.1246890798841168
+    "noul": 0.010892258792795158,
+    "choice": 0.05909628825909472,
+    "multi": 0.011557487042053855,
+    "locate": 0.25249793170062645
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.3177257525083612,
-    "f1": 0.6378653113087674
+    "n": 82,
+    "exact_set": 0.4268292682926829,
+    "f1": 0.6702702702702703
    },
    "locate": {
-    "n": 473,
-    "top1": 0.642706131078224,
-    "top3": 0.8562367864693446
+    "n": 131,
+    "top1": 0.7404580152671756,
+    "top3": 0.9083969465648855
    },
-   "ece_worst": 0.13592801827462125,
+   "ece_worst": 0.25249793170062645,
    "step": 2000
   },
   {
-   "ece": 0.044182182074259085,
-   "accuracy": 0.8309124767225325,
-   "kl": 0.43050666850975905,
+   "ece": 0.03807536552440395,
+   "accuracy": 0.8839634941329857,
+   "kl": 0.33259975850023243,
    "ece_by_type": {
-    "noul": 0.03531643189201053,
-    "choice": 0.053918757863005526,
-    "score": 0.07129591504732762,
-    "multi": 0.009703490670203811,
-    "locate": 0.13904867823733835
+    "noul": 0.028432255791079594,
+    "choice": 0.052143850720789295,
+    "multi": 0.01640025211977481,
+    "locate": 0.197801211635575
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.41471571906354515,
-    "f1": 0.6666666666666666
+    "n": 82,
+    "exact_set": 0.43902439024390244,
+    "f1": 0.6588235294117647
    },
    "locate": {
-    "n": 473,
-    "top1": 0.6490486257928119,
-    "top3": 0.8329809725158562
+    "n": 131,
+    "top1": 0.7480916030534351,
+    "top3": 0.9083969465648855
    },
-   "ece_worst": 0.13904867823733835,
+   "ece_worst": 0.197801211635575,
    "step": 2400
   },
   {
-   "ece": 0.061252941737849856,
-   "accuracy": 0.8405959031657355,
-   "kl": 0.3697396882027976,
+   "ece": 0.05001172654554151,
+   "accuracy": 0.8796175575836592,
+   "kl": 0.35038399462927117,
    "ece_by_type": {
-    "noul": 0.045710157470185656,
-    "choice": 0.0896336348581707,
-    "score": 0.01498017758131024,
-    "multi": 0.03842968270206669,
-    "locate": 0.15707442163633495
+    "noul": 0.02750964254461312,
+    "choice": 0.061888604430626735,
+    "multi": 0.020829151295887208,
+    "locate": 0.2273335669559377
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.4214046822742475,
-    "f1": 0.6655518394648829
+    "n": 82,
+    "exact_set": 0.45121951219512196,
+    "f1": 0.6918918918918919
    },
    "locate": {
-    "n": 473,
-    "top1": 0.6638477801268499,
-    "top3": 0.8520084566596194
+    "n": 131,
+    "top1": 0.7404580152671756,
+    "top3": 0.8854961832061069
    },
-   "ece_worst": 0.15707442163633495,
+   "ece_worst": 0.2273335669559377,
    "step": 2800
   },
   {
-   "ece": 0.04703288564468895,
-   "accuracy": 0.8696461824953445,
-   "kl": 0.4207046497616137,
+   "ece": 0.024922214419310168,
+   "accuracy": 0.884832681442851,
+   "kl": 0.3008875969609121,
    "ece_by_type": {
-    "noul": 0.032132805364076526,
-    "choice": 0.09693201181819894,
-    "score": 0.0036120159758461575,
-    "multi": 0.011076231800743902,
-    "locate": 0.16634205133352925
+    "noul": 0.022913966640349386,
+    "choice": 0.029496132551990004,
+    "multi": 0.018054471943269187,
+    "locate": 0.23492399628722033
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.39464882943143814,
-    "f1": 0.6131907308377896
+    "n": 82,
+    "exact_set": 0.4024390243902439,
+    "f1": 0.6227544910179641
    },
    "locate": {
-    "n": 473,
-    "top1": 0.653276955602537,
-    "top3": 0.8520084566596194
+    "n": 131,
+    "top1": 0.7404580152671756,
+    "top3": 0.8854961832061069
    },
-   "ece_worst": 0.16634205133352925,
+   "ece_worst": 0.23492399628722033,
    "step": 3200
   },
   {
-   "ece": 0.022190351509515167,
-   "accuracy": 0.8558659217877095,
-   "kl": 0.3164360514153051,
+   "ece": 0.061761819010763164,
+   "accuracy": 0.883528900478053,
+   "kl": 0.3827008531392949,
    "ece_by_type": {
-    "noul": 0.018028466137804688,
-    "choice": 0.03951373479248567,
-    "score": 0.005840206808514067,
-    "multi": 0.04691279854434209,
-    "locate": 0.22352832190330135
+    "noul": 0.02961808776342737,
+    "choice": 0.07651488388725412,
+    "multi": 0.010838103700608694,
+    "locate": 0.1739095914682359
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.42474916387959866,
-    "f1": 0.6719492868462758
+    "n": 82,
+    "exact_set": 0.4268292682926829,
+    "f1": 0.6477272727272727
    },
    "locate": {
-    "n": 473,
-    "top1": 0.6765327695560254,
-    "top3": 0.8625792811839323
+    "n": 131,
+    "top1": 0.732824427480916,
+    "top3": 0.8702290076335878
    },
-   "ece_worst": 0.22352832190330135,
+   "ece_worst": 0.1739095914682359,
    "step": 3600
   },
   {
-   "ece": 0.069219513423394,
-   "accuracy": 0.8659217877094972,
-   "kl": 0.4470552618293776,
+   "ece": 0.05192355551910321,
+   "accuracy": 0.8870056497175142,
+   "kl": 0.3260574637551758,
    "ece_by_type": {
-    "noul": 0.040290713495062316,
-    "choice": 0.1168103584046226,
-    "score": 0.006301072902149659,
-    "multi": 0.010713162241653367,
-    "locate": 0.244927298246113
+    "noul": 0.024573245484341935,
+    "choice": 0.06365283860063581,
+    "multi": 0.01987091985068224,
+    "locate": 0.21463545377700383
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.3879598662207358,
-    "f1": 0.6637554585152838
+    "n": 82,
+    "exact_set": 0.43902439024390244,
+    "f1": 0.6629213483146067
    },
    "locate": {
-    "n": 473,
-    "top1": 0.6849894291754757,
-    "top3": 0.86892177589852
+    "n": 131,
+    "top1": 0.7709923664122137,
+    "top3": 0.9007633587786259
    },
-   "ece_worst": 0.244927298246113,
+   "ece_worst": 0.21463545377700383,
    "step": 4000
   },
   {
-   "ece": 0.05948257677159943,
-   "accuracy": 0.8718808193668529,
-   "kl": 0.3561913303995695,
+   "ece": 0.048803524847188175,
+   "accuracy": 0.8826597131681877,
+   "kl": 0.3162171217149705,
    "ece_by_type": {
-    "noul": 0.045462051750153526,
-    "choice": 0.08355872508429696,
-    "score": 0.002550995018747075,
-    "multi": 0.004772070312946493,
-    "locate": 0.2339455685548752
+    "noul": 0.0249218225479126,
+    "choice": 0.05664422360958177,
+    "multi": 0.017802322303086097,
+    "locate": 0.21297926814046525
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.4214046822742475,
-    "f1": 0.6772655007949125
+    "n": 82,
+    "exact_set": 0.4024390243902439,
+    "f1": 0.6390532544378699
    },
    "locate": {
-    "n": 473,
-    "top1": 0.6997885835095138,
-    "top3": 0.8562367864693446
+    "n": 131,
+    "top1": 0.7480916030534351,
+    "top3": 0.8778625954198473
    },
-   "ece_worst": 0.2339455685548752,
+   "ece_worst": 0.21297926814046525,
    "step": 4400
   },
   {
-   "ece": 0.06958188118215376,
-   "accuracy": 0.8793296089385475,
-   "kl": 0.47218957146293733,
+   "ece": 0.056028510321227964,
+   "accuracy": 0.8904823989569752,
+   "kl": 0.35131262505583605,
    "ece_by_type": {
-    "noul": 0.05673471245654793,
-    "choice": 0.09375713293444474,
-    "score": 4.632605446697191e-06,
-    "multi": 0.006023360609353883,
-    "locate": 0.19416455115658285
+    "noul": 0.024519424284658142,
+    "choice": 0.0699597617802035,
+    "multi": 0.009826288135015314,
+    "locate": 0.18423080808333767
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.4280936454849498,
-    "f1": 0.6900763358778625
+    "n": 82,
+    "exact_set": 0.45121951219512196,
+    "f1": 0.6666666666666666
    },
    "locate": {
-    "n": 473,
-    "top1": 0.6871035940803383,
-    "top3": 0.864693446088795
+    "n": 131,
+    "top1": 0.7404580152671756,
+    "top3": 0.8854961832061069
    },
-   "ece_worst": 0.19416455115658285,
+   "ece_worst": 0.18423080808333767,
    "step": 4800
   },
   {
-   "ece": 0.027856978851767654,
-   "accuracy": 0.8804469273743016,
-   "kl": 0.27598698180347553,
+   "ece": 0.04416336725971491,
+   "accuracy": 0.8952629291612343,
+   "kl": 0.31947377288769796,
    "ece_by_type": {
-    "noul": 0.021502879654714306,
-    "choice": 0.054533660865614925,
-    "score": 0.0011856890387005566,
-    "multi": 0.0210933645078741,
-    "locate": 0.18430742945907985
+    "noul": 0.02635544807680188,
+    "choice": 0.05534155872792722,
+    "multi": 0.012024839462320639,
+    "locate": 0.1877896335857515
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.4013377926421405,
-    "f1": 0.6754966887417219
+    "n": 82,
+    "exact_set": 0.45121951219512196,
+    "f1": 0.6923076923076923
    },
    "locate": {
-    "n": 473,
-    "top1": 0.7040169133192389,
-    "top3": 0.864693446088795
+    "n": 131,
+    "top1": 0.732824427480916,
+    "top3": 0.8854961832061069
    },
-   "ece_worst": 0.18430742945907985,
+   "ece_worst": 0.1877896335857515,
    "step": 5200
   },
   {
-   "ece": 0.05113539095253472,
-   "accuracy": 0.8770949720670391,
-   "kl": 0.31856371701035285,
+   "ece": 0.05495522255122487,
+   "accuracy": 0.8948283355063016,
+   "kl": 0.3489468937737143,
    "ece_by_type": {
-    "noul": 0.035099779912667654,
-    "choice": 0.08265257217756519,
-    "score": 0.00013569825225401289,
-    "multi": 0.02330089433369494,
-    "locate": 0.21228816207443177
+    "noul": 0.03301019309669409,
+    "choice": 0.06774300191745326,
+    "multi": 0.0088773566603521,
+    "locate": 0.1752706635885566
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.43478260869565216,
-    "f1": 0.6953846153846154
+    "n": 82,
+    "exact_set": 0.4634146341463415,
+    "f1": 0.7106598984771574
    },
    "locate": {
-    "n": 473,
-    "top1": 0.7019027484143763,
-    "top3": 0.8604651162790697
+    "n": 131,
+    "top1": 0.7251908396946565,
+    "top3": 0.8854961832061069
    },
-   "ece_worst": 0.21228816207443177,
+   "ece_worst": 0.1752706635885566,
    "step": 5600
   },
   {
-   "ece": 0.05545587268843555,
-   "accuracy": 0.8856610800744878,
-   "kl": 0.3611638357698157,
-   "ece_by_type": {
-    "noul": 0.04219939842704649,
-    "choice": 0.08952805843863466,
-    "score": 2.1930535634373527e-05,
-    "multi": 0.005115650817968373,
-    "locate": 0.16475388922789636
-   },
-   "multi": {
-    "n": 299,
-    "exact_set": 0.43812709030100333,
-    "f1": 0.6994047619047619
-   },
-   "locate": {
-    "n": 473,
-    "top1": 0.7167019027484144,
-    "top3": 0.8752642706131079
-   },
-   "ece_worst": 0.16475388922789636,
-   "step": 6000
-  },
-  {
-   "ece": 0.03992676394167781,
-   "accuracy": 0.8797020484171322,
-   "kl": 0.27221902697534306,
-   "ece_by_type": {
-    "noul": 0.02855869882790612,
-    "choice": 0.062238834350688,
-    "score": 0.00024786525302467055,
-    "multi": 0.01387046155792535,
-    "locate": 0.18695985988948105
-   },
-   "multi": {
-    "n": 299,
-    "exact_set": 0.4414715719063545,
-    "f1": 0.6963434022257552
-   },
-   "locate": {
-    "n": 473,
-    "top1": 0.7103594080338267,
-    "top3": 0.8604651162790697
-   },
-   "ece_worst": 0.18695985988948105,
-   "step": 6400
-  },
-  {
-   "ece": 0.0632922098956295,
-   "accuracy": 0.8774674115456238,
-   "kl": 0.31662386085554023,
-   "ece_by_type": {
-    "noul": 0.054821201488953225,
-    "choice": 0.08167373351599455,
-    "score": 7.418096065525681e-05,
-    "multi": 0.004320802969974134,
-    "locate": 0.14180077557268908
-   },
-   "multi": {
-    "n": 299,
-    "exact_set": 0.43478260869565216,
-    "f1": 0.6994047619047619
-   },
-   "locate": {
-    "n": 473,
-    "top1": 0.7061310782241015,
-    "top3": 0.8668076109936576
-   },
-   "ece_worst": 0.14180077557268908,
-   "step": 6800
-  },
-  {
-   "ece": 0.04556644258108444,
-   "accuracy": 0.8901303538175046,
-   "kl": 0.291600836968954,
-   "ece_by_type": {
-    "noul": 0.03058912555376689,
-    "choice": 0.07358632229973752,
-    "score": 1.3902783393859863e-05,
-    "multi": 0.010994426898891654,
-    "locate": 0.17843958861480053
-   },
-   "multi": {
-    "n": 299,
-    "exact_set": 0.45484949832775917,
-    "f1": 0.707936507936508
-   },
-   "locate": {
-    "n": 473,
-    "top1": 0.7251585623678647,
-    "top3": 0.8816067653276956
-   },
-   "ece_worst": 0.17843958861480053,
-   "step": 7200
-  },
-  {
-   "ece": 0.05064941646333513,
-   "accuracy": 0.8927374301675978,
-   "kl": 0.30319393561262215,
-   "ece_by_type": {
-    "noul": 0.03500594524450086,
-    "choice": 0.08044114248988067,
-    "score": 9.04566711845689e-06,
-    "multi": 0.009510180027899825,
-    "locate": 0.1831915795204504
-   },
-   "multi": {
-    "n": 299,
-    "exact_set": 0.41471571906354515,
-    "f1": 0.6889226100151745
-   },
-   "locate": {
-    "n": 473,
-    "top1": 0.7293868921775899,
-    "top3": 0.8942917547568711
-   },
-   "ece_worst": 0.1831915795204504,
-   "step": 7600
-  },
-  {
-   "ece": 0.045957804880550686,
-   "accuracy": 0.8931098696461824,
-   "kl": 0.2877154694214733,
+   "ece": 0.05066511603553936,
+   "accuracy": 0.8978704910908301,
+   "kl": 0.339644631326081,
    "ece_by_type": {
-    "noul": 0.03478641362153288,
-    "choice": 0.07468352052900527,
-    "score": 1.958012580871582e-05,
-    "multi": 0.008941575302629724,
-    "locate": 0.19766248705901257
+    "noul": 0.029168567606197876,
+    "choice": 0.06227849002783697,
+    "multi": 0.011348272917833769,
+    "locate": 0.1809499859241129
    },
    "multi": {
-    "n": 299,
-    "exact_set": 0.4414715719063545,
-    "f1": 0.7003154574132492
+    "n": 82,
+    "exact_set": 0.47560975609756095,
+    "f1": 0.711340206185567
    },
    "locate": {
-    "n": 473,
-    "top1": 0.7399577167019028,
-    "top3": 0.8879492600422833
+    "n": 131,
+    "top1": 0.732824427480916,
+    "top3": 0.8854961832061069
    },
-   "ece_worst": 0.19766248705901257,
-   "step": 7830
+   "ece_worst": 0.1809499859241129,
+   "step": 5783
   }
  ],
  "args": {
   "model": "Qwen/Qwen3-1.7B-Base",
-  "data": "data/v2_mix_train.jsonl",
-  "out": "runs/circuit-1.7b-v2",
+  "data": "data/v22_mix_train.jsonl",
+  "out": "runs/circuit-1.7b-v2.2",
   "epochs": 1,
   "batch": 4,
-  "lr": 0.0001,
-  "head_lr": 0.001,
+  "lr": 5e-05,
+  "head_lr": 0.0005,
   "rank": 16,
   "max_length": 4096,
   "val_frac": 0.1,
@@ -551,11 +416,14 @@
   "micro": 2,
   "grad_checkpoint": true,
   "wandb": "s1proto",
-  "run_name": "circuit-1.7b-v2",
+  "run_name": "circuit-1.7b-v2.2",
   "resume": false,
+  "init": "runs/circuit-1.7b-v2.0",
+  "freeze_adapter": false,
   "head": "pointer",
   "load_4bit": false,
   "modality": "text",
+  "masked": false,
   "exclude_family": null
  }
 }

head.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7604903ab87c2b8c9ad96fafd721afb12bc15b67354bbb35ae857e4c32a5a74b
+oid sha256:0200c6dd48b4565dec1f1d9ad3d927a8675d0a5bb6270306510bbcfe84a7c966
 size 8391160