Spaces:

Rushhaabhhh
/

HONEST-RL-Calibrator

Running

App Files Files Community

HONEST-RL-Calibrator / docs /RUNBOOK.md

Rushhaabhhh

HONEST-RL-Calibrator-v0

3040767 verified 6 months ago

preview code

Raw History Blame Contribute Delete

25.6 kB

HONEST · Operational Runbook

End-to-end pipeline: data → RL training → OOD eval → comparison → success metrics → MCP deployment → self-learning verification.

Every step is a single command. Every command produces a JSON or markdown artifact that the next step consumes. There are no implicit dependencies between steps; each one fails fast if its inputs are missing.

┌───────────────┐  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐
│ 1 · Ingestion │─▶│ 2 · Baseline   │─▶│ 3 · GRPO train │─▶│ 4 · Full eval  │
│   (data/)     │  │   (anchor)     │  │   (LoRA out)   │  │  (ID + OOD)    │
└───────────────┘  └────────────────┘  └────────────────┘  └────────────────┘
                                                                    │
┌───────────────┐  ┌────────────────┐  ┌────────────────┐           │
│ 7 · Self-     │◀─│ 6 · MCP serve  │◀─│ 5 · Compare    │◀──────────┘
│   learning    │  │   (deploy)     │  │   (Δ + CI)     │
└───────────────┘  └────────────────┘  └────────────────┘

0 · Prerequisites

# From the project root
python3 -m venv venv
venv/bin/pip install -r requirements.txt

# Optional: log in to W&B and HuggingFace
venv/bin/wandb login
venv/bin/huggingface-cli login

# Activate (we'll prefix commands with ./venv/bin/python below)
source venv/bin/activate          # POSIX shells

Verify the environment is healthy before running anything expensive:

make test                # 438 unit + integration tests should pass
make smoke-train         # train_grpo --dry-run --hindsight --replay-priority --self-mutate --self-play
make mcp-smoke           # offline MCP self-test

1 · Data ingestion

The unified sampler reads data/processed/{math,code_mbpp,code_apps,logic}.jsonl. If any of those files is missing, train_grpo.py fails fast at startup with a clear error.

# Math (Hendrycks MATH, 7 subjects, 5 difficulty levels)
PYTHONPATH=. python data/ingestion/ingest_hendrycks_math.py
# → data/processed/math.jsonl  (~12.5k problems)

# Code · MBPP (sandboxed verifier)
PYTHONPATH=. python data/ingestion/ingest_mbpp.py
# → data/processed/code_mbpp.jsonl  (~427 problems)

# Code · APPS (streamed JSONL shards from HuggingFace)
PYTHONPATH=. python -m data.ingestion.ingest_apps
# → data/processed/code_apps.jsonl

# Logic · ZebraLogic-style CSP puzzles (regenerated, unique-solution)
PYTHONPATH=. python data/ingestion/regenerate_zebralogic.py
# → data/processed/logic.jsonl  (~75 problems)

Every ingestion script prints a JSON summary on the last line — capture it if you want to assert dataset shape in CI.

Sanity check

./venv/bin/python -c "
from data.sampler.unified_sampler import get_sampler
s = get_sampler()
print('total:', s.total_count())
print('buckets (domain, difficulty):')
for k, v in sorted(s.bucket_counts().items()):
    print(' ', k, '=', v)
"

You should see ~13k problems spread across (math, 1..5), (code, 3..5), (logic, 1..5). A bucket with 0 count will not error — the controller will just never dispense that condition.

OOD data (5-slice transfer suite)

OOD data is fetched from public HuggingFace datasets and is never seen during training. The five-slice suite spans the difficulty range that small and medium models can engage with:

slice	source (HF dataset)	floor	tiny-model headroom
`commonsense`	`tau/commonsense_qa` (validation)	0.20	~30-45 %
`science_easy`	`allenai/ai2_arc` ARC-Easy (test)	0.25	~45-60 %
`science_hard`	`cais/mmlu` astronomy (test)	0.25	~25-35 %
`medical`	`cais/mmlu` professional_medicine (val)	0.25	floor (medium+ only)
`legal`	AGIEval LSAT-LR (with MMLU law fallback)	0.20	floor (medium+ only)

Why five slices instead of two? The transferability claim ("RL-trained calibration generalises to OOD") is empirically unprovable on slices where the model sits at the random-MCQ floor: ECE/Brier deltas collapse into bootstrap noise. Tiny models (Qwen-0.5B, Llama-1B) only have measurable headroom on commonsense/science_easy/science_hard, so the tier-aware fetcher and full_eval.py --ood-slices auto skip the hard slices for those models.

# Tier-aware fetch (recommended — picks slices appropriate to your tier)
PYTHONPATH=. python eval/ood/fetch_ood_data.py --tier tiny     # 3 slices: commonsense + science_easy + science_hard
PYTHONPATH=. python eval/ood/fetch_ood_data.py --tier small    # 4 slices: tiny + medical
PYTHONPATH=. python eval/ood/fetch_ood_data.py --tier medium   # 5 slices: full suite
PYTHONPATH=. python eval/ood/fetch_ood_data.py --tier all      # alias for medium

# Or pick exactly what you want:
PYTHONPATH=. python eval/ood/fetch_ood_data.py --slices commonsense,science_easy --n 200

# Default (no flags) = full medium suite, backward-compatible with prior reports.
PYTHONPATH=. python eval/ood/fetch_ood_data.py --n 200

Outputs (one JSONL per slice in eval/ood/, auto-discovered by full_eval.py): commonsense_qa_sample.jsonl, arc_easy_sample.jsonl, mmlu_astronomy_sample.jsonl, medqa_sample.jsonl, lsat_sample.jsonl.

2 · Baseline characterization

Before training, anchor the pre-RL behaviour of the same model on the same evaluation conditions. Without this anchor, the post-RL numbers are uninterpretable.

# Default: 100 samples × 3 domains × 5 difficulties (~1500 generations)
./venv/bin/python eval/baseline_eval.py \
    --model         Qwen/Qwen2.5-3B-Instruct \
    --model-preset  qwen3b \
    --output        eval/baseline_results.json

# Headline run (200 samples per condition, ~3000 generations)
./venv/bin/python eval/baseline_eval.py \
    --model         Qwen/Qwen2.5-3B-Instruct \
    --model-preset  qwen3b \
    --samples       200 \
    --output        eval/baseline_results.json

Outputs (eval/baseline_results.json):

{
  "model_id":        "Qwen/Qwen2.5-3B-Instruct",
  "preset":          "qwen3b",
  "n_samples":       100,
  "metrics": {
    "ece": 0.18,    "ace": 0.21,    "mce": 0.42,
    "brier": 0.27,  "nll": 0.71,
    "auroc": 0.62,  "auprc": 0.55,
    "format_rate": 0.82,    "abstain_rate": 0.04
  },
  "per_domain":  { "math": {...}, "code": {...}, "logic": {...} },
  "per_difficulty": { "1": {...}, ..., "5": {...} }
}

If format_rate < 0.70, the strict XML parser is rejecting too many completions. Run the optional Stage-2 format SFT first:

./venv/bin/python training/format_sft.py \
    --model-id     Qwen/Qwen2.5-3B-Instruct \
    --output-dir   ./honest-qwen-3b-sft

Then resume from the SFT adapter (--resume-from ./honest-qwen-3b-sft) in Step 3.

3 · GRPO calibration training

Single command. Defaults come from calibration_profiles.py; CLI flags override per-run.

3a · Vanilla GRPO (recommended starting point)

./venv/bin/python training/train_grpo.py \
    --model-preset    qwen3b \
    --colab-profile   l4 \
    --max-steps       350 \
    --output-dir      ./honest-qwen-3b-grpo

3a-bis · Tiny models (Qwen-0.5B, Llama-1B) — SFT first, GRPO second

Tiny models cannot reliably emit the 3-tag XML format from the system prompt alone. Without a Calibration SFT warmup, ~97 % of GRPO rollouts hit the malformed-penalty floor and the GRPO advantage signal stays at zero. The provided one-command recipe handles both phases:

# Qwen-0.5B (Colab T4 friendly, ~10 min SFT + ~50 min GRPO on T4)
./bin/run_calibration_pipeline.sh Qwen/Qwen2.5-0.5B-Instruct

# Llama-1B
./bin/run_calibration_pipeline.sh meta-llama/Llama-3.2-1B-Instruct

# Override anything by appending GRPO flags after the model id:
./bin/run_calibration_pipeline.sh Qwen/Qwen2.5-0.5B-Instruct \
    --max-steps 200 --replay-priority --self-mutate

The script picks tier-aware defaults from calibration_profiles.py: 1500 SFT examples × 2 epochs at max_difficulty=2 with hindsight tag on half the examples, followed by GRPO with --init-adapter pointing at the SFT output and --hindsight-mode legacy (the tractable head for tiny models).

For deeper detail and what to expect from the metrics see docs/SELF_LEARNING.md §2.6.

To reproduce the SFT phase manually (e.g. to swap in a different mix):

./venv/bin/python training/calibration_sft.py \
    --model-id Qwen/Qwen2.5-0.5B-Instruct \
    --output-dir ./sft-qwen-0.5b \
    --n-examples 1500 --epochs 2 --hindsight-frac 0.5

./venv/bin/python training/train_grpo.py \
    --model-id Qwen/Qwen2.5-0.5B-Instruct \
    --init-adapter ./sft-qwen-0.5b \
    --hindsight --hindsight-mode legacy \
    --max-steps 250 --output-dir ./honest-qwen-0-5b-grpo

3b · GRPO + self-learning (recommended for the headline result)

./venv/bin/python training/train_grpo.py \
    --model-preset    qwen3b \
    --colab-profile   l4 \
    --max-steps       350 \
    --hindsight \
    --self-mutate \
    --replay-priority \
    --output-dir      ./honest-qwen-3b-grpo

--self-play is supported but disabled by default — turn it on only after the first three pillars show a clean Δ ECE.

3c · What you get

honest-qwen-3b-grpo/
├── final_adapters/             ← LoRA adapter (load with --adapter-path)
├── difficulty_state.json       ← rolling controller state per domain
├── smc_state.json              ← (if --self-mutate) ceiling per domain
├── replay_state.json           ← (if --replay-priority) buffer snapshot
├── checkpoint-50/              ← intermediate checkpoints
├── checkpoint-100/
└── trainer_state.json

3d · Live monitoring

W&B (or stdout if --no-wandb) reports the metrics that matter for a healthy GRPO run:

Metric	Healthy range
`train/reward`	rising, plateauing > 0
`train/reward_std`	> 1e-4 (else dead-batch guard fires)
`train/kl`	< 0.05 (else AdaptiveBetaCallback)
`train/grad_norm`	< `max_grad_norm = 1.0`
`controller/<domain>/target`	tracking accuracy band (0.30–0.70)
`smc/<domain>/max_unlocked`	promotes only when ready (≥ d=6+)
`replay/buffer_size`	rises to ~ buffer_size after warmup
`replay/priority_entropy`	> 1.0 (else replay disables itself)

3d.1 · Audit whether hindsight is firing

If you are using --hindsight, verify the head is actually contributing signal — the legacy v1 head can be silently zero on every step when reasoning_mode does not teach the <hindsight> tag. After the run:

./venv/bin/python bin/audit_hindsight.py \
    --trainer-state ./honest-qwen-1-5b-grpo/trainer_state.json

The script reports:

The fraction of steps where the hindsight reward channel was exactly 0 (= the model never emitted a parseable hindsight tag in that step's group).
The non-zero magnitude (compared against Brier as a sanity ratio).
A verdict: "structurally silent" / "intermittent" / "healthy".

If the verdict is "structurally silent", relaunch with --hindsight-mode refined (which auto-promotes --reasoning-mode refined so the prompt teaches the new <critique> and <refined_confidence> tags). See docs/SELF_LEARNING.md §2.5 for the design rationale.

./venv/bin/python training/train_grpo.py \
    --model-preset qwen1.5b --colab-profile a100 \
    --hindsight --hindsight-mode refined \
    --max-steps 250 \
    --output-dir ./honest-qwen-1-5b-grpo-refined

3e · Render committed training plots

After training, regenerate the committed evidence PNGs from trainer_state.json:

make plots TRAINER_STATE=./honest-qwen-3b-grpo/trainer_state.json
# or directly:
./venv/bin/python bin/plot_training_curves.py \
    --trainer-state ./honest-qwen-3b-grpo/trainer_state.json \
    --out docs/training \
    --label "qwen3b · 350 steps · L4"

This overwrites docs/training/loss_curve.png, reward_curve.png, and kl_curve.png — the same images embedded in the project README. Commit them so the submission carries real training evidence:

git add docs/training/*.png
git commit -m "docs: refresh training curves from completed run"

3f · Multi-model recipe

For Llama-3B and Phi-4-mini (~3.8B), use L4 instead of A100:

# Llama-3.2-3B
./venv/bin/python training/train_grpo.py \
    --model-preset llama3b --colab-profile l4 \
    --max-steps 400 --hindsight --self-mutate \
    --output-dir ./honest-llama-3b-grpo

# Phi-4-mini-instruct
./venv/bin/python training/train_grpo.py \
    --model-preset phi4mini --colab-profile l4 \
    --max-steps 400 --hindsight --self-mutate \
    --output-dir ./honest-phi4mini-grpo

4 · Full evaluation (in-distribution + OOD)

Same metrics battery as Step 2, run on the trained adapter, with a tier-aware OOD pass that auto-discovers JSONL slices written by fetch_ood_data.py.

# Headline run (medium tier, all 5 OOD slices)
./venv/bin/python eval/full_eval.py \
    --model-id          Qwen/Qwen2.5-3B-Instruct \
    --adapter-path      ./honest-qwen-3b-grpo/final_adapters \
    --baseline-results  eval/baseline_results.json \
    --ood-dir           eval/ood \
    --ood-slices        auto \
    --samples           100 \
    --output            eval/full_results.json

# Tiny-model run (Qwen-0.5B / Llama-1B): tier-auto picks the 3
# small-model-friendly slices (commonsense, science_easy, science_hard)
# and skips medical+legal.
./venv/bin/python eval/full_eval.py \
    --model-id        Qwen/Qwen2.5-0.5B-Instruct \
    --adapter-path    ./grpo-qwen-0.5b/final_adapters \
    --ood-slices      auto \
    --output          eval/full_results_qwen05b.json

Important: to make the calibration-transfer claim provable in Step 5, also run full_eval.py without --adapter-path against the same --ood-slices and save it as eval/baseline_full.json. The compare_runs.py transfer report needs OOD samples on both sides to show before/after deltas. Without the no-adapter run, the transfer table will only have post-RL OOD numbers.

Output (eval/full_results.json):

{
  "model_id":     "Qwen/Qwen2.5-3B-Instruct",
  "adapter_path": "./honest-qwen-3b-grpo/final_adapters",
  "preset":       "qwen3b",
  "ood_slices":   ["commonsense", "science_easy", "science_hard", "medical", "legal"],
  "in_distribution": { "math_1": {...}, ..., "logic_5": {...} },
  "ood": {
    "commonsense":  { "ece": ..., "brier": ..., "random_floor": 0.20, "samples": [...] },
    "science_easy": { "ece": ..., "brier": ..., "random_floor": 0.25, "samples": [...] },
    "science_hard": { "ece": ..., "brier": ..., "random_floor": 0.25, "samples": [...] },
    "medical":      { "ece": ..., "brier": ..., "random_floor": 0.25, "samples": [...] },
    "legal":        { "ece": ..., "brier": ..., "random_floor": 0.20, "samples": [...] }
  },
  "overall": { "ece": ..., "brier": ..., "auroc": ..., "accuracy": ..., ... }
}

--ood-slices accepts:

auto (default): tier-aware default for --model-id / --model-preset.
all: full registry (5 slices).
a comma-separated subset, e.g. commonsense,science_easy.

To skip ID or OOD individually: --skip-indist / --skip-ood. To debug locally without a GPU: --dry-run.

5 · Comparison & success metrics

# Recommended: pass two full_eval JSONs (one before-RL, one after-RL).
# Both need OOD samples for the calibration-transfer table.
./venv/bin/python eval/compare_runs.py \
    --baseline    eval/baseline_full.json \
    --after       eval/full_results.json \
    --output      eval/comparison.md \
    --plot --plot-output eval/plots/comparison.png

# Backward-compat: a baseline_eval.py JSON (in-dist only) also works,
# but the transfer table will be empty for the baseline column.
./venv/bin/python eval/compare_runs.py \
    --baseline    eval/baseline_results.json \
    --after       eval/full_results.json \
    --output      eval/comparison.md

comparison.md is the deliverable artefact — paste it directly into a report or pitch deck. It now contains six sections:

Headline table (ECE, Brier, AUROC) before vs after, with Δ and a 95% bootstrap CI on Δ Brier.
Per-domain breakdown (math / code / logic).
In-distribution vs OOD (after training) — generalization gap row.
Calibration Transfer (HEADLINE CLAIM) — per-slice ΔECE table with 95% paired-bootstrap CIs, status flags (✓ transferred / ~ partial / ⚠ at floor / ✗ no transfer), plus a single transfer-ratio number summarising how much of the in-distribution calibration gain carried to OOD.
Confidence histogram (text bars, before vs after).
Operating-mode shifts (format/abstain/malformed rate deltas).

Plus the reliability diagram PNG when you pass --plot.

Success criteria

A pillar/run "ships" only if all the following are met. These are the pass/fail gates for a publishable headline.

Gate	Threshold
Δ ECE (in-distribution)	≤ -0.03 (lower is better)
Δ Brier (in-distribution)	≤ -0.02, with 95% CI excluding 0
Calibration transfer ratio	≥ 0.5× on slices clear of floor
Δ ECE on ≥2 OOD slices	< 0 with 95% CI upper bound < 0 (✓ transferred)
AUROC (in-distribution)	≥ 0.65 (no discrimination collapse)
Format rate	≥ 0.90 (parsing did not regress)
Abstain rate at d=5	> abstain rate at d=1

For tiny models, the OOD transfer gate applies to the commonsense / science_easy / science_hard slices — medical and legal are at the random-MCQ floor at this scale and the transfer ratio averages exclude them automatically (status ⚠ at floor).

If a gate fails on the headline run but passes per-domain (e.g. math improves but code regresses), report per-domain and investigate the losing slice — usually a verifier sharpness or a curriculum balance issue, not a fundamental training failure.

6 · MCP deployment

The trained adapter is a self-contained artifact: it can run in any process that can load the base model + adapter. The MCP server is a thin, stateless wrapper that exposes that artifact to MCP clients.

6a · Pre-flight (offline, no GPU)

make mcp-smoke         # offline self-test (no model load)
make mcp-health        # config preflight: are model + adapter + calibration present?

make mcp-health should print:

[ok] model_id                Qwen/Qwen2.5-3B-Instruct
[ok] adapter_path            ./honest-qwen-3b-grpo/final_adapters
[ok] calibration_info        eval/full_results.json

6b · Generate a Claude Desktop config

HONEST_MODEL_ID=Qwen/Qwen2.5-3B-Instruct \
HONEST_ADAPTER_PATH=$PWD/honest-qwen-3b-grpo/final_adapters \
HONEST_CALIBRATION_INFO=$PWD/eval/full_results.json \
make mcp-config

Paste the printed JSON snippet into Claude Desktop's ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows).

For Cursor and LangGraph, see mcp_server/README.md.

6c · Launch (manual)

./venv/bin/python -m mcp_server \
    --model-id          Qwen/Qwen2.5-3B-Instruct \
    --adapter-path      ./honest-qwen-3b-grpo/final_adapters \
    --calibration-info  eval/full_results.json

The server speaks MCP over stdio (the standard transport). Once connected, the client can call:

// ask_with_calibrated_confidence
{
  "answer":           "42",
  "confidence":       0.83,
  "calibration_note": "Confidence is empirically calibrated: ECE = 0.04, Brier = 0.18.",
  "abstained":        false,
  "malformed":        false,
  "raw":              "<reasoning>...</reasoning><answer>42</answer><confidence>0.83</confidence>"
}

// get_calibration_info
{
  "available": true,
  "model":     "Qwen/Qwen2.5-3B-Instruct",
  "preset":    "qwen3b",
  "metrics":   { "ece": 0.04, "brier": 0.18, "auroc": 0.71 },
  "ood":       { "medical": {...}, "legal": {...} }
}

6d · One-shot installer

bin/install-mcp.sh
# - installs serving deps if missing
# - runs smoke + health
# - prints a paste-ready Claude Desktop config

7 · Self-learning verification

If you ran Step 3b (with self-learning flags), confirm each pillar contributed and didn't degrade the run.

7a · Hindsight (HCR)

In eval/full_results.json look for the per-completion hindsight emission rate. The model should emit <hindsight> tags on at least ~30 % of episodes by step 200, and the average |hindsight − correctness| should be lower than |confidence − correctness|.

./venv/bin/python -c "
import json
r = json.load(open('eval/full_results.json'))
print('hindsight emission rate:', r['in_distribution']['metrics'].get('hindsight_rate'))
print('mean |c-y|:           ', r['in_distribution']['metrics'].get('mean_calibration_error'))
print('mean |hindsight-y|:   ', r['in_distribution']['metrics'].get('mean_hindsight_error'))
"

7b · Replay (CPR)

./venv/bin/python -c "
import json
s = json.load(open('honest-qwen-3b-grpo/replay_state.json'))
print('buffer_size:        ', s['size'])
print('priority entropy:   ', s['priority_entropy'])   # > 1.0 → diverse, healthy
print('top-1 priority share:', s['top1_share'])         # < 0.05 → no over-replay
"

7c · Self-mutating curriculum (SMC)

./venv/bin/python -c "
import json
s = json.load(open('honest-qwen-3b-grpo/smc_state.json'))
for d, st in s.items():
    print(f'{d}: max_unlocked = {st[\"max_unlocked_difficulty\"]}, '
          f'episodes_at_max = {st[\"episodes_at_max\"]}')
"

A pillar "shipped" if its dedicated metric moved in the right direction and the headline ECE/Brier didn't regress vs the vanilla run.

8 · Notebook variant

For interactive Colab / Kaggle runs, training/train_colab.ipynb mirrors Step 3 with cell-by-cell narration. The notebook respects all the CLI flags, just edit the args = "--model-preset qwen3b ..." cell at the top.

9 · Reproducibility

Every random sampler is seeded from --seed (default 42) → difficulty controller, sampler shuffle, replay buffer init, generator stub.
Eval is seeded by eval/eval_seeds.json; pin the seeds before any comparison run.
The adapter directory contains both trainer_state.json and the full set of CLI args under training_args.json.
Re-running the full pipeline from a fresh checkout against the same seeds reproduces the headline numbers within 95 % bootstrap CI.

10 · Troubleshooting

Symptom	First thing to check
`Unified sampler is empty` at training startup	Re-run §1 ingestion scripts.
`format_rate` < 0.70 in baseline	Run optional Stage-2 format SFT (`training/format_sft.py`) first.
`train/reward_std` flat near 0	`RewardHealthCallback` already disabled the bad batch — keep going.
`train/kl` blowing up (> 0.2)	`AdaptiveBetaCallback` will raise β. If still bad, lower `--learning-rate`.
OOM at step 0	Drop `--num-generations` by 2 or `--max-completion-length` by 128.
MCP client says "tool not found"	`make mcp-health` and confirm the adapter path is absolute.
Δ ECE positive after training	Curriculum imbalance — pin `--domain-weights 0.5,0.35,0.15` and retry.
Replay buffer sampling the same prompt repeatedly	`priority_entropy < log(2)` → CPR auto-disables for 50 steps.
SMC ceiling oscillating	Increase `--smc-min-episodes-at-max` from 20 to 40.

For server-specific MCP issues see the troubleshooting section in mcp_server/README.md.