Symbiote critical-span scorer v2 β recorded 21 October 2026 model
Five DistilBERT span-scoring checkpoints from the completed ANLP course study.
This is a custom PyTorch span classifier, not a chat model, token-classification
AutoModel, or standalone prompt compressor. Use the supplied load_scorer.py.
No new training was performed to prepare this release.
Model and data
Each checkpoint includes the entire DistilBERT encoder plus a mean-pooled-span linear head. The sigmoid predicts the fraction of eligible labelers whose answer failed after masking a span, conditional on answering the original correctly. This is not a certificate of semantic necessity or compressed-input sufficiency.
Training used recovered saved responses: 38,603 unique model/span votes, 37,979 eligible; 13,966 consensus positions over 912 questions/778 families. The supplied team export was already consensus-filtered, so full raw worker coverage is not established. Five family-disjoint folds, soft targets, unweighted BCE, four epochs, batch 8, LR 3e-5, max length 512, seed 42.
On 13,855 shared held-out positions: ROC AUC .907385, average precision .713346, critical F1 .634404, soft-target Brier .060593. The feature ridge AUC was .821711. 111 random-control spans lacked full tokenizer coverage and were excluded from the shared comparison. See scorer_report.json and evidence/feature_report.json.
Five folds are part of the result
These are cross-validation models, not five seeds of a full-data deployment model. Reproduce a recorded question using its held-out assignment in split.json. Do not choose the best-looking checkpoint on the test set or substitute an ensemble for the reported held-out-fold predictions. For a genuinely new input, a declared fold or ensemble is a deployment choice whose performance was not measured here.
Contents and provenance
- fold0/ through fold4/: scorer.pt, encoder config, category prior and selection metadata.
- tokenizer/: the saved local fast tokenizer.
- split.json, training_metadata.json, scorer_report.json, scorer_oof_predictions.csv.
- load_scorer.py: the original architecture/span-mask/scoring definitions plus local loading helpers.
- source/: exact inference-source snapshot, including the original scorer implementation.
- evidence/: small verified comparison reports and scientific-audit metadata.
- release_manifest.json: every payload file's size/SHA256, training/inference fingerprints.
Model source directory: models/span_scorer_v2_20261021 in the project backup. Project code, data scope, run history, baseline adaptations, API settings and full reproduction guide: https://github.com/dhyeythummar09/ANLP_Project_PromptCompression The inference snapshot includes later baseline/provider corrections; original training metadata retains its distinct training-time fingerprint. Both are recorded rather than presenting a Git commit with uncommitted changes as exact source.
Local usage
Download this repository at a fixed full commit revision. From that downloaded folder, install requirements-inference.txt in a suitable PyTorch environment. The measured GPU environment used Python 3.13.15, Torch 2.11.0+cu128 and Transformers 5.16.1; choose a Torch build appropriate for your actual CPU/CUDA. No base encoder/tokenizer download is needed when loading a local fold.
python load_scorer.py
This verifies hashes without loading weights. Then, from the downloaded directory:
import torch
from load_scorer import load_fold, score_spans, fold_for_uid
# Arbitrary fold 0 for a NEW-input demonstration, not the reported OOF evaluation.
model, tokenizer = load_fold(0, device="cpu")
text = "Mira has 3 apples."
spans = [{"span_start": 9, "span_end": 10, "span_text": "3", "span_category": "number"}]
scored = score_spans(text, spans, model, tokenizer, torch.device("cpu"))
print(scored[0]["criticality_score"])
# For a study question instead: load_fold(fold_for_uid("source_dataset|example_id"))
Offsets are Python character positions, end-exclusive. Call verify_bundle() before use. Scoring follows the original conservative fallback: spans without complete token coverage retain score 1.0; do not interpret those as neural predictions.
Compression outcome and limitations
Good span-label prediction did not establish a neural downstream compression gain. Grouped hybrid versus budgeted LLMLingua-2 on the full 160-question companion:
| Reader | Retained-token ceiling | Hybrid accuracy | LLMLingua-2 accuracy |
|---|---|---|---|
| GPT-OSS-120B | .50 | 36.875% | 60.000% |
| GPT-OSS-120B | .75 | 73.125% | 90.000% |
| Qwen-3.8-27B-FP8 | .50 | 46.250% | 69.375% |
| Qwen-3.8-27B-FP8 | .75 | 73.125% | 95.625% |
The full-cohort companion counts provider/parse failures as incorrect and was added after failure review. Original common-success tables are retained in the project: Qwen .50 uses 158 questions there. Qwen retains two empty outputs and its all-responses-successful technical gate remains false. Hybrid/prior ablation intervals include zero. No SOTA, causal sufficiency, or unseen-reader-family transfer is claimed. Masking one span differs from joint deletion. Target LLMs, LLMLingua-2, DistilGPT2 and spaCy are separate components, not included weights.
The optional seven-day follow-up is not part of these weights/results. Preserve this dated release when publishing any later model. Team Symbiote, IIIT Hyderabad.
Model tree for Pranav-Sriv/reasoning-critical-span-scorer-v2
Base model
distilbert/distilbert-base-uncased