Symbiote critical-span scorer v2 β€” recorded 21 October 2026 model

Five DistilBERT span-scoring checkpoints from the completed ANLP course study. This is a custom PyTorch span classifier, not a chat model, token-classification AutoModel, or standalone prompt compressor. Use the supplied load_scorer.py. No new training was performed to prepare this release.

Model and data

Each checkpoint includes the entire DistilBERT encoder plus a mean-pooled-span linear head. The sigmoid predicts the fraction of eligible labelers whose answer failed after masking a span, conditional on answering the original correctly. This is not a certificate of semantic necessity or compressed-input sufficiency.

Training used recovered saved responses: 38,603 unique model/span votes, 37,979 eligible; 13,966 consensus positions over 912 questions/778 families. The supplied team export was already consensus-filtered, so full raw worker coverage is not established. Five family-disjoint folds, soft targets, unweighted BCE, four epochs, batch 8, LR 3e-5, max length 512, seed 42.

On 13,855 shared held-out positions: ROC AUC .907385, average precision .713346, critical F1 .634404, soft-target Brier .060593. The feature ridge AUC was .821711. 111 random-control spans lacked full tokenizer coverage and were excluded from the shared comparison. See scorer_report.json and evidence/feature_report.json.

Five folds are part of the result

These are cross-validation models, not five seeds of a full-data deployment model. Reproduce a recorded question using its held-out assignment in split.json. Do not choose the best-looking checkpoint on the test set or substitute an ensemble for the reported held-out-fold predictions. For a genuinely new input, a declared fold or ensemble is a deployment choice whose performance was not measured here.

Contents and provenance

  • fold0/ through fold4/: scorer.pt, encoder config, category prior and selection metadata.
  • tokenizer/: the saved local fast tokenizer.
  • split.json, training_metadata.json, scorer_report.json, scorer_oof_predictions.csv.
  • load_scorer.py: the original architecture/span-mask/scoring definitions plus local loading helpers.
  • source/: exact inference-source snapshot, including the original scorer implementation.
  • evidence/: small verified comparison reports and scientific-audit metadata.
  • release_manifest.json: every payload file's size/SHA256, training/inference fingerprints.

Model source directory: models/span_scorer_v2_20261021 in the project backup. Project code, data scope, run history, baseline adaptations, API settings and full reproduction guide: https://github.com/dhyeythummar09/ANLP_Project_PromptCompression The inference snapshot includes later baseline/provider corrections; original training metadata retains its distinct training-time fingerprint. Both are recorded rather than presenting a Git commit with uncommitted changes as exact source.

Local usage

Download this repository at a fixed full commit revision. From that downloaded folder, install requirements-inference.txt in a suitable PyTorch environment. The measured GPU environment used Python 3.13.15, Torch 2.11.0+cu128 and Transformers 5.16.1; choose a Torch build appropriate for your actual CPU/CUDA. No base encoder/tokenizer download is needed when loading a local fold.

python load_scorer.py

This verifies hashes without loading weights. Then, from the downloaded directory:

import torch
from load_scorer import load_fold, score_spans, fold_for_uid

# Arbitrary fold 0 for a NEW-input demonstration, not the reported OOF evaluation.
model, tokenizer = load_fold(0, device="cpu")
text = "Mira has 3 apples."
spans = [{"span_start": 9, "span_end": 10, "span_text": "3", "span_category": "number"}]
scored = score_spans(text, spans, model, tokenizer, torch.device("cpu"))
print(scored[0]["criticality_score"])

# For a study question instead: load_fold(fold_for_uid("source_dataset|example_id"))

Offsets are Python character positions, end-exclusive. Call verify_bundle() before use. Scoring follows the original conservative fallback: spans without complete token coverage retain score 1.0; do not interpret those as neural predictions.

Compression outcome and limitations

Good span-label prediction did not establish a neural downstream compression gain. Grouped hybrid versus budgeted LLMLingua-2 on the full 160-question companion:

Reader Retained-token ceiling Hybrid accuracy LLMLingua-2 accuracy
GPT-OSS-120B .50 36.875% 60.000%
GPT-OSS-120B .75 73.125% 90.000%
Qwen-3.8-27B-FP8 .50 46.250% 69.375%
Qwen-3.8-27B-FP8 .75 73.125% 95.625%

The full-cohort companion counts provider/parse failures as incorrect and was added after failure review. Original common-success tables are retained in the project: Qwen .50 uses 158 questions there. Qwen retains two empty outputs and its all-responses-successful technical gate remains false. Hybrid/prior ablation intervals include zero. No SOTA, causal sufficiency, or unseen-reader-family transfer is claimed. Masking one span differs from joint deletion. Target LLMs, LLMLingua-2, DistilGPT2 and spaCy are separate components, not included weights.

The optional seven-day follow-up is not part of these weights/results. Preserve this dated release when publishing any later model. Team Symbiote, IIIT Hyderabad.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Pranav-Sriv/reasoning-critical-span-scorer-v2

Finetuned
(12,911)
this model