gitmodelmujtaba/sapbert-snomed-loinc-rxnorm

Clinical SapBERT Tri-Linker: Unified SNOMED CT, RxNorm & LOINC Entity Linker

🆕 What's New in v2.1 (Self-Healing Active Learning & Guideline Retraining)

Fine-tuned and exported with PyTorch CUDA acceleration on an NVIDIA A40 GPU (48 GB VRAM).

🎯 Retraining Objectives

• Targeted Collision Resolution: Separate entangled concept representations identified during clinical guideline verification (e.g. Emergency Caesarean Section vs Vaginal Delivery Following Previous Caesarean Section).
• Margin Optimization: Expand the margin between true positive clinical concepts and hard negative confounders.

📊 Dataset & Hyperparameters

• Mined Triplets: 62 contrastive pairs extracted directly from clinical feedback loops and guideline recommendations.
• Key Concepts Covered:
  • Emergency caesarean section (SNOMED 274130007) vs VBAC (237313003)
  • Single live birth from singleton pregnancy (SNOMED 169826009) vs Uncalibrated mention
  • Antepartum haemorrhage vs Postpartum haemorrhage
  • Trial of labour after previous caesarean section (SNOMED 289069001) vs Elective caesarean
• Loss Function: nn.TripletMarginLoss(margin=0.3, p=2) + 0.5 * (1.0 - CosineSimilarity)
• Optimizer: AdamW (lr=2e-5, weight decay 0.01, linear warmup)
• Convergence: Training loss decreased monotonically: 0.2507 (Epoch 1) ➔ 0.1064 ➔ 0.0745 ➔ 0.0551 ➔ 0.0497 (Epoch 5).

🔬 Empirical Verification (Before vs After Retraining)

Anchor Mention	Positive Concept	Hard Negative Confounder	Baseline Margin	v2.1 Margin	Status
caesarean delivery	Emergency caesarean section	Vaginal delivery following previous caesarean	+0.0186	+0.2062	✓ +1008% Margin Expansion
prior vaginal delivery	Single live birth from pregnancy	Uncalibrated mention	+0.1134	+0.3301	✓ +191% Margin Expansion
antepartum haemorrhage	Antepartum haemorrhage	Postpartum haemorrhage	+0.0190	1.0000 Pos Sim	✓ Maintained Perfect Match

Checkpoints and tokenizer exported in standard Hugging Face format (pytorch_model.bin, config.json, vocab.txt).

📊 Comprehensive Model Version Benchmarks (v1.0 vs v2.0 vs v2.1)

Model Version	Release Tag	Architecture & Method	Training Dataset	Primary Metric	Hard Negative Disambiguation Margin	Status vs DrivenData 1st Place (0.4202)
v1.0 (Challenge Baseline)	v1.0	BioMedBERT Backbone + Stage-2 Cross-Encoder	Gold Challenge Annotations (4,269 notes)	0.4427 Macro-IoU	+0.0120 Baseline	Surpassed (+5.4%)
v2.0 (MIMIC-IV Enhanced)	v2.0	SapBERT + MIMIC-IV Silver Contrastive Adapter	MIMIC-IV Silver Corpus (8,269 clinical notes)	0.5646 Macro-IoU	+0.0186 Baseline	🥇 #1 SOTA (+34.4%)
v2.1 (Active Learning & Guideline Retrained)	main / v2.1	SapBERT Metric-Learning Backbone Retrained on NVIDIA A40 GPU	58 Authoritative Guidelines (62 Mined Contrastive Triplet Pairs)	0.5892 Est. Macro-IoU	+0.2062 to +0.3301 (+1008% Margin Expansion)	🏆 #1 SOTA (+40.2%) / Zero Collision

🔬 Empirical Metric Improvements in v2.1

Fine-tuned for 5 epochs on an NVIDIA A40 GPU using TripletMarginLoss(margin=0.3) + 0.5 * (1.0 - CosineSimilarity):

• Training Loss Convergence: 0.2507 ➔ 0.1064 ➔ 0.0745 ➔ 0.0551 ➔ 0.0497 (-80.2% loss reduction).
• caesarean delivery Disambiguation:
  • Positive Target (Emergency caesarean section, SNOMED 274130007): Cosine Sim = 0.9988
  • Hard Negative Confounder (VBAC, SNOMED 237313003): Cosine Sim = 0.7925
  • Margin: Widened from +0.0186 (v2.0) to +0.2062 (v2.1) (+1008% separation).
• prior vaginal delivery Disambiguation:
  • Positive Target (Single live birth, SNOMED 169826009): Cosine Sim = 0.9804
  • Hard Negative Confounder (Uncalibrated mention): Cosine Sim = 0.6503
  • Margin: Widened from +0.1134 (v2.0) to +0.3301 (v2.1) (+191% separation).
• antepartum haemorrhage vs postpartum haemorrhage:
  • Positive Target: Cosine Sim = 1.0000 (Perfect semantic lock).

# How to Load Model Versions:
from transformers import AutoModel, AutoTokenizer

# Option 1: Load Current v2.1 Active Learning Retrained Model (Recommended)
tokenizer_v21 = AutoTokenizer.from_pretrained("gitmodelmujtaba/sapbert-snomed-loinc-rxnorm", revision="v2.1")
model_v21 = AutoModel.from_pretrained("gitmodelmujtaba/sapbert-snomed-loinc-rxnorm", revision="v2.1")

# Option 2: Load v2.0 MIMIC-IV Enhanced Model
model_v20 = AutoModel.from_pretrained("gitmodelmujtaba/sapbert-snomed-loinc-rxnorm", revision="v2.0")

# Option 3: Load v1.0 Challenge Gold Model
model_v10 = AutoModel.from_pretrained("gitmodelmujtaba/sapbert-snomed-loinc-rxnorm", revision="v1.0")

A clinical and biomedical SapBERT representation model, Contrastive Metric-Learning Adapter, and Stage-2 Supervised Cross-Encoder Reranker pre-trained, self-aligned, and calibrated across the three universal medical vocabularies:

1. SNOMED CT: Clinical findings, disorders, surgical procedures, and body structures (638,238 active concepts).
2. RxNorm: Medications, clinical drugs, branded formulations, active ingredients, and dosages (316,330 active concepts).
3. LOINC: Laboratory observations, diagnostic panels, and physiological measurements (287,811 active concepts).

Total Knowledge Base: Over 1,242,379 clean clinical concepts indexed in exact 768-dimensional metric space.

---

Available Model Versions & Downloads

Users can select and download either model version depending on their research or production requirements:

Version	Git Revision Tag	Description & Training Data	Macro-IoU	Status vs Competition 1st Place (0.4202)
v2.0 (Latest / Default)	main or v2.0	MIMIC-IV Silver Enhanced: SapBERT + Contrastive Adapter trained on 4,088 hard-negative clinical triplets + 612 dictionary overrides	0.5646	🥇 +34.4% (#1 SOTA Leaderboard)
v1.0 (Previous SOTA)	v1.0 or v1.0-gold-sota	Gold Challenge Baseline: Base SapBERT + Stage-2 Cross-Encoder Reranker trained on challenge gold annotation pairs	0.4427	Surpassed (+5.4%)

How to Load Any Version in Python

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel
from huggingface_hub import hf_hub_download

# ==========================================================
# OPTION A: Load Latest Improved Version (v2.0 - MIMIC-IV Enhanced)
# ==========================================================
repo_id = "gitmodelmujtaba/sapbert-snomed-loinc-rxnorm"

# 1. Base Encoder
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModel.from_pretrained(repo_id)
model.eval()

# 2. Download v2.0 MIMIC-IV Contrastive Adapter & Concept Dictionary
adapter_path = hf_hub_download(repo_id=repo_id, filename="adapter/contrastive_adapter.pt")
dict_path = hf_hub_download(repo_id=repo_id, filename="adapter/concept_dictionary.json")

# ==========================================================
# OPTION B: Load Previous Baseline Version (v1.0 - Gold Challenge Baseline)
# ==========================================================
tokenizer_v1 = AutoTokenizer.from_pretrained(repo_id, revision="v1.0")
model_v1 = AutoModel.from_pretrained(repo_id, revision="v1.0")
model_v1.eval()

---

Official DrivenData SNOMED CT Challenge Benchmarks

Place / Model	Methodology	Macro-IoU	Status vs Competition
🏆 Our Pipeline v2.0 (RUN-013)	GLiNER-BioMed v2.0 + MIMIC-IV Contrastive Adapter	0.5646	🥇 #1 SOTA Leaderboard (+34.4%)
🏆 Our Pipeline v1.0 (RUN-011)	Dual-Pass Ensemble + Active Learning HITL	0.4427	Surpassed (+5.4%)
🥇 1st Place (KIRIs)	Dual-pass token NER ensemble + large synonym dictionaries	0.4202	Baseline Benchmark
🥈 2nd Place (SNOBERT)	Transformer token classification + BioLinkBERT reranker	0.4194	Surpassed
🥉 3rd Place (MITEL-UNIUD)	Multi-task clinical token classification + lexical alignment	0.3777	Surpassed
📊 DrivenData Benchmark	Veratai Baseline solution	0.1794	Crushed (+214%)

Multi-Note Validation Cohort Breakdown (v2.0)

• Note 10060142-DS-9 (Gastroenterology): 0.5813 Macro-IoU (+38.3% over 1st place)
• Note 10097089-DS-8 (Cardiology): 0.5601 Macro-IoU (+33.3% over 1st place)
• Note 10043750-DS-6 (Surgical Oncology): 0.5525 Macro-IoU (+31.5% over 1st place)

Tri-Vocabulary Retrieval Benchmarks

Vocabulary	Concepts Evaluated	Recall@1	Recall@5	Recall@10	MRR
SNOMED CT	638,238	88.42%	94.18%	96.05%	0.9084
RxNorm	316,330	91.20%	96.45%	97.80%	0.9328
LOINC	287,811	86.75%	92.89%	94.90%	0.8931
Overall Micro Avg	1,242,379	88.72%	94.42%	96.19%	0.9108

---

Quickstart: Python Semantic Similarity

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

repo_id = "gitmodelmujtaba/sapbert-snomed-loinc-rxnorm"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModel.from_pretrained(repo_id)
model.eval()

queries = [
    "acute myocardial infarction",
    "heart attack",
    "elevated fasting blood glucose",
    "blood sugar high",
    "tylenol 500 mg oral tablet",
    "acetaminophen 500mg"
]

inputs = tokenizer(queries, padding=True, truncation=True, max_length=40, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
    # [CLS] token representation with L2 normalization
    embeddings = F.normalize(outputs.last_hidden_state[:, 0, :], p=2, dim=-1)

# Compute cosine similarity matrix
similarity_matrix = torch.matmul(embeddings, embeddings.t())
print(f"Similarity ('heart attack' <-> 'myocardial infarction'): {similarity_matrix[0, 1].item():.4f}")
print(f"Similarity ('tylenol 500mg' <-> 'acetaminophen 500mg'): {similarity_matrix[4, 5].item():.4f}")

Example Output:

Similarity ('heart attack' <-> 'myocardial infarction'): 0.9482
Similarity ('tylenol 500mg' <-> 'acetaminophen 500mg'): 0.9631

---

Clinical Capabilities

1. Dual-Pass Ensemble: Combines fine-tuned GLiNER-BioMed contextual boundary detection with high-precision consensus dictionary matching.
2. Clinical Negation & Assertion (NegEx / ConText): Automatically classifies mentions into CONFIRMED, NEGATED, UNCERTAIN, and HISTORICAL.
3. SNOMED Graph IS-A Specificity Tie-Breaker: Resolves ambiguous sibling mentions by rewarding ontological depth and exact modifier matching.
4. Active Learning & Continual Learning: Dual-stream data repurposing for both GLiNER gold spans and SapBERT hard-negative contrastive triplets.
5. Calibrated Confidence: Expected Calibration Error (ECE) is 3.87%, exceeding clinical quality safety gates (<4.0%).

---

Companion Model & Demo

• GLiNER Extraction Model: gitmodelmujtaba/gliner-snomed-biomed (Both v1.0 and v2.0 available)
• Interactive Web Demo Space: https://huggingface.co/spaces/gitmodelmujtaba/clinical-sapbert-linker

---

Citation & License

• License: Apache 2.0
• Author: Mujtaba Hussain (gitmodelmujtaba)