Upload RadBERT German CTRate multi-label classifier

README.md

@@ -0,0 +1,145 @@
+---
+language:
+- de
+license: mit
+library_name: transformers
+pipeline_tag: text-classification
+tags:
+- radiology
+- medical-imaging
+- chest-ct
+- multi-label-classification
+- radbert
+- german
+- ctrate
+base_model: zhihets/RadBERT-RoBERTa-Base
+---
+
+# RadBERT German CTRate Classifier
+
+A **RadBERT**-based multi-label classifier for predicting 18 pathology labels from **German-language** radiology reports.  
+The training data consists of German-translated reports from the [CTRate](https://huggingface.co/datasets/ibrahimhamamci/CT-RATE) dataset, translated using Qwen 2.5 9B.
+
+## Model Details
+
+| Property | Value |
+|---|---|
+| **Base model** | RadBERT (RoBERTa-base architecture, pre-trained on radiology text) |
+| **Task** | Multi-label text classification (18 labels) |
+| **Language** | German (`de`) |
+| **Framework** | 🤗 Transformers + PyTorch |
+| **Problem type** | `multi_label_classification` |
+
+## Labels (18 pathologies)
+
+| ID | Label |
+|----|-------|
+| 0  | Medical material |
+| 1  | Arterial wall calcification |
+| 2  | Cardiomegaly |
+| 3  | Pericardial effusion |
+| 4  | Coronary artery wall calcification |
+| 5  | Hiatal hernia |
+| 6  | Lymphadenopathy |
+| 7  | Emphysema |
+| 8  | Atelectasis |
+| 9  | Lung nodule |
+| 10 | Lung opacity |
+| 11 | Pulmonary fibrotic sequela |
+| 12 | Pleural effusion |
+| 13 | Mosaic attenuation pattern |
+| 14 | Peribronchial thickening |
+| 15 | Consolidation |
+| 16 | Bronchiectasis |
+| 17 | Interlobular septal thickening |
+
+## Quick Start
+
+### Installation
+
+```bash
+pip install transformers torch
+```
+
+### Loading the model
+
+```python
+from transformers import AutoTokenizer, AutoConfig
+from modeling_radbert import RadBertForSequenceClassification
+import torch
+
+repo_id = "suitch/radbert-german-ctrate-classifier"
+
+# Download the custom model class (or copy modeling_radbert.py locally)
+from huggingface_hub import hf_hub_download
+import sys, os
+
+modeling_path = hf_hub_download(repo_id=repo_id, filename="modeling_radbert.py")
+sys.path.insert(0, os.path.dirname(modeling_path))
+
+# Load config, model, and tokenizer
+config = AutoConfig.from_pretrained(repo_id)
+model = RadBertForSequenceClassification.from_pretrained(repo_id, config=config)
+tokenizer = AutoTokenizer.from_pretrained(repo_id)
+
+model.eval()
+```
+
+### Inference example
+
+```python
+text = "Das Herz ist leicht vergrößert. Es zeigt sich ein kleiner Pleuraerguss links."
+
+inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=512)
+
+with torch.no_grad():
+    logits = model(**inputs)
+
+probabilities = torch.sigmoid(logits).squeeze()
+threshold = 0.5
+predicted_labels = [
+    config.id2label[i] for i, p in enumerate(probabilities) if p >= threshold
+]
+
+print("Predicted labels:", predicted_labels)
+print("Probabilities:")
+for i, p in enumerate(probabilities):
+    print(f"  {config.id2label[i]}: {p:.4f}")
+```
+
+## Training Details
+
+- **Base checkpoint**: RadBERT (RoBERTa-base weights pre-trained on radiology corpora)
+- **Training data**: German translations of CTRate radiology reports (translated with Qwen 2.5 9B)
+- **Classification head**: Linear layer on top of the `[CLS]` / pooler output
+- **Loss**: Binary Cross-Entropy with Logits (per-label sigmoid)
+
+## Limitations
+
+- This model is trained for **label inference from report text only** — it does **not** process images.
+- It should **not** be treated as a clinical decision support system.
+- Performance is limited by the quality of the machine-translated training data.
+
+## Citation
+
+If you use this model, please cite the CTRate dataset and RadBERT:
+
+```bibtex
+@article{hamamci2024ctrate,
+  title={CT-RATE: A Large-Scale Computed Tomography Report-Image Dataset for AI in Radiology},
+  author={Hamamci, Ibrahim Ethem and others},
+  journal={arXiv preprint},
+  year={2024}
+}
+
+@article{yan2022radbert,
+  title={RadBERT: Adapting Transformer-based Language Models to Radiology},
+  author={Yan, Di and others},
+  journal={Radiology: Artificial Intelligence},
+  year={2022}
+}
+```
+
+## License
+
+MIT

config.json

@@ -0,0 +1,71 @@
+{
+  "architectures": [
+    "RadBertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "auto_map": {
+    "AutoModelForSequenceClassification": "modeling_radbert.RadBertForSequenceClassification"
+  },
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "Medical material",
+    "1": "Arterial wall calcification",
+    "2": "Cardiomegaly",
+    "3": "Pericardial effusion",
+    "4": "Coronary artery wall calcification",
+    "5": "Hiatal hernia",
+    "6": "Lymphadenopathy",
+    "7": "Emphysema",
+    "8": "Atelectasis",
+    "9": "Lung nodule",
+    "10": "Lung opacity",
+    "11": "Pulmonary fibrotic sequela",
+    "12": "Pleural effusion",
+    "13": "Mosaic attenuation pattern",
+    "14": "Peribronchial thickening",
+    "15": "Consolidation",
+    "16": "Bronchiectasis",
+    "17": "Interlobular septal thickening"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "Arterial wall calcification": 1,
+    "Atelectasis": 8,
+    "Bronchiectasis": 16,
+    "Cardiomegaly": 2,
+    "Consolidation": 15,
+    "Coronary artery wall calcification": 4,
+    "Emphysema": 7,
+    "Hiatal hernia": 5,
+    "Interlobular septal thickening": 17,
+    "Lung nodule": 9,
+    "Lung opacity": 10,
+    "Lymphadenopathy": 6,
+    "Medical material": 0,
+    "Mosaic attenuation pattern": 13,
+    "Peribronchial thickening": 14,
+    "Pericardial effusion": 3,
+    "Pleural effusion": 12,
+    "Pulmonary fibrotic sequela": 11
+  },
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "problem_type": "multi_label_classification",
+  "transformers_version": "4.57.6",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 50265
+}

export_metadata.json

@@ -0,0 +1,25 @@
+{
+  "base_model": "/vol/ideadata/ac54awik/Medical_report_evaluation/radbert_local",
+  "checkpoint": "classifier_training/classifier_de/RadBertClassifier_best.pth",
+  "labels": [
+    "Medical material",
+    "Arterial wall calcification",
+    "Cardiomegaly",
+    "Pericardial effusion",
+    "Coronary artery wall calcification",
+    "Hiatal hernia",
+    "Lymphadenopathy",
+    "Emphysema",
+    "Atelectasis",
+    "Lung nodule",
+    "Lung opacity",
+    "Pulmonary fibrotic sequela",
+    "Pleural effusion",
+    "Mosaic attenuation pattern",
+    "Peribronchial thickening",
+    "Consolidation",
+    "Bronchiectasis",
+    "Interlobular septal thickening"
+  ],
+  "tokenizer_source": "/vol/ideadata/ac54awik/Medical_report_evaluation/radbert_local"
+}

modeling_radbert.py

@@ -0,0 +1,35 @@
+import torch
+import torch.nn as nn
+from transformers import RobertaModel, RobertaConfig, PreTrainedModel
+
+
+class RadBertForSequenceClassification(PreTrainedModel):
+    config_class = RobertaConfig
+    base_model_prefix = "model"
+
+    def __init__(self, config):
+        super().__init__(config)
+        num_labels = getattr(config, "num_labels", 2)
+        self.model = RobertaModel(config)
+        self.classifier = nn.Linear(config.hidden_size, num_labels)
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids=None,
+        attention_mask=None,
+        token_type_ids=None,
+        **kwargs,
+    ):
+        outputs = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            **kwargs,
+        )
+
+        pooled_output = outputs.pooler_output
+        if pooled_output is None:
+            pooled_output = outputs.last_hidden_state[:, 0]
+
+        return self.classifier(pooled_output)

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed6c293735a097bb023a10c7a5bcb1c2cc395bdac5fbedba902f341143f3359b
+size 498704519

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,59 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50264": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "do_lower_case": true,
+  "eos_token": "</s>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "RobertaTokenizer",
+  "trim_offsets": true,
+  "unk_token": "<unk>"
+}