---
language:
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-classification
base_model: FacebookAI/roberta-base
datasets:
- heyongxin233/DETree
tags:
- text-classification
- ai-generated-text-detection
- llm-detection
- machine-generated-text
- roberta
- linguistic-features
- adversarial-robustness
metrics:
- accuracy
- f1
- roc_auc
model-index:
- name: hybrid-roberta-linguistic-llm-detector
  results:
  - task:
      type: text-classification
      name: Machine-Generated Text Detection
    dataset:
      name: DETree (RAID subset, 6 attack types)
      type: heyongxin233/DETree
    metrics:
    - type: accuracy
      value: 0.9507
      name: Accuracy (document-level)
    - type: f1
      value: 0.9505
      name: Macro F1 (document-level)
    - type: roc_auc
      value: 0.9772
      name: ROC AUC (document-level)
---

# Hybrid RoBERTa + Linguistic-Features Detector

Best checkpoint from the master's thesis on hybrid machine learning approaches for detection of LLM-generated English texts by Bohdan Zhvalevskyi. 
It combines a fine-tuned `roberta-base` CLS embedding with 25 normalized
linguistic features through a feature-attention module and a learned gated
fusion, followed by a 2-class classifier (`0 = human`, `1 = machine`).

## Model details

- **Developed by:** Bohdan Zhvalevskyi (master's thesis)
- **Model type:** Hybrid transformer + hand-crafted linguistic-feature classifier
- **Task:** Binary detection of machine-generated vs. human-written English text
- **Language:** English
- **Base model:** [`FacebookAI/roberta-base`](https://huggingface.co/FacebookAI/roberta-base) (bottom 6 layers + embeddings frozen)
- **License:** Apache-2.0
- **Architecture:** RoBERTa CLS embedding + 25 linguistic features → feature
  attention → linear projection → learned gated fusion → 2-class head
  (see `model.py`).
- **The original training pipeline is found on github for reproducibility: [`link to github repo`](https://github.com/foggyghost0/Hybrid-Machine-Learning-Approaches-for-Detection-of-LLM-Generated-Texts)**

## Intended uses & limitations

**Intended use.** Research on LLM-generated text detection and adversarial
robustness; scoring English documents/paragraphs as human- or machine-written.

**Limitations & biases.**
- **English only.** Trained on English (DETree, RAID subset); not validated on
  other languages or code.
- **Domain/generator shift.** Trained on a fixed set of generators and attack
  types (see below). Newer models or unseen attacks may degrade performance.
- **Not for high-stakes decisions.** No detector is reliable enough to be the
  sole basis for consequential judgements (e.g. academic misconduct). Treat
  outputs as a probabilistic signal, not proof.
- **Requires the exact feature pipeline.** Predictions are only valid when the 25
  features are extracted with `features.py` and normalized with the shipped
  `ling_scaler.pkl`; a raw/zero feature vector produces meaningless scores.

## Files

| File | Purpose |
|------|---------|
| `hybrid_model_best.pt` | PyTorch `state_dict` for `HybridClassifier` (~500 MB). |
| `model.py` | Self-contained architecture definition + `load_model()` helper. |
| `features.py` | Self-contained linguistic feature extraction (normalize → chunk → 25 features). |
| `ling_scaler.pkl` | Fitted training `StandardScaler` used to normalize the 25 features. **Required** for valid predictions. |
| `example_usage.py` | Runnable end-to-end scoring example (raw text → prediction). |

## Install

```bash
pip install torch transformers spacy scikit-learn
python -m spacy download en_core_web_lg
```

## Quick start

Score raw text end-to-end (extraction + normalization + document-level scoring
are handled for you):

```bash
python example_usage.py --text "Your transcript goes here."
python example_usage.py --file document.txt
```

Or from Python:

```python
import torch
from transformers import RobertaTokenizer, GPT2LMHeadModel, GPT2TokenizerFast
import spacy, pickle
from model import load_model, CONFIG
from features import extract_raw_features, prepare_document
from example_usage import score_text

device = "cuda" if torch.cuda.is_available() else "cpu"
model = load_model("hybrid_model_best.pt", device=device)
tokenizer = RobertaTokenizer.from_pretrained(CONFIG["roberta_model"])
nlp = spacy.load("en_core_web_lg", disable=["ner"])
gpt2_tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
gpt2_model = GPT2LMHeadModel.from_pretrained("gpt2").to(device).eval()
scaler = pickle.load(open("ling_scaler.pkl", "rb"))

p_llm, chunk_probs = score_text(
    "Your transcript goes here.",
    model, tokenizer, nlp, gpt2_model, gpt2_tokenizer, scaler, device=device,
)
print(p_llm, "->", "machine" if p_llm >= 0.5 else "human")
```

## Linguistic features

The 25 features, in the exact order the model expects, are extracted by
`features.py` and normalized with the fitted training scaler (`ling_scaler.pkl`):

```
msttr, avg_word_len, hapax_ratio, function_ratio, punct_density, char_entropy,
burstiness, repetition_ratio, avg_sent_len, sent_len_std, noun_ratio,
verb_ratio, adj_ratio, adv_ratio, pron_ratio, pos_diversity, avg_tree_depth,
max_tree_depth, sub_clause_ratio, dm_density, sent_len_cv, fp_ratio,
num_sentences, words_per_sent, perplexity
```

`features.py` reproduces the training/testing pipeline exactly: `normalize_text`
→ `sliding_window_chunk` (450-word windows, 350-word stride) → 24 spaCy features
+ GPT-2 perplexity → `StandardScaler.transform`. Document-level scores are the
mean of per-chunk `P(machine)`.

## Training data

Trained on the [**DETree**](https://huggingface.co/heyongxin233/DETree) dataset
(RAID subset), balanced across six
conditions to study adversarial robustness: `no_attack` (clean), `synonym`,
`polish`, `paraphrase`, `perplexity_attack`, and `paraphrase_by_llm`. Labels are
`0 = human`, `1 = machine`. Documents are lowercased/cleaned, split by document
id (no document leakage across train/val/test), then chunked with a 450-word
sliding window (350-word stride). See `01_data_preprocessing_v2.py`.

## Training procedure

- Base `roberta-base` with embeddings and the bottom 6 encoder layers frozen.
- 25 linguistic features (`StandardScaler`-normalized) passed through a
  feature-attention module and projected to 768-d, then fused with the RoBERTa
  CLS embedding via a learned sigmoid gate.
- Dropout 0.15; max sequence length 512; seed 42.

## Evaluation results

Document-level metrics on the held-out test set (**6,292 documents / 6,806
chunks**), aggregation = mean chunk probability per document, thresholded at 0.5.

| Model | Accuracy | Macro-F1 | AUC |
|---|---|---|---|
| **Hybrid (this model)** | **0.9507** | **0.9505** | 0.9772 |
| RoBERTa-only baseline | 0.9170 | 0.9158 | 0.9824 |
| Feature-only MLP baseline | 0.8462 | 0.8459 | 0.9213 |

**Robustness (Macro-F1 by attack type).** The hybrid's margin over RoBERTa is
largest on the hardest attacks — Polish (0.896 vs 0.798) and Paraphrase (0.926 vs
0.861) — because the linguistic features restore specificity that RoBERTa loses
on attacked human text. The improvement over RoBERTa is statistically significant
(McNemar p < 1e-12; ΔMacro-F1 = +0.035, 95% CI [+0.028, +0.041]).

## Notes

- The checkpoint is a plain `state_dict`; load it with the `HybridClassifier`
  in `model.py` (see `load_model()`).
- Long documents were chunked to 512 tokens at training time; the reported
  document-level score is the mean chunk probability per document.
- `return_gate=True` in `forward()` also returns the fusion gate values and the
  feature-attention weights for interpretability.