Docs | Blog | Slack | GitHub

Vela Reranker

Vela Reranker ranks passages by their relevance to a query.

307M parameters · Input capacity: 32,768 tokens, including special tokens.

Higher scores indicate greater relevance. The default uses the 22-layer, 768-dimensional exit.

Evaluation

Compared with the original mmBERT Reranker on the same fixed development subsets. Scores are ×100; higher is better.

Evaluation	Metric	Original mmBERT	Vela
MIRACL · 320 queries	nDCG@10	80.47	81.92
Natural Questions · 128 queries	nDCG@10	97.86	97.97
SciFact · 48 queries	nDCG@10	87.85	87.52
QASPER · 64 queries	nDCG@10	56.73	41.62
Controlled 4K–32K context · 288 pairs	Pair accuracy	55.21	60.07
32K end-position subset · 24 pairs	Pair accuracy	54.17	50.00

Controlled context · pair accuracy	Original mmBERT	Vela
4K · 72 pairs	58.33	68.06
8K · 72 pairs	54.17	59.72
16K · 72 pairs	51.39	54.17
32K · 72 pairs	56.94	58.33

Both models use 22 layers, 768 dimensions, FP32, identical tokenization and candidate pools, and complete inputs without truncation. MIRACL covers Arabic, Spanish, Japanese and Chinese (80 queries each). Controlled context places relevant text at different positions; the 32K end-position row is a subset. These development sets informed Vela checkpoint selection; they are not independent tests or full benchmark leaderboards. Original model training overlap is unknown.

SciFact uses the same first-stage top-100 candidates for both models; two queries have no relevant candidate in that pool.

Quick start

Install torch, transformers, and safetensors.

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "llm-semantic-router/Vela-1.0-Encoder-307M-Reranker"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval()
inputs = tokenizer(["When does the library open?"], ["The library opens in the morning."], padding=True, truncation=False, return_tensors="pt")
with torch.inference_mode():
    scores = model(**inputs).logits
print(scores)