Fix transformers 5.x loading and add Sentence Transformers support

#1
by tomaarsen HF Staff - opened

Hello!

The MultiVectorEncoder class ships in the next Sentence Transformers release, planned for around the 18th, so for now the install below pulls from source. I would love to feature this model in that release's blog post and documentation, especially once it loads without the revision pin (that is, once this PR is merged).

Heads up, this PR was AI-generated and human-reviewed. Here's a summary of the changes as reported by my agent:

Pull Request overview

  • Fix five small transformers 5.x incompatibilities that stop the model from loading.
  • Add a Sentence Transformers loading path (multi-vector, ColBERT-style late interaction) via MultiVectorEncoder.

Details

This model shares the five transformers 5.x compatibility fixes from the tomoro-colqwen3-embed-4b PR (the modeling files are byte-identical between the two repositories, so the patch is identical): a sub_configs dataclass-field clash, sub-config init order, and three tie_weights / mm_token_type_ids signature mismatches in configuration_colqwen3.py and modeling_colqwen3.py. Without them, AutoModel.from_pretrained(..., trust_remote_code=True) fails on transformers 5.x. The weights are untouched and existing AutoModel plus AutoProcessor usage is unchanged apart from now working on current transformers.

The model also loads directly into Sentence Transformers as a Transformer(retrieval) -> MultiVectorMask pipeline (its own forward already handles the projection, normalization and masking), so it needs only the standard ST config files and no custom module. I verified it against AutoModel plus AutoProcessor (in float16, since it does not fit in 24 GB at float32): MaxSim scores agree to 3.3e-3, query embeddings to 1.9e-3, document embeddings are bit-identical, and each query retrieves its own page.

pip install "sentence-transformers[image] @ git+https://github.com/huggingface/sentence-transformers.git"
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("TomoroAI/tomoro-colqwen3-embed-8b", revision="refs/pr/1", trust_remote_code=True)

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
documents = [
    f"https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc{i}.jpg"
    for i in range(1, 5)
]

query_embeddings = model.encode_query(queries, convert_to_tensor=True)
document_embeddings = model.encode_document(documents, convert_to_tensor=True)
print(tuple(query_embeddings[0].shape), tuple(document_embeddings[0].shape))
# (23, 320) (1251, 320)

print(model.similarity(query_embeddings, document_embeddings))
# tensor([[12.4336,  8.5479,  6.7305,  5.1968],
#         [ 4.5449, 11.1719,  4.8320,  4.5195]])
  • Tom Aarsen
Tomoro AI Ltd org

Hi Tom, it looks good to me, thanks for fixing this.

hxssgaa changed pull request status to open
hxssgaa changed pull request status to merged

@hxssgaa I'm glad you like it! I've also opened a PR against the 4B variant: https://huggingface.co/TomoroAI/tomoro-colqwen3-embed-4b/discussions/4
I'll also update my Sentence Transformers MultiVectorEncoder blogpost draft to report that this model can load directly from the official model repository 🤗 It should release in ~4 days!

  • Tom Aarsen

Sign up or log in to comment