ABot-ER-9B

ABot-ER unifies embodied question answering, spatial reasoning, point grounding and task planning. This checkpoint uses the Task-Conditioned Adaptive Training Strategy described in the technical report: shared backbone projections with sequence-level, task-conditioned low-rank residuals.

Checkpoint

Final joint post-training checkpoint: 300.
Residual layers (zero-based): [19, 23, 27, 31].
Residual rank and alpha: 16.
Adapted projections: query and value projections in selected full-attention blocks.
Tokenizer, route-token IDs and tensor values are preserved from the evaluated checkpoint.

Loading

Use a clean Python environment, install huggingface_hub for downloading, and install the dependencies from requirements.txt. Set HF_REVISION to this repository's immutable commit SHA. Authentication is needed only if the destination repository requires it. Review the included Python files before enabling custom code. The example below is a text-only loading and routing smoke test, not a visual benchmark example; real image/video inputs must follow the processor and benchmark contracts in the project documentation.

import os
import sys
from huggingface_hub import snapshot_download

path = snapshot_download(repo_id="acvlab/ABot-ER-9B", revision=os.environ["HF_REVISION"])
sys.path.insert(0, path)
from abot_er import load_model, load_processor, route_messages

model = load_model(path, dtype='bfloat16', device_map='auto')
processor = load_processor(path)
messages = route_messages([
    {'role': 'user', 'content': [{'type': 'text', 'text': 'Describe the types of embodied questions you can answer.'}]}
], task_family='2d_qa')
inputs = processor.apply_chat_template(
    messages, tokenize=True, return_dict=True, return_tensors='pt',
    add_generation_prompt=True, enable_thinking=False,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(output[:, inputs['input_ids'].shape[1]:], skip_special_tokens=True))

Direct Transformers loading must use AutoModelForImageTextToText.from_pretrained(path, trust_remote_code=True). Loading as an ordinary backbone without the included custom implementation would omit the conditional branches and is unsupported.

Task routing

The first user message must contain exactly one route token:

[2D_QA]: all embodied QA, including VABench-P and Where2Place point grounding.
[TASK_PLAN]: task planning.

These are the actual tokenizer strings. Preserve them exactly; do not insert both. route_messages adds the marker without modifying the benchmark-specific prompt. Use one active request per model instance; concurrent interleaved decoding on a shared instance is not supported by the route cache. Use greedy/sampling generation with num_beams=1.

Scope and terms

This card does not grant a new license or supersede the original backbone's applicable terms. The dataset is maintained separately in acvlab/ABot-ER-Training-Data; referenced image and video binaries are not bundled with this model.

See LICENSE for component terms and CITATION.bib for the technical report.