README.md

metadata

base_model: Qwen/Qwen3.5-2B
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
tags:
  - token-efficient
  - efficient-thinking
  - reasoning
  - grpo
  - lora
  - peft

ThinkingCap-Qwen3.5-2B (loss-level variant)

A merged, ready-to-serve fine-tune of Qwen3.5-2B trained with GRPO to reach the same answers with 43% fewer thinking tokens — while improving accuracy by +13 to +17 points, because the dominant failure of the base model is not verbosity but never finishing its reasoning at all.

This adapter is the loss-level variant of a controlled study on how to compress chain-of-thought at small scale. It combines three modifications to the GRPO objective — Dr.GRPO normalization, positional advantage decay inside the reasoning span, and KL restricted to that span — on top of a correctness-gated length penalty. The companion study that isolates each component is linked at the bottom.

---

In-domain results

Measured under a natural-length protocol: greedy decoding, a 4,096-token budget, and no forced truncation of thinking. A response whose reasoning span never closes contains no answer and is scored incorrect — so token reduction and accuracy are read off the same generations.

Benchmark	Base Acc	Ours Acc	Base Thinking Tokens	Ours Thinking Tokens	Reduction
Held-out dev (300 problems)	69.0	82.3	2,922	1,656	↓ 43.3%
GSM8K	50.0	66.7	3,545	2,438	↓ 31.2%
ARC-Challenge	70.8	87.5	2,216	1,247	↓ 43.7%

Out-of-domain

Benchmark	Base Acc	Ours Acc	Base Thinking Tokens	Ours Thinking Tokens	Reduction
MATH-500	20.8	20.8	4,096	3,320	↓ 19.0%

Out of distribution the adapter preserves accuracy exactly while cutting a fifth of the reasoning budget. Compression transfers; the accuracy gains do not — a 2B model has little headroom on competition mathematics either way.

Failure modes

The base model's real pathology is non-termination: it finds the answer early, then loops on self-verification and never emits a closing marker.

Metric (out-of-domain)	Base	Ours
Reasoning never closes within budget	100%	75%
Terminates normally (EOS)	0%	25%

On in-domain data the effect is far stronger — unclosed reasoning drops from 58% to 21%. Most of the accuracy gain is simply answers that now exist.

Training

Base model	Qwen/Qwen3.5-2B (thinking mode)
Method	GRPO + LoRA (rank 16, α 32, bf16), 200 optimizer steps
Reward	correctness-gated length penalty; zero reward when the answer is wrong or the reasoning span never closes
Objective modifications	Dr.GRPO normalization · positional advantage decay over the thinking span · KL restricted to the thinking span
Data	10.7K verifiable problems (GSM8K + ARC training splits), deduplicated against every evaluation set
Hardware	1× NVIDIA RTX 5080 (16 GB)

No generation-time intervention is used at any point — no forced stopping, no length cap on thinking. The model learns to terminate purely from the reward.

Usage

The repository root holds the merged model — load it like any Qwen3.5 checkpoint, no PEFT required.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "drlee1/ThinkingCap-Qwen3.5-2B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "If a train travels 120 km in 1.5 hours, what is its average speed?"}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

out = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

enable_thinking=True is required — Qwen3.5-2B runs in non-thinking mode by default, and this model is trained on the thinking-mode chat template.

Honest scope

• Single training seed, 200 optimizer steps. Numbers are a screening result, not a multi-seed benchmark. Evaluation uses 300 held-out problems for the dev set and 24 items per public benchmark.
• A simpler recipe is stronger. In the companion ablation, dropping all three loss-level modifications and keeping only the reward-level length penalty reaches 79% token reduction at 90.7% dev accuracy — better on both axes than this model. Each objective modification measurably costs compression in this regime (2B, LoRA, small rollout groups). This model is released as the loss-level arm of that comparison, not as the recommended configuration.
• Findings are specific to this scale and setup; they are not a general verdict on the underlying techniques in their original settings.

Citation

@misc{reducing-think-token-2026,
  title  = {Reducing Thinking Tokens via Reinforcement Learning:
            A Controlled Study on Loss-Level Interventions},
  author = {DONGRYEOLLEE1},
  year   = {2026},
  url    = {https://github.com/DONGRYEOLLEE1/reducing-think-token}
}

Acknowledgements. Inspired by ThinkingCap-Qwen3.6-27B (BottleCapAI). Built on Qwen3.5-2B, TRL, and PEFT. The full study — including the component-wise ablation with confidence intervals — is on GitHub.