dinushiTJ/action-gemma-2-2b-it-lora

ActionGemma 2B: Gemma 2 fine-tuned for function calling

In one sentence: a small, free, open model (Gemma 2 2B) trained to turn plain-English requests into structured function calls, so tool-using AI can run privately on an ordinary computer instead of a paid cloud API.

This repository holds the LoRA adapter. Two ready-to-deploy versions are also available:

Format	Use it for	Repo
LoRA adapter (this repo)	further fine-tuning; loading with PEFT or Unsloth	dinushiTJ/action-gemma-2-2b-it-lora
GGUF (Q4_K_M 1.71 GB, Q5_K_M, Q6_K, Q8_0, F16)	laptops and CPUs via llama.cpp, Ollama or LM Studio	dinushiTJ/action-gemma-2-2b-it-gguf
Merged FP16 weights	GPU serving with vLLM or Transformers	dinushiTJ/action-gemma-2-2b-it-vllm-f16

Datasets: training · evaluation · full collection

---

Results

ActionGemma was compared with the base Gemma 2 2B instruction-tuned model on 812 function-calling examples adapted from the Berkeley Function Calling Leaderboard. None of these functions appear in the training data.

Model	Environment	Precision	Recall	Macro F1	Hallucination rate ↓	Tokens per call ↓
Gemma 2 2B IT	Local GPU	82.27	85.25	85.92	16.03%	57.63
ActionGemma 2B	Local GPU	90.52	94.37	93.74	6.68%	47.64
Gemma 2 2B IT	Kaggle T4	84.73	83.03	83.58	18.04%	57.02
ActionGemma 2B	Kaggle T4	94.84	94.30	94.30	6.01%	47.70

• Accuracy: +7.82% to +10.72% macro F1 over the base model.
• Reliability: made-up (hallucinated) function calls fall from about 16–18% to about 6%.
• Efficiency: about 17% fewer tokens per function call. Peak inference memory is about 7.5 GB on a consumer GPU with no optimisation.

Compared with a frontier cloud model

To check that the model isn't just memorising function names, the evaluation was repeated with every function name replaced by a random string. Claude 3.5 Sonnet was run on the same set as a reference.

ActionGemma sits between the base model and Claude, closer to Claude (88.04 vs 95.99 F1), while using less than half the tokens (55.9 vs 117.5 per call) at similar latency.

---

How to use

With PEFT / Transformers

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

repo = "dinushiTJ/action-gemma-2-2b-it-lora"
model = AutoPeftModelForCausalLM.from_pretrained(repo, device_map="auto")  # needs bitsandbytes
tokenizer = AutoTokenizer.from_pretrained(repo)

functions = [{
    "name": "get_weather",
    "description": "Get the current weather in a given location",
    "parameters": {
        "type": "object",
        "properties": {"location": {"type": "string", "description": "City, e.g. Hamilton, NZ"}},
        "required": ["location"],
    },
}]
messages = [{"role": "user", "content": "What's the weather like in Hamilton right now?"}]

prompt = tokenizer.apply_chat_template(
    messages, tools=functions, chat_template="tool_use",
    tokenize=False, add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# Output looks like: <functioncall> {"name": "get_weather", "arguments": {"location": "Hamilton, NZ"}} </functioncall>

Output format

The model wraps each call in <functioncall> tags, so it is easy to extract with a regex:

<functioncall> {"name": "function_name", "arguments": {"arg_1": "value_1"}} </functioncall>

Send the function's result back as a message with the function_response role, and the model will write the final answer. The tokenizer ships four chat templates:

• tool_use: injects the function list and validates role order.
• default
• restrictive
• non_restrictive

---

How it was built

1. Data preparation

The source data was hypervariance/function-calling-sharegpt: 86,864 multi-turn conversations. It was prepared as follows:

• Fixed role order: merged consecutive assistant turns (affecting 8.49% of conversations) and renamed roles to match Gemma's format.
• Extracted structure: pulled each conversation's function definitions and function calls out into their own JSON columns.
• Removed invalid rows: dropped 14 conversations that call functions not in their own function list.
• Split with stratification: made an 80/20 train/test split stratified by each conversation's sorted combination of function-call names, so rare combinations appear in both splits.

Before (ShareGPT)	After (Gemma chat format)

Evaluation set: built from the Berkeley Function Calling Leaderboard's "simple", "multiple", "irrelevance" and "chatable" categories.

• Function definitions were converted to valid JSON Schema, and 227 examples with invalid schemas were removed.
• Every function was checked by signature (name plus sorted argument names and types) so that none overlaps with the training data.
• This left 812 examples.

2. Fine-tuning

The method was supervised fine-tuning (SFT) with QLoRA on the 4-bit base model, using Unsloth and TRL. It ran for 1 epoch on a single 16 GB Tesla T4 GPU (Kaggle) and took 15 h 12 m, with peak memory of 10.52 GB. Runs were tracked in Weights & Biases.

Setting	Value
LoRA rank / alpha	16 / 16
Target modules	q, k, v, o, gate, up, down projections
Learning rate	2e-4
Epochs	1 (checkpoint every 500 steps)
Seed	3407

---

Limitations

• Evaluation checks function signatures (name, argument names and types), not whether argument values are correct or whether calls execute successfully.
• Scores drop by about 6% when function names are anonymised. This suggests the model partly relies on how function names are worded, not just their descriptions.
• The training data was not independently filtered for quality.
• The model is English only.

License

Use of this model is subject to the Gemma Terms of Use. The training and evaluation datasets are released under Apache 2.0.

Citation

@mastersthesis{jayasinghe2024actiongemma,
  author = {Dinushi Thathsarani Jayasinghe},
  title  = {ActionGemma: Supervised Fine-Tuning of Google Gemma 2 for Function Calling},
  school = {University of Waikato},
  year   = {2024}
}

Author: Dinushi Jayasinghe · GitHub · Supervised by Dr. Hongyu Wang, University of Waikato.