skeole commited on
Commit
e86ab95
·
verified ·
1 Parent(s): 6cee5e8

remove "commentary" from the valid channel list when the request contains no tools

Browse files

The default jinja chat template includes "commentary" as a valid output channel even when no tools are present in the request. This one template mismatch breaks the model on nearly every non-tool-call task. Removing "commentary" from valid channels when no tools are defined fixes the problem. The accuracy gains are dramatic.

On the chat API with fc_model=0 (BFCL_v4 tool calling benchmark, AST parsing, no tools in request):

| Test | Default Jinja | Fixed Jinja | Change |
|------|:------------:|:-----------:|:------:|
| live_relevance | 3.1% | 41.4% | +38.3% |
| live_simple | 3.1% | 36.7% | +33.7% |
| simple_java | 18.0% | 60.5% | +42.5% |
| simple_javascript | 13.0% | 56.5% | +43.5% |
| simple_python | 1.8% | 36.6% | +34.8% |
| multi_turn_base | 0.2% | 14.6% | +14.3% |
| irrelevance | 99.9% | 93.8% | -6.1% |

The one metric that drops is irrelevance (99.9% to 93.8%). With the fixed template, the model actually tries to answer questions instead of producing silent failures. The irrelevance test marks those attempts as wrong, so the score goes down even though the system is doing more.

Removing one channel from the chat template moves accuracy from 3% to 40% on live tests. This is not a minor tuning adjustment. The default template shipped with the inference stack was making the model non-functional for basic tasks. Mismatches between training structure and inference template produce silent failures that look like model incompetence.

Files changed (1) hide show
  1. chat_template.jinja +4 -2
chat_template.jinja CHANGED
@@ -216,9 +216,11 @@
216
  {%- endfor %}
217
  {{- render_builtin_tools(available_builtin_tools.browser, available_builtin_tools.python) }}
218
  {%- endif -%}
219
- {{- "# Valid channels: analysis, commentary, final. Channel must be included for every message." }}
220
- {%- if tools -%}
221
  {{- "\nCalls to these tools must go to the commentary channel: 'functions'." }}
 
 
222
  {%- endif -%}
223
  {%- endmacro -%}
224
 
 
216
  {%- endfor %}
217
  {{- render_builtin_tools(available_builtin_tools.browser, available_builtin_tools.python) }}
218
  {%- endif -%}
219
+ {%- if tools is defined and tools -%}
220
+ {{- "# Valid channels: analysis, commentary, final. Channel must be included for every message." }}
221
  {{- "\nCalls to these tools must go to the commentary channel: 'functions'." }}
222
+ {%- else -%}
223
+ {{- "# Valid channels: analysis, final. Channel must be included for every message." }}
224
  {%- endif -%}
225
  {%- endmacro -%}
226