happyz
/
llama.cpp
mirror of https://github.com/ggerganov/llama.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
llama.cpp/docs/autoparser.md

610 lines
28 KiB
Markdown
Raw Blame History

# Unified Auto-Parser Architecture
The auto-parser automatically analyzes chat templates to determine how to parse model outputs, including content, reasoning, and tool calls.
## Overview
The unified auto-parser uses a **pure differential, compositional approach** to analyze chat templates:
**Core Philosophy**:
- **Zero Hardcoded Patterns**: All markers extracted through template comparison (the **only heuristic** is JSON detection)
- **Compositional Architecture**: Separate parsers for reasoning, content, and tools that compose cleanly
**Four-Phase Analysis**:
1. **Phase 1: Reasoning Analysis** (R1-R3) - Detects reasoning markers and mode
2. **Phase 2: Content Analysis** (C1) - Detects content wrapping markers
3. **Phase 3: Tool Call Analysis** (T1-T7) - Extracts tool section, function, and call ID markers
4. **Phase 4: Argument Analysis** (A1-A2) - Extracts argument name/value markers (TAG_WITH_TAGGED only)
## Data Structures
### diff_analysis_result
The result of differential analysis contains all extracted markers and format classifications:
```cpp
struct diff_analysis_result {
// Classification results
reasoning_mode reasoning = reasoning_mode::NONE;
content_mode content = content_mode::PLAIN;
tool_format tools = tool_format::NONE;
// All extracted markers (see marker_registry below)
marker_registry markers;
// JSON field names (for JSON_NATIVE format)
bool fun_name_is_key = false; // Function name is the JSON key: {"func_name": {...}}
std::string function_field = "function"; // Outer object key (e.g., "function" in "function.name")
std::string name_field = "name";
std::string args_field = "arguments";
std::string id_field; // String call ID field (e.g., "id")
std::string gen_id_field; // Generated integer call ID field (e.g., "tool_call_id")
std::vector<std::string> parameter_order; // Order of JSON fields for parsing
// Call ID position (for non-JSON formats)
call_id_position call_id_pos = call_id_position::NONE;
// Flags
bool supports_tools = false;
bool supports_parallel_calls = false;
bool requires_nonnull_content = false;
bool tools_array_wrapped = false; // Tool calls wrapped in JSON array [...]
// Preserved tokens for tokenizer (union of all non-empty markers)
std::vector<std::string> preserved_tokens;
};
```
### Enums
**`reasoning_mode`**: How the template handles reasoning/thinking blocks.
| Value | Description |
|----------------------|-------------------------------------------------------------------------------|
| `NONE` | No reasoning markers detected |
| `TAG_BASED` | Standard tag-based: `<think>...</think>` |
| `DELIMITER` | Delimiter-based: reasoning ends at delimiter (e.g., `[BEGIN FINAL RESPONSE]`) |
| `FORCED_OPEN` | Template ends with open reasoning tag (empty start, non-empty end) |
| `FORCED_CLOSED` | Both tags when disabled; only start tag when enabled |
| `TOOLS_ONLY` | Reasoning only appears when tool calls are present |
**`content_mode`**: How the template wraps content.
| Value | Description |
|----------------------------|------------------------------------------------------|
| `PLAIN` | No content markers |
| `ALWAYS_WRAPPED` | Content always wrapped: `<response>...</response>` |
| `WRAPPED_WITH_REASONING` | Content wrapped only when reasoning is present |
**`tool_format`**: Classification of tool call structure.
| Value | Description |
|--------------------|------------------------------------------------------------------|
| `NONE` | No tool support detected |
| `JSON_NATIVE` | Pure JSON: `{"name": "X", "arguments": {...}}` |
| `TAG_WITH_JSON` | Tag-based with JSON args: `<function=X>{...}</function>` |
| `TAG_WITH_TAGGED` | Tag-based with tagged args: `<param=key>value</param>` |
**`call_id_position`**: Where call IDs appear relative to function name and arguments (for non-JSON formats).
| Value | Description |
|----------------------------|------------------------------------------|
| `NONE` | No call ID support detected |
| `PRE_FUNC_NAME` | Before function name |
| `BETWEEN_FUNC_AND_ARGS` | Between function name and arguments |
| `POST_ARGS` | After arguments |
### marker_registry
All markers are extracted via differential analysis without hardcoded patterns:
```cpp
struct marker_registry {
// === Reasoning markers (from R1-R3) ===
std::string reasoning_start; // e.g., "<think>", "[THINK]", "<|START_THINKING|>", ""
std::string reasoning_end; // e.g., "</think>", "[BEGIN FINAL RESPONSE]", "<|END_THINKING|>"
// === Content markers (from C1) ===
std::string content_start; // e.g., "<response>", ""
std::string content_end; // e.g., "</response>", ""
// === Tool section markers (from T1-T2) ===
std::string tool_section_start; // e.g., "<tool_call>", "[TOOL_CALLS]"
std::string tool_section_end; // e.g., "</tool_call>", ""
std::string per_call_start; // e.g., "<|tool_call_begin|>" (for multi-call templates)
std::string per_call_end; // e.g., "<|tool_call_end|>"
std::string call_separator; // e.g., ",", "\n"
// === Function markers (from T3-T6) ===
std::string func_name_prefix; // e.g., "<function=", "functions."
std::string func_name_suffix; // e.g., ">", ":0"
std::string func_close; // e.g., "</function>"
std::string args_start; // e.g., "{"
std::string args_end; // e.g., "}"
// === Argument markers (from A1-A2, for TAG_WITH_TAGGED) ===
std::string arg_name_prefix; // e.g., "<param=", "<arg_key>"
std::string arg_name_suffix; // e.g., ">", "</arg_key>"
std::string arg_value_prefix; // e.g., "", "<arg_value>"
std::string arg_value_suffix; // e.g., "</param>", "</arg_value>"
std::string arg_separator; // e.g., "", "\n"
// === Call ID markers (from T7) ===
std::string call_id_prefix; // e.g., "[CALL_ID]"
std::string call_id_suffix; // e.g., "[ARGS]"
// === Special markers ===
std::string code_block_marker; // e.g., "Action:" (for markdown code block format)
std::string code_block_language; // e.g., "json"
std::string function_namespace; // e.g., "functions." (for prefixed-indexed format)
};
```
## Tool Calling Formats
The auto-parser recognizes three tool calling formats.
### JSON_NATIVE
**Structure**: The entire tool call (function name, arguments, and values) is in JSON format. There may be enclosing tags around the tool calling section.
**Characteristics**:
- Function name is a JSON field: `"name": "function_name"`
- Arguments are a JSON object: `"arguments": {"key": "value"}`
- May be wrapped in section markers like `<tool_call>...</tool_call>` or `[TOOL_CALLS]...]`
**Examples**:
Standard OpenAI-style:
```json
<tool_call>
{"name": "get_weather", "arguments": {"location": "Paris", "unit": "celsius"}}
</tool_call>
```
Mistral Nemo with array wrapper:
```json
[TOOL_CALLS]
[{"name": "calculate", "arguments": {"expr": "2+2"}}]
```
**Detection**: Function name found inside a JSON structure (determined by JSON parse attempt).
---
### TAG_WITH_JSON
**Structure**: The function name is outside the JSON structure, typically within quasi-XML markers. Arguments are still provided as a JSON object.
**Characteristics**:
- Function name appears in tag attributes: `<function=function_name>` or `<tool_name>function_name</tool_name>`
- Arguments are a JSON object following the tag
- Has closing tags: `</function>` or `</tool_call>`
- Arguments remain valid JSON
**Examples**:
Functionary v3.1:
```xml
<function=get_weather>{"location": "Paris", "unit": "celsius"}</function>
```
MiniMax:
```xml
<minimax:tool_call>
<tool_name>calculate</tool_name>
<arguments>{"expr": "2+2"}</arguments>
</minimax:tool_call>
```
**Detection**: Function name not in JSON, but arguments are JSON (args_start is `{`).
---
### TAG_WITH_TAGGED
**Structure**: Both the function name AND argument names are in XML-style tags. Argument values may be JSON or unquoted primitives depending on schema type.
**Characteristics**:
- Function name in tag: `<function=name>` or `<invoke=name>`
- Each argument has its own tag: `<param=key>value</param>`
- String values are **unquoted** (raw text content of the tag)
- Non-string values (objects, arrays, numbers, booleans) are still JSON-formatted
- Supports streaming: partial arguments can be parsed incrementally
**Examples**:
Qwen/Hermes XML format:
```xml
<function=get_weather>
<param=location>Paris</param>
<param=unit>celsius</param>
</function>
```
Note how string values (`Paris`, `celsius`) are unquoted inside the tags.
Mixed types example:
```xml
<function=calculate>
<param=expr>2+2</param>
<param=precision>2</param>
<param=options>{"round": true}</param>
</function>
```
Here:
- `expr` and `precision` are strings (unquoted)
- `options` is an object (JSON-formatted inside the tag)
**Detection**: `arg_name_prefix` is non-empty, arguments use tagged format rather than JSON object.
---
## Analysis Flow
```text
Template
|
v
differential_analyzer::analyze(tmpl)
|
|-- Phase 1: analyze_reasoning(tmpl, result)
| |-- R1: compare_reasoning_presence() — with/without reasoning_content field
| |-- R2: compare_thinking_enabled() — enable_thinking=false vs true
| '-- R3: compare_reasoning_scope() — reasoning with content vs with tools
|
|-- Phase 2: analyze_content(tmpl, result)
| '-- C1: compare_content_values() — content vs tools vs reasoning
|
|-- Phase 3: analyze_tools(tmpl, result)
| |-- T1: analyze_tool_calls() — no tools vs with tools + format classification
| |-- T2: check_per_call_markers() — per-section vs per-call markers
| |-- T3: extract_call_separator() — separator between multiple calls
| |-- T4: extract_function_markers() — func_alpha vs func_beta
| |-- T5: extract_argument_separator() — 1 arg vs 2 args
| |-- T6: extract_args_markers() — no args vs with args
| '-- T7: extract_call_id_markers() — call_id "call00001" vs "call99999"
|
|-- Phase 4: analyze_arguments(tmpl, result) [TAG_WITH_TAGGED only]
| |-- A1: extract_argument_name_markers() — "first" arg vs "second" arg
| '-- A2: extract_argument_value_markers() — value "XXXX" vs "YYYY"
|
'-- collect_preserved_tokens(result)
|
v
diff_analysis_result
|
v
universal_peg_generator::generate_parser(tmpl, inputs, analysis)
|-- build_parser(analysis, inputs, ...) — builds PEG parser arena
| |-- Reasoning parser (based on reasoning_mode)
| |-- Content parser (based on content_mode)
| '-- Tool parser (dispatches by tool_format):
| |-- build_tool_parser_json_native()
| |-- build_tool_parser_tag_json()
| '-- build_tool_parser_tag_tagged()
|
|-- Build GBNF grammar (if tools present)
'-- Set grammar triggers from tool markers
|
v
common_chat_params (prompt, parser, grammar, triggers, preserved_tokens)
```
## Entry Point
The auto-parser is invoked in `common/chat.cpp` in `common_chat_templates_apply_jinja`. A few specialized templates are handled first (Ministral/Magistral Large 3, GPT-OSS, Functionary v3.2), then the auto-parser handles everything else:
```cpp
try {
LOG_DBG("Using differential autoparser\n");
auto auto_params = universal_peg_generator::generate_parser(tmpl, params);
return auto_params;
} catch (const std::exception & e) {
LOG_WRN("Automatic parser generation failed: %s\n", e.what());
}
```
## Algorithm Details
### Core Mechanism: Differential Comparison
All analysis phases use the same factorized comparison function:
```cpp
compare_variants(tmpl, params_A, params_modifier)
```
This creates variant B by applying a modifier lambda to a copy of params_A, renders both through the template, and computes a `diff_split`:
```cpp
struct diff_split {
std::string prefix; // Common prefix between A and B
std::string suffix; // Common suffix between A and B
std::string left; // Unique to variant A
std::string right; // Unique to variant B
};
```
The diff is computed via `calculate_diff_split()`, which uses longest-common-prefix/suffix with iterative tag boundary fixing — it moves incomplete `<...>` or `[...]` markers from prefix/suffix into the left/right parts until stable.
Text is segmentized into markers and non-marker fragments using `segmentize_markers()`, which splits on `<...>` and `[...]` boundaries.
### Phase 1: Reasoning Analysis
Three comparisons extract reasoning markers and classify the reasoning mode:
**R1 — `compare_reasoning_presence()`**: Compares assistant message with vs without a `reasoning_content` field.
- Segmentizes `diff.right` to find markers around the reasoning content
- 3+ segments → `TAG_BASED` (start marker, content, end marker)
- 2 segments → `DELIMITER` (content followed by delimiter)
- Special case: markers found in prefix/suffix → `FORCED_CLOSED`
**R2 — `compare_thinking_enabled()`**: Compares `enable_thinking=false` vs `true`.
- Detects `FORCED_OPEN`: template adds opening tag when thinking enabled
- Detects `FORCED_CLOSED`: disable mode has both markers, enable mode has only start
- Handles reverse patterns (e.g., GLM-4.6 where disabled adds empty block)
**R3 — `compare_reasoning_scope()`**: Compares reasoning with content vs with tool calls.
- Detects `TOOLS_ONLY`: reasoning appears only when tool calls are present
- Extracts reasoning markers from tool call output by segmentizing
### Phase 2: Content Analysis
**C1 — `compare_content_values()`**: Compares content-only output vs tools output vs reasoning output.
- Creates two comparisons: content→tools and content→reasoning
- Finds content text position in diff to extract surrounding markers
- Classifies:
- `ALWAYS_WRAPPED`: content has start/end markers in both comparisons
- `WRAPPED_WITH_REASONING`: markers only when reasoning is present
- `PLAIN`: no wrapping markers detected
### Phase 3: Tool Call Analysis
**T1 — `analyze_tool_calls()`**: Compares no-tools vs with-tools output.
- Calls `analyze_tool_call_format()` to classify the format using the **only heuristic**: a JSON parse attempt
- `in_json_haystack()` checks whether the function name appears inside a JSON structure
- If function name is in JSON → `JSON_NATIVE` → `analyze_tool_call_format_json_native()`:
- Parses JSON structure, matches needle values to extract field names
- Detects `fun_name_is_key`, `function_field`, `name_field`, `args_field`, `id_field`, `gen_id_field`
- Detects `tools_array_wrapped` by checking for `[` before JSON
- Builds `parameter_order` by sorting fields by position
- Extracts `tool_section_start`/`tool_section_end`
- If function name is not in JSON → `analyze_tool_call_format_non_json()`:
- Segmentizes the haystack into markers and text
- Uses symmetry: counts opening markers, matches with closing markers
- Extracts `tool_section_start`, `tool_section_end`, `per_call_start`, `per_call_end`
**T2 — `check_per_call_markers()`**: Compares 1 call vs 2 calls.
- If the second call starts with `tool_section_start`, markers are per-call not per-section
- Moves tool_section markers to per_call markers, clears section markers
**T3 — `extract_call_separator()`**: Compares 1 call vs 2 calls.
- Finds separator between calls using `until_common_prefix(diff.right, ...)` with the two function names as anchors
**T4 — `extract_function_markers()`**: Compares function name "foofoo" vs "barbar".
- Finds function name in diff, segmentizes to extract prefix/suffix markers
- Extracts `func_name_prefix`, `func_name_suffix`
- Searches for closing marker after args to extract `func_close`
**T5 — `extract_argument_separator()`**: Compares 1 argument vs 2 arguments.
- Uses `until_common_prefix()` with argument names as anchors to find the separator
**T6 — `extract_args_markers()`**: Compares 0 arguments vs 1 argument.
- Uses `until_common_prefix()` and `after_common_suffix()` to find container markers
- Extracts `args_start`, `args_end`
**T7 — `extract_call_id_markers()`**: Compares call IDs "call00001" vs "call99999".
- Determines position relative to function name and arguments
- Classifies as `PRE_FUNC_NAME`, `BETWEEN_FUNC_AND_ARGS`, or `POST_ARGS`
- Extracts `call_id_prefix`, `call_id_suffix`
### Phase 4: Argument Analysis (TAG_WITH_TAGGED only)
Only runs when Phase 3 detected TAG_WITH_TAGGED or TAG_WITH_JSON format with non-JSON argument structures.
**A1 — `extract_argument_name_markers()`**: Compares argument name "first" vs "second".
- Finds common prefix of diff.left/right to extract marker structure
- Extracts `arg_name_prefix`, `arg_name_suffix`
**A2 — `extract_argument_value_markers()`**: Compares value "XXXX" vs "YYYY".
- Segmentizes prefix/suffix around value to find markers
- Extracts `arg_value_prefix`, `arg_value_suffix`
### Parser Building
The parser generator (`universal_peg_generator`) takes the analysis result and builds a PEG parser arena. The entry point is `generate_parser(tmpl, inputs)`, which:
1. Runs `differential_analyzer::analyze(tmpl)` to get the analysis result
2. Calls `build_parser(analysis, inputs, ...)` to construct the PEG parser
3. Builds a GBNF grammar if tools are present (for constrained decoding)
4. Sets grammar triggers from `tool_section_start` or `per_call_start`
#### Reasoning Parser Construction
Built inline in `build_parser()` based on `reasoning_mode`:
| Mode | Parser |
|-----------------------------------|---------------------------------------------------------------------------------------------|
| `FORCED_OPEN` / `FORCED_CLOSED` | `reasoning(until(end)) + end` — expects reasoning immediately (opening tag was in template) |
| `TAG_BASED` / `TOOLS_ONLY` | `optional(start + reasoning(until(end)) + end)` |
| `DELIMITER` | `optional(reasoning(until(end)) + end)` — no start marker, reasoning ends at delimiter |
#### Content Parser Construction
| Condition | Parser |
|------------------------------------|---------------------------------------------------------------------------|
| `json_schema` present | `reasoning + space() + content(schema(json(), ...)) + end()` |
| Tools present | Dispatches to tool parser builder |
| `ALWAYS_WRAPPED` with reasoning | `reasoning + start + content(until(end)) + end + end()` |
| `ALWAYS_WRAPPED` without reasoning | `content(until(start)) + start + content(until(end)) + end + end()` |
| Default | `reasoning + content(rest()) + end()` |
#### Tool Parser Construction
`build_tool_parser()` dispatches by `tool_format`:
**`build_tool_parser_json_native()`**: Uses the `standard_json_tools()` builder helper which has three internal modes:
- `build_json_tools_function_is_key()` — function name is the JSON key: `{"get_weather": {"location": "Paris"}}`
- `build_json_tools_nested_keys()` — nested object: `{"function": {"name": "X", "arguments": {...}}}`
- `build_json_tools_flat_keys()` — flat object: `{"name": "X", "arguments": {...}}`
Handles content wrappers, array wrapping, parallel calls, and section markers.
**`build_tool_parser_tag_json()`**: For each tool, builds:
```text
tool_open(prefix + tool_name(literal(name)) + suffix) +
call_id_section +
tool_args(schema(json(), tool_schema))
```
Wraps in per-call or section markers. Handles parallel calls.
**`build_tool_parser_tag_tagged()`**: For each tool, builds per-argument parsers:
- String types: `tool_arg_string_value(schema(until(suffix), ...))`
- JSON types: `tool_arg_json_value(schema(json(), ...))`
- Required vs optional arguments
- Arguments joined with `space()` between them
Handles `func_close`, `peek()` for partial parsing safety, and call_id sections.
All three return: `reasoning + optional(content(until(trigger))) + tool_calls + end()`
### Mapper
The `common_chat_peg_unified_mapper` maps PEG parse results (AST nodes) into `common_chat_msg` structures. Key design:
- **Buffered arguments**: Before `tool_name` is known, argument text goes to `args_buffer`; once name is set, the buffer is flushed to `current_tool->arguments`
- **`args_target()`**: Returns a reference to whichever destination is active, eliminating branching
- **`closing_quote_pending`**: Tracks whether a closing `"` needs to be appended when a string argument value is finalized
- **Quote normalization**: Python-style quotes (`'key': 'value'`) are converted to JSON (`"key": "value"`)
- **Brace auto-closing**: At tool close, unclosed `{` braces are closed automatically (tracked via `json_brace_depth()`)
## Files
| File | Purpose |
|-------------------------------------------|-------------------------------------------------------------------|
| `common/chat-auto-parser.h` | `universal_peg_generator` class and `templates_params` struct |
| `common/chat-auto-parser-generator.cpp` | Parser generator implementation |
| `common/chat-diff-analyzer.h` | Analysis result types, enums, and `differential_analyzer` class |
| `common/chat-diff-analyzer.cpp` | Differential analysis implementation |
| `common/chat-auto-parser-helpers.h/cpp` | `calculate_diff_split()`, `segmentize_markers()`, string helpers |
| `common/chat-peg-parser.h/cpp` | PEG builder and mapper classes |
| `common/chat.cpp` | Entry point: `common_chat_templates_apply_jinja()` |
| `tools/parser/debug-template-parser.cpp` | Debug tool for template analysis |
| `tools/parser/template-analysis.cpp` | Template analysis tool |
## Testing & Debugging
### Debug Tools
**Template Debugger**: `tools/parser/debug-template-parser.cpp`
- Usage: `./bin/llama-debug-template-parser path/to/template.jinja`
- Shows detected format, markers, generated parser, and GBNF grammar
**Template Analysis**: `tools/parser/template-analysis.cpp`
- Usage: `./bin/llama-template-analysis path/to/template.jinja`
**Debug Logging**: Enable with `LLAMA_LOG_VERBOSITY=2`
- Shows detailed analysis steps, pattern extraction results, and generated parser structure
**PEG Test Builder**: Fluent API for creating test cases in `tests/test-chat.cpp`:
```cpp
auto tst = peg_tester("models/templates/Template.jinja");
tst.test("input text")
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.tools({tool_json})
.parallel_tool_calls(true)
.enable_thinking(true)
.expect(expected_message)
.run();
```
### Tested Templates
The following templates have active tests in `tests/test-chat.cpp`:
| Template | Format | Notes |
| -------- | ------ | ----- |
| Ministral-3-14B-Reasoning | Reasoning | `[THINK]...[/THINK]` tags |
| NVIDIA-Nemotron-3-Nano-30B | TAG_WITH_TAGGED | Reasoning + tools |
| CohereForAI Command-R7B | JSON_NATIVE | `<\|START_THINKING\|>`/`<\|START_RESPONSE\|>` markers |
| Google Gemma 2 2B | Content only | No tool support |
| Qwen-QwQ-32B | Reasoning | Forced-open thinking |
| NousResearch Hermes 2 Pro | JSON_NATIVE | `<tool_call>` wrapper |
| IBM Granite 3.3 | JSON_NATIVE | `<think></think>` + `<response></response>` |
| ByteDance Seed-OSS | TAG_WITH_TAGGED | Custom `<seed:think>` and `<seed:tool_call>` tags |
| Qwen3-Coder | TAG_WITH_TAGGED | XML-style tool format |
| DeepSeek V3.1 | JSON_NATIVE | Forced thinking mode |
| GLM-4.6 | TAG_WITH_TAGGED | `<tool_call>name\n<arg_key>...<arg_value>...` format |
| GLM-4.7-Flash | TAG_WITH_TAGGED | Updated GLM format |
| Kimi-K2-Thinking | JSON_NATIVE | Reasoning + JSON tools |
| Apertus-8B-Instruct | JSON_NATIVE | Function name as JSON key |
| MiniMax-M2 | TAG_WITH_JSON | XML invoke with JSON args |
| NVIDIA-Nemotron-Nano-v2 | JSON_NATIVE | `<TOOLCALL>` wrapper (nested) |
| CohereForAI Command-R Plus | JSON_NATIVE | Markdown code block format |
| Mistral-Nemo-Instruct-2407 | JSON_NATIVE | `[TOOL_CALLS]` wrapper with ID field |
| Functionary v3.1 | TAG_WITH_JSON | `<function=X>` format |
| Functionary v3.2 | Specialized | `>>>` recipient delimiter (dedicated handler) |
| Fireworks Firefunction v2 | TAG_WITH_JSON | Fireworks tool format |
| DeepSeek R1 Distill (Llama/Qwen) | Reasoning | Forced-open thinking |
| llama-cpp-deepseek-r1 | Reasoning | Forced-open thinking |
| Kimi-K2 / Kimi-K2-Instruct | JSON_NATIVE | JSON tools with special markers |
| Llama 3.1/3.2/3.3 | JSON_NATIVE | Standard Llama tool format |
| OpenAI GPT-OSS | Specialized | Channel-based (dedicated handler) |
| Apriel 1.5 | JSON_NATIVE | `<tool_calls>` wrapper with JSON array |
| Apriel 1.6 Thinker | Reasoning | Implicit reasoning start |
| Mistral Small 3.2 | JSON_NATIVE | `[TOOL_CALLS]func[ARGS]{...}` with call ID |
| Devstral | JSON_NATIVE | `[TOOL_CALLS]func[ARGS]{...}` without call ID |
| StepFun 3.5 Flash | TAG_WITH_TAGGED | `<function=X><parameter=Y>` format |
## Adding Support for New Templates
To support a new template format:
1. **If it follows standard patterns** - The auto-parser should detect it automatically using the three formats (JSON_NATIVE, TAG_WITH_JSON, TAG_WITH_TAGGED)
2. **If differential analysis doesn't extract markers correctly** - Add a workaround in the workarounds array in `chat-diff-analyzer.cpp`
3. **If it needs fundamentally different handling** - Add a dedicated handler in `chat.cpp` before the auto-parser block (as done for GPT-OSS, Functionary v3.2, and Ministral)
## Edge Cases and Quirks
1. **Forced Thinking**: If `enable_thinking` is true but the model has already started a thought block (e.g., ended the prompt with `<think>`), the parser enters "forced thinking" mode where it immediately expects reasoning content.
2. **Per-Call vs Per-Section Markers**: Some templates wrap each tool call individually (`per_call_start`/`per_call_end`), others wrap the entire tool section (`tool_section_start`/`tool_section_end`). T2 disambiguates by checking if the second call in a two-call output starts with the section marker.
3. **Double Wrapping**: Some templates (e.g., Functionary) use the same string for both the tool section start and the function prefix (e.g., `<function=`). The analyzer detects this overlap and prevents double-wrapping in the generated parser.
4. **Null Content Rendering**: Some templates render `null` content as Python "None" string. The analyzer detects this and patches content to empty string.
5. **Tag Boundary Fixing**: The `calculate_diff_split()` function iteratively adjusts the prefix/suffix boundary to avoid splitting `<tag>` or `[marker]` tokens, ensuring clean extraction.
6. **Workarounds**: A workaround array in `chat-diff-analyzer.cpp` applies post-analysis patches for templates whose differential analysis produces incomplete or incorrect results (e.g., old Qwen thinking, Granite 3.3, Cohere Command-R+, Functionary, DeepSeek-R1-Distill-Qwen).
Reference in New Issue View Git Blame Copy Permalink