Mungert

/

KAT-Dev-GGUF

like 0

Ok it will take some time. My vps server is slow. This is the template thats in the config : "chat_template": "{%- if tools %}\n {{- '<|im_start|>system\n' }}\n {%- if messages[0]['role'] == 'system' %}\n {{- messages[0]['content'] }}\n {%- else %}\n {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n {%- endif %}\n {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }}\n {%- for tool in tools %}\n {{- "\n" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n<|im_end|>\n" }}\n{%- else %}\n {%- if messages[0]['role'] == 'system' %}\n {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}\n {%- else %}\n {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}\n {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}\n {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}\n {%- elif message.role == "assistant" %}\n {{- '<|im_start|>' + message.role }}\n {%- if message.content %}\n {{- '\n' + message.content }}\n {%- endif %}\n {%- for tool_call in message.tool_calls %}\n {%- if tool_call.function is defined %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\n\n{"name": "' }}\n {{- tool_call.name }}\n {{- '", "arguments": ' }}\n {{- tool_call.arguments | tojson }}\n {{- '}\n' }}\n {%- endfor %}\n {{- '<|im_end|>\n' }}\n {%- elif message.role == "tool" %}\n {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\n\n' }}\n {{- message.content }}\n {{- '\n' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}\n {{- '<|im_end|>\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\n' }}\n{%- endif %}\n",
.... the model is based on Qwen2, the template looks pretty standard.

I used template and it is working fine in lmstudio too
https://huggingface.co/Kwaipilot/KAT-Dev/blob/main/chat_template.jinja

I wonder what adapter I need to get PCIe 4.0 5.0 16x To SlimSAS SFF-8654 ? There is one with external ports that I need, hmm any info on that?

I used template and it is working fine in lmstudio too
https://huggingface.co/Kwaipilot/KAT-Dev/blob/main/chat_template.jinja

I wonder what adapter I need to get PCIe 4.0 5.0 16x To SlimSAS SFF-8654 ? There is one with external ports that I need, hmm any info on that?

This is off topic : https://www.eproductsolutions.com/adapters-and-converters/slimeline-sas-sff-8654

There you go Bartowski has already done the whole thing : https://huggingface.co/bartowski/Kwaipilot_KAT-Dev-72B-Exp-GGUF...before my server has even created the imatrix file

I trust only you to do the quant, you do way better with your script and I tested the 33B version from Bartowski and others too they were not working properly, errors with tool calls etc.
So please take time, but do make your version please.

Somebody still needs to dig into tool calling problems with roocode.

My lastest template fix (or rather KAT-DEV fix itself) still gives these errors:

{% macro render_extra_keys(json_dict, handled_keys) %}
{%- if json_dict is mapping %}
{%- for json_key in json_dict if json_key not in handled_keys %}
{%- if json_dict[json_key] is mapping or (json_dict[json_key] is sequence and json_dict[json_key] is not string) %}
{{- '\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | tojson | safe) ~ '</' ~ json_key ~ '>' }}
{%- else %}
{{- '\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | string) ~ '</' ~ json_key ~ '>' }}
{%- endif %}
{%- endfor %}
{%- endif %}
{% endmacro %}

{%- if messages[0]["role"] == "system" %}
{%- set system_message = messages[0]["content"] %}
{%- set loop_messages = messages[1:] %}
{%- else %}
{%- set loop_messages = messages %}
{%- endif %}

{%- if not tools is defined %}
{%- set tools = [] %}
{%- endif %}

{%- if system_message is defined %}
{{- "system\n" + system_message + "" }}
{%- else %}
{%- if tools is iterable and tools | length > 0 %}
{{- "system\nYou are a helpful AI assistant that can interact with a computer to solve tasks." }}
{%- endif %}
{%- endif %}

{%- if tools is iterable and tools | length > 0 %}
{{- "\n\n# Tools\n\nYou have access to the following functions:\n\ntools>" }}
{%- for tool in tools %}
{%- if tool.function is defined %}
{%- set tool = tool.function %}
{%- endif %}
{{- "\n\n" ~ tool.name ~ "" }}
{%- if tool.description is defined %}
{{- '\n' ~ (tool.description | trim) ~ '' }}
{%- endif %}
{{- '\n' }}
{%- if tool.parameters is defined and tool.parameters is mapping and tool.parameters.properties is defined and tool.parameters.properties is mapping %}
{%- for param_name, param_fields in tool.parameters.properties|items %}
{{- '\n' }}
{{- '\n' ~ param_name ~ '' }}
{%- if param_fields.type is defined %}
{{- '\n' ~ (param_fields.type | string) ~ '' }}
{%- endif %}
{%- if param_fields.description is defined %}
{{- '\n' ~ (param_fields.description | trim) ~ '' }}
{%- endif %}
{%- set handled_keys = ['name', 'type', 'description'] %}
{{- render_extra_keys(param_fields, handled_keys) }}
{{- '\n' }}
{%- endfor %}
{%- endif %}
{% set handled_keys = ['type', 'properties'] %}
{{- render_extra_keys(tool.parameters, handled_keys) }}
{{- '\n' }}
{%- set handled_keys = ['type', 'name', 'description', 'parameters'] %}
{{- render_extra_keys(tool, handled_keys) }}
{{- '\n' }}
{%- endfor %}
{{- "\n" }}
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n\n\n\n\n\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...> block must be nested within XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your content, do not tell the function calls\n' }}
{%- endif %}

{%- if system_message is defined %}
{{- '\n' }}
{%- else %}
{%- if tools is iterable and tools | length > 0 %}
{{- '\n' }}
{%- endif %}
{%- endif %}

{%- for message in loop_messages %}
{%- if message.role == "assistant" and message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}
{{- '' + message.role }}
{%- if message.content is defined and message.content is string and message.content | trim | length > 0 %}
{{- '\n' + message.content | trim + '\n' }}
{%- endif %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '\n\n<function=' + tool_call.name + '>\n' }}
{%- if tool_call.arguments is defined %}
{%- for args_name, args_value in tool_call.arguments|items %}
{{- '<parameter=' + args_name + '>\n' }}
{%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
{{- args_value }}
{{- '\n\n' }}
{%- endfor %}
{%- endif %}
{{- '\n' }}
{%- endfor %}
{{- '\n' }}
{%- elif message.role == "user" %}
{{- 'user\n' + message.content + '\n' }}
{%- elif message.role == "system" %}
{{- 'system\n' + message.content + '\n' }}
{%- elif message.role == "assistant" %}
{{- 'assistant\n' + message.content + '\n' }}
{%- elif message.role == "tool" %}
{%- set tool_content = message.content %}
{%- if tool_content.startswith('[tool_call] ') %}
{%- set tool_content = tool_content[10:] | trim }
{%- elif tool_content.startswith('[tool_name] ') %}
{%- set tool_content = tool_content[11:] | trim }
{%- endif %}
{{- 'tool>\n' + tool_content + '\n\n' }}
{%- else %}
{{- '' + message.role + '\n' + message.content + '\n' }}
{%- endif %}
{%- endfor %}

{%- if add_generation_prompt %}
{{- 'assistant\n' }}
{%- endif %}

when debug mode listing for file that does not exist:

Error
Roo tried to use search_files without value for required parameter 'path'. Retrying...

code mode

Roo said
Now I'll create the manifest.json file compatible with Chrome v109. Chrome v109 requires manifest version 3, which has several important differences from version 2:

Checkpoint

Error
Roo tried to use write_to_file without value for required parameter 'path'. Retrying...

roo code mode tries to read folder as file:

Roo said
Now I need to update the background.js file. In Chrome v109 with Manifest V3, the background script runs as a service worker, which has some differences from the previous event pages. The current code should work mostly as-is, but I need to make sure it follows service worker best practices:

Roo wants to read this file "project folder"

Error
Roo tried to use read_file without value for required parameter 'args (containing valid file paths)'. Retrying...

Error
Roo tried to use apply_diff without value for required parameter 'path'. Retrying...

Error
Roo tried to use apply_diff without value for required parameter 'path'. Retrying...

update

It seems no errors are present when running:

Jan.ai v0.7.1 recent versions are unstable errors above
Mungert-kat-Dev https://huggingface.co/Mungert/KAT-Dev-GGUF
No jinja template at all.
Context set to 90k
GPU layers 100
Others settings to default

roocode v3.28.15
vscode:
Version: 1.105.1 (user setup)
Commit: 7d842fb85a0275a4a8e4d7e040d2625abbf7f084
Date: 2025-10-14T22:33:36.618Z
Electron: 37.6.0
ElectronBuildId: 12502201
Chromium: 138.0.7204.251
Node.js: 22.19.0
V8: 13.8.258.32-electron.0
OS: Windows_NT x64 10.0.26100

If you decide to test Jan.ai with other jinja templates you need to restart it and vscode close apps and start fresh, otherwise false positives happen.

No template has nice formatting, somehow it gets removed with even official template

This needs permanent fix

I can't see what is different between my GGUF files and Bartowski's in terms of the chat template we have both used the template that comes with original model. I only tweak the quants. I have done the model you requested with the original models template : ReadyForQuantum/KAT-Dev-72B-Exp-GGUF . It might be helpful to the model maintiners if you pass on your findings :)

Thank you for your hard work and time. I don't know either I just shared my observations. Everything is updated every day, and it can be bugs in software like Roocode or other agent.
I extracted template from your KAT-Dev-32B Q8 and it seem exactly the same as official, if I add it to Jan.ai it changes the output of model, no fancy icons, different output etc.
I tried to tweak it with Ai and it does not work reliably, then I downgraded the roocode to older v3.28.15 and it started to work, no more errors reading/writing files or doing inline diff edits. I registered bugs in github for Jan.ai and Roocode.

Somehow it does not work reliably with LMstudio, Cline, KiloCode.
The Mungert-Qwen3-Coder-30B-A3B-Instruct-GGUF somehow works only with Cline. If using Jan.ai then you need to paste jinja template, if LMstudio Jinja tempalate is required too.

The Jinja template was not required (default worked) for BasedBase Qwen3-30B-A3B-Instruct-Coder-480B-Distill-v2-Q8_0.gguf
This was the only no error working model of Qwen3-30B-A3B in LMstudio. It is a shame since author was removed from HuggingFace, guess no democracy there.
The model might be the same but it worked, and official did not. If the only difference was embedded Jinja template that made it work, so be it.

IMHO Until we start seeing models in torrents the Ai is not valuable yet enough. We need to see models in torrents pirated obviously, that will mean scene is mature and gives value.

Thank you for your work again.