SicariusSicariiStuff

/

Sweet_Dreams_12B

like 17

I'll make iMatrix HA, for normal iMatrix I hope that one of the quanters will do them, space is sadly limited now for me.

NL_HA and ARM_HA being uploaded now :)

NL_HA and ARM_HA being uploaded now :)

Thx

Also - does my account count as having space? I could upload something on my account, but I've never done it. I also have 50GB of free space on Google Drive if that helps.

Just to add. For research purposes I do study IQ4_NL, ARM Q4_0, ARM_HA, NL_HA - Q4 and Q8.

This is currently not a big issue, but it will be in the longer term, it's a matter of scale.
Meanwhile I'll just be offering a bit less quant options to save space (so EXL3 quants that not many people use won't be uploaded, or at least not with that many bpw options, etc)

Thank you for the willingness to help though, it is appreciated.

I don't mind. Especially since I do plan on:

1. Benchmarking certain models for use on Phones. Again, I was a bit surprised when I saw Q8_0 with High Attention be as fast as Q6 with High Attention.
2. I already benched a lot of models, not using Layla AI's inbuilt benchmark tool. Run to run variance is much smaller.
3. I'm a horrible programmer. Last time I programmed seriously was in C/C++ 25 years ago. I managed to do a simple calculator. Last time (and only time) I produced any serious software was on Borland Turbo-Pascal - it was missing a lot of it's libraries, so I wrote a mouse driver to new back then Windows 95. I guess I would need a GitHub repo.
4. I want to write 100% human made stories. Including naughty ones. Because I plan to finetune one of your models (Unaligned). I just don't know how to do this yet and if I should do full finetune or LoRA type. Not that I would know how to do this. Heck, I would need to learn how to give you a credit.
5. I found out why Lexi Q4 was giving me so few refusals in LM Studio. I didn't know that you could set up a Llama 3 prompt, so I went with default LM Studio - Template (Jinja), which looks like this:

{{- bos_token }}
{%- if custom_tools is defined %}
{%- set tools = custom_tools %}
{%- endif %}
{%- if not tools_in_user_message is defined %}
{%- set tools_in_user_message = true %}
{%- endif %}
{%- if not date_string is defined %}
{%- set date_string = "26 Jul 2024" %}
{%- endif %}
{%- if not tools is defined %}
{%- set tools = none %}
{%- endif %}

{#- This block extracts the system message, so we can slot it into the right place. #}
{%- if messages[0]['role'] == 'system' %}
{%- set system_message = messages[0]['content']|trim %}
{%- set messages = messages[1:] %}
{%- else %}
{%- set system_message = "" %}
{%- endif %}

{#- System message + builtin tools #}
{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
{%- if builtin_tools is defined or tools is not none %}
{{- "Environment: ipython\n" }}
{%- endif %}
{%- if builtin_tools is defined %}
{{- "Tools: " + builtin_tools | reject('equalto', 'code_interpreter') | join(", ") + "\n\n"}}
{%- endif %}
{{- "Cutting Knowledge Date: December 2023\n" }}
{{- "Today Date: " + date_string + "\n\n" }}
{%- if tools is not none and not tools_in_user_message %}
{{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
{{- "Do not use variables.\n\n" }}
{%- for t in tools %}
{{- t | tojson(indent=4) }}
{{- "\n\n" }}
{%- endfor %}
{%- endif %}
{{- system_message }}
{{- "<|eot_id|>" }}

{#- Custom tools are passed in a user message with some extra guidance #}
{%- if tools_in_user_message and not tools is none %}
{#- Extract the first user message so we can plug it in here #}
{%- if messages | length != 0 %}
{%- set first_user_message = messages[0]['content']|trim %}
{%- set messages = messages[1:] %}
{%- else %}
{{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
{%- endif %}
{{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
{{- "Given the following functions, please respond with a JSON for a function call " }}
{{- "with its proper arguments that best answers the given prompt.\n\n" }}
{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
{{- "Do not use variables.\n\n" }}
{%- for t in tools %}
{{- t | tojson(indent=4) }}
{{- "\n\n" }}
{%- endfor %}
{{- first_user_message + "<|eot_id|>"}}
{%- endif %}

{%- for message in messages %}
{%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
{{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
{%- elif 'tool_calls' in message %}
{%- if not message.tool_calls|length == 1 %}
{{- raise_exception("This model only supports single tool-calls at once!") }}
{%- endif %}
{%- set tool_call = message.tool_calls[0].function %}
{%- if builtin_tools is defined and tool_call.name in builtin_tools %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
{{- "<|python_tag|>" + tool_call.name + ".call(" }}
{%- for arg_name, arg_val in tool_call.arguments | items %}
{{- arg_name + '="' + arg_val + '"' }}
{%- if not loop.last %}
{{- ", " }}
{%- endif %}
{%- endfor %}
{{- ")" }}
{%- else %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
{{- '{"name": "' + tool_call.name + '", ' }}
{{- '"parameters": ' }}
{{- tool_call.arguments | tojson }}
{{- "}" }}
{%- endif %}
{%- if builtin_tools is defined %}
{#- This means we're in ipython mode #}
{{- "<|eom_id|>" }}
{%- else %}
{{- "<|eot_id|>" }}
{%- endif %}
{%- elif message.role == "tool" or message.role == "ipython" %}
{{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
{%- if message.content is iterable and message.content is not string %}
{{- message.content | tojson }}
{%- else %}
{{- message.content }}
{%- endif %}
{{- "<|eot_id|>" }}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
{%- endif %}

6. I plan to obtain Red Magic 11 Pro 24GB+1TB, probably just after New Years. Though cost is just 30-50% lower than retail.
7. I plan to make an educational video on Various quants and how various versions of quants are accelerated on Qualcomm's silicon.