Shisa-K-sakurization

This is a merge of pre-trained language models created using mergekit.

An experimental merge of LoRa adapter to further boost Shisa-K-12B roleplaying capabilities with essence of PocketDoc/Dans-SakuraKaze-V1.0.0-12b.

Can occasionaly output japanese symbols, potentially mitigated by lowering TOP_P to 90 and increasing MIN_P to 0.1.

Uses ChatML.

Oh, and I am planning to use this model as layer range for next KansenSakura update

Merge Details

Merge Method

This model was merged using the Linear merge method using ./retokenized_SHK as a base.

Models Merged

The following models were included in the merge:

• ./retokenized_SHK + ./lora_Dans-SakuraKaze-V1.0.0-12b-64d

Configuration

The following YAML configuration was used to produce this model:

merge_method: linear
base_model: ./retokenized_SHK
models:
  - model: ./retokenized_SHK
    parameters: 
      weight: 0.0
  - model: ./retokenized_SHK+./lora_Dans-SakuraKaze-V1.0.0-12b-64d
    parameters: 
      weight: 1.0
dtype: bfloat16
out_dtype: bfloat16
tokenizer_source: Retreatcost/KansenSakura-Radiance-RP-12b