File size: 4,195 Bytes
66b8b82 b8a8a59 66b8b82 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 |
---
datasets:
- glaiveai/reasoning-v1-20m
language:
- en
base_model:
- facebook/galactica-1.3b
tags:
- reasoning
- text-generation-inference
- medical
---
## What is Galactic Reasoning?
The Galactic Reasoning adapters are a collection of LoRA adapters, trained for the various sizes of the Facebook/Galactica models. These LoRAs enable the OPT architecture based Galactica models to use reasoning, inspired by more modern models like DeepSeek and OpenAI's O3.
To achieve this, the [glaiveai/reasoning-v1-20m](https://huggingface.co/datasets/glaiveai/reasoning-v1-20m) dataset was used for both training and evalulation of points.
| Size | Parameters | Galactic Reasoning Adapter |
|:-----------:|:-----------:|:--------------------------:|
| `mini` | 125 M | Too few neurons for reason |
| `base` | 1.3 B | You are here :) |
| `standard` | 6.7 B | Coming Soon™ |
| `large` | 30 B | Coming Soon™ |
| `huge` | 120 B | Short of a GPU grant, unlikely to happen. |
## How were these adapters developed?
In addition to the adapters, I will be releasing the training script I used soon on GitHub. The script supports the finetuning of a specified base model with a specified dataset for any number of steps, using a wide range of optional quantization.
Included in the GitHub training repo will be a batch file to replicate the exact arguments and seed passed to said script used to create this adapter.
## How do I prompt this galactic thinker?
A proper inference script will be provided eventually™ but for the time being, refer to the following code snippet.
```python
import torch
from peft import PeftModel
from transformers import AutoTokenizer, OPTForCausalLM
ADAPTER_PATH = "C:\\Users\\TitleOS\Downloads\GalacticReasoning-1.3b" # Change to point to your downloaded adapter of course.
BASE_MODEL_NAME = "facebook/Galactica-1.3b" # Use the right adapter for the right sized Galactica.
new_special_tokens = ["<think>", "</think>"]
new_pad_token = "<PAD>"
model = OPTForCausalLM.from_pretrained(
BASE_MODEL_NAME,
load_in_8bit=False,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_NAME)
print(f"Original vocab size: {len(tokenizer)}")
# Add the special tokens and the new pad token
special_tokens_dict = {'additional_special_tokens': new_special_tokens, 'pad_token': new_pad_token}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
print(f"Number of tokens added: {num_added_toks}")
print(f"New vocab size: {len(tokenizer)}")
# Resize the model's token embeddings to match the new tokenizer
model.resize_token_embeddings(len(tokenizer))
print("Resized model's token embeddings to match the new tokenizer. This is critical for the model to recognize the thinking tokens and the new pad token.")
print(f"New embed_tokens shape: {model.get_input_embeddings().weight.shape}")
print(f"New lm_head shape: {model.get_output_embeddings().weight.shape}")
print("\nLoading adapter...")
model.load_adapter(ADAPTER_PATH, adapter_name="default", device_map="auto")
print("Adapter loaded successfully!")
def evaluate(instruction, input=None):
prompt = "Do androids dream of electric sheep?"
inputs = tokenizer(prompt, return_tensors="pt")
input_ids = inputs["input_ids"].to(model.device)
generation_output = model.generate(
input_ids=input_ids,
return_dict_in_generate=True,
output_scores=True,
do_sample=True,
max_length=1024,
temperature=0.7,
top_k=50,
top_p=0.95,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id
)
s = generation_output.sequences[0]
output = tokenizer.decode(s, skip_special_tokens=False)
print(output)
```
## Credits
* Credit to Meta/Facebook for the Galactica OPT Based models.
* Credit to GlaiveAi for the reasoning-v1-20m dataset.
* Finally, credit to my highly overworked Tesla M40 who ran for days straight to produce this. |