leafspark commited on
Commit
4df1f10
·
verified ·
1 Parent(s): fd57c06

docs: add model card

Browse files
Files changed (1) hide show
  1. README.md +55 -3
README.md CHANGED
@@ -1,3 +1,55 @@
1
- ---
2
- license: unknown
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: unknown
3
+ pipeline_tag: text-generation
4
+ language:
5
+ - en
6
+ - zh
7
+ library_name: transformers
8
+ tags:
9
+ - mergekit
10
+ - qwen2
11
+ ---
12
+
13
+ # FeatherQwen2-72B-v0.1
14
+
15
+ ## Model Description
16
+ FeatherQwen is a 72B parameter language model created through a merge of Qwen2-72B-Instruct, calme2.1-72b, and magnum-72b-v1 using `model_stock`.
17
+
18
+ ## Features
19
+ - 72 billion parameters
20
+ - Comes in 1,043 individual safetensor files
21
+ - Combines Magnum prose with Calam smarts
22
+
23
+ ## Technical Specifications
24
+
25
+ ### Architecture
26
+ - Models: Qwen2-72B-Instruct (base), calme2.1-72b, magnum-72b-v1
27
+ - Merged layers: 79
28
+ - Total tensors: 1,043
29
+
30
+ ### Tensor Distribution
31
+ - Attention layers: 333 files
32
+ - MLP layers: 333 files
33
+ - Layer norms: 166 files
34
+ - Miscellaneous (embeddings, output): 211 files
35
+
36
+ ### Merging
37
+ Custom script utilizing safetensors library.
38
+
39
+ ## Usage
40
+
41
+ ### Loading the Model
42
+ ```python
43
+ from transformers import AutoModelForCausalLM, AutoTokenizer
44
+ import torch
45
+
46
+ model = AutoModelForCausalLM.from_pretrained("leafspark/FeatherQwen2-72B-v0.1",
47
+ device_map="auto",
48
+ torch_dtype=torch.float16)
49
+ tokenizer = AutoTokenizer.from_pretrained("leafspark/FeatherQwen2-72B-v0.1")
50
+
51
+ ```
52
+
53
+ ### Hardware Requirements
54
+ - Minimum ~140GB of storage
55
+ - ~140GB VRAM