File size: 4,600 Bytes
693de0b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 |
---
license: apache-2.0
base_model:
- mistralai/Mistral-7B-Instruct-v0.3
pipeline_tag: text2text-generation
---
# Elastic models
Elastic models are the models produced by TheStage AI ANNA: Automated Neural Networks Accelerator.
ANNA allows you to control model size, latency and quality with a simple slider movement. For each model, ANNA produces a series of optimized models:
* __XL__: Mathematically equivalent neural network, optimized with our DNN compiler.
* __L__: Near lossless model, with less than 1% degradation obtained on corresponding benchmarks.
* __M__: Faster model, with accuracy degradation less than 1.5%.
* __S__: The fastest model, with accuracy degradation less than 2%.
__Goals of elastic models:__
* Provide flexibility in cost vs quality selection for inference
* Provide clear quality and latency benchmarks
* Provide interface of HF libraries: transformers and diffusers with a single line of code
* Provide models supported on a wide range of hardware, which are pre-compiled and require no JIT.
> It's important to note that specific quality degradation can vary from model to model. For instance, with an S model, you can have 0.5% degradation as well.
## Inference
To infer our models, you just need to replace `transformers` import with `elastic_models.transformers`:
```python
import torch
from transformers import AutoTokenizer
from elastic_models.transformers import AutoModelForCausalLM
model_name = "mistralai/Mistral-7B-Instruct-v0.3"
token = ''
device = torch.device("cuda")
tokenizer = AutoTokenizer.from_pretrained(model_name, token=token)
model = AutoModelForCausalLM.from_pretrained(
model_name,
token=token,
cache_dir="/mnt/rnd/huggingface_cache",
torch_dtype=torch.bfloat16,
attn_implementation="sdpa"
).to(device)
model.generation_config.pad_token_id = tokenizer.eos_token_id
prompt = "Describe basics of DNNs quantization."
inputs = tokenizer(prompt, return_tensors="pt")
inputs.to(device)
generate_ids = model.generate(**inputs, max_length=500)
input_len = inputs['input_ids'].shape[1]
generate_ids = generate_ids[:, input_len:]
output = tokenizer.batch_decode(
generate_ids,
skip_special_tokens=True,
clean_up_tokenization_spaces=False
)[0]
print(f"# Q:\n{prompt}\n")
print(f"# A:\n{output}\n")
```
---
### System requirements
__GPUs__: H100, L40s
__OS__: Linux #TODO
__Python__: 3.10-3.12
---
### Installation
```shell
pip install thestage
pip install elastic_models
```
Then go to app.thestage.ai, login and generate API token from your profile page.
Set up API token as follows:
```shell
thestage config set --api-token <YOUR_API_TOKEN>
```
Congrats, now you can use accelerated models!
## Benchmarks
Benchmarking is one of the most important procedures during model acceleration. We aim to provide clear performance metrics for models using our algorithms. The `W8A8, int8 column` indicates that we applied W8A8 quantization with int8 data type to all linear layers and used the same calibration data as for ANNA. The S model achieves practically identical speed but much higher quality, as ANNA knows how to improve quantization quality on sensitive layers!
### Quality benchmarks
For quality evaluation we have used: #TODO link to github
| Metric/Model | S | M | L | XL | Original | W8A8, int8 |
|---------------|---|---|---|----|----------|------------|
| MMLU | 0 | 0 | 0 | 0 | 0 | 0 |
| PIQA | 0 | 0 | 0 | 0 | 0 | 0 |
| Arc Challenge | 0 | 0 | 0 | 0 | 0 | 0 |
| Winogrande | 0 | 0 | 0 | 0 | 0 | 0 |
> __MMLU__: Evaluates/shows ...
> __MMLU__: Evaluates/shows ...
> __Arc Challenge__: Evaluates/shows ...
> __PIQA__: Evaluates/shows ...
### Latency benchmarks
We have profiled models in different scenarios:
__100 input/300 output tok/s__:
| GPU/Model | S | M | L | XL | Original | W8A8, int8 |
|-----------|-----|---|---|----|----------|------------|
| H100 | 189 | 0 | 0 | 0 | 48 | 0 |
| L40s | 79 | 0 | 0 | 0 | 42 | 0 |
__1000 input/1000 output tok/s__:
| GPU/Model | S | M | L | XL | Original | W8A8, int8 |
|-----------|-----|---|---|----|----------|------------|
| H100 | 189 | 0 | 0 | 0 | 48 | 0 |
| L40s | 79 | 0 | 0 | 0 | 42 | 0 |
## Links
* __Platform__: [app.thestage.ai](app.thestage.ai)
* __Elastic models Github__: [app.thestage.ai](app.thestage.ai)
* __Subscribe for updates__: [TheStageAI X](https://x.com/TheStageAI)
* __Contact email__: contact@thestage.ai
|