---
quantized_by: bobchenyx
base_model:
- deepseek-ai/DeepSeek-V3-0324
base_model_relation: quantized
license: mit
tags:
- deepseek_v3
- deepseek
- transformers
- GGUF
pipeline_tag: text-generation
---

## llama.cpp Quantizations of DeepSeek-V3-0324 (MLA version)

This page is going to be deprecated. For other quantized versions, please refer to [moxin-org/DeepSeek-V3-0324-Moxin-GGUF](https://huggingface.co/moxin-org/DeepSeek-V3-0324-Moxin-GGUF) for more details.


All quants made based on [moxin-org/CC-MoE](https://github.com/moxin-org/CC-MoE).
```
- IQ1_S : 129.94 GiB (1.66 BPW)
- IQ1_M : 144.24 GiB (1.85 BPW)
- Q2_K_L : 222.01 GiB (2.84 BPW)
- Q4_K_L : 381.64 GiB (4.89 BPW)
```

## Smallest Compression (103GB) 

For our smallest compressed version. Please refer to 
[tflsxyy/DeepSeek-V3-0324-E192](https://huggingface.co/tflsxyy/DeepSeek-V3-0324-MoE-Pruner-E192-bf16) 
and [bobchenyx/DeepSeek-V3-0324-508B-A32B-MLA-GGUF](https://huggingface.co/bobchenyx/DeepSeek-V3-0324-508B-A32B-MLA-GGUF)
for more details.

---
## Download Guide

```
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "bobchenyx/DeepSeek-V3-0324-MLA-GGUF",
    local_dir = "bobchenyx/DeepSeek-V3-0324-MLA-GGUF",
    allow_patterns = ["*IQ1_M*"],
)
```