Update README.md

README.md

@@ -4,106 +4,185 @@ language:
 - en
 pipeline_tag: text-generation
 tags:
-- diffusion
-- text generation
+- text diffusion model
+- language model
 - code generation
 ---
-# CoDA-v0-Base
+# CoDA: Coding LM via Diffusion Adaptation
 
-## Overview 🎯
-CoDA is Salesforce AI Research's open diffusion language model.
+**CoDA-1.7B** is a lightweight diffusion language model for code generation developed by Salesforce AI Research. Unlike traditional autoregressive models, CoDA leverages discrete diffusion processes to enable bidirectional context understanding and efficient code completion.
 
+- 📄 [Technical Report](https://github.com/SalesforceAIResearch/CoDA/blob/main/technical_report.pdf)
+- 💻 [Code Repository](https://github.com/SalesforceAIResearch/CoDA/)
 
-[Technical Report](https://github.com/SalesforceAIResearch/CoDA/blob/main/technical_report.pdf)
+## 📊 Model Details
 
-[Code](https://github.com/SalesforceAIResearch/CoDA/)
+- **Model Size**: 1.7B parameters
+- **Architecture**: Diffusion-based language model
+- **Training**: TPU-based pre-training with GPU fine-tuning
+- **Primary Use**: Code generation and completion tasks
 
-The code repo contains a unified training pipeline from pre-training to post-training, evaluation harnesses, and a simple Fast-API based serving backend.
+## ✨ Key Features
 
+- **Bidirectional Context**: Diffusion modeling enables understanding of both past and future tokens
+- **Confidence-Guided Sampling**: Maintains competitive inference latency through intelligent sampling
+- **Lightweight Design**: Achieves strong performance with fewer parameters than comparable models
+- **Open Training Pipeline**: Fully reproducible training from pre-training to fine-tuning
 
+## 📈 Performance
 
-## Requirements 📦
-```
-torch==2.8.0
-transformers>=4.47.1
-flash-attn==2.8.3
-```
+CoDA-1.7B-Instruct demonstrates competitive performance on standard code generation benchmarks:
+
+| Model | HumanEval | HumanEval+ | MBPP | MBPP+ | EvalPlus |
+|-------|-----------|------------|------|-------|----------|
+| **CoDA-Base** | 29.3 | 23.8 | 35.2 | 46.0 | 34.9 |
+| **CoDA-Instruct** | **54.3** | **47.6** | 47.2 | **63.2** | **55.4** |
+| Dream-Base | 56.7 | 50.0 | 68.7 | 57.4 | 53.7 |
+| Dream-7B-Instruct | 57.9 | 53.7 | 68.3 | 56.1 | 54.9 |
+| LLaDA-8B-Instruct | 35.4 | 31.7 | 31.5 | 28.6 | 30.2 |
+
+**🎯 Key Finding**: CoDA-1.7B-Instruct matches or surpasses diffusion models up to 7B parameters while maintaining significantly lower computational requirements. CoDA offers an advantageous balance between inference speed and accuracy compared to larger diffusion models.
+
+## 🎓 Training Methodology
+
+CoDA employs a three-stage training process:
+
+*Three-stage training: (1) Pre-training with bidirectional masking, (2) Post-training with instruction format, (3) Inference with progressive denoising.*
+
+## 🛠️ Usage
+
+### 🚀 Quick Start
 
-## Quickstart 🚀
-Here is a code snippet for loading the model, tokenizer and run unmasking for a partially finished code.
 ```python
-import torch
-from transformers import AutoModel, AutoTokenizer
-
-model_name = "Salesforce/CoDA-v0-Base"
-device = "cuda"
-
-model = AutoModel.from_pretrained(model_name, torch_dtype=torch.bfloat16, trust_remote_code=True).to(device)
-tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
-model.eval()
-
-prompt = """```python
-from typing import List
-
-class Solution:
-    def twoSum(self, nums: List[int], target: int) -> List[int]:
-        # Create a dictionary to store the numbers and their indices
-        num_to_index = {}
-
-        # Iterate over the list of numbers
-        for index, num in enumerate(nums):
-            # Calculate the complement
-            complement = target - num
-
-            # Check if the complement is already in the dictionary
-            if complement in num_to_index:
-                # If found, return the indices of the complement and the current number
-                return [num_to_index[complement], index]
-
-            # Otherwise, add the current number and its index to the dictionary
-            num_to_index[num] = index
-```"""
-input_ids = tokenizer.encode(prompt, return_tensors="pt")
-mask = torch.rand(input_ids.shape) < 0.4
-masked_input_ids = input_ids.clone()
-masked_input_ids[mask] = tokenizer.mask_token_id
-generated_ids = model.diffusion_generate(
-    inputs=masked_input_ids.to(model.device),
-    max_new_tokens=1,
-    steps=128,
-    top_p=0.95,
-    temperature=0.2,
-    alg="entropy",
-    alg_temp=0.2,
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_name = "Salesforce/CoDA-v0-Instruct"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name)
+
+# Generate code
+prompt = "Write a Python function to calculate fibonacci numbers"
+inputs = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(
+    **inputs,
+    max_tokens=256,
+    diffusion_steps=128,
+    temperature=0.0
 )
-generated_ids = [
-    output_ids[:-1] for output_ids in generated_ids
-]
-unmasked_output = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+print(tokenizer.decode(outputs[0]))
+```
+
+### 🚀 Deployment
+
+For production deployment, we provide serving with OpenAI-compatible APIs:
 
+```bash
+# Clone the repository
+git clone https://github.com/SalesforceAIResearch/CoDA
+cd CoDA
 
+# Set up environment
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r serving/requirements.txt
+
+# Export your Hugging Face token
+export HF_TOKEN="hf_..."
+
+# Start the server
+bash serving/fast-api/start_server.sh
 ```
 
-## Benchmark 📊
-Comparison of code-generation performance across standard and plus-enhanced benchmarks. Evalplus is computed as the mean pass@1 on enhanced variants. Bold marks results where CoDA produces the strongest diffusion-model performance.
+The server will listen on `http://localhost:8000`.
 
-| Model | Humaneval Instruct | Humaneval Plus | MBPP Instruct | MBPP Plus | Evalplus |
-| --- | --- | --- | --- | --- | --- |
-| CoDA-Base | 29.3 | 23.8 | 35.2 | 46.0 | 34.9 |
-| CoDA-Instruct | 54.3 | 47.6 | 47.2 | **63.2** | **55.4** |
-| Dream-Base | 56.7 | 50.0 | 68.7 | 57.4 | 53.7 |
-| Dream-7B-Instruct | 57.9 | 53.7 | 68.3 | 56.1 | 54.9 |
-| LLaDA-8B-Instruct | 35.4 | 31.7 | 31.5 | 28.6 | 30.2 |
-| Qwen3-1.7B | 66.5 | 61.6 | 46.2 | 65.9 | 63.8 |
-| Qwen2.5-Coder-1.5B | 43.9 | 36.6 | 69.2 | 58.6 | 47.6 |
-| Qwen2.5-Coder-1.5B-Instruct | 70.7 | 66.5 | 69.2 | 59.4 | 62.3 |
-| Gemma-3-1B-it | 39.6 | 35.4 | 39.4 | 63.5 | 49.5 |
-| LLaMA-3.2-1B-Instruct | 35.4 | 31.1 | 24.4 | 53.7 | 42.4 |
+### 💬 Interactive CLI
+
+```bash
+python serving/fast-api/chat_cli.py \
+  --base-url http://localhost:8000 \
+  --model Salesforce/CoDA-v0-Instruct \
+  --stream \
+  --show-meta
+```
+
+### ⚙️ Generation Hyperparameters
 
-## Deployment 🛠️
-Checkout our [Deployment Guide](https://github.com/SalesforceAIResearch/CoDA?tab=readme-ov-file#deployment-guide-%EF%B8%8F)!
+Customize generation behavior with environment variables:
 
-## Citation 📚
+```bash
+export MAX_TOKENS=512          # Maximum tokens to generate
+export TEMPERATURE=0.7         # Sampling temperature
+export TOP_P=0.9              # Nucleus sampling threshold
+export STEPS=128              # Number of diffusion steps
+export ALG="entropy"          # Sampling algorithm
+export ALG_TEMP=0.1           # Algorithm temperature
+export BLOCK_LENGTH=32        # Block size for processing
 ```
-coming soon
-```
+
+**Recommended Settings**:
+- **Fast inference**: `STEPS=64`, `TEMPERATURE=0.0`
+- **Quality generation**: `STEPS=128`, `TEMPERATURE=0.7`, `TOP_P=0.9`
+- **High quality**: `STEPS=256`, `TEMPERATURE=0.5`, `TOP_P=0.95`
+
+## 🔧 Training from Scratch
+
+The complete training pipeline is available in our [repository](https://github.com/SalesforceAIResearch/CoDA):
+
+```bash
+# Clone the repository
+git clone https://github.com/SalesforceAIResearch/CoDA
+cd CoDA
+```
+
+### 🧠 Pre-training on TPU
+```bash
+# Configure TPU environment
+cd pre-train
+cp env.example .env  # Add your TPU metadata
+bash setup_tpu.sh
+
+# Launch pre-training
+bash recipes/midtrain_v4_512.sh
+```
+
+### 🎯 Supervised Fine-tuning
+```bash
+# Set up fine-tuning environment
+cd post-train/LLaMA-Factory
+pip install -r requirements.txt
+
+# Configure dataset and run fine-tuning
+bash ../../run_sft.sh
+```
+
+### 📊 Evaluation
+```bash
+cd evaluation/lm_eval
+bash eval_mbpp_humaneval.sh
+```
+## 📚 Citation
+
+Technical report coming soon. For now, please cite:
+
+```bibtex
+@misc{coda2025,
+  title={CoDA: Coding LM via Diffusion Adaptation},
+  author={Chen, Haolin and Wang, Shiyu and Qin, Can and Pang, Bo and Liu, Zuxin and Qiu, Jielin and Zhang, Jianguo and Zhou, Yingbo and Chen, Zeyuan and Xu, Ran and Heinecke, Shelby and Savarese, Silvio and Xiong, Caiming and Wang, Huan and Yao, Weiran},
+  year={2025},
+  publisher={Salesforce AI Research}
+}
+```
+
+## 🔗 Resources
+
+- 📄 **Technical Report**: [technical_report.pdf](https://github.com/SalesforceAIResearch/CoDA/blob/main/technical_report.pdf)
+- 💻 **Code Repository**: [github.com/SalesforceAIResearch/CoDA](https://github.com/SalesforceAIResearch/CoDA)
+- 🤗 **Model Hub**: [Salesforce CoDA collection](https://huggingface.co/collections/Salesforce/coda-68d627d87921c0e28a69e340)
+
+## 🙏 Acknowledgements
+
+We thank Lingpeng Kong for insightful discussions and Jialei Chen for technical support with TPU infrastructure.
+
+---
+
+*🏢 Developed by Salesforce AI Research*