---
language:
- en
license: mit
tags:
- text-generation
- llama
- tinystories
- storytelling
datasets:
- roneneldan/TinyStories
widget:
- text: "Once upon a time, there was a"
  example_title: "Story Beginning"
- text: "One day, Lily met a"
  example_title: "Character Introduction"
- text: "The little boy was very happy because"
  example_title: "Story Continuation"
---

# TinyStories Llama Model

## Model Description

This is a small Llama-architecture language model trained on the [TinyStories dataset](https://huggingface.co/datasets/roneneldan/TinyStories). 
The model is designed to generate simple, coherent children's stories using a vocabulary and concepts that a typical 3-4 year old would understand.

**Model Architecture:** Llama 2  
**Training Framework:** PyTorch  
**Implementation:** Based on [llama2.c](https://github.com/karpathy/llama2.c)

## Model Details

### Architecture Hyperparameters

- **Dimension:** 288
- **Number of Layers:** 6
- **Number of Attention Heads:** 6
- **Number of KV Heads:** 6
- **Vocabulary Size:** 32,000 (Llama 2 tokenizer)
- **Maximum Sequence Length:** 256 tokens
- **Dropout:** 0.0
- **Hidden Dimension Multiple:** 32

**Total Parameters:** ~15M

### Training Hyperparameters

- **Batch Size:** 128 (micro-batch)
- **Gradient Accumulation Steps:** 4
- **Effective Batch Size:** 512
- **Learning Rate:** 5e-4 (max)
- **Learning Rate Schedule:** Cosine decay with warmup
- **Warmup Iterations:** 1,000
- **Total Training Iterations:** 100,000
- **Weight Decay:** 0.1
- **Beta1:** 0.9
- **Beta2:** 0.95
- **Gradient Clipping:** 1.0
- **Optimizer:** AdamW
- **Precision:** bfloat16 (with mixed precision training)

**Tokens per Iteration:** ~65,536 (4 grad accum × 1 process × 64 batch × 256 seq len)

## Intended Use

This model is intended for:
- Generating simple children's stories
- Educational demonstrations of small-scale language model training
- Research into emergent capabilities in small language models
- Experimentation with efficient inference (e.g., pure C implementation)

## Limitations

- **Domain-Specific:** The model is trained exclusively on simple stories and will not perform well on general text generation tasks
- **Vocabulary:** Limited to concepts and language appropriate for very young children
- **Context Length:** Maximum sequence length of 256 tokens limits story length
- **No Instruction Following:** This is a base model without instruction tuning

## Training Data

The model was trained on the [TinyStories dataset](https://huggingface.co/datasets/roneneldan/TinyStories), which consists of short stories generated to contain only words that a typical 3-4 year old would understand. The dataset was created to study the capabilities of small language models.

**Dataset Size:** ~2.1M stories  
**Vocabulary:** Words understandable by 3-4 year olds  
**Content:** Simple narratives, common objects, basic emotions and actions

## Example Outputs

**Prompt:** "Once upon a time, there was a little girl named Lily."

**Generation (temperature=0.8, top_p=0.9):**
```
She loved to play outside in the park. One day, she saw a big, red ball. 
She wanted to play with it, but it was too high. Lily's mom said, "Let's 
go get it together!" They worked together and got the ball down. Lily was 
so happy! She played with the ball all day long.
```

## Citation

If you use this model or the llama2.c implementation, please cite:

```bibtex
@misc{llama2c,
  author = {Andrej Karpathy},
  title = {llama2.c: Inference Llama 2 in one file of pure C},
  year = {2023},
  publisher = {GitHub},
  url = {https://github.com/karpathy/llama2.c}
}

@article{eldan2023tinystories,
  title={TinyStories: How Small Can Language Models Be and Still Speak Coherent English?},
  author={Eldan, Ronen and Li, Yuanzhi},
  journal={arXiv preprint arXiv:2305.07759},
  year={2023}
}
```

## License

MIT License - See the [LICENSE](LICENSE) file for details.

## Acknowledgments

- Model architecture and training code adapted from [llama2.c](https://github.com/karpathy/llama2.c) by Andrej Karpathy
- Trained on the [TinyStories dataset](https://huggingface.co/datasets/roneneldan/TinyStories) by Ronen Eldan and Yuanzhi Li
- Based on the Llama 2 architecture by Meta AI