---
title: Intelligent Document Assistant
emoji: 🏆
colorFrom: yellow
colorTo: red
sdk: gradio
sdk_version: 5.46.1
app_file: app.py
pinned: false
license: apache-2.0
short_description: Transform your documents into intelligent conversations
---

# 🏆 Intelligent Document Assistant

**Transform your documents into interactive conversations!** This advanced RAG system combines the power of semantic search with intuitive chat interfaces, allowing you to explore your knowledge base through natural dialogue.

Built with cutting-edge AI technologies, this production-ready assistant turns static documents into dynamic, searchable conversations. Whether you're researching, analyzing, or simply exploring your content, experience the future of document interaction through intelligent, context-aware responses.

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Python](https://img.shields.io/badge/Python-3.8%2B-blue.svg)](https://www.python.org/)
[![Gradio](https://img.shields.io/badge/Gradio-5.44.0-orange.svg)](https://gradio.app/)

## ✨ **What Makes This Special?**

### 🎯 **Intelligent Conversation System**
- **Natural Chat Interface**: Talk to your documents like chatting with an expert
- **Context-Aware Responses**: Remembers conversation flow and provides relevant follow-ups
- **Multiple Response Modes**:
  - 💬 **Conversation**: Natural dialogue with your documents
  - 📚 **RAG**: Information-focused responses with source citations
  - 🔀 **Hybrid**: Best of both worlds (recommended)

### 🧠 **Advanced RAG Technology**
- **Hybrid Search Engine**: Combines vector similarity + keyword search (BM25)
- **Smart Re-ranking**: Cross-encoder models for improved relevance
- **Intelligent Chunking**: Context-aware document segmentation
- **Source Attribution**: Always know where information comes from

## 🌟 **Core Features**

### 💬 **Conversational AI**
- **Smart Chat Interface**: Natural dialogue with your documents
- **Follow-up Suggestions**: AI-generated next questions to guide conversation
- **Context Memory**: Maintains conversation flow and context
- **Greeting Recognition**: Responds naturally to greetings and casual conversation
- **Fallback Handling**: Graceful responses when documents don't contain requested information

### 📁 **Document Processing**
- **Multi-format Support**: PDF, DOCX, and TXT files (up to 50MB each)
- **Intelligent Chunking**: Context-aware text segmentation with overlap
- **Metadata Extraction**: Automatic extraction and indexing of document properties
- **Batch Processing**: Upload and process multiple documents simultaneously
- **Progress Tracking**: Real-time processing status with detailed feedback

### 🔍 **Advanced Search System**
- **Hybrid Search Engine**: Vector similarity + BM25 keyword search
- **Cross-Encoder Re-ranking**: Improved relevance scoring for better results
- **Configurable Weights**: Adjust vector vs keyword search importance
- **Metadata Filtering**: Filter by document properties using JSON or key:value format
- **Search Modes**: Vector-only, keyword-only, or hybrid approaches

### 🎨 **Professional Interface**
- **5-Tab Organization**:
  - 💬 **Chat**: Conversational document assistant
  - 📁 **Upload**: Document processing and management
  - 🔍 **Search**: Advanced technical search interface
  - 📚 **Documents**: Document library management
  - 📊 **Analytics**: Usage statistics and system performance
- **Modern Design**: Clean, responsive UI with custom themes
- **Real-time Updates**: Live progress indicators and status feedback
- **Interactive Elements**: Click suggestions, source links, and quick actions

### ⚡ **Performance & Reliability**
- **Multi-level Caching**: Embeddings, queries, and model caching for speed
- **Memory Optimization**: Efficient processing for resource-constrained environments
- **Error Recovery**: Comprehensive error handling with user-friendly messages
- **Session Management**: Maintains user sessions and conversation state
- **Analytics Dashboard**: Built-in usage tracking and performance monitoring

## 🚀 **Quick Start**

### **Option 1: Online Demo** (Recommended)
Visit the [Hugging Face Space](https://huggingface.co/spaces/) for an instant demo - no installation required!

### **Option 2: Local Installation**

#### **Prerequisites**
- Python 3.8 or higher
- 4GB+ RAM (8GB+ recommended for larger documents)
- Internet connection for downloading models

#### **Installation Steps**

1. **Clone the repository**:
   ```bash
   git clone <repository-url>
   cd RAG_ChatBot
   ```

2. **Create virtual environment** (recommended):
   ```bash
   python -m venv .venv
   # Windows
   .venv\Scripts\activate
   # Linux/Mac
   source .venv/bin/activate
   ```

3. **Install dependencies**:
   ```bash
   pip install -r requirements.txt
   ```

4. **Run the application**:
   ```bash
   python app.py
   ```

5. **Open your browser** to `http://localhost:7860`

### **First Time Setup**
1. The system will automatically download required models on first run
2. Upload sample documents to test the system
3. Try both chat and search interfaces to explore capabilities

### Configuration

The system uses YAML configuration files:
- `config.yaml`: Production configuration
- `config-local.yaml`: Local development overrides

Key configuration sections:
```yaml
models:
  embedding:
    name: "sentence-transformers/all-MiniLM-L6-v2"
    batch_size: 32
  reranker:
    name: "cross-encoder/ms-marco-MiniLM-L-6-v2"
    enabled: true

search:
  default_k: 10
  vector_weight: 0.7
  bm25_weight: 0.3
```

## 📖 **Usage Guide**

### 🎯 **Getting Started (Recommended Flow)**

1. **📁 Upload Documents** → Upload your PDF, DOCX, or TXT files
2. **💬 Start Chatting** → Ask questions naturally about your documents
3. **🔍 Deep Search** → Use advanced search for specific research needs
4. **📊 Monitor Analytics** → Track usage and system performance

---

### 💬 **Chat Tab - Your AI Assistant**

**Perfect for**: Natural conversation, quick questions, exploration

```
👤 "What are the main topics in my documents?"
🤖 "Based on your uploaded documents, I can see several key themes..."

👤 "Tell me more about topic X"
🤖 "From document.pdf, here's what I found about topic X..."
```

**Features**:
- **Response Modes**: Choose conversation, RAG, or hybrid mode
- **Follow-up Suggestions**: AI suggests related questions
- **Source Citations**: See exactly where information comes from
- **Context Memory**: Remembers conversation flow

**Tips**:
- Start with greetings: "Hi, what can you help me with?"
- Ask open questions: "What are the key points about X?"
- Use follow-up suggestions to explore deeper
- Toggle sources on/off based on your needs

---

### 📁 **Document Upload Tab**

**Perfect for**: Adding and managing your document library

**Supported Formats**:
- 📄 PDF files (research papers, reports, manuals)
- 📝 DOCX files (Word documents, proposals)
- 📃 TXT files (notes, transcripts, code)

**Process**:
1. **Select Files**: Choose up to 50MB per file
2. **Process**: Click "🚀 Process Documents"
3. **Monitor**: Watch real-time progress
4. **Verify**: Check results and error reports

**Pro Tips**:
- Upload related documents together for better context
- Check file sizes before upload (large files take longer)
- Review processing results for any errors

---

### 🔍 **Search Tab - Technical Research**

**Perfect for**: Detailed research, finding specific information, analysis

**Search Options**:
- **Search Mode**:
  - `hybrid` (recommended): Best overall results
  - `vector`: Semantic similarity search
  - `bm25`: Keyword-based search
- **Results Count**: 1-20 results
- **Re-ranking**: Enable for improved relevance
- **Metadata Filters**:
  ```json
  {"source": "document.pdf"}
  {"author": "John Doe", "year": "2023"}
  ```

**Advanced Features**:
- **JSON View**: See raw search data and scores
- **Processing Time**: Monitor search performance
- **Query Analysis**: Understand how your query was processed

---

### 📚 **Documents Tab - Library Management**

**Perfect for**: Organizing and maintaining your document collection

**Features**:
- **Document List**: View all uploaded documents
- **Statistics**: See chunks, file size, processing date
- **Individual Removal**: Delete specific documents
- **Bulk Clear**: Remove all documents at once
- **Refresh**: Update document list

---

### 📊 **Analytics Tab - System Insights**

**Perfect for**: Understanding usage patterns and system health

**Metrics Available**:
- **Query Patterns**: Most common search terms
- **Search Modes**: Usage distribution
- **Performance**: Response times and system load
- **Document Stats**: Processing metrics
- **Conversation Analytics**: Chat session insights

**Use Cases**:
- Track which documents are most useful
- Optimize search strategies
- Monitor system performance
- Understand user behavior patterns

## 🏗️ **Technical Architecture**

### **System Overview**
```
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Gradio UI     │    │   RAG Engine     │    │  Vector Store   │
│  (Chat & Web)   │◄──►│  (Intelligence)  │◄──►│   (Memory)      │
└─────────────────┘    └──────────────────┘    └─────────────────┘
        │                       │                       │
        ▼                       ▼                       ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Analytics     │    │   Document       │    │   Cache         │
│   (Insights)    │    │   Processor      │    │   (Speed)       │
└─────────────────┘    └──────────────────┘    └─────────────────┘
```

### **Core Components**

```
src/
├── 🤖 rag_system.py           # Main RAG orchestrator & conversation logic
├── 🧠 enhanced_rag_system.py  # Advanced conversational capabilities
├── 📄 document_processor.py   # Multi-format parsing (PDF/DOCX/TXT)
├── 🔤 embedding_manager.py    # HuggingFace embeddings with caching
├── 🗄️ vector_store.py         # In-memory vector storage & retrieval
├── 🔍 search_engine.py        # Hybrid search (vector + BM25)
├── 📊 reranker.py             # Cross-encoder re-ranking
├── ⚡ cache_manager.py        # Multi-level caching system
├── 🛡️ error_handler.py        # Comprehensive error management
├── 📈 analytics.py            # Usage tracking & performance metrics
└── 🗣️ conversation/           # Conversational AI components
    ├── manager.py             # Conversation flow management
    ├── intent_classifier.py   # Intent recognition
    └── context_tracker.py     # Context maintenance

ui/
├── 🖥️ main_interface.py      # Primary Gradio interface
├── 💬 conversation_components.py # Chat UI components
├── 🧩 components.py          # Reusable UI components
├── 🎨 themes.py              # Custom styling & CSS
└── 🔧 utils.py               # UI utilities & helpers
```

### **Key Technologies**
- **🤖 AI/ML**: HuggingFace Transformers, Sentence Transformers, Cross-encoders
- **🔍 Search**: Vector similarity, BM25, Hybrid ranking
- **💾 Storage**: In-memory vector store with persistence options
- **🖥️ Interface**: Gradio with custom themes and components
- **⚡ Performance**: Multi-level caching, batch processing, optimization

## 🔧 **Advanced Configuration**

### **Environment Variables**
```bash
# Optional performance tuning
export GRADIO_SERVER_PORT=7860
export CUDA_VISIBLE_DEVICES=0  # For GPU acceleration
export TRANSFORMERS_CACHE=/path/to/cache  # Model cache location
```

### **Configuration Files**
- `config.yaml`: Production settings
- `config-local.yaml`: Local development overrides

**Key Settings**:
```yaml
# Model Configuration
models:
  embedding:
    name: "sentence-transformers/all-MiniLM-L6-v2"
    batch_size: 32
    device: "auto"  # auto, cpu, cuda
  reranker:
    name: "cross-encoder/ms-marco-MiniLM-L-6-v2"
    enabled: true

# Search Configuration
search:
  default_k: 10
  vector_weight: 0.7
  bm25_weight: 0.3
  enable_reranking: true

# Chat Configuration
conversation:
  max_context_length: 10
  enable_suggestions: true
  enable_sources: true
  response_modes: ["conversation", "rag", "hybrid"]
```

---

## 🧪 **Testing & Development**

### **Running Tests**
```bash
# Install test dependencies
pip install -r requirements-dev.txt

# Run all tests
pytest

# Run with coverage report
pytest --cov=src --cov-report=html

# Run specific test categories
pytest tests/test_conversation_manager.py  # Conversation tests
pytest tests/test_vector_store.py          # Search tests
pytest tests/test_document_processor.py    # Document tests
```

### **Development Mode**
```bash
# Development server with auto-reload
python app.py --debug --config config-local.yaml

# Enable verbose logging
python app.py --log-level DEBUG

# Test with sample documents
python scripts/test_with_samples.py
```

### **Performance Testing**
```bash
# Benchmark document processing
python scripts/benchmark_processing.py

# Test search performance
python scripts/benchmark_search.py

# Memory usage analysis
python scripts/profile_memory.py
```

---

## 🤝 **Contributing**

We welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for details.

### **Development Workflow**
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

### **Code Standards**
- Follow PEP 8 for Python code style
- Add type hints for new functions
- Include docstrings for public methods
- Write tests for new features
- Update documentation as needed

---

## 📞 **Support & Community**

- **🐛 Bug Reports**: [GitHub Issues](https://github.com/user/repo/issues)
- **💡 Feature Requests**: [GitHub Discussions](https://github.com/user/repo/discussions)
- **📖 Documentation**: [Project Wiki](https://github.com/user/repo/wiki)
- **💬 Community**: [Discord Server](https://discord.gg/your-server)

---

## 📄 **License**

This project is licensed under the **Apache License 2.0** - see the [LICENSE](LICENSE) file for details.

### **Third-Party Libraries**
- [Gradio](https://gradio.app/) - Web interface framework
- [HuggingFace Transformers](https://huggingface.co/transformers/) - AI models
- [Sentence Transformers](https://www.sbert.net/) - Embedding models
- See [requirements.txt](requirements.txt) for complete list

---

## 🌟 **Acknowledgments**

- **HuggingFace** for providing excellent AI models and infrastructure
- **Gradio** team for the intuitive interface framework
- **Sentence Transformers** community for embedding models
- **Open Source Community** for inspiration and contributions

---

<div align="center">

**⭐ If you find this project helpful, please consider giving it a star! ⭐**

[🚀 **Try Demo**](https://huggingface.co/spaces/) | [📖 **Documentation**](https://github.com/user/repo/wiki) | [🐛 **Report Issues**](https://github.com/user/repo/issues)

</div>