Spaces:
Sleeping
Sleeping
Enhance Dockerfile and Streamlit app for comprehensive environment setup and permission testing
98aae70
| title: Docling | |
| emoji: 🚀 | |
| colorFrom: red | |
| colorTo: red | |
| sdk: docker | |
| app_port: 8501 | |
| tags: | |
| - streamlit | |
| pinned: false | |
| short_description: Streamlit template space | |
| # Welcome to Streamlit! | |
| Edit `/src/streamlit_app.py` to customize this app to your heart's desire. :heart: | |
| If you have any questions, checkout our [documentation](https://docs.streamlit.io) and [community | |
| forums](https://discuss.streamlit.io). | |
| # Medical Document Parser & Redactor | |
| A sophisticated medical document processing application that uses **Docling** (structure-aware parser) to parse PDF medical documents and automatically redact medication information using AI-powered analysis. | |
| ## 🎯 Overview | |
| This application provides a Streamlit-based interface for uploading medical PDF documents, parsing them with Docling to extract structured content, and using Azure OpenAI to intelligently identify and redact formal medication lists while preserving clinical context. | |
| ## 🏗️ Project Structure | |
| ``` | |
| docling/ | |
| ├── src/ # Main source code | |
| │ ├── processing/ # Core processing logic | |
| │ │ ├── __init__.py | |
| │ │ ├── document_processor.py # Main document processing pipeline | |
| │ │ ├── llm_extractor.py # Azure OpenAI integration for medication detection | |
| │ │ └── sections.py # Section extraction and redaction logic | |
| │ ├── utils/ # Utility functions | |
| │ │ ├── __init__.py | |
| │ │ └── logging_utils.py # Logging configuration and handlers | |
| │ └── streamlit_app.py # Main Streamlit application interface | |
| ├── temp_files/ # Temporary file storage (auto-created) | |
| ├── .env # Environment variables (Azure OpenAI credentials) | |
| ├── requirements.txt # Python dependencies | |
| ├── pyproject.toml # Project configuration | |
| ├── Dockerfile # Container configuration | |
| └── README.md # This file | |
| ``` | |
| ## 📁 File Responsibilities | |
| ### Core Processing Files | |
| #### `src/processing/document_processor.py` | |
| **Purpose**: Main document processing pipeline that orchestrates the entire workflow. | |
| **Key Classes**: | |
| - `DocumentResult`: Data class holding processed results | |
| - `DocumentProcessor`: Main processing class | |
| **Key Functions**: | |
| - `process(file_path)`: Main processing method | |
| - `_export_redacted_markdown()`: Generates redacted markdown | |
| - `_reconstruct_markdown_from_filtered_texts()`: Reconstructs markdown from filtered content | |
| **Responsibilities**: | |
| - Document conversion using Docling | |
| - Section redaction coordination | |
| - Markdown generation and reconstruction | |
| - File persistence and logging | |
| #### `src/processing/llm_extractor.py` | |
| **Purpose**: Azure OpenAI integration for intelligent medication detection. | |
| **Key Classes**: | |
| - `AzureO1MedicationExtractor`: LLM-based medication extractor | |
| **Key Functions**: | |
| - `extract_medication_sections(doc_json)`: Main extraction method | |
| - `__init__()`: Azure OpenAI client initialization | |
| **Responsibilities**: | |
| - Azure OpenAI API communication | |
| - Medication section identification | |
| - Structured JSON response generation | |
| - Error handling and logging | |
| #### `src/processing/sections.py` | |
| **Purpose**: Section extraction and redaction logic. | |
| **Key Classes**: | |
| - `ReasoningSectionExtractor`: AI-powered section extractor | |
| - `SectionDefinition`: Section definition data class | |
| - `SectionExtractor`: Traditional regex-based extractor | |
| **Key Functions**: | |
| - `remove_sections_from_json()`: JSON-based section removal | |
| - `remove_sections()`: Text-based section removal (fallback) | |
| **Responsibilities**: | |
| - Section identification and removal | |
| - JSON structure manipulation | |
| - Text processing and redaction | |
| - Reasoning logging and transparency | |
| ### Interface Files | |
| #### `src/streamlit_app.py` | |
| **Purpose**: Main Streamlit web application interface. | |
| **Key Functions**: | |
| - `save_uploaded_file()`: File upload handling | |
| - `cleanup_temp_files()`: Temporary file management | |
| - `create_diff_content()`: Diff view generation | |
| **Responsibilities**: | |
| - User interface and interaction | |
| - File upload and management | |
| - Visualization and diff display | |
| - Session state management | |
| - Download functionality | |
| ### Utility Files | |
| #### `src/utils/logging_utils.py` | |
| **Purpose**: Logging configuration and management. | |
| **Key Functions**: | |
| - `get_log_handler()`: Creates in-memory log handlers | |
| - Log buffer management for UI display | |
| **Responsibilities**: | |
| - Logging setup and configuration | |
| - In-memory log capture | |
| - Log display in UI | |
| ## 🔧 Detailed Function Documentation | |
| ### Document Processing Pipeline | |
| #### `DocumentProcessor.process(file_path: str) -> DocumentResult` | |
| **Purpose**: Main entry point for document processing. | |
| **Parameters**: | |
| - `file_path`: Path to the PDF file to process | |
| **Returns**: | |
| - `DocumentResult`: Object containing all processing results | |
| **Process Flow**: | |
| 1. Converts PDF using Docling | |
| 2. Exports structured markdown and JSON | |
| 3. Applies section redaction if extractor is provided | |
| 4. Persists results to temporary files | |
| 5. Returns comprehensive result object | |
| **Example Usage**: | |
| ```python | |
| processor = DocumentProcessor(section_extractor=extractor) | |
| result = processor.process("document.pdf") | |
| print(f"Original: {len(result.structured_markdown)} chars") | |
| print(f"Redacted: {len(result.redacted_markdown)} chars") | |
| ``` | |
| #### `AzureO1MedicationExtractor.extract_medication_sections(doc_json: Dict) -> Dict` | |
| **Purpose**: Uses Azure OpenAI to identify medication sections for redaction. | |
| **Parameters**: | |
| - `doc_json`: Docling-generated JSON structure | |
| **Returns**: | |
| - Dictionary with indices to remove and reasoning | |
| **Process Flow**: | |
| 1. Analyzes document structure | |
| 2. Sends structured prompt to Azure OpenAI | |
| 3. Parses JSON response | |
| 4. Validates and limits results | |
| 5. Returns structured analysis | |
| **Example Usage**: | |
| ```python | |
| extractor = AzureO1MedicationExtractor(endpoint, api_key, version, deployment) | |
| result = extractor.extract_medication_sections(doc_json) | |
| print(f"Removing {len(result['indices_to_remove'])} elements") | |
| ``` | |
| #### `ReasoningSectionExtractor.remove_sections_from_json(doc_json: Dict) -> Dict` | |
| **Purpose**: Removes identified sections from JSON structure. | |
| **Parameters**: | |
| - `doc_json`: Original document JSON structure | |
| **Returns**: | |
| - Redacted JSON structure | |
| **Process Flow**: | |
| 1. Calls LLM extractor for analysis | |
| 2. Logs detailed reasoning | |
| 3. Removes identified text elements | |
| 4. Updates document structure | |
| 5. Returns redacted JSON | |
| ## 🚨 Troubleshooting | |
| ### Permission Error: `[Errno 13] Permission denied: '/.cache'` | |
| **Problem**: When deploying to Hugging Face Spaces, you may encounter a permission error where the application tries to create cache directories in the root filesystem (`/.cache`). | |
| **Root Cause**: Hugging Face Hub and other ML libraries try to create cache directories in the root filesystem by default, but containers in Hugging Face Spaces don't have permission to write to the root directory. | |
| **Solution**: This application includes comprehensive environment variable configuration to redirect all cache directories to writable locations: | |
| 1. **Environment Variables**: All cache directories are redirected to `/tmp/docling_temp/` | |
| 2. **Lazy Initialization**: DocumentConverter is initialized lazily to ensure environment variables are set first | |
| 3. **Startup Script**: Docker container uses a startup script that sets all necessary environment variables | |
| 4. **Test Script**: `test_permissions.py` verifies the environment setup | |
| **Files Modified**: | |
| - `src/streamlit_app.py`: Environment variables set at the very beginning | |
| - `src/processing/document_processor.py`: Lazy initialization of DocumentConverter | |
| - `Dockerfile`: Environment variables and startup script | |
| - `test_permissions.py`: Environment verification script | |
| **Testing**: Run the test script to verify the environment: | |
| ```bash | |
| python test_permissions.py | |
| ``` | |
| **Expected Output**: | |
| ``` | |
| ✅ ALL TESTS PASSED | |
| 🎉 All tests passed! The environment is ready for Docling. | |
| ``` | |
| ### Other Common Issues | |
| #### Memory Issues | |
| - **Problem**: Large PDF files may cause memory issues | |
| - **Solution**: The application includes automatic cleanup of temporary files and memory management | |
| #### Azure OpenAI Configuration | |
| - **Problem**: Missing or incorrect Azure OpenAI credentials | |
| - **Solution**: Ensure `.env` file contains: | |
| ``` | |
| AZURE_OPENAI_ENDPOINT=your_endpoint | |
| AZURE_OPENAI_KEY=your_key | |
| AZURE_OPENAI_VERSION=your_version | |
| AZURE_OPENAI_DEPLOYMENT=your_deployment | |
| ``` | |
| #### File Upload Issues | |
| - **Problem**: Files not uploading or processing | |
| - **Solution**: Check file size limits and ensure PDF format is supported | |
| ## 🔧 Development and Deployment | |
| ### Local Development | |
| 1. Clone the repository | |
| 2. Install dependencies: `pip install -r requirements.txt` | |
| 3. Set up environment variables in `.env` | |
| 4. Run the test script: `python test_permissions.py` | |
| 5. Start the app: `streamlit run src/streamlit_app.py` | |
| ### Hugging Face Spaces Deployment | |
| 1. Push code to repository | |
| 2. Ensure `Dockerfile` is present | |
| 3. Set environment variables in Spaces settings | |
| 4. Deploy and monitor logs for any issues | |
| ### Environment Variables | |
| The application uses these environment variables to control cache directories: | |
| ```bash | |
| # Core temp directory | |
| TEMP_DIR=/tmp/docling_temp | |
| # Hugging Face Hub | |
| HF_HOME=/tmp/docling_temp/huggingface | |
| HF_CACHE_HOME=/tmp/docling_temp/huggingface_cache | |
| HF_HUB_CACHE=/tmp/docling_temp/huggingface_cache | |
| # ML Libraries | |
| TRANSFORMERS_CACHE=/tmp/docling_temp/transformers_cache | |
| HF_DATASETS_CACHE=/tmp/docling_temp/datasets_cache | |
| TORCH_HOME=/tmp/docling_temp/torch | |
| TENSORFLOW_HOME=/tmp/docling_temp/tensorflow | |
| KERAS_HOME=/tmp/docling_temp/keras | |
| # XDG Directories | |
| XDG_CACHE_HOME=/tmp/docling_temp/cache | |
| XDG_CONFIG_HOME=/tmp/docling_temp/config | |
| XDG_DATA_HOME=/tmp/docling_temp/data | |
| ``` | |
| ## 📊 Performance and Monitoring | |
| ### Memory Management | |
| - Automatic cleanup of temporary files | |
| - Session state management | |
| - File size monitoring | |
| ### Logging | |
| - Comprehensive logging throughout the application | |
| - In-memory log capture for UI display | |
| - Error tracking and debugging information | |
| ### Caching | |
| - Hugging Face model caching in temp directories | |
| - Document processing result caching | |
| - Session state persistence | |