File size: 5,359 Bytes

50466ce

# Self-Organizing Map (SOM) Model for Document Clustering

A trained Self-Organizing Map model for clustering and visualizing high-dimensional document embeddings. This model was trained on technical documentation and can be used for document similarity analysis, topic discovery, and semantic clustering.

## 📊 Model Details

- **Model Type**: Self-Organizing Map (SOM)
- **Training Data**: 11,412 records
- **Embedding Dimension**: 3,072 (OpenAI Large Embedding Model)
- **Number of Clusters**: 625
- **Grid Size**: 25x25
- **Learning Rate**: 0.1
- **Sigma**: 1.0

## 🎯 Use Cases

- **Document Clustering**: Group similar documents based on semantic similarity
- **Topic Discovery**: Identify common themes and topics in large document collections
- **Semantic Search**: Find related documents through vector similarity
- **Data Visualization**: Interactive visualization of document relationships
- **Knowledge Organization**: Structure and organize large knowledge bases

## 📁 Model Files

- `som_model.pkl`: Trained SOM model weights and parameters
- `cluster_assignments.json`: Document-to-cluster assignments for all 11,412 records
- `cluster_analysis.json`: Detailed analysis of each cluster including keywords and topics
- `interactive_som_map.html`: Interactive visualization of the SOM grid with cluster information

## 🚀 Quick Start

### Installation

```bash
pip install numpy scikit-learn matplotlib plotly
```

### Loading and Using the Model

```python
import pickle
import json
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

# Load the trained SOM model
with open('som_model.pkl', 'rb') as f:
    som_model = pickle.load(f)

# Load cluster assignments
with open('cluster_assignments.json', 'r') as f:
    cluster_assignments = json.load(f)

# Load cluster analysis
with open('cluster_analysis.json', 'r') as f:
    cluster_analysis = json.load(f)

# Example: Get cluster for a new document embedding
def get_cluster_for_embedding(embedding, som_model):
    """Get the cluster assignment for a new document embedding"""
    # Find the best matching unit (BMU)
    bmu = som_model.winner(embedding)
    return f"{bmu[0]},{bmu[1]}"

# Example: Find similar documents
def find_similar_documents(embedding, cluster_assignments, top_k=5):
    """Find similar documents based on cluster membership"""
    cluster = get_cluster_for_embedding(embedding, som_model)
    
    # Get all documents in the same cluster
    cluster_docs = [doc for doc, doc_cluster in cluster_assignments.items() 
                   if doc_cluster == cluster]
    
    return cluster_docs[:top_k]
```

### Interactive Visualization

Open `interactive_som_map.html` in a web browser to explore the SOM grid interactively. The visualization shows:

- Cluster sizes and distributions
- Top keywords for each cluster
- Topic analysis
- Document counts per cluster

## 📈 Model Performance

Based on the cluster analysis:

- **Total Documents**: 11,412
- **Total Clusters**: 625 (25x25 grid)
- **Silhouette Score**: -0.0078
- **Calinski-Harabasz Score**: 13.69
- **Davies-Bouldin Score**: 2.33

## 🔍 Cluster Analysis

The model identifies meaningful clusters with distinct topics. For example, one of the largest clusters (659 documents) focuses on:

- **Keywords**: connector, anypoint, mule, studio, connectors
- **Topics**: Configuration, API integration, MuleSoft platform usage

## 🛠️ Advanced Usage

### Custom Clustering

```python
# Train a new SOM with different parameters
from minisom import MiniSom

def train_custom_som(embeddings, grid_size=(20, 20), sigma=1.0, learning_rate=0.1):
    som = MiniSom(grid_size[0], grid_size[1], embeddings.shape[1], 
                  sigma=sigma, learning_rate=learning_rate, random_seed=42)
    som.train_random(embeddings, 100)
    return som
```

### Cluster Analysis

```python
def analyze_cluster(cluster_key, cluster_analysis):
    """Get detailed information about a specific cluster"""
    for cluster in cluster_analysis['top_clusters']:
        if cluster['cluster_key'] == cluster_key:
            return {
                'size': cluster['size'],
                'keywords': cluster['keywords'],
                'topics': cluster['topics']
            }
    return None
```

## 📚 Dependencies

- `numpy`: Numerical computations
- `scikit-learn`: Machine learning utilities
- `minisom`: Self-Organizing Map implementation
- `matplotlib`: Static plotting
- `plotly`: Interactive visualizations
- `pandas`: Data manipulation

## 🤝 Contributing

This model is part of a larger document processing and clustering pipeline. For questions or contributions, please refer to the main project repository.

## 📄 License

This model is provided for research and educational purposes. Please ensure compliance with the original data source licenses when using this model.

## 🔗 Related Resources

- [Self-Organizing Maps Tutorial](https://en.wikipedia.org/wiki/Self-organizing_map)
- [MiniSom Documentation](https://github.com/JustGlowing/minisom)
- [OpenAI Embeddings](https://platform.openai.com/docs/guides/embeddings)

---

**Note**: This model was trained on technical documentation and may be most effective for similar types of content. For best results, ensure your input documents are in the same domain or consider fine-tuning the model on your specific data.