File size: 5,809 Bytes
cb9905b
2197ab7
 
cb9905b
 
 
2197ab7
cb9905b
5b40ec9
 
2197ab7
 
 
5b40ec9
 
 
 
 
 
 
2197ab7
 
 
5b40ec9
 
 
 
 
2197ab7
 
 
5b40ec9
2197ab7
5b40ec9
 
 
 
 
 
 
 
2197ab7
 
5b40ec9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2197ab7
 
 
5b40ec9
2197ab7
5b40ec9
2197ab7
5b40ec9
 
2197ab7
 
5b40ec9
2197ab7
5b40ec9
 
2197ab7
 
5b40ec9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2197ab7
 
 
5b40ec9
 
 
2197ab7
 
5b40ec9
2197ab7
5b40ec9
2197ab7
 
5b40ec9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2197ab7
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
---
title: fabric_to_espanso
app_file: gradio_app_query_only.py
sdk: gradio
sdk_version: 5.12.0
---
# Fabric to Espanso Converter

A Python application that bridges Fabric prompts with Espanso and Obsidian Textgenerator by managing and converting prompts through a vector database. It enables semantic search and efficient management of prompts while providing a modern web interface for easy interaction.
There's also a seperate gradio app that can be hosted on Hugging Face Spaces to provide a query-only interface.

## Features

- **Vector Database Integration**: Store and manage Fabric prompts in a Qdrant vector database with semantic search capabilities
- **Automated Conversion**: Convert stored prompts into Espanso YAML format for system-wide usage
- **Change Detection**: Automatically detect and process changes in the Fabric patterns folder
- **Web Interface**: Modern Gradio-based interface for easy prompt searching and management
- **Semantic Search**: Find relevant prompts based on their meaning, not just exact matches
- **Clipboard Integration**: Quick copying of prompts directly to clipboard
- **Logging System**: Comprehensive logging for tracking operations and debugging

## Prerequisites

- Python 3.11 or higher
- Fabric (https://github.com/danielmiessler/fabric)
- Qdrant vector database (local or cloud instance)
- Obsidian with TextGenerator plugin (https://github.com/obsidianmd/obsidian-textgenerator)
- Linux/WSL2 or Windows with WSL2

## Installation

1. **Environment Setup**:
   ```bash
   # Clone the repository
   git clone [repository-url]
   cd fabric_to_espanso

   # Install PDM if not already installed
   pip install pdm

   # Install dependencies
   pdm install
   ```

2. **Configuration**:
   - Copy `.env.example` to `.env`
   - Set your Qdrant API key in `.env`:
     ```
     QDRANT_API_KEY=your_api_key_here
     ```

3. **Obsidian Setup**:
   - Install Obsidian and the TextGenerator plugin
   - Create the folder structure:
     ```
     Extra/
     └── FabricPatterns/
         β”œβ”€β”€ Official/  # Official Fabric patterns
         └── Own/       # Custom patterns
     ```

4. **Fabric Setup**:
   - Install Fabric, see https://github.com/danielmiessler/fabric

5. **QDRANT Setup**:
   - Install Qdrant, see https://qdrant.io/en/
   - Start Qdrant server

6. **Parameters**:
   - Set all the parameters in the file `parameters.py`.

7. **Optional**:
   - Create a Powershell script to run the Streamlit app


## Usage

### Starting the Application

#### Linux/WSL2
```bash
# Start the Gradio interface
python gradio_app_query_only.py
```

#### Windows (with WSL2)
```powershell
# Use the provided PowerShell script
./start_app.ps1
```

### Core Operations

1. **Search Prompts**:
   - Enter your search query in the search box
   - Results are ranked by semantic similarity
   - Click on a result to view its contents

2. **Copy Prompts**:
   - Select a prompt from the results
   - Click "Copy to Clipboard" to copy the prompt text

3. **Update Database**:
   - Run `python main.py` to process changes in the Fabric patterns folder
   - New and modified prompts are automatically added to the database
   - Deleted prompts are removed from the database

## Project Structure

```
fabric_to_espanso/
β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ fabrics_processor/    # Core processing logic
β”‚   └── search_qdrant/        # Search functionality
β”œβ”€β”€ gradio_app_query_only.py  # Web interface
β”œβ”€β”€ main.py                   # CLI entry point
└── parameters.py            # Configuration parameters
```

## Dependencies

Core dependencies are managed through PDM:

- gradio >= 5.12.0
- qdrant-client >= 1.12.1
- fastembed >= 0.4.2
- python-dotenv
- pyperclip >= 1.9.0
- pyyaml >= 6.0.2
- regex >= 2024.11.6

## TODO

The following items need to be addressed to improve code quality, maintainability, and functionality:

### Database Optimization
- Check the database for any points with exactly the same vector or nearly the same. Remove those to reduce redundancy and improve search efficiency.

### Metadata Enhancement
- If available, use the readme.md file from the fabrics folder to fill the "purpose" field in the database entries.
- If readme.md is not available in the fabrics folder, create the "purpose" field from an LLM response that summarizes the goal of the fabric file.

### UI/UX Improvements
- Add a compare interface to the gradio app to allow side-by-side comparison of prompts.
- Remove the streamlit_only_query app as it's being replaced by the gradio interface.

### Code Refactoring
- Implement proper error handling for database operations.
- Add comprehensive logging throughout the application.
- Create unit tests for core functionality.
- Implement type hints consistently across all Python files.
- Add input validation for all user-provided data.
- Refactor the database operations into a dedicated class.
- Implement connection pooling for better database performance.
- Add docstrings to all functions and classes.
- Create a configuration class to handle all settings.
- Add proper cleanup of resources in error cases.

### Documentation
- Add API documentation for all public interfaces.
- Include examples for common use cases.
- Document the database schema and vector space organization.
- Add contribution guidelines.
- Include troubleshooting section.

### Security
- Implement proper environment variable handling.
- Add input sanitization for all user inputs.
- Implement rate limiting for the web interface.
- Add proper authentication for the web interface.

### Performance
- Implement caching for frequently accessed prompts.
- Optimize vector similarity search parameters.
- Add batch processing for large-scale operations.

## License

This project is licensed under the MIT License.