giacomorossojakala/paraphrase-multilingual-mpnet-base-v2-eutekne-filtri-materia-lv1-2-desc-cascata

Model Description

This model is a fine-tuned paraphrase-multilingual-mpnet-base-v2 for a multi-label text classification on Eutekne domande_fitlri_materie train dataset.

The multi-label text classification task is faced as a retrieval problem: given a query and a list of description of labels, they are encoded with the same ambedder and the neareast neighbors are retrieved.

Intended Use

This model is intended for classifying legal questions into categories. These categories are based on the Eutekne domande_fitlri_materie.

Training Details

Experiment

The Training is preformed using following settings: contrastive_learning - ['level_1_2', 'descrizione_riassunto_cascata']

Base Model

This model is a finetuning of: paraphrase-multilingual-mpnet-base-v2

Training Data

{
  "train_samples": 4044,
  "val_samples": 867,
  "test_samples": 867
}

Training Hyperparameters

{
  "batch_size": 16,
  "learning_rate": 2e-05,
  "num_epochs": 30,
  "max_length": 256
}

Evaluation Results

|              |   validation_results |   test_results |
|:-------------|---------------------:|---------------:|
| accuracy_1   |             0.681661 |       0.663206 |
| accuracy_3   |             0.83045  |       0.817762 |
| accuracy_5   |             0.889273 |       0.861592 |
| accuracy_10  |             0.929642 |       0.916955 |
| precision_1  |             0.681661 |       0.663206 |
| precision_3  |             0.532488 |       0.517878 |
| precision_5  |             0.396078 |       0.376471 |
| precision_10 |             0.233449 |       0.227682 |
| recall_1     |             0.195662 |       0.199829 |
| recall_3     |             0.422025 |       0.41992  |
| recall_5     |             0.495023 |       0.479549 |
| recall_10    |             0.555928 |       0.547916 |
| f1_1         |             0.284624 |       0.286278 |
| f1_3         |             0.434482 |       0.426173 |
| f1_5         |             0.405207 |       0.386656 |
| f1_10        |             0.305581 |       0.297389 |
| mrr_1        |             0.681661 |       0.663206 |
| mrr_3        |             0.749519 |       0.731642 |
| mrr_5        |             0.76313  |       0.741619 |
| mrr_10       |             0.768567 |       0.749094 |
| r_precision  |             0.475959 |       0.468916 |

How to Use

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

repo_id = f"giacomorossojakala/paraphrase-multilingual-mpnet-base-v2-eutekne-filtri-materia-lv1-2-desc-cascata"
model = SentenceTransformer(repo_id, device=device)

downloaded_path = hf_hub_download(repo_id=HF_REPO_ID, filename="label_description.json", token=HF_TOKEN)
label_descriptions = json.load(open(downloaded_path, "r"))

# Get predictions
custom_text = "acquista casa nel 2025 lavori ristrutturazione ma andrà ad abitare nel 2026, detrazioni 50%?"

def classify_text(text, model, label_descriptions, top_k=5):
    '''
    Classify a text by computing similarity with all label descriptions.

    Args:
        text: Input text to classify
        model: Trained SentenceTransformer model
        label_descriptions: Dict mapping label codes to descriptions
        top_k: Number of top predictions to return

    Returns:
        List of (label, similarity_score) tuples, sorted by score (descending)
    '''

    # Get label list and descriptions
    label_list = sorted(list(label_descriptions.keys()))
    label_texts = [label_descriptions[label] for label in label_list]

    # Encode question and all labels
    question_embedding = model.encode(text, convert_to_numpy=True)
    label_embeddings = model.encode(label_texts, convert_to_numpy=True, show_progress_bar=False)

    # Compute cosine similarities
    similarities = cosine_similarity([question_embedding], label_embeddings)[0]

    # Get top-k
    top_k_indices = np.argsort(similarities)[::-1][:top_k]
    predictions = [(label_list[idx], similarities[idx]) for idx in top_k_indices]

    return predictions



predictions = classify_text(custom_text, model, label_descriptions, top_k=5)