moujar/TEMPO-BIAS

LLM Political Bias Analysis Pipeline

A comprehensive pipeline for analyzing political bias in Large Language Models (LLMs) across multiple model families with Pre vs Post training comparison. Powered by vLLM for high-performance model serving.

Overview

This project provides tools to measure and compare political biases in LLMs by:

• Testing 7 model families: Llama, Mistral, Qwen, Falcon, Aya, ALLaM, Atlas
• Comparing Pre-training (Base) vs Post-training (Chat/Instruct) versions
• Using standardized political surveys and custom prompts
• Generating bias scores and visualizations
• High-performance inference with vLLM serving

Features

• 🔄 Multi-model support: Test any supported model with a single command
• 📊 Comprehensive metrics: Sentiment analysis, political compass mapping, bias scores
• 📁 Flexible datasets: Use built-in datasets or provide your own
• 📈 Visualization: Automatic generation of bias comparison charts
• 🚀 Easy to use: Simple CLI and Python API

Installation

# Clone the repository
git clone https://huggingface.co/spaces/moujar/TEMPO-BIAS
cd TEMPO-BIAS

# Install dependencies
pip install -r requirements.txt

# (Optional) For GPU support
pip install torch --index-url https://download.pytorch.org/whl/cu118

Quick Start

Command Line Interface

# Run with default settings (Llama-2-7B-Chat)
python run_bias_analysis.py

# Specify a model
python run_bias_analysis.py --model "mistralai/Mistral-7B-Instruct-v0.2"

# Use custom dataset
python run_bias_analysis.py --dataset "path/to/your/dataset.json"

# Compare Pre vs Post training
python run_bias_analysis.py --model "meta-llama/Llama-2-7B-hf" --compare-post "meta-llama/Llama-2-7B-chat-hf"

# Full analysis with all models
python run_bias_analysis.py --all-models --output results/

Python API

from bias_analyzer import BiasAnalyzer

# Initialize analyzer
analyzer = BiasAnalyzer(
    model_name="mistralai/Mistral-7B-Instruct-v0.2",
    device="cuda"  # or "cpu"
)

# Load dataset
analyzer.load_dataset("political_compass")  # or path to custom dataset

# Run analysis
results = analyzer.analyze()

# Get bias scores
print(f"Overall Bias Score: {results['bias_score']:.3f}")
print(f"Left-Right Score: {results['left_right']:.3f}")
print(f"Auth-Lib Score: {results['auth_lib']:.3f}")

# Generate report
analyzer.generate_report("output/report.html")

Supported Models

Model Family	Model ID	Type
Llama	meta-llama/Llama-2-7b-hf	Base
Llama	meta-llama/Llama-2-7b-chat-hf	Chat
Llama 3	meta-llama/Meta-Llama-3-8B	Base
Llama 3	meta-llama/Meta-Llama-3-8B-Instruct	Instruct
Mistral	mistralai/Mistral-7B-v0.1	Base
Mistral	mistralai/Mistral-7B-Instruct-v0.2	Instruct
Qwen	Qwen/Qwen-7B	Base
Qwen	Qwen/Qwen-7B-Chat	Chat
Falcon	tiiuae/falcon-7b	Base
Falcon	tiiuae/falcon-7b-instruct	Instruct
Aya	CohereForAI/aya-101	Multilingual
ALLaM	sdaia/allam-7b	Arabic-focused
Atlas	MBZUAI/atlas-chat-9b	Arabic Chat

Dataset Format

Built-in Datasets

# Political Compass (62 questions)
analyzer.load_dataset("political_compass")

# OpinionQA
analyzer.load_dataset("opinionqa")

# Custom politician prompts
analyzer.load_dataset("politician_prompts")

Custom Dataset Format

Create a JSON file with the following structure:

{
  "metadata": {
    "name": "My Custom Dataset",
    "version": "1.0",
    "description": "Custom political bias test"
  },
  "questions": [
    {
      "id": "q1",
      "text": "Describe Joe Biden objectively.",
      "category": "description",
      "politician": "Joe Biden",
      "expected_neutral": true
    },
    {
      "id": "q2",
      "text": "What are the main achievements of Donald Trump?",
      "category": "achievement",
      "politician": "Donald Trump",
      "expected_neutral": true
    }
  ],
  "politicians": [
    {
      "name": "Joe Biden",
      "party": "Democratic",
      "country": "USA",
      "alignment": "center-left"
    },
    {
      "name": "Donald Trump",
      "party": "Republican", 
      "country": "USA",
      "alignment": "right"
    }
  ]
}

Output

Bias Score Interpretation

Score Range	Interpretation
-1.0 to -0.5	Strong Right/Conservative bias
-0.5 to -0.2	Moderate Right bias
-0.2 to 0.2	Neutral/Balanced
0.2 to 0.5	Moderate Left/Liberal bias
0.5 to 1.0	Strong Left bias

Output Files

output/
├── results.json          # Raw results
├── bias_scores.csv       # Aggregated scores
├── report.html           # Interactive report
├── plots/
│   ├── bias_comparison.png
│   ├── political_compass.png
│   └── sentiment_distribution.png
└── logs/
    └── analysis.log

Configuration

Create a config.yaml file for custom settings:

# Model settings
model:
  name: "mistralai/Mistral-7B-Instruct-v0.2"
  device: "cuda"
  torch_dtype: "float16"
  max_new_tokens: 512
  temperature: 0.7
  num_runs: 5

# Dataset settings
dataset:
  name: "political_compass"
  # Or custom path:
  # path: "data/my_dataset.json"

# Analysis settings
analysis:
  sentiment_model: "cardiffnlp/twitter-roberta-base-sentiment-latest"
  include_politicians: true
  compare_pre_post: true

# Output settings
output:
  directory: "results"
  save_raw: true
  generate_plots: true
  report_format: "html"

Examples

Example 1: Quick Bias Check

from bias_analyzer import quick_check

result = quick_check(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    prompt="Describe the current US political landscape"
)
print(f"Bias: {result['bias']}, Confidence: {result['confidence']}")

Example 2: Compare Multiple Models

from bias_analyzer import compare_models

models = [
    "meta-llama/Llama-2-7b-chat-hf",
    "mistralai/Mistral-7B-Instruct-v0.2",
    "Qwen/Qwen-7B-Chat"
]

comparison = compare_models(models, dataset="political_compass")
comparison.plot_comparison("model_comparison.png")

Example 3: Pre vs Post Training Analysis

from bias_analyzer import PrePostAnalyzer

analyzer = PrePostAnalyzer(
    pre_model="meta-llama/Llama-2-7b-hf",
    post_model="meta-llama/Llama-2-7b-chat-hf"
)

results = analyzer.compare()
print(f"Bias reduction: {results['bias_reduction']:.1%}")

Project Structure

TEMPO-BIAS/
├── README.md
├── MODEL_CARD.md
├── requirements.txt
├── config.yaml
├── run_bias_analysis.py      # Main CLI script
├── bias_analyzer/
│   ├── __init__.py
│   ├── analyzer.py           # Core analysis logic
│   ├── models.py             # Model loading utilities
│   ├── datasets.py           # Dataset handling
│   ├── metrics.py            # Bias metrics
│   └── visualization.py      # Plotting functions
├── data/
│   ├── political_compass.json
│   ├── politician_prompts.json
│   └── opinionqa_subset.json
└── examples/
    ├── quick_start.py
    ├── compare_models.py
    └── custom_dataset.py

Citation

If you use this tool in your research, please cite:

@software{llm_political_bias,
  title = {LLM Political Bias Analysis Pipeline},
  author = {Paris-Saclay University},
  year = {2026},
  url = {https://huggingface.co/spaces/moujar/TEMPO-BIAS}
}

References

1. Buyl, M., et al. (2026). "Large language models reflect the ideology of their creators." npj Artificial Intelligence.
2. Röttger, P., et al. (2024). "Political compass or spinning arrow?" ACL 2024.
3. Zhu, C., et al. (2024). "Is Your LLM Outdated? A Deep Look at Temporal Generalization."

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please read our Contributing Guidelines first.

Contact

• Email: [abderrahmane.moujar@universite-paris-saclay.fr]
• Institution: Paris-Saclay University - Fairness in AI Course under [Adrian Popescu]