src/MCP_README.md

MCP & Agent Integration for Laban Movement Analysis

This project provides comprehensive MCP (Model Context Protocol) integration and agent-ready APIs for professional movement analysis with pose estimation, AI action recognition, and automation capabilities.

🚀 Quick Start

1. Install All Dependencies

# Clone the repository
git clone https://github.com/[your-repo]/labanmovementanalysis
cd labanmovementanalysis

# Install core dependencies
pip install -r backend/requirements.txt

# Install MCP and enhanced features
pip install -r backend/requirements-mcp.txt

2. Start the MCP Server

# Start MCP server for AI assistants
python -m backend.mcp_server

3. Configure Your AI Assistant

Add to your Claude Desktop or other MCP-compatible assistant configuration:

{
  "mcpServers": {
    "laban-movement-analysis": {
      "command": "python",
      "args": ["-m", "backend.mcp_server"],
      "env": {
        "PYTHONPATH": "/path/to/labanmovementanalysis"
      }
    }
  }
}

🛠️ Enhanced MCP Tools

1. analyze_video

Comprehensive video analysis with enhanced features including SkateFormer AI and multiple pose models.

Parameters:

• video_path (string): Path or URL to video (supports YouTube, Vimeo, local files)

• model (string, optional): Advanced pose model selection:

  • MediaPipe: mediapipe-lite, mediapipe-full, mediapipe-heavy
  • MoveNet: movenet-lightning, movenet-thunder
  • YOLO: yolo-v8-n/s/m/l, yolo-v11-n/s/m/l

• enable_visualization (boolean, optional): Generate annotated video

• include_keypoints (boolean, optional): Include raw keypoint data

• use_skateformer (boolean, optional): Enable AI action recognition

Examples:

Analyze the dance video at https://youtube.com/watch?v=dQw4w9WgXcQ using SkateFormer AI
Analyze movement in video.mp4 using yolo-v11-s model with visualization
Process the exercise video with mediapipe-full and include keypoints

2. get_analysis_summary

Get human-readable summaries with enhanced AI insights.

Parameters:

• analysis_id (string): ID from previous analysis

Enhanced Output Includes:

• SkateFormer action recognition results
• Movement quality metrics (rhythm, complexity, symmetry)
• Temporal action segmentation
• Video source metadata (YouTube/Vimeo titles, etc.)

Example:

Get a detailed summary of analysis dance_2024-01-01T12:00:00 including AI insights

3. list_available_models

Comprehensive list of all 20+ pose estimation models with detailed specifications.

Enhanced Model Information:

• Performance characteristics (speed, accuracy, memory usage)
• Recommended use cases (real-time, research, production)
• Hardware requirements (CPU, GPU, memory)
• Keypoint specifications (17 COCO, 33 MediaPipe)

Example:

What pose estimation models are available for real-time processing?
List all YOLO v11 model variants with their specifications

4. batch_analyze

Enhanced batch processing with parallel execution and progress tracking.

Parameters:

• video_paths (array): List of video paths/URLs (supports mixed sources)
• model (string, optional): Pose estimation model for all videos
• parallel (boolean, optional): Enable parallel processing
• use_skateformer (boolean, optional): Enable AI analysis for all videos
• output_format (string, optional): Output format ("summary", "structured", "full")

Enhanced Features:

• Mixed source support (local files + YouTube URLs)
• Progress tracking and partial results
• Resource management and optimization
• Failure recovery and retry logic

Examples:

Analyze all dance videos in the playlist with SkateFormer AI
Batch process exercise videos using yolo-v11-s with parallel execution

5. compare_movements

Advanced movement comparison with AI-powered insights.

Parameters:

• analysis_id1 (string): First analysis ID
• analysis_id2 (string): Second analysis ID
• comparison_type (string, optional): Type of comparison ("basic", "detailed", "ai_enhanced")

Enhanced Comparison Features:

• SkateFormer action similarity analysis
• Movement quality comparisons (rhythm, complexity, symmetry)
• Temporal pattern matching
• Statistical significance testing

Example:

Compare the movement patterns between the two dance analyses with AI insights
Detailed comparison of exercise form between beginner and expert videos

6. real_time_analysis (New)

Start/stop real-time WebRTC analysis.

Parameters:

• action (string): "start" or "stop"
• model (string, optional): Real-time optimized model
• stream_config (object, optional): WebRTC configuration

Example:

Start real-time movement analysis using mediapipe-lite

7. filter_videos_advanced (New)

Advanced video filtering with AI-powered criteria.

Parameters:

• video_paths (array): List of video paths/URLs
• criteria (object): Enhanced filtering criteria including:
  • Traditional LMA metrics (direction, intensity, fluidity)
  • SkateFormer actions (dancing, jumping, etc.)
  • Movement qualities (rhythm, complexity, symmetry)
  • Temporal characteristics (duration, segment count)

Example:

Filter videos for high-energy dance movements with good rhythm
Find exercise videos with proper form (high fluidity and symmetry)

🤖 Enhanced Agent API

Comprehensive Python Agent API

from gradio_labanmovementanalysis import LabanMovementAnalysis
from gradio_labanmovementanalysis.agent_api import (
    LabanAgentAPI,
    PoseModel,
    MovementDirection,
    MovementIntensity,
    analyze_and_summarize
)

# Initialize with all features enabled
analyzer = LabanMovementAnalysis(
    enable_skateformer=True,
    enable_webrtc=True,
    enable_visualization=True
)

agent_api = LabanAgentAPI(analyzer=analyzer)

Advanced Analysis Workflows

# YouTube video analysis with AI
result = agent_api.analyze(
    "https://youtube.com/watch?v=...",
    model=PoseModel.YOLO_V11_S,
    use_skateformer=True,
    generate_visualization=True
)

# Enhanced batch processing
results = agent_api.batch_analyze(
    ["video1.mp4", "https://youtube.com/watch?v=...", "https://vimeo.com/..."],
    model=PoseModel.YOLO_V11_S,
    parallel=True,
    use_skateformer=True
)

# AI-powered movement filtering
filtered = agent_api.filter_by_movement_advanced(
    video_paths,
    skateformer_actions=["dancing", "jumping"],
    movement_qualities={"rhythm": 0.8, "complexity": 0.6},
    traditional_criteria={
        "direction": MovementDirection.UP,
        "intensity": MovementIntensity.HIGH,
        "min_fluidity": 0.7
    }
)

# Real-time analysis control
agent_api.start_realtime_analysis(model=PoseModel.MEDIAPIPE_LITE)
live_metrics = agent_api.get_realtime_metrics()
agent_api.stop_realtime_analysis()

Enhanced Quick Functions

from gradio_labanmovementanalysis import (
    quick_analyze_enhanced,
    analyze_and_summarize_with_ai,
    compare_videos_detailed
)

# Enhanced analysis with AI
data = quick_analyze_enhanced(
    "https://youtube.com/watch?v=...",
    model="yolo-v11-s",
    use_skateformer=True
)

# AI-powered summary
summary = analyze_and_summarize_with_ai(
    "dance_video.mp4",
    include_skateformer=True,
    detail_level="comprehensive"
)

# Detailed video comparison
comparison = compare_videos_detailed(
    "video1.mp4", 
    "video2.mp4",
    include_ai_analysis=True
)

🌐 Enhanced Gradio 5 Agent Features

Comprehensive API Endpoints

The unified Gradio 5 app exposes these endpoints optimized for agents:

1. /analyze_standard - Basic LMA analysis
2. /analyze_enhanced - Advanced analysis with all features
3. /analyze_agent - Agent-optimized structured output
4. /batch_analyze - Efficient multiple video processing
5. /filter_videos - Movement-based filtering
6. /compare_models - Model performance comparison
7. /real_time_start - Start WebRTC real-time analysis
8. /real_time_stop - Stop WebRTC real-time analysis

Enhanced Gradio Client Usage

from gradio_client import Client

# Connect to unified demo
client = Client("http://localhost:7860")

# Enhanced single analysis
result = client.predict(
    video_input="https://youtube.com/watch?v=...",
    model="yolo-v11-s",
    enable_viz=True,
    use_skateformer=True,
    include_keypoints=False,
    api_name="/analyze_enhanced"
)

# Agent-optimized batch processing
batch_results = client.predict(
    files=["video1.mp4", "video2.mp4"],
    model="yolo-v11-s",
    api_name="/batch_analyze"
)

# Advanced movement filtering
filtered_results = client.predict(
    files=video_list,
    direction_filter="up",
    intensity_filter="high", 
    fluidity_threshold=0.7,
    expansion_threshold=0.5,
    api_name="/filter_videos"
)

# Model comparison analysis
comparison = client.predict(
    video="test_video.mp4",
    model1="mediapipe-full",
    model2="yolo-v11-s",
    api_name="/compare_models"
)

📊 Enhanced Output Formats

AI-Enhanced Summary Format

🎭 Movement Analysis Summary for "Dance Performance"
Source: YouTube (10.5 seconds, 30fps)
Model: YOLO-v11-S with SkateFormer AI

📊 Traditional LMA Metrics:
• Primary direction: up (65% of frames)
• Movement intensity: high (80% of frames)  
• Average speed: fast (2.3 units/frame)
• Fluidity score: 0.85/1.00 (very smooth)
• Expansion score: 0.72/1.00 (moderately extended)

🤖 SkateFormer AI Analysis:
• Detected actions: dancing (95% confidence), jumping (78% confidence)
• Movement qualities:
  - Rhythm: 0.89/1.00 (highly rhythmic)
  - Complexity: 0.76/1.00 (moderately complex)
  - Symmetry: 0.68/1.00 (slightly asymmetric)
  - Smoothness: 0.85/1.00 (very smooth)
  - Energy: 0.88/1.00 (high energy)

⏱️ Temporal Analysis:
• 7 movement segments identified
• Average segment duration: 1.5 seconds
• Transition quality: smooth (0.82/1.00)

🎯 Overall Assessment: Excellent dance performance with high energy, 
good rhythm, and smooth transitions. Slightly asymmetric but shows 
advanced movement complexity.

Enhanced Structured Format

{
    "success": true,
    "video_metadata": {
        "source": "youtube",
        "title": "Dance Performance",
        "duration": 10.5,
        "platform_id": "dQw4w9WgXcQ"
    },
    "model_info": {
        "pose_model": "yolo-v11-s",
        "ai_enhanced": true,
        "skateformer_enabled": true
    },
    "lma_metrics": {
        "direction": "up",
        "intensity": "high",
        "speed": "fast",
        "fluidity": 0.85,
        "expansion": 0.72
    },
    "skateformer_analysis": {
        "actions": [
            {"type": "dancing", "confidence": 0.95, "duration": 8.2},
            {"type": "jumping", "confidence": 0.78, "duration": 2.3}
        ],
        "movement_qualities": {
            "rhythm": 0.89,
            "complexity": 0.76,
            "symmetry": 0.68,
            "smoothness": 0.85,
            "energy": 0.88
        },
        "temporal_segments": 7,
        "transition_quality": 0.82
    },
    "performance_metrics": {
        "processing_time": 12.3,
        "frames_analyzed": 315,
        "keypoints_detected": 24
    }
}

Comprehensive JSON Format

Complete analysis including frame-by-frame data, SkateFormer attention maps, movement trajectories, and statistical summaries.

🏗️ Enhanced Architecture

┌─────────────────────────────────────────────────────────────┐
│                    AI Assistant Integration                  │
│          (Claude, GPT, Local Models via MCP)               │
└─────────────────────┬───────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────────┐
│                   MCP Server                                │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐│
│  │   Video     │ │   Enhanced  │ │      Real-time          ││
│  │  Analysis   │ │    Batch    │ │      WebRTC             ││
│  │   Tools     │ │ Processing  │ │     Analysis            ││
│  └─────────────┘ └─────────────┘ └─────────────────────────┘│
└─────────────────────┬───────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────────┐
│              Enhanced Agent API Layer                       │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐│
│  │  Movement   │ │ AI-Enhanced │ │     Advanced            ││
│  │  Filtering  │ │ Comparisons │ │    Workflows            ││
│  └─────────────┘ └─────────────┘ └─────────────────────────┘│
└─────────────────────┬───────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────────┐
│             Core Analysis Engine                            │
│                                                             │
│  📹 Video Input    🤖 Pose Models   🎭 SkateFormer AI      │
│  ┌─────────────┐   ┌─────────────┐   ┌─────────────────────┐│
│  │Local Files  │   │MediaPipe(3) │   │  Action Recognition ││
│  │YouTube URLs │   │MoveNet(2)   │   │Movement Qualities   ││
│  │Vimeo URLs   │   │YOLO(8)      │   │Temporal Segments    ││
│  │Direct URLs  │   │            │   │Attention Analysis   ││
│  └─────────────┘   └─────────────┘   └─────────────────────┘│
│                                                             │
│  📊 LMA Engine     📹 WebRTC       🎨 Visualization        │
│  ┌─────────────┐   ┌─────────────┐   ┌─────────────────────┐│
│  │Direction    │   │Live Camera  │   │  Pose Overlays      ││
│  │Intensity    │   │Real-time    │   │  Motion Trails      ││
│  │Speed/Flow   │   │Sub-100ms    │   │  Metric Displays    ││
│  │Expansion    │   │Adaptive FPS │   │  AI Visualizations  ││
│  └─────────────┘   └─────────────┘   └─────────────────────┘│
└─────────────────────────────────────────────────────────────┘

📝 Advanced Agent Workflows

1. Comprehensive Dance Analysis Pipeline

# Multi-source dance video analysis
videos = [
    "local_dance.mp4",
    "https://youtube.com/watch?v=dance1",
    "https://vimeo.com/dance2"
]

# Batch analyze with AI
results = agent_api.batch_analyze(
    videos,
    model=PoseModel.YOLO_V11_S,
    use_skateformer=True,
    parallel=True
)

# Filter for high-quality performances
excellent_dances = agent_api.filter_by_movement_advanced(
    videos,
    skateformer_actions=["dancing"],
    movement_qualities={
        "rhythm": 0.8,
        "complexity": 0.7,
        "energy": 0.8
    },
    traditional_criteria={
        "intensity": MovementIntensity.HIGH,
        "min_fluidity": 0.75
    }
)

# Generate comprehensive report
report = agent_api.generate_analysis_report(
    results,
    include_comparisons=True,
    include_recommendations=True
)

2. Real-time Exercise Form Checker

# Start real-time analysis
agent_api.start_realtime_analysis(
    model=PoseModel.MEDIAPIPE_FULL,
    enable_skateformer=True
)

# Monitor form in real-time
while exercise_in_progress:
    metrics = agent_api.get_realtime_metrics()
    
    # Check form quality
    if metrics["fluidity"] < 0.6:
        send_feedback("Improve movement smoothness")
    
    if metrics["symmetry"] < 0.7:
        send_feedback("Balance left and right movements")
    
    time.sleep(0.1)  # 10Hz monitoring

# Stop and get session summary
agent_api.stop_realtime_analysis()
session_summary = agent_api.get_session_summary()

3. Movement Pattern Research Workflow

# Large-scale analysis for research
research_videos = get_research_dataset()

# Batch process with comprehensive analysis
results = agent_api.batch_analyze(
    research_videos,
    model=PoseModel.YOLO_V11_L,  # High accuracy for research
    use_skateformer=True,
    include_keypoints=True,  # Full data for research
    parallel=True
)

# Statistical analysis
patterns = agent_api.extract_movement_patterns(
    results,
    pattern_types=["temporal", "spatial", "quality"],
    clustering_method="hierarchical"
)

# Generate research insights
insights = agent_api.generate_research_insights(
    patterns,
    include_visualizations=True,
    statistical_tests=True
)

🔧 Advanced Configuration & Customization

Environment Variables

# Core configuration
export LABAN_DEFAULT_MODEL="mediapipe-full"
export LABAN_CACHE_DIR="/path/to/cache"
export LABAN_MAX_WORKERS=4

# Enhanced features
export LABAN_ENABLE_SKATEFORMER=true
export LABAN_ENABLE_WEBRTC=true
export LABAN_SKATEFORMER_MODEL_PATH="/path/to/skateformer"

# Performance tuning
export LABAN_GPU_ENABLED=true
export LABAN_BATCH_SIZE=8
export LABAN_REALTIME_FPS=30

# Video download configuration
export LABAN_YOUTUBE_QUALITY="720p"
export LABAN_MAX_DOWNLOAD_SIZE="500MB"
export LABAN_TEMP_DIR="/tmp/laban_downloads"

Custom MCP Tools

# Add custom MCP tool
from backend.mcp_server import server

@server.tool("custom_movement_analysis")
async def custom_analysis(
    video_path: str,
    custom_params: dict
) -> dict:
    """Custom movement analysis with specific parameters."""
    # Your custom implementation
    return results

# Register enhanced filters
@server.tool("filter_by_sport_type")
async def filter_by_sport(
    videos: list,
    sport_type: str
) -> dict:
    """Filter videos by detected sport type using SkateFormer."""
    # Implementation using SkateFormer sport classification
    return filtered_videos

WebRTC Configuration

# Custom WebRTC configuration
webrtc_config = {
    "video_constraints": {
        "width": 1280,
        "height": 720,
        "frameRate": 30
    },
    "processing_config": {
        "max_latency_ms": 100,
        "quality_adaptation": True,
        "model_switching": True
    }
}

agent_api.configure_webrtc(webrtc_config)

🤝 Contributing to Agent Features

Adding New MCP Tools

1. Define tool in backend/mcp_server.py
2. Implement core logic in agent API
3. Add comprehensive documentation
4. Include usage examples
5. Write integration tests

Extending Agent API

1. Add methods to LabanAgentAPI class
2. Ensure compatibility with existing workflows
3. Add structured output formats
4. Include error handling and validation
5. Update documentation

Enhancing SkateFormer Integration

1. Extend action recognition types
2. Add custom movement quality metrics
3. Implement temporal analysis features
4. Add visualization components
5. Validate with research datasets

📚 Resources & References

• MCP Specification
• SkateFormer Research Paper
• Gradio 5 Documentation
• Unified Demo Application
• Core Component Code

🎯 Production Deployment

Docker Deployment

FROM python:3.9-slim

COPY . /app
WORKDIR /app

RUN pip install -r backend/requirements.txt
RUN pip install -r backend/requirements-mcp.txt

EXPOSE 7860 8080

CMD ["python", "-m", "backend.mcp_server"]

Kubernetes Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: laban-mcp-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: laban-mcp
  template:
    metadata:
      labels:
        app: laban-mcp
    spec:
      containers:
      - name: mcp-server
        image: laban-movement-analysis:latest
        ports:
        - containerPort: 8080
        env:
        - name: LABAN_MAX_WORKERS
          value: "2"
        - name: LABAN_ENABLE_SKATEFORMER
          value: "true"

---

🤖 Transform your AI assistant into a movement analysis expert with comprehensive MCP integration and agent-ready automation.