docs/planning/007_record.md

User Story 007: Clean Record API Implementation

Story

As a robotics developer building teleoperation recording systems
I want to record robot motor positions and control data using a clean record() function API
So that I can capture teleoperation sessions for training AI models, analysis, and replay without dealing with complex class-based APIs or mixed concerns

Background

A community contributor has provided a recording implementation in this PR branch, which includes a comprehensive LeRobotDatasetRecorder class with video recording, data export, and LeRobot dataset format support. However, the current implementation violates several of our core conventions and doesn't match the clean API patterns established by calibrate(), teleoperate(), and findPort().

Current Implementation Problems

The existing LeRobotDatasetRecorder implementation has several architectural issues:

• Missing Standard Library Pattern: Uses class instantiation instead of simple function call like our other APIs
• Library vs Demo Separation Violation: Mixes hardware recording (library concern) with video streams, export formats, and UI (demo concerns)
• Teleoperator Integration Issues: Recording logic deeply embedded in BaseWebTeleoperator with complex state management
• Complex Constructor Anti-Pattern: Requires pre-configured teleoperators and video streams, violating our "direct library usage" principle
• Export API Complexity: ZIP, HuggingFace, and S3 upload belong in demo code, not standard library
• No Clean Process API: Doesn't follow our consistent start()/stop()/result pattern
• Redundant Event System: Uses dispatchMotorPositionChanged events that aren't consumed and duplicate callback functionality
• Artificial Polling: 100ms polling in teleoperate instead of immediate callbacks when motors change

Convention Alignment Needed

Our established patterns from calibrate(), teleoperate(), and findPort() follow these principles:

• Simple Function API: const process = await record(config)
• Clean Process Objects: Consistent start(), stop(), getState(), result interface
• Hardware-Only Library: Standard library handles only robotics hardware, not UI/storage/export
• Demo Handles UI: Examples handle video, export formats, browser storage, file downloads
• Immediate Callbacks: Real-time updates via callbacks, not polling or unused events
• Direct Usage: End users call library functions directly without complex setup

Acceptance Criteria

Core Functionality

• Standard Library API: Clean record(config) function matching our established patterns
• Process Object Interface: Consistent RecordProcess with start(), stop(), getState(), result methods
• Hardware-Only Recording: Library captures only robot motor positions and teleoperation data
• Real-Time Callbacks: Immediate onDataUpdate and onStateUpdate callbacks, no polling
• Device-Agnostic: Works with any robot type through configuration, not hardcoded values
• Clean Teleoperator Integration: Recording subscribes to teleoperation changes without embedding in teleoperator classes

User Experience

• Simple Integration: Easy to add recording to existing teleoperation workflows
• Consistent API: Same patterns as calibrate() and teleoperate() for familiar developer experience
• Immediate Feedback: Real-time recording state and data updates for responsive UI
• Error Handling: Clear error messages for recording failures or invalid configurations
• Resource Management: Proper cleanup of recording resources on stop/disconnect

Technical Requirements

• Library/Demo Separation: Move video, export, and storage logic to examples/demo layer
• Remove Event System: Eliminate unused dispatchMotorPositionChanged events, use callbacks only
• Extract from Teleoperators: Remove recording state and logic from BaseWebTeleoperator
• TypeScript: Fully typed with proper interfaces for recording configuration and data
• No Code Duplication: Reuse existing teleoperation and motor communication infrastructure
• Performance: Immediate callbacks when data changes, no unnecessary polling

Expected User Flow

Basic Robot Recording

import { record } from "@lerobot/web";

// Clean API matching our conventions
const recordProcess = await record({
  robot: connectedRobot,
  options: {
    fps: 30,
    onDataUpdate: (data) => {
      // Real-time recording data for UI feedback
      console.log(`Recorded ${data.frameCount} frames`);
      updateRecordingUI(data);
    },
    onStateUpdate: (state) => {
      // Recording state changes
      console.log(`Recording: ${state.isActive}`);
      updateRecordingStatus(state);
    },
  },
});

// Consistent process interface
recordProcess.start();

// Recording runs automatically while teleoperation is active
setTimeout(() => {
  recordProcess.stop();
}, 30000);

// Get pure robot recording data
const robotData = await recordProcess.result;
console.log("Episodes:", robotData.episodes);
console.log("Metadata:", robotData.metadata);

Recording with Teleoperation

import { teleoperate, record } from "@lerobot/web";

// Start teleoperation
const teleoperationProcess = await teleoperate({
  robot: connectedRobot,
  teleop: { type: "keyboard" },
  calibrationData: calibrationData,
  onStateUpdate: (state) => {
    updateTeleoperationUI(state);
  },
});

// Add recording to existing teleoperation
const recordProcess = await record({
  robot: connectedRobot,
  options: {
    onDataUpdate: (data) => {
      console.log(`Recording frame ${data.frameCount}`);
    },
  },
});

// Both run independently
teleoperationProcess.start();
recordProcess.start();

// Control independently
setTimeout(() => {
  recordProcess.stop(); // Stop recording, keep teleoperation
}, 60000);

setTimeout(() => {
  teleoperationProcess.stop(); // Stop teleoperation
}, 120000);

Demo-Layer Dataset Export

// In examples/demo - NOT in standard library
import { record } from "@lerobot/web";
import { DatasetExporter } from "./dataset-exporter"; // Demo code

const recordProcess = await record({ robot, options });
recordProcess.start();

// ... recording session ...

recordProcess.stop();
const robotData = await recordProcess.result;

// Demo handles complex export logic
const exporter = new DatasetExporter({
  robotData,
  videoStreams: cameraStreams, // Demo manages video
  taskDescription: "Pick and place task",
});

// Export options handled by demo
await exporter.downloadZip();
await exporter.uploadToHuggingFace({ apiKey, repoName });
await exporter.uploadToS3({ credentials });

Component Integration

// React component - direct library usage like calibration
const [recordingState, setRecordingState] = useState<RecordingState>();
const [recordingData, setRecordingData] = useState<RecordingData>();
const recordProcessRef = useRef<RecordProcess | null>(null);

useEffect(() => {
  const initRecording = async () => {
    const process = await record({
      robot,
      options: {
        onStateUpdate: setRecordingState,
        onDataUpdate: setRecordingData,
      },
    });
    recordProcessRef.current = process;
  };
  initRecording();
}, [robot]);

const handleStartRecording = () => {
  recordProcessRef.current?.start();
};

const handleStopRecording = async () => {
  recordProcessRef.current?.stop();
  const data = await recordProcessRef.current?.result;
  // Handle recorded data
};

Implementation Details

File Structure Refactoring

packages/web/src/
├── record.ts                         # NEW: Clean record() function
├── types/
│   └── recording.ts                  # NEW: Recording-specific types
├── utils/
│   └── recording-manager.ts          # NEW: Internal recording logic
├── teleoperators/
│   └── base-teleoperator.ts          # UPDATED: Remove recording logic
└── [MOVED TO EXAMPLES]
    ├── LeRobotDatasetRecorder.ts      # Complex export logic
    ├── dataset-exporter.ts            # Video + export functionality
    └── upload-handlers.ts             # HuggingFace, S3 upload logic

Key Dependencies

No New Dependencies for Standard Library

• Existing: Reuse all current dependencies (motor communication, teleoperation integration)
• Architecture Only: Pure refactoring to clean up existing functionality

Demo Dependencies (Moved)

• Video/Export: parquet-wasm, apache-arrow, jszip - moved to examples
• Upload: @huggingface/hub, AWS SDK - moved to examples

Core Functions to Implement

Clean Record API

// record.ts - New clean API
interface RecordConfig {
  robot: RobotConnection;
  options?: {
    fps?: number; // Default: 30
    onDataUpdate?: (data: RecordingData) => void;
    onStateUpdate?: (state: RecordingState) => void;
  };
}

interface RecordProcess {
  start(): void;
  stop(): void;
  getState(): RecordingState;
  result: Promise<RobotRecordingData>;
}

interface RecordingState {
  isActive: boolean;
  frameCount: number;
  episodeCount: number;
  duration: number; // milliseconds
  lastUpdate: number;
}

interface RecordingData {
  frameCount: number;
  currentEpisode: number;
  recentFrames: MotorPositionFrame[]; // Last few frames for UI
}

interface RobotRecordingData {
  episodes: MotorPositionFrame[][]; // Pure motor data only
  metadata: {
    fps: number;
    robotType: string;
    startTime: number;
    endTime: number;
    totalFrames: number;
    totalEpisodes: number;
  };
}

// Main function - matches our conventions
export async function record(config: RecordConfig): Promise<RecordProcess>;

Recording Manager (Internal)

// utils/recording-manager.ts - Internal implementation
class RecordingManager {
  private robot: RobotConnection;
  private isActive: boolean = false;
  private episodes: MotorPositionFrame[][] = [];
  private currentEpisode: MotorPositionFrame[] = [];
  private startTime: number = 0;
  private frameCount: number = 0;

  constructor(
    robot: RobotConnection,
    private options: RecordOptions,
    private onDataUpdate?: (data: RecordingData) => void,
    private onStateUpdate?: (state: RecordingState) => void
  ) {
    this.robot = robot;
  }

  start(): void {
    if (this.isActive) return;

    this.isActive = true;
    this.startTime = Date.now();

    // Subscribe to teleoperation changes (NO events, just callbacks)
    this.subscribeToRobotChanges();

    this.notifyStateUpdate();
  }

  stop(): void {
    if (!this.isActive) return;

    this.isActive = false;
    this.finishCurrentEpisode();
    this.unsubscribeFromRobotChanges();

    this.notifyStateUpdate();
  }

  private subscribeToRobotChanges(): void {
    // Listen to existing teleoperation callbacks - no new events needed
    // This integrates with the existing onStateUpdate mechanism
  }

  private recordFrame(motorConfigs: MotorConfig[]): void {
    const frame: MotorPositionFrame = {
      timestamp: Date.now() - this.startTime,
      motorPositions: motorConfigs.map((config) => ({
        id: config.id,
        name: config.name,
        position: config.currentPosition,
      })),
      frameIndex: this.frameCount++,
    };

    this.currentEpisode.push(frame);

    if (this.onDataUpdate) {
      this.onDataUpdate({
        frameCount: this.frameCount,
        currentEpisode: this.episodes.length,
        recentFrames: this.currentEpisode.slice(-10), // Last 10 frames
      });
    }
  }

  getState(): RecordingState {
    return {
      isActive: this.isActive,
      frameCount: this.frameCount,
      episodeCount: this.episodes.length,
      duration: this.isActive ? Date.now() - this.startTime : 0,
      lastUpdate: Date.now(),
    };
  }

  async getResult(): Promise<RobotRecordingData> {
    return {
      episodes: [...this.episodes],
      metadata: {
        fps: this.options.fps || 30,
        robotType: this.robot.robotType || "unknown",
        startTime: this.startTime,
        endTime: Date.now(),
        totalFrames: this.frameCount,
        totalEpisodes: this.episodes.length,
      },
    };
  }
}

Updated Teleoperate Integration

// teleoperate.ts - Remove 100ms polling, add immediate callbacks
export async function teleoperate(
  config: TeleoperateConfig
): Promise<TeleoperationProcess> {
  const teleoperator = await createTeleoperatorProcess(config);

  return {
    start: () => {
      teleoperator.start();

      // NO MORE 100ms polling! Use immediate callbacks
      if (config.onStateUpdate) {
        teleoperator.setStateUpdateCallback(config.onStateUpdate);
      }
    },
    // ... rest of interface
  };
}

Clean Teleoperator Base

// teleoperators/base-teleoperator.ts - Remove recording logic
export abstract class BaseWebTeleoperator extends WebTeleoperator {
  protected port: MotorCommunicationPort;
  public motorConfigs: MotorConfig[] = [];
  protected isActive: boolean = false;

  // REMOVED: All recording-related properties
  // REMOVED: dispatchMotorPositionChanged events
  // REMOVED: recordedMotorPositions, episodeIndex, etc.

  private stateUpdateCallback?: (state: TeleoperationState) => void;

  setStateUpdateCallback(callback: (state: TeleoperationState) => void): void {
    this.stateUpdateCallback = callback;
  }

  protected motorPositionsChanged(): void {
    // Call immediately when motors change - no events, no 100ms delay
    if (this.stateUpdateCallback) {
      const state = this.buildCurrentState();
      this.stateUpdateCallback(state);
    }
  }

  // Clean implementation without recording concerns
}

Technical Considerations

Migration Strategy

Preserve Existing Functionality:

1. Move Complex Logic: LeRobotDatasetRecorder moves to examples/ as demo code
2. Extract Clean Core: Create new record() function for standard library
3. Update Examples: Cyberpunk demo uses new API with demo-layer export functionality
4. Remove Event System: Clean up unused dispatchMotorPositionChanged events
5. Fix Polling: Replace 100ms polling with immediate callbacks

Performance Improvements

• Remove Polling: Eliminate artificial 100ms delays in favor of immediate callbacks
• Event-Driven: Only fire callbacks when robot state actually changes
• Memory Efficiency: No unused event listeners or redundant data structures
• Responsive UI: Immediate feedback for recording status and data updates

Future Extensibility

The clean architecture supports advanced recording features as demo enhancements:

// Future: Advanced demo features (NOT in standard library)
class AdvancedDatasetExporter extends DatasetExporter {
  // Video synchronization, multi-camera support
  // Cloud storage, data preprocessing
  // Visualization, playback, analysis tools
}

Definition of Done

• Clean Record API: record(config) function implemented matching our established patterns
• Process Interface: RecordProcess with consistent start(), stop(), getState(), result methods
• Hardware-Only Library: Standard library captures only robot motor data, no video/export complexity
• Demo Separation: Video recording, export formats, and UI logic moved to examples layer
• Remove Events: dispatchMotorPositionChanged events eliminated, callbacks used exclusively
• Fix Polling: 100ms artificial polling replaced with immediate callbacks when motors change
• Clean Teleoperators: Recording logic extracted from BaseWebTeleoperator and teleoperator classes
• TypeScript Coverage: Full type safety with proper interfaces for all recording functionality
• Performance: Immediate, event-driven updates with no unnecessary polling or unused listeners
• Integration: Easy integration with existing teleoperation workflows using familiar patterns
• Example Updates: Cyberpunk demo updated to use new clean API with demo-layer export features
• No Regression: All existing recording functionality preserved through demo layer
• Documentation: Clear examples showing standard library vs demo separation