Operational_Instructions/Live_Challenge_Day_and_Dry_Test_Instructions.md

Live Challenge Day (May 12)

1. We will conduct two live sessions on May 12 as part of the Live Challenge Day (calendar invites will be sent shortly):

   • Session 1: 7:00 – 9:00 UTC
   • Session 2: 15:00 – 17:00 UTC

2. Just before your session start time, your team leader will receive an email containing the DataMorgana generated Question file (500 questions) in JSONL format, where each line contains a Question JSON object — see Question JSON schema and example

3. You must generate and submit your Answer file in JSONL format, where each line contains an Answer JSON object — see Answer JSON schema and example — within 2 hours from your session start time
   3.1 JSONL format: Each line must be a single, complete JSON object, with no line breaks within an object, and no multi-line formatting or indentation
   3.2 Here is a simple script for generating and verifying a valid Answer file
   3.2 It is highly recommended to parallelize and use retries when sending requests to Falcon — see a simple script of parallel batch request with retries using AI71 Falcon
   3.3 Submission instructions
   3.4 All submissions made by the same team members will be considered together
   3.5 Submissions made outside your designated Session time window will be disqualified

4. Please refer to the LiveRAG Challenge Evaluation Guidelines for important information about the evaluation process

5. You must share your RAG system Git repository with us by end-of-day AoE May 13 via email at sigir2025-liverag-gen@tii.ae to enable result reproduction

6. Additional information:
   6.1 The automatic evaluation results leaderboard will be published on the HuggingFace Challenge page once the evaluation process is completed
   6.2 The top-performing teams will undergo manual evaluation to determine the final winners, who will be announced on July 17 during the SIGIR 2025 LiveRAG Workshop

Dry Test (May 5)

1. To ensure a smooth Live Challenge Day, we will conduct a Dry Test on May 5 (calendar invites will be sent shortly)

2. The Dry Test DataMorgana generated Question file will contain 50 questions

3. Your answers will not be evaluated, and no leaderboard will be published for the Dry Test

4. Note that instructions 1–4 from the Live Challenge Day still apply