Spaces:

---

LiveRAG

/

Challenge

like 0

Running

June 15:

LiveRAG Challenge - Finalists Announcement

We are delighted to list below (in alphabetical order) the finalists of the SIGIR'2025 LiveRAG Challenge.

The prize winners will be announced at the LiveRAG Workshop in Padua, Italy on July 17.

Note that prizes will be awarded only to teams that have at least one team member registered to and present at the workshop.

The finalists were identified after a thorough validation and assessment of the teams' artifacts. This included the Correctness and Faithfulness scores, as computed by DataMorgana following the official evaluation guidelines, manual examination of results by annotators, PC members reviews of the teams' reports, and code repositories' inspection.

June 12:

The Notifications were sent with indication of Oral or Poster/Spotlight slots to all those who submitted reports on time. But Easychair has been repeatedly down today. We sent the notifications between down times, but we are not sure all went through.

• Please email us immediately if you did not get an Easychair notification
  
• Remember that you absolutely have to register to the LiveRAG Workshop and be present in person be part of the program (deadline for regular rates is tomorrow June 13!)
  
  Finalists to be announced soon here!

June 1:

Please note that the AWS accounts and pre-built indices will be decommissioned by June 5.

May 16:

Here are the leaderboards (one for each session), where teams can find their temporary results for the Correctness and Faithfulness metrics, as calculated by DataMorgana following the official evaluation guidelines. Notes that rows are sorted according to the Correctness metric.

Disclaimers

• These results are only temporary as they are still pending validation and manual assessment. Final results and finalists will be announced on June 12.
• Teams are encouraged to use these automatic scores in the preparation of their short papers, which must be submitted via EasyChair by May 23. These scores may be updated for the final presentation at the LiveRAG workshop at SIGIR in Padua on July 17.
• Please note that without a timely short paper submission and registration to the SIGIR LiveRAG workshop, teams will not be eligible for prizes or for presenting (oral or poster) at the workshop.

Session 1 - May 12, 07:00 - 09:00 UTC

Session 2 - May 12, 15:00 - 17:00 UTC

May 15:

We have updated the short paper submission instructions to allow unlimited pages for references, appendices, and the discussion of ethical considerations.

May 14:

Detailed instructions for the LiveRAG short paper submission are now available.

May 12:

The Live Challenge Day Sessions are now over. Thanks to all participants!

Automatic evaluation results will be published by the end of the week.

Please remember to share your Git repositories with us by end-of-day (AoE) May 13 via sigir2025-liverag-gen@tii.ae

May 11:

Friendly reminder: The LiveRAG Live Challenge Day events will take place tomorrow, May 12. Please RSVP to your invitation and follow the operational instructions and the submission instructions (see also "Submission Information" on the left rail).

Please note: (1) Session invites were sent to team leaders only; (2) Submissions made by any team member will be considered as part of the team's overall submissions; and (3) Submissions made outside your designated session time window will be disqualified.

Don't forget to check your Spam folder for LiveRAG emails!

May 4:

Friendly reminder: The LiveRAG Dry Test event will take place tomorrow, May 5. Please RSVP to your invitation and follow the operational instructions and the submission instructions (see also "Submission Information" on the left rail).

May 2:

Submission instructions are now available!

See also "Submission information" on the left rail.

April 29:

A couple of useful scripts for generating and verifying Answer files, as well as properly accessing AI71 Falcon, have been added to the operational instructions.

April 28:

Operational instructions for the Live Challenge Day and Dry Test events are now available!

• Please note that details about the exact Answer file upload procedure will be provided soon.

LiveRAG survey ends by AoE April 28. Don't miss the opportunity to provide your feedback.

• Extra AWS credits will be awarded to the top informative responders.

April 24:

From now on you may generate Q&A bulk requests using AI71 DataMorgana Sandbox. See the revised Sandbox manual for more details.

Don't forget to complete our survey by April 28.

April 21:

Please take a few minutes to complete this short survey by April 28. We truly appreciate your feedback!

April 17:

For those who were unable to attend the LiveRAG Online Session, the video recording and presentation are now available for download.

April 7:

You're all invited to a 1-hour online event taking place on Wednesday, April 16th at 15:00 UTC!

Session agenda:
1. Opening remarks by Dr. Yoelle Maarek
2. DataMorgana overview by Dr. Simone Filice
3. Open Q&A session

The event will be recorded and made available on the LiveRAG Hugging Face webpage for those who cannot attend live.

We hope to virtually see you all next Tuesday!

Meeting details:
MS Teams link, Meeting ID: 334 425 107 882, Passcode: th9WK28b

To add the event to your calendar and RSVP, please download LiveRAG_Online_Session.ics and import it into your preferred calendar application.

April 4:

We are pleased to announce that DataMorgana now supports multi-document Q&A generation! You can now configure DataMorgana calls to generate Q&A spanning one or two documents. We encourage you to train your RAG systems also with multi-document Q&A, as this will be tested during the Live Challenge Day. For more details, please refer to the updated API documentation and the revised Sandbox manual.