Spaces:

---

LiveRAG

/

Challenge

like 0

Sleeping

I don't know whether it is the right place to post a question.
But here's my short question.

When using prebuilt index of opensearch using the described method in https://huggingface.co/spaces/LiveRAG/Challenge/blob/main/Operational_Instructions/Indices_Usage_Examples_for_LiveRAG.ipynb
I've got error regarding timeout as following

opensearchpy.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeout(HTTPSConnectionPool(host='search-index01-zc4xlabgpncm3uqmerfedq5nx4.us-east-1.es.amazonaws.com', port=443): Read timed out. (read timeout=10))

Should I increase the timeout value or is there other workarounds?

You should increase the timeout, yes.
You can do so by setting client.search(..., timeout=)
When querying opensearch for many documents it sometimes takes longer to reply. It's still more efficient than query one by one.

You should increase the timeout, yes.
You can do so by setting client.search(..., timeout=)
When querying opensearch for many documents it sometimes takes longer to reply. It's still more efficient than query one by one.

Thank you for your response :D
With timeout it works now.

Has anyone encountered the same problem as me? When using the pinecone usage section in https://huggingface.co/spaces/LiveRAG/Challenge/blob/main/Operational_Instructions/Indices_Usage_Examples_for_LiveRAG.ipynb
, the pre-built indexes do not seem to exist.
I changed the api_key part to the credit key I applied for on the pinecone platform.

Has anyone encountered the same problem as me? When using the pinecone usage section in https://huggingface.co/spaces/LiveRAG/Challenge/blob/main/Operational_Instructions/Indices_Usage_Examples_for_LiveRAG.ipynb
, the pre-built indexes do not seem to exist.
I changed the api_key part to the credit key I applied for on the pinecone platform.

@WinstonCHEN1 Looks like the issues is that you changed the api_key.
The pre-built indexes are hosted under the account created by the competition organizers. When using them you therefore must use the API key stored under the readonly shared AWS account using get_ssm_secret("/pinecone/ro_token").

If you build your own index then of course you need a different key but if you choose to use the prebuilt index you must use this api_key.

During the Team's call, in the Resources section, it was stated that we would have free access to Falcon-10B-Instruct API via the AI71 platform using the same account as is receiving the 10K DataMorgana Q&A pairs.

Upon using the AI71 client to access that model, I note a usage charge of $0.01 (after 5 response generations).

Did I misunderstand the presentation in the call, or am I misusing the AI71 API for the model?

During the Team's call, in the Resources section, it was stated that we would have free access to Falcon-10B-Instruct API via the AI71 platform using the same account as is receiving the 10K DataMorgana Q&A pairs.
Upon using the AI71 client to access that model, I note a usage charge of $0.01 (after 5 response generations).
Did I misunderstand the presentation in the call, or am I misusing the AI71 API for the model?

Hi @dfisher-osc your understanding is correct, I will check internally and update.

Is there any problem with bulk generation with DataMorgana? I am constantly getting error with provided documentation: Unprocessable Entity for url: https://api.ai71.ai/v1/bulk_generation

Also, what is the query / hour limit of the API so we don't face 429 (rate limit) error?

Is there any problem with bulk generation with DataMorgana? I am constantly getting error with provided documentation: Unprocessable Entity for url: https://api.ai71.ai/v1/bulk_generation

Hi @alireza7 we are looking into the matter and get back to you soon

During the Team's call, in the Resources section, it was stated that we would have free access to Falcon-10B-Instruct API via the AI71 platform using the same account as is receiving the 10K DataMorgana Q&A pairs.

Upon using the AI71 client to access that model, I note a usage charge of $0.01 (after 5 response generations).

Did I misunderstand the presentation in the call, or am I misusing the AI71 API for the model?

@dfisher-osc here's the response from AI71: To take advantage of the free credit benefits, the API key must be assigned to the whitelisted user. Based on their checks, the API key that your probably used was not assigned.
Click API Keys in the left bar and choose your API key, click the three dots (...) and assign it to the whitelisted use (your user I believe).

As a side-note, every user gets $25 free credits. Whitelisted users are allowed to use Falcon10 without limits and DataMorgana (for the duration of the competition)

Is there any problem with bulk generation with DataMorgana? I am constantly getting error with provided documentation: Unprocessable Entity for url: https://api.ai71.ai/v1/bulk_generation

Hi @alireza7 we are looking into the matter and get back to you soon

Hi @alireza7 the DataMorgana bulk generation service is up and running now. Please give it a try

Hi, when we want to generate a question with the document ID in Datamorgana Sandbox, we also encounter an error. When we try the same configuration with which we created the question before, no result is returned. Is there a problem?

@dfisher-osc here's the response from AI71: To take advantage of the free credit benefits, the API key must be assigned to the whitelisted user. Based on their checks, the API key that your probably used was not assigned.
Click API Keys in the left bar and choose your API key, click the three dots (...) and assign it to the whitelisted use (your user I believe).

As a side-note, every user gets $25 free credits. Whitelisted users are allowed to use Falcon10 without limits and DataMorgana (for the duration of the competition)

Thank you, I will try that.

Hi, when we want to generate a question with the document ID in Datamorgana Sandbox, we also encounter an error. When we try the same configuration with which we created the question before, no result is returned. Is there a problem?

Repeating your query without a doc ID generates a Q&A. Can you share the doc ID that causes the error? It might be that the document includes inappropriate content

Hi, I’m trying to watch the recording of the recent meeting. The link opens, but the video doesn’t seem to load. Is anyone else experiencing the same issue, or is it just on my end? Thanks.

Hi, I’m trying to watch the recording of the recent meeting. The link opens, but the video doesn’t seem to load. Is anyone else experiencing the same issue, or is it just on my end? Thanks.

@suveyda-genaius please try to download the video to you machine before watching it

I cannot successfully execute the code after the new cross-document generation feature appears.
It will prompt an error：requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://api.ai71.ai/v1/bulk_generation

I cannot successfully execute the code after the new cross-document generation feature appears.
It will prompt an error：requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://api.ai71.ai/v1/bulk_generation

@Silver000213 In case the issue repeats please send us your AI71 credentials and the request body to sigir2025-liverag-tech@tii.ae

I also get this bulk generation error: requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://api.ai71.ai/v1/bulk_generation.
Seems like there's a limit in n_questions as I decreased the size to small number it did not throw this error.
What is the maximum size you allow for bulk generation?

@teknology Just wanted to chime in from my experience. If a user already has a job queued or running, this kind of result might occur. I recommend trying to capture the full error details by wrapping your call in a try/except block like this :

try:
    results = bulk_generate(...)
except requests.exceptions.HTTPError as e:
    print("Status Code:", e.response.status_code)
    print("Error details:", e.response.text)  # # Should provide more context about the error

This should help you get a clearer picture of the error. Additionally, you can use the following function to check your active or past requests:

def get_all_requests():
    resp = requests.get(
        f"{BASE_URL}get_all_requests",
        headers={"Authorization": f"Bearer {get_api_key()}"},
    )
    resp.raise_for_status()
    print(json.dumps(resp.json(), indent=4))

Sometimes an existing task might still be running, which could explain the behavior you're seeing. Admin probably has deeper insights into this error. Hope this helps!

Thank you for providing the input and response json schemas.

It is unclear from the task description if the intent is to provide a separate input file for each individual question, or to present all the questions in a json array.

It is unclear from the task description if the intent is to provide a separate output file for each individual response, or to present all the responses in a json array.

Could you please clarify?

Hello, why does AI71 api_key still show insufficient balance even though I have been assigned a whitelist user?

Do we provide doc_id or chunk_id?

Thank you for providing the live day instructions.
However, it'd be great if you could clarify more on the answer file json format described at: https://huggingface.co/spaces/LiveRAG/Challenge/blob/main/Operational_Instructions/Answer_File_Example.json

According to the provided example json file, within the passages list, we need to provide each passage and their doc_ids. Do you mean chunk_id here? I don't think one passage/chunk can have multiple documents as its source...
So in the passage field, we provide a retrieved "chunk" as a value, and in the doc_ids field do we put a source document id?

Hello, it seems like our AWS account is inaccessible. Can anybody help how to resolve this?
The console shows that "We are unable to determine the status of this account. Contact your administrator for assistance".

Never mind.. please ignore my issue. I just logged-in on wrong account XD

Hello, I'm trying the ai71 api for falcon model completions.
And I'm getting the "Error code: 429 - {'detail': 'Rate limit exceeded. Try again later.'}" rate limit errors.
I wonder what is the exact rate limit value of the ai71 endpoints, and would it be unlimited on the dry run & challenge day? or not?

Hello, I'm trying the ai71 api for falcon model completions.
And I'm getting the "Error code: 429 - {'detail': 'Rate limit exceeded. Try again later.'}" rate limit errors.
I wonder what is the exact rate limit value of the ai71 endpoints, and would it be unlimited on the dry run & challenge day? or not?

I get the same HTTP error when using falcon through the a171 api, while the API key is assigned to the whitelisted user (me).

    {
  "id": 17,
  "question": "What is the capital of France?",
  "passages": [
    {
      "passage": "Paris is the capital and most populous city of France.",
      "doc_IDs": ["<urn:uuid:1234abcd-5678-efgh-9101-ijklmnopqrst>", "<urn:uuid:1234abcd-5678-efgh-9202-ijklmnopqrst>"]
    },
    {
      "passage": "France is located in Western Europe.",
      "doc_IDs": ["<urn:uuid:1234abcd-5678-efgh-9101-ijklmnopqrst>"]
    }
  ],
  "final_prompt": "Using the following - Paris is the capital and most populous city of France - and - France is located in Western Europe - answer the question: What is the capital of France?",
  "answer": "Paris" 
} 

As mentioned in this answer format, I cannot understand the "doc_IDs" field of a passage element in "passages" list.
As far as I understand, the "passages" are the list of retrieved documents using the query.
Then, how could a single passage have multiple doc_IDs?
Did you mean something that concatenation of the two documents could be a single passage, which makes the llm to answer multi-document sourced queries?

@Marcushwang - when your passage is extracted from a single doc, please provide a list with this single doc-id. If your passage is generated from multiple docs (e.g. a multi-doc summary), please provide the list of doc-ids used for generating the passage.

Thank you for providing the input and response json schemas.

It is unclear from the task description if the intent is to provide a separate input file for each individual question, or to present all the questions in a json array.

It is unclear from the task description if the intent is to provide a separate output file for each individual response, or to present all the responses in a json array.

Could you please clarify?

@dfisher-osc All questions will be provided in a single Question file, and all answers should be submitted in a single Answer file. Apologies for the confusion.

Hello, why does AI71 api_key still show insufficient balance even though I have been assigned a whitelist user?

@SSLNN The additional user should be invited to your AI71 organization and send us their credentials so we can whitelist them. If this has already been done, please let us know.

Do we provide doc_id or chunk_id?

Thank you for providing the live day instructions.
However, it'd be great if you could clarify more on the answer file json format described at: https://huggingface.co/spaces/LiveRAG/Challenge/blob/main/Operational_Instructions/Answer_File_Example.json

According to the provided example json file, within the passages list, we need to provide each passage and their doc_ids. Do you mean chunk_id here? I don't think one passage/chunk can have multiple documents as its source...
So in the passage field, we provide a retrieved "chunk" as a value, and in the doc_ids field do we put a source document id?

@teknology Please refer to @davidcarmel answer to a similar question "when your passage is extracted from a single doc, please provide a list with this single doc-id. If your passage is generated from multiple docs (e.g. a multi-doc summary), please provide the list of doc-ids used for generating the passage."

Hello, we need a quick clarification on the answer schema, in cases where our system doesn't use any "passages", is it correct to return "passages": [] or is there any other preferred structure?
Thanks!

Hello, we need a quick clarification on the answer schema, in cases where our system doesn't use any "passages", is it correct to return "passages": [] or is there any other preferred structure?
Thanks!

@mmaddipatla Yes, using "passages": [] is acceptable, and {"passage": "", "doc_IDs": []} is also fine. However, keep in mind that submitting answers without supporting passages will negatively impact your evaluation (See evaluation guidelines)

Do we provide doc_id or chunk_id?

Thank you for providing the live day instructions.
However, it'd be great if you could clarify more on the answer file json format described at: https://huggingface.co/spaces/LiveRAG/Challenge/blob/main/Operational_Instructions/Answer_File_Example.json

According to the provided example json file, within the passages list, we need to provide each passage and their doc_ids. Do you mean chunk_id here? I don't think one passage/chunk can have multiple documents as its source...
So in the passage field, we provide a retrieved "chunk" as a value, and in the doc_ids field do we put a source document id?

@teknology Please refer to @davidcarmel answer to a similar question "when your passage is extracted from a single doc, please provide a list with this single doc-id. If your passage is generated from multiple docs (e.g. a multi-doc summary), please provide the list of doc-ids used for generating the passage."

So, just to confirm we got it right.
If we just use the retrieved chunks, without modification, each would be a passage, right?

What if we have multiple (let's say 3) chunks from the same document? Should we pass each as a passage and the same doc_id multiple times (3)?
Or can/should we combine these chunks into a single "passage" and pass it with a single doc_id?

Do we provide doc_id or chunk_id?

Thank you for providing the live day instructions.
However, it'd be great if you could clarify more on the answer file json format described at: https://huggingface.co/spaces/LiveRAG/Challenge/blob/main/Operational_Instructions/Answer_File_Example.json

According to the provided example json file, within the passages list, we need to provide each passage and their doc_ids. Do you mean chunk_id here? I don't think one passage/chunk can have multiple documents as its source...
So in the passage field, we provide a retrieved "chunk" as a value, and in the doc_ids field do we put a source document id?

@teknology Please refer to @davidcarmel answer to a similar question "when your passage is extracted from a single doc, please provide a list with this single doc-id. If your passage is generated from multiple docs (e.g. a multi-doc summary), please provide the list of doc-ids used for generating the passage."

So, just to confirm we got it right.
If we just use the retrieved chunks, without modification, each would be a passage, right?

What if we have multiple (let's say 3) chunks from the same document? Should we pass each as a passage and the same doc_id multiple times (3)?
Or can/should we combine these chunks into a single "passage" and pass it with a single doc_id?

@ThePun-isher The answer to both questions is yes. Yes, a passage—whether original or processed—is considered a passage. And yes, each passage should be accompanied by the document(s) it is based on. So, if you have three passages derived from the same document, that document should be referenced with each passage.

With regard to the answer schema, the prompt is a simple string. Our system differentiates the system prompt from the user prompt.

When submitting, should we simply catenate our system prompt and user prompt, dropping the role information, to fill the final_prompt?

Just for clarification: Are we limited to 50 RPM (as per the Falcon 3 10b docs at https://platform.ai71.ai/) or not? Getting 429s and wondering if encouraging 10 parallel requests (as in https://huggingface.co/spaces/LiveRAG/Challenge/blob/main/Operational_Instructions/Falcon_Ai71_Usage.ipynb) really makes sense if we are.

With regard to the answer schema, the prompt is a simple string. Our system differentiates the system prompt from the user prompt.

When submitting, should we simply catenate our system prompt and user prompt, dropping the role information, to fill the final_prompt?

@dfisher-osc Please include the prompt you used to call the Falcon LLM in your Answer file.

Just for clarification: Are we limited to 50 RPM (as per the Falcon 3 10b docs at https://platform.ai71.ai/) or not? Getting 429s and wondering if encouraging 10 parallel requests (as in https://huggingface.co/spaces/LiveRAG/Challenge/blob/main/Operational_Instructions/Falcon_Ai71_Usage.ipynb) really makes sense if we are.

@mkrueger We're looking into the matter and will get back to you shortly.

Just for clarification: Are we limited to 50 RPM (as per the Falcon 3 10b docs at https://platform.ai71.ai/) or not? Getting 429s and wondering if encouraging 10 parallel requests (as in https://huggingface.co/spaces/LiveRAG/Challenge/blob/main/Operational_Instructions/Falcon_Ai71_Usage.ipynb) really makes sense if we are.

Sorry for the delay, the team is working on it and we were waiting for their updates.

There’s a rate limit in place but the team will increase it just for the competing teams.
We still don’t have the final numbers though.

Regardless you should apply retries with reasonable sleeps as demonstrated in the example notebook.

Just for clarification: Are we limited to 50 RPM (as per the Falcon 3 10b docs at https://platform.ai71.ai/) or not? Getting 429s and wondering if encouraging 10 parallel requests (as in https://huggingface.co/spaces/LiveRAG/Challenge/blob/main/Operational_Instructions/Falcon_Ai71_Usage.ipynb) really makes sense if we are.

Good news, the team at AI71 increased the default limits for participants to:

Up to 10 concurrent requests
And up to 500 per minute

Were the test sets identical or different during the liveday in Session 1 and Session 2?

No, the two benchmarks are not identical. Each session got a different set of 500 Q&As. About one quarter of the questions are identical, while the rest are different. All Q&As were generated by DataMorgana, using the same question/user categories. We built the two benchmarks with the goal of keeping the difficulty level as similar as possible. The joint subset of identical Q&As will be used for final calibration between the sessions. We will elaborate on the benchmark generation process In the final report .