Increase storage limit

#1
by NohTow - opened
LightOn AI org

Hello,
As detailed here, the 1GB storage limit is pretty though to do demo for database search (MS MARCO with multi vector is 27GB even with pooling factor of 1)
Would it be possible to have higher storage?
Thanks!

Hey @NohTow , Spaces already have 50GB of ephemeral storage. If you are trying to add some large files directly to the Space repo, then we can suggest uploading the datasets to a dataset repo and downloading them at the startup time of the Space. Let me know if this works for you?

Hey,

The files are not parquet-like files, they are a combination of .pt, .json and .sqlite files.
Does it work anyways? Could you share some pointer on how to use this?

Edit: else I can make it so the indexes are built at startup, but it would be quite slow and would require GPU. I think it's best to use the indexes I pre-computed.

LightOn AI org

The leetcode split of BRIGHT is 4.2Go, could be an interesting demo as well.

Tagging @hysts for his views and help on this.

Sign up or log in to comment