asdf

.gitattributes

@@ -34,6 +34,3 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 scale-hf-logo.png filter=lfs diff=lfs merge=lfs -text
 *.png filter=lfs diff=lfs merge=lfs -text
-*.mp4 filter=lfs diff=lfs merge=lfs -text
-git filter=lfs diff=lfs merge=lfs -text
-add filter=lfs diff=lfs merge=lfs -text

README.md

@@ -4,37 +4,55 @@ emoji: 🥇
 colorFrom: green
 colorTo: indigo
 sdk: gradio
-hf_oauth: true
 app_file: app.py
 pinned: true
 license: apache-2.0
 short_description: FormulaOne Leaderboard
-sdk_version: 5.42.0
-python_version: 3.12.0
+sdk_version: 5.19.0
 ---
 
+# Start the configuration
 
-### Competition terms
+Most of the variables to change for a default leaderboard are in `src/env.py` (replace the path for your leaderboard) and `src/about.py` (for tasks).
 
-- By submitting, you agree to the **FormulaOne Submission Agreement
-(v1.2)** and our **Privacy Notice**.
-- Your uploaded file remains yours; we only use it to evaluate, score,
-and contact you about your result.
-**Licensing for the public benchmark assets (informational)**
-- **Evaluator code:** Apache License 2.0
-- **Problem statements & public tests:** Creative Commons **CC BY 4.0**
-See the project's **README licence section** and full texts: `LICENSE-
-APACHE2`, `LICENSE-CC-BY` in our GitHub repo.
-**Platform**
-- Your use of Hugging Face is also governed by Hugging Face's Terms and
-Privacy Policy.
-- 
+Results files should have the following format and be stored as json files:
+```json
+{
+    "config": {
+        "model_dtype": "torch.float16", # or torch.bfloat16 or 8bit or 4bit
+        "model_name": "path of the model on the hub: org/model",
+        "model_sha": "revision on the hub",
+    },
+    "results": {
+        "task_name": {
+            "metric_name": score,
+        },
+        "task_name2": {
+            "metric_name": score,
+        }
+    }
+}
+```
 
-# FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming
+Request files are created automatically by this tool.
 
-This is the official Hugging Face space for the paper:
+If you encounter problem on the space, don't hesitate to restart it to remove the create eval-queue, eval-queue-bk, eval-results and eval-results-bk created folder.
 
-**FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming** <br>
-*Gal Beniamini, Yuval Dor, Alon Vinnikov, Shir Granot Peled, Or Weinstein, Or Sharir, Noam Wies, Tomer Nussbaum, Nadav Schweiger, Ido Ben Shaul, Tomer Zekharya, Yoav Levine, Shai Shalev-Shwartz, Amnon Shashua* <br>
-**AAI, July 2025**
+# Code logic for more complex edits
 
+You'll find 
+- the main table' columns names and properties in `src/display/utils.py`
+- the logic to read all results and request files, then convert them in dataframe lines, in `src/leaderboard/read_evals.py`, and `src/populate.py`
+- the logic to allow or filter submissions in `src/submission/submit.py` and `src/submission/check_validity.py`
+
+
+# Setting up the environment
+If you encounter issues with SentencePiece package, see this thread:
+https://github.com/google/sentencepiece/issues/1083
+
+Specifically:
+```
+brew install cmake pkg-config
+export CMAKE_POLICY_VERSION_MINIMUM=3.5
+pip install sentencepiece --no-cache-dir
+```

SECURITY.md

@@ -1,4 +0,0 @@
-To report a vulnerability or security concern, email
-security@formulaone-ai.org.
-Please include steps to reproduce and any relevant logs. We will
-acknowledge receipt within 3 business days.

TRADEMARKS.md

@@ -1,5 +0,0 @@
-"FormulaOne" is used solely to identify this research benchmark.
-You may use the name to refer to the benchmark, but you may not use it:
-- to imply sponsorship or endorsement of your project or results, or
-- as part of your own product or service name without written
-permission.

app.py

@@ -1,83 +1,68 @@
-# app.py
-
-import math
+from functools import partial
 
 import gradio as gr
+from gradio_leaderboard import Leaderboard, ColumnFilter, SelectColumns
 import pandas as pd
-import plotly.graph_objects as go
 from apscheduler.schedulers.background import BackgroundScheduler
-from gradio_leaderboard import Leaderboard, SelectColumns
-from huggingface_hub import whoami
-
-# HTML is split so we can inject Gradio media (images/video) where needed.
-from src.about import WHAT_IS_F1_HTML_AFTER_VIDEO  # text immediately after the video
-from src.about import WHAT_IS_F1_HTML_BOTTOM_A_AFTER_TABS  # text after the heading, before the first figure
-from src.about import WHAT_IS_F1_HTML_BOTTOM_A_BEFORE_TABS  # up to (and including) the "Infinite Well" heading
-from src.about import WHAT_IS_F1_HTML_EVAL_BEFORE_WARMUPFIG  # evaluation section up to before Warmup fig
-from src.about import (  # tail after Tier1 fig; ⬅️ split to insert the tabs right after the heading
+
+# from huggingface_hub import snapshot_download
+
+from src.about import (
     CITATION_BUTTON_LABEL,
     CITATION_BUTTON_TEXT,
     EVALUATION_QUEUE_TEXT,
-    SUBMISSION_TERMS_TEXT,
-    WHAT_IS_F1_HTML_AFTER_TIER1FIG_TAIL,
-    WHAT_IS_F1_HTML_TOP,
+    INTRODUCTION_TEXT,
+    LLM_BENCHMARKS_TEXT,
+    TITLE,
 )
 from src.datamodel.data import F1Data
-from src.display.css_html_js import custom_css
-from src.display.formatting import styled_error
-from src.display.utils import AutoEvalColumn, fields
-from src.envs import API, CODE_PROBLEMS_REPO, REPO_ID, RESULTS_REPO, SUBMISSIONS_REPO
-from src.logger import get_logger
-from src.populate import get_leaderboard_df
-from src.submission.submit import add_new_solutions, fetch_user_info
-from src.validation.validate import MAX_INPUT_LENGTH, MIN_INPUT_LENGTH, is_submission_file_valid, is_valid
-
-logger = get_logger(__name__)
 
-ENSURE_ALL_PRESENT = True
-SPLIT = "hard"  # warmup for debug
+from src.display.css_html_js import custom_css
 
-lbdb = F1Data(
-    cp_ds_name=CODE_PROBLEMS_REPO,
-    sub_ds_name=SUBMISSIONS_REPO,
-    res_ds_name=RESULTS_REPO,
-    split=SPLIT,
+from src.display.utils import (
+    # BENCHMARK_COLS,
+    COLS,
+    EVAL_COLS,
+    EVAL_TYPES,
+    AutoEvalColumn,
+    ModelType,
+    fields,
+    WeightType,
+    Precision,
 )
+from src.envs import API, REPO_ID, TOKEN, CODE_PROBLEMS_REPO, SUBMISSIONS_REPO, RESULTS_REPO
+from src.logger import get_logger
 
-leaderboard_df = None
+from src.populate import get_evaluation_queue_df, get_leaderboard_df
+from src.submission.submit import add_new_solutions
 
-logger.info("Initialized LBDB")
+logger = get_logger(__name__)
+
+SPLIT = "warmup"  # TODO temp
+SKIP_VALIDATION = True  # TODO temp
 
 
 def restart_space():
-    logger.info("Restarting space")
     API.restart_space(repo_id=REPO_ID)
 
 
-def refresh_leaderboard_data():
-    """Refresh the leaderboard data from the latest results"""
-    global leaderboard_df
-    try:
-        logger.info("Loading leaderboard data...")
-        new_leaderboard_df = get_leaderboard_df(RESULTS_REPO)
+lbdb = F1Data(cp_ds_name=CODE_PROBLEMS_REPO, sub_ds_name=SUBMISSIONS_REPO, res_ds_name=RESULTS_REPO, split=SPLIT)
+leaderboard_df = get_leaderboard_df(RESULTS_REPO)
 
-        if new_leaderboard_df is not None:
-            logger.info("Leaderboard data refreshed successfully")
-            leaderboard_df = new_leaderboard_df
-        else:
-            logger.warning("No new leaderboard data found")
-            return None
-    except Exception as e:
-        logger.error(f"Error refreshing leaderboard data: {e}")
-        return None
 
+logger.info("Initialized LBDB")
 
-def init_leaderboard(dataframe: pd.DataFrame):
+# (
+#     finished_eval_queue_df,
+#     running_eval_queue_df,
+#     pending_eval_queue_df,
+# ) = get_evaluation_queue_df(EVAL_REQUESTS_PATH, EVAL_COLS)
 
-    if dataframe is None:
-        raise ValueError("Leaderboard DataFrame is None.")
 
-    lb = Leaderboard(
+def init_leaderboard(dataframe):
+    if dataframe is None or dataframe.empty:
+        raise ValueError("Leaderboard DataFrame is empty or None.")
+    return Leaderboard(
         value=dataframe,
         datatype=[c.type for c in fields(AutoEvalColumn)],
         select_columns=SelectColumns(
@@ -85,510 +70,166 @@ def init_leaderboard(dataframe: pd.DataFrame):
             cant_deselect=[c.name for c in fields(AutoEvalColumn) if c.never_hidden],
             label="Select Columns to Display:",
         ),
-        search_columns=[AutoEvalColumn.system.name, AutoEvalColumn.organization.name],
+        search_columns=[AutoEvalColumn.system.name, AutoEvalColumn.system_type.name],
         hide_columns=[c.name for c in fields(AutoEvalColumn) if c.hidden],
+        filter_columns=[
+            ColumnFilter(AutoEvalColumn.system_type.name, type="checkboxgroup", label="Model types"),
+            # ColumnFilter(AutoEvalColumn.precision.name, type="checkboxgroup", label="Precision"),
+            # ColumnFilter(
+            #     AutoEvalColumn.params.name,
+            #     type="slider",
+            #     min=0.01,
+            #     max=150,
+            #     label="Select the number of parameters (B)",
+            # ),
+            # ColumnFilter(AutoEvalColumn.still_on_hub.name, type="boolean", label="Deleted/incomplete", default=True),
+        ],
         bool_checkboxgroup_label="Hide models",
         interactive=False,
     )
-    lb.col_count = (1, "fixed")
-    return lb
-
-
-def add_solution_cbk(
-    system_name: str,
-    org: str,
-    submission_path: str,
-    profile: gr.OAuthProfile | None,
-    oauth_token: gr.OAuthToken | None,
-):
-    logger.info("Fetching user details for submission")
-    logger.info("PROFILE %s", profile)
-    logger.info("TOKEN %s", oauth_token)
-
-    if profile is None or oauth_token is None:
-        return styled_error("Please sign in with Hugging Face before submitting.")
-
-    # Display handle and display name (may change over time)
-    logger.info(f"User handle: {profile.username}")
-    display_name = profile.name or profile.username
-    logger.info(f"Display name: {display_name}")
-
-    # Stable account id
-    user_info = fetch_user_info(oauth_token)
-    logger.info("Logged in user info: %s", user_info)
-    stable_id = user_info.get("id") if user_info else None
-    logger.info(f"User stable ID: {stable_id}")
-
-    if not stable_id:
-        return styled_error("Could not retrieve your stable user ID. Please try signing in again.")
-    user_id = stable_id
-
-    if not profile.username:
-        return styled_error("Could not retrieve username. Please try signing in again.")
-
-    try:
-        # Validating the submission file.
-        if not submission_path:
-            return styled_error("Please upload JSONL submission file.")
-
-        if not is_submission_file_valid(
-            submission_path,
-            is_warmup_dataset=(SPLIT == "warmup"),
-        ):
-            return styled_error("Failed to read JSONL submission file. Please try again later.")
-
-        # Validating all user-supplied arguments.
-        for val, val_name in [
-            (system_name, "System name"),
-            (org, "Organisation name"),
-        ]:
-            if len(val) == 0:
-                return styled_error(f"Please fill in the '{val_name}' field.")
-
-            if not is_valid(val):
-                return styled_error(
-                    f"{val_name} is invalid! Must only contain characters [a-zA-Z0-9], spaces, "
-                    + "or the special characters '-' and '.', and be of length between "
-                    + f"{MIN_INPUT_LENGTH} and {MAX_INPUT_LENGTH}."
-                )
-    except Exception:
-        logger.warning("Failed to process user submission", exc_info=True)
-        return styled_error("An error occurred. Please try again later.")  # Intentionally vague.
-
-    return add_new_solutions(
-        lbdb,
-        profile.username,
-        user_id,
-        system_name,
-        org,
-        submission_path,
-        is_warmup_dataset=(SPLIT == "warmup"),
-        ensure_all_present=ENSURE_ALL_PRESENT,
-    )
 
 
-def gate_submission(oauth_token: gr.OAuthToken | None):
-    """
-    @brief Toggles the visibility of the login box and submission panel based on the user's login status.
-    """
-    logger.info("GATE TOKEN %s", oauth_token)
-    if oauth_token is None:
-        logger.info("GATE: NO TOKEN")
-        return gr.update(visible=True), gr.update(visible=False)
-    try:
-        whoami(oauth_token.token)
-        logger.info("GATE: TOKEN IS VALID")
-        return gr.update(visible=False), gr.update(visible=True)
-    except Exception:
-        logger.info("GATE: TOKEN HAS EXPIRED")
-        return gr.update(visible=True), gr.update(visible=False)
-
-
-def get_theme():
-    # return gr.themes.Soft(
-    #     primary_hue=gr.themes.colors.blue,
-    #     secondary_hue=gr.themes.colors.sky,
-    #     neutral_hue=gr.themes.colors.gray,
-    # ).set(
-    #     body_background_fill="#FFFFFF",
-    #     panel_background_fill="#f3f4f6",
-    # )
-    return "light"
-
-
-# --- Gradio-based tabs for examples (no JS in HTML) ---
-def _select_example_tab(choice: str):
-    return (
-        gr.update(visible=(choice == "Shallow")),
-        gr.update(visible=(choice == "Deeper")),
-        gr.update(visible=(choice == "Deepest")),
-    )
+# Display image using Markdown
+# banner = "![Leaderboard Banner](file/assets/banner.png)"
 
-
-# === Static, made-up results for the landing chart (not tied to leaderboard) ===
-
-MODEL_RELEASES = {
-    "GPT-5": "2025-08-07",
-    "Gemini 2.5 Pro": "2025-03-25",
-    "Grok 4": "2025-07-09",
-    "Claude Opus 4": "2025-05-22",
-    "o3 Pro": "2025-06-10",
-}
-
-TIER_TOTALS = {"Shallow Tier": 100, "Deeper Tier": 100, "Deepest Tier": 20}
-MODELS_ORDER = ["GPT-5", "Gemini 2.5 Pro", "Grok 4", "Claude Opus 4", "o3 Pro"]
-
-ACCURACY_PCT = {
-    "Shallow Tier": {
-        "GPT-5": 49,
-        "Gemini 2.5 Pro": 30,
-        "Grok 4": 28,
-        "Claude Opus 4": 30,
-        "o3 Pro": 41,
-    },
-    "Deeper Tier": {
-        "GPT-5": 4,
-        "Gemini 2.5 Pro": 0,
-        "Grok 4": 0,
-        "Claude Opus 4": 0,
-        "o3 Pro": 1,
-    },
-    "Deepest Tier": {
-        "GPT-5": 0,
-        "Gemini 2.5 Pro": 0,
-        "Grok 4": 0,
-        "Claude Opus 4": 0,
-        "o3 Pro": 0,
-    },
-}
-
-
-def build_accuracy_figure(tier: str):
-    """Interactive scatter: x = release date (ISO str), y = accuracy (%). Hover shows solved/total."""
-    total = TIER_TOTALS[tier]
-    fig = go.Figure()
-
-    for model in MODELS_ORDER:
-        date_str = MODEL_RELEASES[model]  # e.g., "2025-08-07"
-        y = ACCURACY_PCT[tier][model]  # percent
-        solved = round(y * total / 100)
-
-        fig.add_trace(
-            go.Scatter(
-                x=[date_str],
-                y=[y],
-                mode="markers",
-                opacity=0.85,
-                name=model,  # distinct legend entry & color per model
-                marker=dict(size=8, opacity=0.85, line=dict(width=0.5)),
-                cliponaxis=False,  # let markers render over axes
-                hovertemplate=(
-                    f"<b>{model}</b><br>"
-                    "Release: %{x|%b %d, %Y}<br>"
-                    "Accuracy: %{y:.1f}%<br>"
-                    f"Solved: {solved}/{total}"
-                    "<extra></extra>"
-                ),
-            )
-        )
-
-    fig.update_layout(
-        template="plotly_white",
-        height=420,
-        margin=dict(l=30, r=120, t=10, b=40),  # extra right room for legend
-        xaxis=dict(
-            title="Model Release Date",
-            type="date",
-            tickformat="%b %Y",
-            showgrid=True,
-            title_standoff=10,  # small gap so the label doesn’t crowd the ticks
-        ),
-        yaxis=dict(
-            title="Accuracy (%)",
-            range=[0, 100],  # fixed 0–100
-            tick0=0,
-            dtick=10,
-            showgrid=True,
-            layer="below traces",  # draw axis below points so dots aren't “cut”
-        ),
-        legend=dict(title="Models", orientation="v", y=1, x=1.02, yanchor="top"),
-        hovermode="closest",
+demo = gr.Blocks(css=custom_css)
+with demo:
+    gr.Image(
+        "assets/banner.png",
+        interactive=False,
+        show_label=False,
+        show_download_button=False,
+        container=False,
     )
-    return fig
-
-
-_initial_accuracy_fig = build_accuracy_figure("Deeper Tier")
-
-# Force light theme even if HF user prefers dark
-blocks = gr.Blocks(
-    css=custom_css,
-    theme=get_theme(),
-    js="""
-() => {
-  // Force light theme (your original)
-  document.body.classList.remove('dark');
-  document.documentElement.setAttribute('data-theme','light');
-  document.documentElement.setAttribute('data-color-mode','light');
-
-  // Handle <a data-tab-target="..."> to switch Gradio tabs by panel id
-  document.addEventListener('click', (e) => {
-    const a = e.target.closest('a[data-tab-target]');
-    if (!a) return;
-    e.preventDefault();
-    const id = a.getAttribute('data-tab-target'); // e.g., "what-is"
-    const panel = document.getElementById(id);
-    if (!panel) return;
-
-    // Find the tab header button that controls this panel and click it
-    const btn = document.querySelector(`[role="tab"][aria-controls="${panel.id}"]`);
-    if (btn) btn.click();
-  }, true);
-}
-""",
-)
-with blocks:
-
-    with gr.Tabs(elem_classes="tab-buttons") as tabs:
-        with gr.TabItem("FormulaOne", id=0, elem_id="landing-accuracy-tab"):
-
-            gr.HTML(
-                '<div align="center"><header class="text-center mb-12"><h1 class="text-4xl md:text-5xl font-bold text-gray-900 f1-h1" style="margin:0; display:inline;">FormulaOne</h1><span style="display:inline-block; margin-left:0.5em;"><h3 style="margin:0; display:inline;" class="text-4xl md:text-5xl font-bold text-gray-900 f1-h3 style=">by <a href="https://doubleai.com/">AAI</a></h3></header></div>'
-            )
-            with gr.Row(elem_id="landing-hero-row"):
-                with gr.Column(scale=7, elem_id="landing-hero-left"):
-                    gr.Markdown(
-                        """
-        <div class="f1-container">
-        <p class="f1-hero">
-            A benchmark of novel, expert-level algorithmic problems over graphs that demand deep dynamic
-            programming and logical reasoning. <strong>Shallow</strong> and <strong>Deeper</strong> tiers span moderate through
-            challenging problems, while <strong>Deepest</strong> is research-level.
-        </p>
-        </div>
-        """,
-                        elem_classes="markdown-text",
-                    )
-                with gr.Column(scale=3, elem_id="landing-hero-right"):
-                    learn_more_btn = gr.Button(
-                        "Learn More about FormulaOne",
-                        elem_id="learn-more-pill",
-                        variant="secondary",
-                    )
-
-            # Make the pill switch to the "What is FormulaOne" tab
-            learn_more_btn.click(
-                lambda: gr.Tabs(selected="what-is"),  # switches tabs
-                inputs=None,
-                outputs=tabs,  # 'tabs' is your Tabs handle
-            )
-            # Pill-style selector aligned to the top-right
-            with gr.Row(elem_id="f1-tier-select-row"):
-                tier_selector = gr.Radio(
-                    choices=list(reversed(list(TIER_TOTALS.keys()))),
-                    value="Deeper Tier",
-                    label=None,
-                    show_label=False,
-                    elem_id="f1-tier-select",
-                )
 
-            accuracy_plot = gr.Plot(
-                value=_initial_accuracy_fig,
-                elem_id="f1-accuracy-plot",
-                show_label=False,
-            )
-
-            tier_selector.change(
-                lambda t: build_accuracy_figure(t),
-                inputs=tier_selector,
-                outputs=accuracy_plot,
-            )
-
-            # Footnote (sampling + prompt details)
-            gr.Markdown(
-                """
-<div class="f1-container">
-  <p class="f1-p" style="font-size:0.95rem;color:var(--f1-subtle);">
-    All models were sampled with their highest available reasoning settings and a maximum token budget.
-    We also provided the models with a diverse few-shot prompt that is highly supportive for FormulaOne problems,
-    covering many of the subtle details of state design and maintenance, from a broad array of categories.
-  </p>
-</div>
-""",
-                elem_classes="markdown-text",
-            )
-
-        # Existing "What is FormulaOne" tab
-        with gr.TabItem("What is FormulaOne", id="what-is", elem_id="what-is-tab"):
-            gr.Image(
-                "assets/banner.png",
-                show_label=False,
-                elem_classes=["f1-image"],
-                show_share_button=False,
-                show_download_button=False,
-                show_fullscreen_button=False,
-                width=550,
-            )
-
-            # Top content and categories table
-            gr.HTML(WHAT_IS_F1_HTML_TOP)
-
-            # ---- Bottom content pieces interleaved with real Gradio media ----
-            # Up to and including the "An Infinite Well" heading
-            gr.HTML(WHAT_IS_F1_HTML_BOTTOM_A_BEFORE_TABS)
-
-            # ===== Examples (now right after the “Infinite Well” heading; inner width 710px via CSS) =====
-            with gr.Group(elem_id="f1-examples", elem_classes=["f1-container"]):
-                gr.HTML(
-                    '<div class="f1-tabs-body"><div class="f1-examples-chip">Examples of FormulaOne problems</div></div>'
-                )
+    # gr.Markdown(banner)
+    gr.HTML(
+        """
+        <style>
+            body {
+                background-color: #121212;
+                color: white;
+                margin: 0;  /* Reset browser default */
+            }
+
+            /* Outer container margin & spacing */
+            .gradio-container {
+                max-width: 1100px;
+                margin: 2rem auto;        /* top/bottom spacing + horizontal centering */
+                padding: 2rem;            /* inner spacing */
+                background-color: rgba(0, 0, 0, 0.6);  /* optional: semi-transparent panel */
+                border-radius: 12px;      /* rounded corners */
+            }
+        </style>
+        """
+    )
 
-                _latex = [
-                    {"left": "$$", "right": "$$", "display": True},
-                    {"left": "$", "right": "$", "display": False},
-                    {"left": "\\(", "right": "\\)", "display": False},
-                    {"left": "\\[", "right": "\\]", "display": True},
-                ]
-
-                md_warmup = gr.Markdown(
-                    value=(
-                        '<p style="text-align: center;"><code>Union-of-Paths-and-Cycles</code></p>\n'
-                        "Given a tree-like graph $G=(V,E)$ and a weight function $w:V\\to\\mathbb{N}$, compute the sum of all weights of sets $S\\subseteq V$ such that the induced subgraph $G[S]$ is a <b>disjoint union of paths and cycles</b>."
-                    ),
-                    latex_delimiters=_latex,
-                    elem_classes=["f1-problem-markdown"],
-                )
-                md_tier1 = gr.Markdown(
-                    value=(
-                        '<p style="text-align: center;"><code>Maximal-Union-of-Paths-and-Cycles</code></p>\n'
-                        "Given a tree-like graph $G=(V,E)$ and a weight function $w:V\\to\\mathbb{N}$, compute the sum of all weights of sets $S\\subseteq V$ such that $G[S]$ is a <b>disjoint union of paths and cycles</b> and $S$ is <b>maximal</b> with respect to this property."
-                    ),
-                    visible=False,
-                    latex_delimiters=_latex,
-                    elem_classes=["f1-problem-markdown"],
-                )
-                md_tier2 = gr.Markdown(
-                    value=(
-                        '<p style="text-align: center;"><code>Maximal-Union-of-Cycles</code></p>\n'
-                        "Given a tree-like graph $G=(V,E)$ and a weight function $w:V\\to\\mathbb{N}$, compute the sum of all weights of sets $S\\subseteq V$ such that $G[S]$ is a <b>disjoint union of cycles</b> and $S$ is <b>maximal</b> with respect to this property."
-                    ),
-                    visible=False,
-                    latex_delimiters=_latex,
-                    elem_classes=["f1-problem-markdown"],
-                )
+    gr.HTML(TITLE)
+    gr.Markdown(INTRODUCTION_TEXT, elem_classes="markdown-text")
 
-                tab_radio = gr.Radio(
-                    choices=["Shallow", "Deeper", "Deepest"],
-                    value="Shallow",
-                    label=None,
-                    show_label=False,
-                    elem_id="f1-example-radio",
-                )
-                tab_radio.change(_select_example_tab, inputs=tab_radio, outputs=[md_warmup, md_tier1, md_tier2])
-
-            # Continue the text after the heading (before the first figure)
-            gr.HTML(WHAT_IS_F1_HTML_BOTTOM_A_AFTER_TABS)
-
-            # Video (no autoplay/loop), smaller gap to caption via CSS
-            gr.Video(
-                "assets/DominatingSetAnimation.mp4",
-                autoplay=False,
-                loop=False,
-                show_label=False,
-                interactive=False,
-                elem_classes=["f1-video"],
-                show_share_button=False,
-                show_download_button=False,
-            )
-            gr.HTML(
-                '<div class="f1-figcaption f1-figcaption-video">Brief explanation showcasing the design of a compressed dynamic programming state-space.</div>'
-            )
+    with gr.Tabs(elem_classes="tab-buttons") as tabs:
+        with gr.TabItem("🏅 FormulaOne Leaderboard", elem_id="formulaone-leaderboar-tab-table", id=0):
+            leaderboard = init_leaderboard(leaderboard_df)
 
-            gr.HTML(WHAT_IS_F1_HTML_AFTER_VIDEO)
-
-            # Evaluation: Warmup figure
-            gr.HTML(WHAT_IS_F1_HTML_EVAL_BEFORE_WARMUPFIG, padding=False)
-            gr.Image(
-                "assets/perf_plot.png",
-                width=600,
-                show_label=False,
-                elem_classes=["f1-image"],
-                show_share_button=False,
-                show_download_button=False,
-                show_fullscreen_button=False,
-            )
-            gr.HTML('<div class="f1-figcaption">Performance of frontier models on the FormulaOne dataset.</div>')
-
-            # Tail after Deeper Tier fig
-            gr.HTML(WHAT_IS_F1_HTML_AFTER_TIER1FIG_TAIL)
-
-        # Rename tab to "Leaderboard" and cap at 800px width
-        with gr.TabItem("Leaderboard", elem_id="formulaone-leaderboard-tab-table", id=2):
-            gr.Markdown(
-                """
-                Welcome to the FormulaOne leaderboard. This table tracks performance on the core FormulaOne benchmark, covering the **deeper** and **deepest** tiers (120 problems). 
-                Use the 'Select Columns to Display' dropdown to customize your view, and the search bar to find specific models or organizations.
-                """,
-                elem_classes="markdown-text",
-            )
-            refresh_leaderboard_data()
-            assert leaderboard_df is not None
-            leaderboard_component = init_leaderboard(leaderboard_df)
+        # with gr.TabItem("📝 About", elem_id="llm-benchmark-tab-table", id=1):
+        #     logger.info("Tab about")
+        #     gr.Markdown(LLM_BENCHMARKS_TEXT, elem_classes="markdown-text")
 
-        with gr.TabItem("Submit Solutions", elem_id="formulaone-submit-tab-table", id=3):
+        with gr.TabItem("🚀 Submit here! ", elem_id="llm-benchmark-tab-table", id=2):
             logger.info("Tab submission")
             with gr.Column():
                 with gr.Row():
                     gr.Markdown(EVALUATION_QUEUE_TEXT, elem_classes="markdown-text")
 
+                # with gr.Column():
+                #     with gr.Accordion(
+                #         f"✅ Finished Evaluations ({len(finished_eval_queue_df)})",
+                #         open=False,
+                #     ):
+                #         with gr.Row():
+                #             finished_eval_table = gr.components.Dataframe(
+                #                 value=finished_eval_queue_df,
+                #                 headers=EVAL_COLS,
+                #                 datatype=EVAL_TYPES,
+                #                 row_count=5,
+                #             )
+                #     with gr.Accordion(
+                #         f"🔄 Running Evaluation Queue ({len(running_eval_queue_df)})",
+                #         open=False,
+                #     ):
+                #         with gr.Row():
+                #             running_eval_table = gr.components.Dataframe(
+                #                 value=running_eval_queue_df,
+                #                 headers=EVAL_COLS,
+                #                 datatype=EVAL_TYPES,
+                #                 row_count=5,
+                #             )
+
+                #     with gr.Accordion(
+                #         f"⏳ Pending Evaluation Queue ({len(pending_eval_queue_df)})",
+                #         open=False,
+                #     ):
+                #         with gr.Row():
+                #             pending_eval_table = gr.components.Dataframe(
+                #                 value=pending_eval_queue_df,
+                #                 headers=EVAL_COLS,
+                #                 datatype=EVAL_TYPES,
+                #                 row_count=5,
+                #             )
             with gr.Row():
-                gr.Markdown("# ✉️✨ Submit your solutions", elem_classes="markdown-text")
-
-            gr.Markdown(SUBMISSION_TERMS_TEXT, elem_classes="markdown-text")
-
-            login_box = gr.Group(visible=True, elem_id="f1-login-box")
-            with login_box:
-                gr.Markdown("Please sign in with Hugging Face to submit")
-                gr.LoginButton(elem_id="hf-login-btn")
+                gr.Markdown("# ✉️✨ Submit your solutions here!", elem_classes="markdown-text")
 
-            submit_panel = gr.Group(visible=False, elem_classes="markdown-text")
-            with submit_panel:
-                with gr.Row():
-                    with gr.Column():
-                        system_name_textbox = gr.Textbox(label=AutoEvalColumn.system.name)
-                        org_textbox = gr.Textbox(label=AutoEvalColumn.organization.name)
-                        submission_file = gr.File(label="JSONL solutions file", file_types=[".jsonl"])
-
-                        # Required checkboxes
-                        agreement_checkbox = gr.Checkbox(
-                            label="I agree to the FormulaOne Submission Agreement (v1.2).",
-                            value=False,
-                            elem_classes="markdown-text",
-                        )
-
-                        privacy_checkbox = gr.Checkbox(
-                            label="I have read the Privacy Notice.", value=False, elem_classes="markdown-text"
-                        )
-
-                        security_checkbox = gr.Checkbox(
-                            label="I confirm this submission does not attempt to access private tests or exfiltrate data.",
-                            value=False,
-                            elem_classes="markdown-text",
-                        )
-
-                        privacy_link = "https://huggingface.co/spaces/double-ai/FormulaOne-Leaderboard/blob/main/docs/privacy-policy.md"
-                        submission_agreement_link = "https://huggingface.co/spaces/double-ai/FormulaOne-Leaderboard/blob/main/terms/submission-agreement.md"
-
-                        gr.Markdown(
-                            f'<a href="{privacy_link}" target="_blank" rel="noopener noreferrer">Privacy Notice</a>; '
-                            f'<a href="{submission_agreement_link}" target="_blank" rel="noopener noreferrer">Submission Agreement</a>',
-                            elem_classes="markdown-text",
-                        )
-
-                        logger.info("Submit button")
-                        submit_button = gr.Button("Submit", variant="primary", interactive=False)
-                        submission_result = gr.Markdown()
-
-                # Update submit button interactivity based on checkboxes
-                def update_submit_button(agreement, privacy, security):
-                    return gr.update(interactive=agreement and privacy and security)
-
-                for checkbox in [agreement_checkbox, privacy_checkbox, security_checkbox]:
-                    checkbox.change(
-                        update_submit_button,
-                        inputs=[agreement_checkbox, privacy_checkbox, security_checkbox],
-                        outputs=submit_button,
+            with gr.Row():
+                with gr.Column():
+                    system_name_textbox = gr.Textbox(label=AutoEvalColumn.system.name)
+                    org_textbox = gr.Textbox(label=AutoEvalColumn.organization.name)
+                    # revision_name_textbox = gr.Textbox(label="Revision commit", placeholder="main")
+                    sys_type_dropdown = gr.Dropdown(
+                        choices=[t.to_str(" ") for t in ModelType],
+                        label=AutoEvalColumn.system_type.name,
+                        multiselect=False,
+                        value=ModelType.LLM.to_str(" "),
+                        interactive=True,
                     )
 
-                submit_button.click(
-                    add_solution_cbk,
-                    [
-                        system_name_textbox,
-                        org_textbox,
-                        submission_file,
-                    ],
-                    submission_result,
+                    # with gr.Column():
+                    submission_file = gr.File(label="JSONL solutions file", file_types=[".jsonl"])
+                    # precision = gr.Dropdown(
+                    #     choices=[i.value.name for i in Precision if i != Precision.Unknown],
+                    #     label="Precision",
+                    #     multiselect=False,
+                    #     value="float16",
+                    #     interactive=True,
+                    # )
+                    # weight_type = gr.Dropdown(
+                    #     choices=[i.value.name for i in WeightType],
+                    #     label="Weights type",
+                    #     multiselect=False,
+                    #     value="Original",
+                    #     interactive=True,
+                    # )
+                    # base_model_name_textbox = gr.Textbox(label="Base model (for delta or adapter weights)")
+
+            logger.info("Submit button")
+            submit_button = gr.Button("Submit")
+            submission_result = gr.Markdown()
+
+            def add_solution_cbk(system_name, org, sys_type, submission_path):
+                return add_new_solutions(
+                    lbdb, system_name, org, sys_type, submission_path, skip_validation=SKIP_VALIDATION
                 )
 
+            submit_button.click(
+                add_solution_cbk,
+                [
+                    system_name_textbox,
+                    org_textbox,
+                    sys_type_dropdown,
+                    submission_file,
+                ],
+                submission_result,
+            )
+
     with gr.Row():
         logger.info("Citation")
         with gr.Accordion(CITATION_BUTTON_LABEL, open=False):
@@ -596,17 +237,18 @@ with blocks:
                 value=CITATION_BUTTON_TEXT.strip(),
                 elem_id="citation-block",
             )
-
-    blocks.load(lambda: leaderboard_df, inputs=[], outputs=[leaderboard_component])
-    blocks.load(gate_submission, inputs=None, outputs=[login_box, submit_panel])
-
+            # citation_button = gr.Textbox(
+            #     value=CITATION_BUTTON_TEXT,
+            #     # label=CITATION_BUTTON_LABEL,
+            #     lines=20,
+            #     elem_id="citation-button",
+            #     show_copy_button=True,
+            # )
 
 logger.info("Scheduler")
 scheduler = BackgroundScheduler()
 scheduler.add_job(restart_space, "interval", seconds=1800)
-scheduler.add_job(refresh_leaderboard_data, "interval", seconds=120)
 scheduler.start()
 logger.info("Launch")
-blocks.queue(default_concurrency_limit=40).launch()
-
+demo.queue(default_concurrency_limit=40).launch()
 logger.info("Done")

assets/DominatingSetAnimation.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:1b4f64a0f3d3e4a662214dfda8b1cf3a13dffbc7a2698a70e6bef15b0bddd3f6
-size 10836895

docs/privacy-policy.md

@@ -1,56 +0,0 @@
-FormulaOne AI Coding Challenge - Privacy Notice v1.2
-Last updated: 6 Aug 2025
-
-1. Controller
-FormulaOne Team, <legal entity and address>
-Contact: privacy@formulaone-ai.org
-
-2. Data We Collect
-- GitHub or Hugging Face username and public profile ID
-- Email address (if supplied via OAuth)
-- Uploaded Submission files, compile/runtime logs, numeric scores
-- IP address and browser telemetry for security and rate limiting
-
-3. Purposes and GDPR Legal Bases
-
-| Purpose                                      | Legal basis (GDPR)            |
-|----------------------------------------------|-------------------------------|
-| Evaluate and rank submissions                | Contract (6(1)(b))            |
-| Site security and abuse prevention           | Legitimate interests (6(1)(f))|
-| Academic, non-commercial research on models  | Legitimate interests (6(1)(f))|
-| Contacting participants about results/collab | Legitimate interests (6(1)(f))|
-
-
-4. Public Disclosure
-Your display name, score and rank appear on a public leaderboard.
-You may use a pseudonym.
-5. Data Sharing
-Processors: Hugging Face Inc., GitHub Inc., <cloud host> - under
-written contracts.
-Independent controllers: GitHub and Hugging Face regarding your
-platform accounts.
-6. International Transfers
-Where data moves outside the EEA/UK, we rely on EU Standard
-Contractual Clauses or adequacy decisions.
-7. Retention
-- Leaderboard entries: indefinitely.
-- Raw submission files and logs: up to 5 years, then deleted or
-anonymised.
-- Security logs: 12 months.
-8. Your Rights
-Access, correction, erasure, restriction, portability, and
-objection. Contact us at the address above.
-You may lodge a complaint with your supervisory authority.
-9. Security
-Submissions run in network-restricted containers. Access is limited
-to authorised staff using multi-factor authentication.
-10. Cookies
-If you use only GitHub and Hugging Face, their cookies apply under
-their policies. If you later self-host a site for the leaderboard,
-publish your own cookie notice.
-11. Children
-The Competition is not directed to children under 16. Do not submit
-personal data if you are under 16.
-12. Changes
-We will post updates here and notify registered participants by
-email when material.

dummy_submission.jsonl

@@ -1,2 +0,0 @@
-{"problem_id": "142", "solution": "from dataclasses import dataclass\nfrom typing import TypeAlias\n\nfrom evaluation.graph_structure import Graph\n\nMOD = 1_000_000_007\n\n\n@dataclass(frozen=True)\nclass DPState:\n    \"\"\"\n    Dynamic Programming state for the Dominating Set problem on tree decompositions.\n    This dataclass encapsulates the complete state information needed at each bag\n    during the tree decomposition-based dynamic programming algorithm.\n\n    Fields:\n    ( 1 ) assign_mask : int        Bitmask indicating which vertices in the current bag\n                                   are assigned to the dominating set (1 = IN, 0 = OUT).\n                                   Bit i corresponds to the i-th vertex in the sorted bag.\n\n    ( 2 ) need_mask   : int        Bitmask indicating which OUT vertices in the current bag\n                                   still require domination (1 = needs domination, 0 = already dominated).\n                                   Only meaningful for OUT vertices (assign_mask bit = 0).\n                                   IN vertices never have the need bit set since they dominate themselves.\n\n    State Invariants:\n    - For any bit position i: if (assign_mask >> i) & 1 == 1, then (need_mask >> i) & 1 == 0\n    - The need_mask only tracks vertices that are OUT and not yet dominated by adjacent IN vertices\n    - When a vertex is forgotten, it must not have the need bit set (invalid state otherwise)\n\n    Usage in Algorithm:\n    - LEAF: Initialize with vertex either IN (assign=1, need=0) or OUT (assign=0, need=1)\n    - INTRODUCE: Insert new bit at appropriate position, update domination status\n    - FORGET: Remove bit at appropriate position, reject states with undominated OUT vertices\n    - JOIN: Merge compatible states (same assignment), combine need masks with AND operation\n\n    All fields are immutable and hashable, making this object suitable as a dictionary key.\n    \"\"\"\n\n    assign_mask: int\n    need_mask: int\n\n\n@dataclass\nclass DPValue:\n    \"\"\"\n    Dynamic Programming value representing the computational result for a given state.\n    This dataclass replaces the previous Tuple[int, int] representation with a more\n    structured and self-documenting approach for weighted model counting.\n\n    Fields:\n    ( 1 ) count  : int             Number of distinct vertex subsets (dominating sets) that\n                                   achieve the current state configuration. This counts the\n                                   multiplicity of solutions that lead to the same DP state.\n\n    ( 2 ) weight : int             Total weighted sum across all vertex subsets that achieve\n                                   the current state configuration. Each dominating set contributes\n                                   its total vertex weight sum to this field.\n\n    Implementation Details:\n    - Both fields are maintained modulo MOD (1,000,000,007)\n    - The count field enables tracking the number of valid dominating sets\n    - The weight field accumulates the total weight contribution from all valid sets\n    - When combining values from different DP branches:\n      * Counts are multiplied for independent choices\n      * Weights are combined using inclusion-exclusion principle to avoid double-counting\n\n    This structure enables simultaneous tracking of both solution count and cumulative\n    weight during the tree decomposition DP computation.\n    \"\"\"\n\n    count: int\n    weight: int\n\n\nBag: TypeAlias = set[int]  # a bag is a set of vertices\n\n\ndef insert_bit(\n    mask: int,\n    pos: int,\n    bit: int,\n) -> int:\n    \"\"\"\n    Insert `bit` (0/1) at position `pos` (LSB == position 0) in `mask`\n    shifting higher bits left by one.\n    \"\"\"\n    lower = mask & ((1 << pos) - 1)\n    higher = mask >> pos\n    return lower | (bit << pos) | (higher << (pos + 1))\n\n\ndef remove_bit(\n    mask: int,\n    pos: int,\n) -> int:\n    \"\"\"\n    Delete the bit at position `pos` from `mask`, shifting higher bits right.\n    \"\"\"\n    lower = mask & ((1 << pos) - 1)\n    higher = mask >> (pos + 1)\n    return lower | (higher << pos)\n\n\ndef bag_tuple(bag: Bag) -> tuple[int, ...]:\n    return tuple(sorted(bag))\n\n\ndef bag_selected_weight(\n    assign_mask: int,\n    bag_vertices: tuple[int, ...],\n    vertex_weights: dict[int, int],\n) -> int:\n    \"\"\"Sum of weights of vertices in the bag that are selected (IN).\"\"\"\n    s = 0\n    for idx, v in enumerate(bag_vertices):\n        if (assign_mask >> idx) & 1:\n            s += vertex_weights[v]\n    return s % MOD\n\n\ndef accumulate(\n    table: dict[DPState, DPValue],\n    state: DPState,\n    cnt: int,\n    wsum: int,\n) -> None:\n    \"\"\"Add (cnt, wsum) to existing entry of state inside table (MOD arithmetic).\"\"\"\n    cnt %= MOD\n    wsum %= MOD\n    if state in table:\n        existing = table[state]\n        table[state] = DPValue(count=(existing.count + cnt) % MOD, weight=(existing.weight + wsum) % MOD)\n    else:\n        table[state] = DPValue(count=cnt, weight=wsum)\n\n\ndef leaf_callback(\n    graph: Graph,\n    cur_table: dict[DPState, DPValue],\n    cur_bag_info: tuple[int, Bag],\n    leaf_vertex: int,\n):\n    bag_vertices = bag_tuple(cur_bag_info[1])  # (leaf_vertex,)\n    assert len(bag_vertices) == 1 and bag_vertices[0] == leaf_vertex\n\n    w_v = graph.vertex_weights[leaf_vertex] if graph.vertex_weights else 1\n\n    # Case 1: vertex is IN the dominating set\n    assign_mask = 1  # bit 0 == 1\n    need_mask = 0  # IN vertices never need domination\n    state_in = DPState(assign_mask=assign_mask, need_mask=need_mask)\n    accumulate(cur_table, state_in, cnt=1, wsum=w_v % MOD)\n\n    # Case 2: vertex is OUT - needs domination\n    assign_mask = 0\n    need_mask = 1  # NEEDS_DOMINATION\n    state_out = DPState(assign_mask=assign_mask, need_mask=need_mask)\n    accumulate(cur_table, state_out, cnt=1, wsum=0)\n\n\ndef introduce_callback(\n    graph: Graph,\n    cur_table: dict[DPState, DPValue],\n    cur_bag_info: tuple[int, Bag],\n    child_table: dict[DPState, DPValue],\n    child_bag_info: tuple[int, Bag],\n    introduced_vertex: int,\n):\n    parent_vertices = bag_tuple(cur_bag_info[1])\n\n    # index at which the new vertex was inserted\n    idx_new = parent_vertices.index(introduced_vertex)\n    w_new = graph.vertex_weights[introduced_vertex] if graph.vertex_weights else 1\n\n    # pre-compute adjacency between introduced vertex and vertices in parent bag\n    is_adj = [(v in graph.neighbors(introduced_vertex)) for v in parent_vertices]\n\n    for child_state, dp_value in child_table.items():\n        child_assign, child_need = child_state.assign_mask, child_state.need_mask\n        cnt_child, wsum_child = dp_value.count, dp_value.weight\n        # \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n        # Choice A: new vertex is IN_X\n        # \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n        assign_in = insert_bit(child_assign, idx_new, 1)\n        need_in = insert_bit(child_need, idx_new, 0)\n\n        # when y is IN it may dominate some previously undominated OUT vertices\n        for idx, adj in enumerate(is_adj):\n            if idx == idx_new or not adj:\n                continue\n            # vertex idx is OUT?\n            if (assign_in >> idx) & 1:\n                continue  # IN vertices never carry NEED flag\n            # if it was NEED, clear it\n            if (need_in >> idx) & 1:\n                need_in &= ~(1 << idx)\n\n        cnt_new = cnt_child\n        wsum_new = (wsum_child + cnt_child * w_new) % MOD\n        state_in = DPState(assign_mask=assign_in, need_mask=need_in)\n        accumulate(cur_table, state_in, cnt_new, wsum_new)\n\n        # \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n        # Choice B: new vertex is NOT_IN_X\n        # \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n        assign_out = insert_bit(child_assign, idx_new, 0)\n\n        # Determine if introduced vertex is already dominated by some\n        # IN vertex present in the (extended) bag.\n        dominated = False\n        for idx, adj in enumerate(is_adj):\n            if idx == idx_new or not adj:\n                continue\n            if (assign_out >> idx) & 1:  # neighbor is IN\n                dominated = True\n                break\n        need_bit = 0 if dominated else 1\n        need_out = insert_bit(child_need, idx_new, need_bit)\n\n        # ( no other vertices change status )\n        state_out = DPState(assign_mask=assign_out, need_mask=need_out)\n        accumulate(cur_table, state_out, cnt_child, wsum_child)\n\n\ndef forget_callback(\n    graph: Graph,\n    cur_table: dict[DPState, DPValue],\n    cur_bag_info: tuple[int, Bag],\n    child_table: dict[DPState, DPValue],\n    child_bag_info: tuple[int, Bag],\n    forgotten_vertex: int,\n):\n    child_vertices = bag_tuple(child_bag_info[1])\n    idx_forgot = child_vertices.index(forgotten_vertex)\n\n    for child_state, dp_value in child_table.items():\n        assign_child, need_child = child_state.assign_mask, child_state.need_mask\n        cnt_child, wsum_child = dp_value.count, dp_value.weight\n        bit_assign = (assign_child >> idx_forgot) & 1\n        bit_need = (need_child >> idx_forgot) & 1\n\n        # If forgotten vertex is OUT and still needs domination -> invalid state\n        if bit_assign == 0 and bit_need == 1:\n            continue\n\n        assign_par = remove_bit(assign_child, idx_forgot)\n        need_par = remove_bit(need_child, idx_forgot)\n\n        state_par = DPState(assign_mask=assign_par, need_mask=need_par)\n        accumulate(cur_table, state_par, cnt_child, wsum_child)\n\n\ndef join_callback(\n    graph: Graph,\n    cur_table: dict[DPState, DPValue],\n    cur_bag_info: tuple[int, Bag],\n    left_child_table: dict[DPState, DPValue],\n    left_child_bag_info: tuple[int, Bag],\n    right_child_table: dict[DPState, DPValue],\n    right_child_bag_info: tuple[int, Bag],\n):\n    bag_vertices = bag_tuple(cur_bag_info[1])\n    vertex_weights = graph.vertex_weights\n    assert vertex_weights is not None\n\n    # Group right states by assignment mask for O(|L| + |R|) compatibility\n    right_by_assign: dict[int, list[tuple[int, int, int]]] = {}\n    for right_state, dp_value in right_child_table.items():\n        assign_r, need_r = right_state.assign_mask, right_state.need_mask\n        cnt_r, wsum_r = dp_value.count, dp_value.weight\n        right_by_assign.setdefault(assign_r, []).append((need_r, cnt_r, wsum_r))\n\n    for left_state, dp_value in left_child_table.items():\n        assign_l, need_l = left_state.assign_mask, left_state.need_mask\n        cnt_l, wsum_l = dp_value.count, dp_value.weight\n        if assign_l not in right_by_assign:\n            continue\n        for need_r, cnt_r, wsum_r in right_by_assign[assign_l]:\n            # Merge NEED flags: dominated if dominated in either side\n            need_merge = need_l & need_r  # bitwise AND keeps 1 only if both have NEED\n\n            cnt_merge = (cnt_l * cnt_r) % MOD\n\n            w_bag_sel = bag_selected_weight(assign_l, bag_vertices, vertex_weights)\n            w_merge = (wsum_l * cnt_r + wsum_r * cnt_l - cnt_merge * w_bag_sel) % MOD\n            if w_merge < 0:\n                w_merge += MOD\n\n            state_merge = DPState(assign_mask=assign_l, need_mask=need_merge)\n            accumulate(cur_table, state_merge, cnt_merge, w_merge)\n\n\ndef extract_solution(root_table: dict[DPState, DPValue]) -> int:\n    \"\"\"\n    Sum the total weights of all globally valid dominating sets.\n    Return -1 if none exist.\n    \"\"\"\n    answer = 0\n    found = False\n    for state, dp_value in root_table.items():\n        assign_mask, need_mask = state.assign_mask, state.need_mask\n        cnt, wsum = dp_value.count, dp_value.weight\n        # Bag may be empty or not\n        if assign_mask == 0 and need_mask == 0 and cnt == 0:\n            # Defensive - shouldn't happen\n            continue\n        if need_mask != 0:\n            # some vertex in root bag still needs domination -> invalid\n            continue\n        answer = (answer + wsum) % MOD\n        found = True\n    return answer if found else -1\n"}
-{"problem_id": "120", "solution": "from dataclasses import dataclass\nfrom typing import TypeAlias\n\nfrom evaluation.graph_structure import Graph\n\nMOD = 1_000_000_007\n\n\n@dataclass(frozen=True)\nclass DPState:\n    \"\"\"\n    Dynamic Programming state for the Dominating Set problem on tree decompositions.\n    This dataclass encapsulates the complete state information needed at each bag\n    during the tree decomposition-based dynamic programming algorithm.\n\n    Fields:\n    ( 1 ) assign_mask : int        Bitmask indicating which vertices in the current bag\n                                   are assigned to the dominating set (1 = IN, 0 = OUT).\n                                   Bit i corresponds to the i-th vertex in the sorted bag.\n\n    ( 2 ) need_mask   : int        Bitmask indicating which OUT vertices in the current bag\n                                   still require domination (1 = needs domination, 0 = already dominated).\n                                   Only meaningful for OUT vertices (assign_mask bit = 0).\n                                   IN vertices never have the need bit set since they dominate themselves.\n\n    State Invariants:\n    - For any bit position i: if (assign_mask >> i) & 1 == 1, then (need_mask >> i) & 1 == 0\n    - The need_mask only tracks vertices that are OUT and not yet dominated by adjacent IN vertices\n    - When a vertex is forgotten, it must not have the need bit set (invalid state otherwise)\n\n    Usage in Algorithm:\n    - LEAF: Initialize with vertex either IN (assign=1, need=0) or OUT (assign=0, need=1)\n    - INTRODUCE: Insert new bit at appropriate position, update domination status\n    - FORGET: Remove bit at appropriate position, reject states with undominated OUT vertices\n    - JOIN: Merge compatible states (same assignment), combine need masks with AND operation\n\n    All fields are immutable and hashable, making this object suitable as a dictionary key.\n    \"\"\"\n\n    assign_mask: int\n    need_mask: int\n\n\n@dataclass\nclass DPValue:\n    \"\"\"\n    Dynamic Programming value representing the computational result for a given state.\n    This dataclass replaces the previous Tuple[int, int] representation with a more\n    structured and self-documenting approach for weighted model counting.\n\n    Fields:\n    ( 1 ) count  : int             Number of distinct vertex subsets (dominating sets) that\n                                   achieve the current state configuration. This counts the\n                                   multiplicity of solutions that lead to the same DP state.\n\n    ( 2 ) weight : int             Total weighted sum across all vertex subsets that achieve\n                                   the current state configuration. Each dominating set contributes\n                                   its total vertex weight sum to this field.\n\n    Implementation Details:\n    - Both fields are maintained modulo MOD (1,000,000,007)\n    - The count field enables tracking the number of valid dominating sets\n    - The weight field accumulates the total weight contribution from all valid sets\n    - When combining values from different DP branches:\n      * Counts are multiplied for independent choices\n      * Weights are combined using inclusion-exclusion principle to avoid double-counting\n\n    This structure enables simultaneous tracking of both solution count and cumulative\n    weight during the tree decomposition DP computation.\n    \"\"\"\n\n    count: int\n    weight: int\n\n\nBag: TypeAlias = set[int]  # a bag is a set of vertices\n\n\ndef insert_bit(\n    mask: int,\n    pos: int,\n    bit: int,\n) -> int:\n    \"\"\"\n    Insert `bit` (0/1) at position `pos` (LSB == position 0) in `mask`\n    shifting higher bits left by one.\n    \"\"\"\n    lower = mask & ((1 << pos) - 1)\n    higher = mask >> pos\n    return lower | (bit << pos) | (higher << (pos + 1))\n\n\ndef remove_bit(\n    mask: int,\n    pos: int,\n) -> int:\n    \"\"\"\n    Delete the bit at position `pos` from `mask`, shifting higher bits right.\n    \"\"\"\n    lower = mask & ((1 << pos) - 1)\n    higher = mask >> (pos + 1)\n    return lower | (higher << pos)\n\n\ndef bag_tuple(bag: Bag) -> tuple[int, ...]:\n    return tuple(sorted(bag))\n\n\ndef bag_selected_weight(\n    assign_mask: int,\n    bag_vertices: tuple[int, ...],\n    vertex_weights: dict[int, int],\n) -> int:\n    \"\"\"Sum of weights of vertices in the bag that are selected (IN).\"\"\"\n    s = 0\n    for idx, v in enumerate(bag_vertices):\n        if (assign_mask >> idx) & 1:\n            s += vertex_weights[v]\n    return s % MOD\n\n\ndef accumulate(\n    table: dict[DPState, DPValue],\n    state: DPState,\n    cnt: int,\n    wsum: int,\n) -> None:\n    \"\"\"Add (cnt, wsum) to existing entry of state inside table (MOD arithmetic).\"\"\"\n    cnt %= MOD\n    wsum %= MOD\n    if state in table:\n        existing = table[state]\n        table[state] = DPValue(count=(existing.count + cnt) % MOD, weight=(existing.weight + wsum) % MOD)\n    else:\n        table[state] = DPValue(count=cnt, weight=wsum)\n\n\ndef leaf_callback(\n    graph: Graph,\n    cur_table: dict[DPState, DPValue],\n    cur_bag_info: tuple[int, Bag],\n    leaf_vertex: int,\n):\n    bag_vertices = bag_tuple(cur_bag_info[1])  # (leaf_vertex,)\n    assert len(bag_vertices) == 1 and bag_vertices[0] == leaf_vertex\n\n    w_v = graph.vertex_weights[leaf_vertex] if graph.vertex_weights else 1\n\n    # Case 1: vertex is IN the dominating set\n    assign_mask = 1  # bit 0 == 1\n    need_mask = 0  # IN vertices never need domination\n    state_in = DPState(assign_mask=assign_mask, need_mask=need_mask)\n    accumulate(cur_table, state_in, cnt=1, wsum=w_v % MOD)\n\n    # Case 2: vertex is OUT - needs domination\n    assign_mask = 0\n    need_mask = 1  # NEEDS_DOMINATION\n    state_out = DPState(assign_mask=assign_mask, need_mask=need_mask)\n    accumulate(cur_table, state_out, cnt=1, wsum=0)\n\n\ndef introduce_callback(\n    graph: Graph,\n    cur_table: dict[DPState, DPValue],\n    cur_bag_info: tuple[int, Bag],\n    child_table: dict[DPState, DPValue],\n    child_bag_info: tuple[int, Bag],\n    introduced_vertex: int,\n):\n    parent_vertices = bag_tuple(cur_bag_info[1])\n\n    # index at which the new vertex was inserted\n    idx_new = parent_vertices.index(introduced_vertex)\n    w_new = graph.vertex_weights[introduced_vertex] if graph.vertex_weights else 1\n\n    # pre-compute adjacency between introduced vertex and vertices in parent bag\n    is_adj = [(v in graph.neighbors(introduced_vertex)) for v in parent_vertices]\n\n    for child_state, dp_value in child_table.items():\n        child_assign, child_need = child_state.assign_mask, child_state.need_mask\n        cnt_child, wsum_child = dp_value.count, dp_value.weight\n        # \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n        # Choice A: new vertex is IN_X\n        # \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n        assign_in = insert_bit(child_assign, idx_new, 1)\n        need_in = insert_bit(child_need, idx_new, 0)\n\n        # when y is IN it may dominate some previously undominated OUT vertices\n        for idx, adj in enumerate(is_adj):\n            if idx == idx_new or not adj:\n                continue\n            # vertex idx is OUT?\n            if (assign_in >> idx) & 1:\n                continue  # IN vertices never carry NEED flag\n            # if it was NEED, clear it\n            if (need_in >> idx) & 1:\n                need_in &= ~(1 << idx)\n\n        cnt_new = cnt_child\n        wsum_new = (wsum_child + cnt_child * w_new) % MOD\n        state_in = DPState(assign_mask=assign_in, need_mask=need_in)\n        accumulate(cur_table, state_in, cnt_new, wsum_new)\n\n        # \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n        # Choice B: new vertex is NOT_IN_X\n        # \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n        assign_out = insert_bit(child_assign, idx_new, 0)\n\n        # Determine if introduced vertex is already dominated by some\n        # IN vertex present in the (extended) bag.\n        dominated = False\n        for idx, adj in enumerate(is_adj):\n            if idx == idx_new or not adj:\n                continue\n            if (assign_out >> idx) & 1:  # neighbor is IN\n                dominated = True\n                break\n        need_bit = 0 if dominated else 1\n        need_out = insert_bit(child_need, idx_new, need_bit)\n\n        # ( no other vertices change status )\n        state_out = DPState(assign_mask=assign_out, need_mask=need_out)\n        accumulate(cur_table, state_out, cnt_child, wsum_child)\n\n\ndef forget_callback(\n    graph: Graph,\n    cur_table: dict[DPState, DPValue],\n    cur_bag_info: tuple[int, Bag],\n    child_table: dict[DPState, DPValue],\n    child_bag_info: tuple[int, Bag],\n    forgotten_vertex: int,\n):\n    child_vertices = bag_tuple(child_bag_info[1])\n    idx_forgot = child_vertices.index(forgotten_vertex)\n\n    for child_state, dp_value in child_table.items():\n        assign_child, need_child = child_state.assign_mask, child_state.need_mask\n        cnt_child, wsum_child = dp_value.count, dp_value.weight\n        bit_assign = (assign_child >> idx_forgot) & 1\n        bit_need = (need_child >> idx_forgot) & 1\n\n        # If forgotten vertex is OUT and still needs domination -> invalid state\n        if bit_assign == 0 and bit_need == 1:\n            continue\n\n        assign_par = remove_bit(assign_child, idx_forgot)\n        need_par = remove_bit(need_child, idx_forgot)\n\n        state_par = DPState(assign_mask=assign_par, need_mask=need_par)\n        accumulate(cur_table, state_par, cnt_child, wsum_child)\n\n\ndef join_callback(\n    graph: Graph,\n    cur_table: dict[DPState, DPValue],\n    cur_bag_info: tuple[int, Bag],\n    left_child_table: dict[DPState, DPValue],\n    left_child_bag_info: tuple[int, Bag],\n    right_child_table: dict[DPState, DPValue],\n    right_child_bag_info: tuple[int, Bag],\n):\n    bag_vertices = bag_tuple(cur_bag_info[1])\n    vertex_weights = graph.vertex_weights\n    assert vertex_weights is not None\n\n    # Group right states by assignment mask for O(|L| + |R|) compatibility\n    right_by_assign: dict[int, list[tuple[int, int, int]]] = {}\n    for right_state, dp_value in right_child_table.items():\n        assign_r, need_r = right_state.assign_mask, right_state.need_mask\n        cnt_r, wsum_r = dp_value.count, dp_value.weight\n        right_by_assign.setdefault(assign_r, []).append((need_r, cnt_r, wsum_r))\n\n    for left_state, dp_value in left_child_table.items():\n        assign_l, need_l = left_state.assign_mask, left_state.need_mask\n        cnt_l, wsum_l = dp_value.count, dp_value.weight\n        if assign_l not in right_by_assign:\n            continue\n        for need_r, cnt_r, wsum_r in right_by_assign[assign_l]:\n            # Merge NEED flags: dominated if dominated in either side\n            need_merge = need_l & need_r  # bitwise AND keeps 1 only if both have NEED\n\n            cnt_merge = (cnt_l * cnt_r) % MOD\n\n            w_bag_sel = bag_selected_weight(assign_l, bag_vertices, vertex_weights)\n            w_merge = (wsum_l * cnt_r + wsum_r * cnt_l - cnt_merge * w_bag_sel) % MOD\n            if w_merge < 0:\n                w_merge += MOD\n\n            state_merge = DPState(assign_mask=assign_l, need_mask=need_merge)\n            accumulate(cur_table, state_merge, cnt_merge, w_merge)\n\n\ndef extract_solution(root_table: dict[DPState, DPValue]) -> int:\n    \"\"\"\n    Sum the total weights of all globally valid dominating sets.\n    Return -1 if none exist.\n    \"\"\"\n    answer = 0\n    found = False\n    for state, dp_value in root_table.items():\n        assign_mask, need_mask = state.assign_mask, state.need_mask\n        cnt, wsum = dp_value.count, dp_value.weight\n        # Bag may be empty or not\n        if assign_mask == 0 and need_mask == 0 and cnt == 0:\n            # Defensive - shouldn't happen\n            continue\n        if need_mask != 0:\n            # some vertex in root bag still needs domination -> invalid\n            continue\n        answer = (answer + wsum) % MOD\n        found = True\n    return answer if found else -1\n"}

requirements.txt

@@ -2,7 +2,7 @@ APScheduler
 black
 datasets
 pydantic==2.10.6
-gradio==5.42.0
+gradio
 gradio[oauth]
 gradio_leaderboard==0.0.13
 gradio_client
@@ -14,5 +14,4 @@ python-dateutil
 tqdm
 transformers
 tokenizers>=0.15.0
-sentencepiece
-plotly>=5
+sentencepiece

scripts/upload_f1_dataset.py

@@ -2,7 +2,6 @@ import argparse
 import fnmatch
 import json
 import os
-from typing import Iterator
 
 from datasets import Dataset
 
@@ -14,23 +13,9 @@ logger = get_logger(__name__)
 
 def get_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
-    parser.add_argument(
-        "--input_dir",
-        type=str,
-        help="Dir with .json files",
-        required=True,
-    )
-    parser.add_argument(
-        "--dataset_name",
-        type=str,
-        default=f"{CODE_PROBLEMS_REPO}",
-    )
-    parser.add_argument(
-        "--split",
-        type=str,
-        choices=["hard", "warmup"],
-        default="hard",
-    )
+    parser.add_argument("--input_dir", type=str, help="Dir with .json files", required=True)
+    parser.add_argument("--dataset_name", type=str, default=f"{CODE_PROBLEMS_REPO}")
+    parser.add_argument("--split", type=str, choices=["hard", "warmup"], default="hard")
     return parser.parse_args()
 
 
@@ -41,7 +26,7 @@ def main(args: argparse.Namespace) -> None:
         raise ValueError(f"No .json files in input dir {args.input_dir}")
     logger.info("Found %d code problems in %s", len(input_files), args.input_dir)
 
-    def ds_generator() -> Iterator[dict]:
+    def ds_generator():
         for fname in sorted(input_files):
             formula_name = os.path.splitext(fname)[0]
             cp_path = os.path.join(args.input_dir, fname)
@@ -50,7 +35,7 @@ def main(args: argparse.Namespace) -> None:
             logger.info("Read code problem for formula %s from %s", formula_name, cp_path)
             yield dict(id=code_problem["id"], code_problem=code_problem)
 
-    ds: Dataset = Dataset.from_generator(ds_generator)  # type: ignore
+    ds = Dataset.from_generator(ds_generator)
     logger.info("Created dataset")
 
     ds.push_to_hub(args.dataset_name, split=args.split, private=True)

src/about.py

@@ -1,122 +1,66 @@
-# The paper's URL for linking
-PAPER_URL = "https://arxiv.org/abs/2507.13337"
-
-WHAT_IS_F1_HTML_TOP = f"""
-<div class="f1-container">
-  <div align="center"><header class="text-center mb-12"><h1 class="text-4xl md:text-5xl font-bold text-gray-900 f1-h1" style="margin:0; display:inline;">FormulaOne</h1><span style="display:inline-block; margin-left:0.5em;"><h3 style="margin:0; display:inline;" class="text-4xl md:text-5xl font-bold text-gray-900 f1-h3 style=">by <a href="https://doubleai.com/">AAI</a></h3></header></div>
-  <section>
-    <br/>
-    <p class="text-lg mb-4 f1-p">Frontier AI models have recently demonstrated strong performance on mathematical and algorithmic benchmarks, including earning <a href="https://deepmind.google/discover/blog/advanced-version-of-gemini-with-deep-think-officially-achieves-gold-medal-standard-at-the-international-mathematical-olympiad/" target="_blank" rel="noopener noreferrer" class="f1-a">gold medals in olympiads</a>, and attaining <a href="https://arxiv.org/html/2502.06807v1" target="_blank" rel="noopener noreferrer" class="f1-a">top percentile ratings</a> in competitive programming contests. How well do such benchmarks capture the true depth of algorithmic reasoning, as it arises in real-world research problems?</p>
-
-    <p class="text-lg mb-4 f1-p">We believe that existing benchmarks fail to capture the deep reasoning skills required for complex, research-level algorithmic problems. To address this gap, <a href="{PAPER_URL}" target="_blank" rel="noopener noreferrer" class="f1-a">we introduce <strong>FormulaOne</strong></a>.</p>
-
-    <p class="mb-4 f1-p"><strong>FormulaOne</strong> consists of 220 novel dynamic programming problems over graphs. The problems are organised into three categories, ranging from moderate difficulty and all the way up to research-level.</p>
-
-    <!-- Clean, centered "table" using a single grid -->
-    <div class="f1-grid-wrap" role="region" aria-label="FormulaOne categories">
-      <div class="f1-grid-table" role="table">
-        <div class="f1-grid-row f1-grid-head" role="row">
-          <div class="f1-grid-cell" role="columnheader">Category</div>
-          <div class="f1-grid-cell" role="columnheader">Size</div>
-          <div class="f1-grid-cell" role="columnheader">Description</div>
-        </div>
-        <div class="f1-grid-row" role="row">
-          <div class="f1-grid-cell" role="cell">Shallow</div>
-          <div class="f1-grid-cell" role="cell">100</div>
-          <div class="f1-grid-cell" role="cell">A set of “easier” problems.</div>
-        </div>
-        <div class="f1-grid-row" role="row">
-          <div class="f1-grid-cell" role="cell">Deeper</div>
-          <div class="f1-grid-cell" role="cell">100</div>
-          <div class="f1-grid-cell" role="cell">A set of challenging problems.</div>
-        </div>
-        <div class="f1-grid-row" role="row">
-          <div class="f1-grid-cell" role="cell">Deepest</div>
-          <div class="f1-grid-cell" role="cell">20</div>
-          <div class="f1-grid-cell" role="cell">A set of highly challenging problems.</div>
-        </div>
-      </div>
-    </div>
-  </section>
-</div>
-"""
+from dataclasses import dataclass
+from enum import Enum
 
-# Bottom is split so we can insert real Gradio media (images/video) from app.py.
 
-# Up to (and including) the "An Infinite Well" heading — tabs are inserted immediately after
-WHAT_IS_F1_HTML_BOTTOM_A_BEFORE_TABS = """
-<div class="f1-container">
-  <section>
-    <p class="mb-4 f1-p">The latter category is incredibly demanding, requiring resolution of many points of uncertainty, and involving an array of reasoning steps, including topological and geometric insight, knowledge of mathematical domains such as extremal graph theory and logic, combinatorial considerations, precise implementation, and more.</p>
-    <p class="f1-p">Despite <a href="https://epoch.ai/frontiermath" target="_blank" rel="noopener noreferrer" class="f1-a">impressive</a> <a href="https://artificialanalysis.ai/evaluations/gpqa-diamond" target="_blank" rel="noopener noreferrer" class="f1-a">performance</a> on existing benchmarks, presently <strong>no model solves even a single 'Deepest Tier' problem</strong>.</p>
-  </section>
+@dataclass
+class Task:
+    benchmark: str
+    metric: str
+    col_name: str
 
-  <section>
-    <h2 class="f1-h2">An “Infinite Well” of Problems</h2>
-"""
 
-# After the heading (and after the tabbed examples), before the first figure
-WHAT_IS_F1_HTML_BOTTOM_A_AFTER_TABS = """
-    <p class="mb-4 f1-p">While the problems are often natural to state, their solutions are far from obvious. The solvability of this vast class of problems is guaranteed by an algorithmic <strong>meta-theorem</strong> due to <a href="https://en.wikipedia.org/wiki/Courcelle%27s_theorem" target="_blank" rel="noopener noreferrer" class="f1-a">Courcelle</a>, which broadly states:</p>
-    <blockquote class="my-6 f1-blockquote">
-        “For every sufficiently tree-like graph, any problem definable in an expressive formal logic — <a href="https://en.wikipedia.org/wiki/Monadic_second-order_logic">Monadic Second-Order (MSO) logic</a> — can be solved by a dynamic programming algorithm that operates in time linear in the order of the graph.”
-    </blockquote>
-    <p class="f1-p">The key is to use a structure known as a tree decomposition, which organises the graph's vertices into a series of overlapping sets, or “bags”, that are themselves arranged in a tree.</p>
-    <p class="mb-4 f1-p">An algorithm can then traverse this tree of bags, solving the problem piece by piece using dynamic programming. This process involves designing a “state” that summarises all necessary information about the partial solution within a bag, and then defining how this state transforms as vertices are introduced, forgotten, or bags are merged.</p>
-    <!-- Video inserted via gr.Video in app.py -->
-"""
+# Select your tasks here
+# ---------------------------------------------------
+class Tasks(Enum):
+    # task_key in the json file, metric_key in the json file, name to display in the leaderboard
+    task0 = Task("FormulaOne", "success_rate", "Success Rate (%)")
+    # task1 = Task("logiqa", "acc_norm", "LogiQA")
+
+
+NUM_FEWSHOT = 0  # Change with your few shot
+# ---------------------------------------------------
 
-# Text immediately after the video; opens Evaluation section header/content (up to before Warmup figure)
-WHAT_IS_F1_HTML_AFTER_VIDEO = """
-    <p class="f1-p">The deceptive simplicity of the problem statements belies the <strong>extraordinary difficulty</strong> of discovering the correct dynamic programming solution. This process is riddled with subtle combinatorial and logical pitfalls, demanding a profound understanding of the problem’s underlying structure. For a detailed walkthrough of the fifteen interdependent reasoning steps required to solve a single hard problem &mdash; <code>Maximal-Cluster-Graph</code> &mdash; <a href="https://arxiv.org/pdf/2507.13337#appendix.A" target="_blank" rel="noopener noreferrer" class="f1-a">see the appendix of our paper</a>.</p>
-  </section>
-
-  <section id="evaluation">
-    <h2 class="f1-h2">Evaluation</h2>
-    <p class="mb-4 f1-p">All models were evaluated using their highest available reasoning settings and with the maximum context length permitted. To give models the best possible chance of success, we provide a generous few-shot prompt that covers a broad array of the ideas and techniques involved in solving these problems.</p>
-    <p class="mb-4 f1-p">Each submitted solution is subjected to a rigorous and automated <a href="https://arxiv.org/pdf/2507.13337#section.4" target="_blank" rel="noopener noreferrer" class="f1-a">test suite</a> that measures three key aspects of its validity:</p>
-    <ul class="list-disc list-inside space-y-2 mb-6">
-        <li class="f1-li"><strong>Correctness:</strong> The output of the submitted algorithm must be correct on all graphs.</li>
-        <li class="f1-li"><strong>Consistency:</strong> The solution must produce the same output for a given graph, regardless of the specific tree decomposition.</li>
-        <li class="f1-li"><strong>Efficiency:</strong> The solution must be truly <a href="https://en.wikipedia.org/wiki/Parameterized_complexity" target="_blank" rel="noopener noreferrer" class="f1-a">fixed-parameter linear</a>.</li>
-    </ul>
-    <p class="mb-4 f1-p">To support research and encourage community contributions, the <code>FormulaOne-Shallow</code> ("warmup") dataset is released as a public resource for training and fine-tuning models. The complete test suite for all 100 'Shallow' problems is available, alongside a standalone evaluation environment, in our <a href="https://github.com/double-ai/formulaone-dataset/tree/main" target="_blank" rel="noopener noreferrer" class="f1-a">GitHub repository</a>.</p>
-    <p class="f1-p">To maintain the integrity of the core benchmark, only a minimal subset of tests is released for the Deeper and Deepest Tier problems. Solutions submitted for evaluation on our benchmark are evaluated against a withheld comprehensive test-suite.</p>
-"""
 
-# Evaluation: begins the "Model Accuracy" subsection and the Warmup paragraph, up to (but not including) the Warmup figure.
-WHAT_IS_F1_HTML_EVAL_BEFORE_WARMUPFIG = """
-    <h2 class="f1-h2">Model Accuracy</h2>
-    <p class="mb-4 f1-p">On the <strong>FormulaOne-Shallow</strong> problems, frontier models perform reasonably well. This confirms they have a foundational capability for these types of algorithmic tasks, in other words, the tasks are squarely in-distribution.</p>
-    <p class="mb-4 f1-p">However, as the reasoning depth increases in the <strong>Deeper</strong> tier, and solutions require the discovery and integration of novel and more complex state representations, model performance drops off sharply.</p>
+# Your leaderboard name
+# TITLE = """<h1 align="center" id="space-title">AAI FormulaOne Leaderboard</h1>"""
+
+TITLE = """
+<h1 id="space-title" style="
+    text-align: center;
+    font-family: 'Segoe UI', 'Helvetica Neue', sans-serif;
+    font-weight: 300;
+    letter-spacing: 0.05em;
+    color: white;
+    text-transform: none;
+    margin-top: 2rem;
+    font-size: 2.6rem;
+">
+    FormulaOne Leaderboard
+</h1>
 """
 
-# Tail after Deeper figure (closes evaluation section + container)
-WHAT_IS_F1_HTML_AFTER_TIER1FIG_TAIL = """
-    <p class="f1-p">This trend culminates in <strong>Deepest Tier</strong>, where the difficulty is characteristic of exploratory research problems. On this set of 20 problems, no current frontier model solves even a single one. This result starkly illustrates the gap that remains between high performance on existing benchmarks and the deep algorithmic reasoning required for truly complex problems.</p>
-  </section>
-</div>
+# What does your leaderboard evaluate?
+INTRODUCTION_TEXT = """
+Welcome to the official leaderboard for the paper:
+
+**FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming** <br>
+*Gal Beniamini, Yuval Dor, Alon Vinnikov, Shir Granot Peled, Or Weinstein, Or Sharir, Noam Wies, Tomer Nussbaum, Ido Ben Shaul, Tomer Zekharya, Yoav Levine, Shai Shalev-Shwartz, Amnon Shashua* <br>
+**AAI, July 2025**
+
+FormulaOne is a new benchmark designed to challenge frontier AI models. The benchmark is constructed from a vast and conceptually diverse family of dynamic programming problems derived from Monadic Second-Order (MSO) logic on graphs, a framework with profound connections to theoretical computer science.
 """
 
+# Which evaluations are you running? how can people reproduce what you have?
+LLM_BENCHMARKS_TEXT = f"""
+## How it works
+
+## Reproducibility
+To reproduce our results, here is the commands you can run:
 
-SUBMISSION_TERMS_TEXT = """
-### Competition terms
-- By submitting, you agree to the **FormulaOne Submission Agreement
-(v1.2)** and our **Privacy Notice**.
-- Your uploaded file remains yours; we only use it to evaluate, score,
-and contact you about your result.
-**Licensing for the public benchmark assets (informational)**
-- **Evaluator code:** Apache License 2.0
-- **Problem statements &amp; public tests:** Creative Commons **CC BY 4.0**
-See the project&#39;s **README licence section** and full texts: `LICENSE-
-APACHE2`, `LICENSE-CC-BY` in our GitHub repo.
-**Platform**
-- Your use of Hugging Face is also governed by Hugging Face&#39;s Terms and
-Privacy Policy.
 """
 
 EVALUATION_QUEUE_TEXT = """
-## Submitting to the FormulaOne Leaderboard
+## 🧪 Submitting to the FormulaOne Leaderboard
 
 This leaderboard evaluates systems on the FormulaOne core dataset. Submissions consist of a .jsonl file with solution code for each problem.
 
@@ -134,7 +78,7 @@ Your submission must be a .jsonl file with one entry per problem:
 - solution: A Python code implementing the required callback functions.
 
 📄 Full list of problem_ids:
-View the [FormulaOne core dataset](https://github.com/double-ai/formulaone-dataset-release/tree/main/dataset/formulaone) for the complete list of problem IDs.
+View the [FormulaOne core dataset](https://github.com/double-ai/formulaone-dataset-release/dataset/formulaone) for the complete list of problem IDs.
 
 ⚠️ Validation Rules:
 Submissions must:
@@ -151,7 +95,7 @@ Submissions must:
    - **Organization**
    - **System Type**
 -  Click **Submit**.
-
+	
 ### ⏱️ After Submission
 
 Submissions are validated and evaluated within ~24 hours. Results will appear on the leaderboard once processed.
@@ -161,12 +105,12 @@ Submissions are validated and evaluated within ~24 hours. Results will appear on
 CITATION_BUTTON_LABEL = """📚 How to cite FormulaOne"""
 CITATION_BUTTON_TEXT = r"""
 @misc{beniamini2025formulaonemeasuringdepthalgorithmic,
-      title={FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming},
-      author={Gal Beniamini and Yuval Dor and Alon Vinnikov and Shir Granot Peled and Or Weinstein and Or Sharir and Noam Wies and Tomer Nussbaum and Nadav Schweiger and Ido Ben Shaul and Tomer Zekharya and Yoav Levine and Shai Shalev-Shwartz and Amnon Shashua},
+      title={FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming}, 
+      author={Gal Beniamini and Yuval Dor and Alon Vinnikov and Shir Granot Peled and Or Weinstein and Or Sharir and Noam Wies and Tomer Nussbaum and Ido Ben Shaul and Tomer Zekharya and Yoav Levine and Shai Shalev-Shwartz and Amnon Shashua},
       year={2025},
       eprint={2507.13337},
       archivePrefix={arXiv},
       primaryClass={cs.AI},
-      url={https://arxiv.org/abs/2507.13337},
+      url={https://arxiv.org/abs/2507.13337}, 
 }
 """

src/datamodel/data.py

@@ -3,20 +3,14 @@ import time
 
 from datasets import load_dataset
 
-from src.envs import CODE_PROBLEMS_REPO, RESULTS_REPO, SUBMISSIONS_REPO, TOKEN
+from src.envs import TOKEN, CODE_PROBLEMS_REPO, RESULTS_REPO, SUBMISSIONS_REPO
 from src.logger import get_logger
 
 logger = get_logger(__name__)
 
 
 class F1Data:
-    def __init__(
-        self,
-        cp_ds_name: str,  # Name of the dataset. Fixed.
-        sub_ds_name: str,  # Name of subdataset. Fixed.
-        res_ds_name: str,  # Name of results repository. Fixed.
-        split: str = "hard",  # Split is either 'hard' or 'easy'.
-    ):
+    def __init__(self, cp_ds_name: str, sub_ds_name: str, res_ds_name: str, split: str = "hard"):
         self.cp_dataset_name = cp_ds_name
         self.submissions_dataset_name = sub_ds_name
         self.results_dataset_name = res_ds_name
@@ -25,16 +19,16 @@ class F1Data:
         self._initialize()
 
     def _initialize(self):
-        logger.info(f"Initialize F1Data TOKEN='{TOKEN}'")
+        logger.info("Initialize F1Data TOKEN='%s'", TOKEN)
         start_time = time.monotonic()
-        cp_ds = load_dataset(
+        cp_ds = load_dataset(self.cp_dataset_name, split=self.split, token=TOKEN)
+        logger.info(
+            "Loaded code-problems dataset from %s in %f sec",
             self.cp_dataset_name,
-            split=self.split,
-            token=TOKEN,
+            time.monotonic() - start_time,
         )
-        logger.info(f"Loaded code-problems dataset from {self.cp_dataset_name} in {time.monotonic() - start_time} sec")
-        self.code_problems = {r["id"]: r["code_problem"] for r in cp_ds}  # id string -> code problem.
-        logger.info(f"Loaded {len(self.code_problems)} code problems")
+        self.code_problems: dict[str, str] = {r["id"]: r["code_problem"] for r in cp_ds}
+        logger.info(f"Loaded %d code problems {len(self.code_problems)}")
 
     @functools.cached_property
     def code_problem_ids(self) -> set[str]:
@@ -43,11 +37,6 @@ class F1Data:
 
 if __name__ == "__main__":
     split = "hard"
-    f1_data = F1Data(
-        cp_ds_name=CODE_PROBLEMS_REPO,
-        sub_ds_name=SUBMISSIONS_REPO,
-        res_ds_name=RESULTS_REPO,
-        split=split,
-    )
+    f1_data = F1Data(cp_ds_name=CODE_PROBLEMS_REPO, sub_ds_name=SUBMISSIONS_REPO, res_ds_name=RESULTS_REPO, split=split)
 
     print(f"Found {len(f1_data.code_problem_ids)} code problems in {split} split of {f1_data.cp_dataset_name}")

src/display/css_html_js.py

@@ -1,219 +1,99 @@
 custom_css = """
-:root, [data-theme="light"] {
-  --f1-text: #111827;
-  --f1-subtle: #6b7280;
-  --f1-border: #e5e7eb;
-  --f1-bg: #ffffff;
-  --f1-bg-muted: #f9fafb;
 
-  /* Nuke Gradio chrome that interfered with colors */
-  --block-border-color: transparent !important;
-  --background-fill-primary: transparent !important;
+.markdown-text {
+    font-size: 16px !important;
 }
 
-/* Width caps */
-.f1-container { max-width: 800px; margin: 0 auto; padding: 0 16px; }
-.markdown-text { font-size: 16px !important; max-width: 800px; margin: 0 auto; }
-#what-is-tab { max-width: 800px; margin-left: auto; margin-right: auto; }
-/* requested 710px */
-#f1-examples { max-width: 710px; margin: 0 auto; }
-
-/* NEW: landing tab width + tier selector alignment */
-/* Landing tab width + tier selector alignment */
-#landing-accuracy-tab { max-width: 800px; margin-left: auto; margin-right: auto; }
-
-/* Right-align the switcher row; transparent background & border */
-#f1-tier-select-row {
-  justify-content: flex-end;
-  margin-bottom: 6px;
-  background: white !important;
-  border: none !important;
+#models-to-add-text {
+    font-size: 18px !important;
 }
 
-/* Make the control look like "pills" and force RTL layout */
-#f1-tier-select { direction: rtl; }  /* RTL direction (equivalent effect to dir="rtl") */
-#f1-tier-select .wrap {
-  display: inline-flex;
-  gap: 6px;
-  padding: 4px;
-  background: #ffffff;                /* white background for the switcher itself */
-  border: 1px solid var(--f1-border);
-  border-radius: 999px;
-}
-#f1-tier-select input[type="radio"] { display: none; }
-#f1-tier-select label {
-  border: none;
-  border-radius: 999px;
-  padding: 6px 12px;
-  background: transparent;
-  cursor: pointer;
-}
-#f1-tier-select input[type="radio"]:checked + span {
-  background: #eef2ff;                /* only selected pill is tinted */
-  border-radius: 999px;
-  padding: 6px 12px;
-  box-shadow: 0 1px 2px rgba(0,0,0,0.04);
+#citation-button span {
+    font-size: 16px !important;
 }
 
-#f1-tier-select-row div { border: none; }
-
-#f1-tier-select { background: white; }
-
-#f1-accuracy-plot label { display: false; }
-
-#learn-more-btn, #learn-more-btn button {
-  background: transparent !important;
-  border: none !important;
-  color: #2563eb !important;
-  font-weight: 700 !important;
-  font-size: 1.05rem !important;
-  padding: 0 !important;
-  box-shadow: none !important;
-}
-#learn-more-btn button:hover { text-decoration: underline !important; background: transparent !important; }
-
-#landing-hero-row {
-  max-width: 800px;
-  margin-left: auto;
-  margin-right: auto;
-  margin-bottom: 8px;
-  align-items: center;
-}
-#landing-hero-left { padding-right: 8px; }
-#landing-hero-right { display: flex; justify-content: flex-end; }
-
-/* Blurb text */
-.f1-hero { margin: 0; font-size: 1.02rem; color: var(--f1-text); }
-
-/* Learn More pill — compact */
-#learn-more-pill,
-#learn-more-pill button {
-  width: auto !important;
-  min-width: 0 !important;
-  background: #2563eb !important;
-  color: #ffffff !important;
-  border: 0 !important;
-  border-radius: 999px !important;
-  padding: 6px 12px !important;
-  box-shadow: 0 1px 2px rgba(0,0,0,0.06) !important;
-  font-weight: 700 !important;
-  font-size: 0.95rem !important;
-  white-space: nowrap !important;
-}
-#learn-more-pill button:hover { filter: brightness(0.95) !important; }
-
-#landing-hero-left { min-width: 75%; }
-
-/* Text */
-.f1-p, .f1-li { line-height: 1.75; color: #374151; text-wrap: pretty; overflow-wrap: break-word; hyphens: auto; }
-
-/* Headings */
-.f1-h1 { font-weight: 700; font-size: 2.25rem; line-height: 2.5rem; color: var(--f1-text); text-align: center; margin-bottom: 1.25rem !important; }
-.f1-h2 { font-weight: 700; border-bottom: 1px solid var(--f1-border); padding-bottom: 0.45rem; margin-top: 1.75rem; margin-bottom: 0.9rem; color: var(--f1-text); font-size: 1.5rem; line-height: 2rem; }
-
-/* Links */
-.f1-a { color: #2563eb; text-decoration: none; font-weight: 500; }
-.f1-a:hover { text-decoration: underline; }
-
-/* Captions (centered + dark) */
-.f1-figcaption { margin-top: 4px; font-size: 0.875rem; color: #111827; text-align: center; }
-.f1-figcaption-video { margin-top: 2px; } /* tighter under the video */
-
-/* Problem name — force center from first render; code bg color #f9fafb */
-#f1-examples .f1-problem-markdown .markdown p:first-child { text-align: center !important; margin: 0 0 8px 0; }
-.f1-problem-markdown p code,
-#f1-examples .f1-problem-markdown .markdown p:first-child code {
-  display: inline-block; background: #f9fafb !important; padding: 2px 8px; border-radius: 6px; margin-left: auto; margin-right: auto;
+#citation-button textarea {
+    font-size: 16px !important;
 }
 
-.f1-problem-markdown > * { margin-top: 7px; margin-bottom: 7px; line-height: 1.8; }
-
-/* Problem body: padding + comfortable line-height inside Markdown + small space before tabs */
-#f1-examples .f1-problem-markdown .markdown { background: var(--f1-bg-muted); border: 1px solid var(--f1-border); border-radius: 8px; padding: 18px; margin: 10px 14px 10px 14px; line-height: 1.8; }
-#f1-examples .f1-problem-markdown .markdown p { margin: 0.35rem 0; }
-
-/* Pills (Radio) — compact spacing at bottom */
-#f1-example-radio { border-top: 1px solid var(--f1-border); padding: 8px 10px 4px 10px; margin: 0 8px 2px; }
-#f1-example-radio input[type="radio"] { display: none; }
-#f1-example-radio .wrap { display: flex; gap: 6px; flex-wrap: wrap; justify-content: flex-start; }
-#f1-example-radio label { border: 1px solid var(--f1-border); border-radius: 999px; padding: 6px 10px; cursor: pointer; background: #f3f4f6; }
-#f1-example-radio input[type="radio"]:checked + span { background: #e5e7eb; border-color: var(--f1-border); border-radius: 999px; padding: 6px 10px; }
-
-/* Examples card chrome */
-#f1-examples { background: var(--f1-bg-muted); border: 1px solid var(--f1-border); border-radius: 10px; box-shadow: 0 1px 2px rgba(0,0,0,0.04); margin-bottom: 12px; }
-#f1-examples .form { background: transparent !important; }  /* no bg on inner form container */
-.f1-tabs-body { padding-top: 12px; text-align: center; }
-.f1-examples-chip { display: inline-block; background: #e5e7eb; color: #111827; padding: 6px 12px; border-radius: 999px; font-weight: 700; }
-
-/* Figures via Gradio components (centered, rounded, shadow) */
-.f1-image img, .f1-video video { width: 100%; max-width: 42rem; display: block; margin: 0 auto; border-radius: 12px; box-shadow: 0 2px 8px rgba(0,0,0,0.08); }
-
-/* Categories "table" — equal width per column set; compact first two, flexible third */
-.f1-grid-wrap { text-align: center; margin: 10px auto 8px auto; }
-.f1-grid-table {
-  display: inline-grid;
-  grid-template-columns: max-content 12ch minmax(360px, 1fr);  /* 2nd col fixed width; 3rd grows */
-  border: 1px solid var(--f1-border);
-  background: var(--f1-bg);
-  border-radius: 8px;
-  overflow: hidden;
-}
-.f1-grid-row { display: contents; }
-.f1-grid-cell { padding: 8px 12px; text-align: left; border-left: 1px solid var(--f1-border); border-top: 1px solid var(--f1-border); white-space: normal; }
-.f1-grid-cell:nth-child(3n+1) { border-left: none; }
-.f1-grid-head .f1-grid-cell { font-weight: 600; text-align: center; border-top: none; }
+#citation-button > label > button {
+    margin: 6px;
+    transform: scale(1.3);
+}
 
-#f1-login-box { width: 800px; margin: auto; }
+#leaderboard-table {
+    margin-top: 15px
+}
 
-/* Leaderboard: cap width at 800px and center; keep your chain rule */
-#formulaone-leaderboard-tab-table { max-width: 800px; margin-left: auto; margin-right: auto; }
-#formulaone-leaderboard-tab-table .column .row .column { min-width: 80% !important; }
-#formulaone-leaderboard-tab-table .row, #formulaone-leaderboard-tab-table .column { width: 100% !important; max-width: 100% !important; }
-#formulaone-leaderboard-tab-table [data-testid="dropdown"], #formulaone-leaderboard-tab-table input[type="text"] { width: 100% !important; }
+#leaderboard-table-lite {
+    margin-top: 15px
+}
 
-/* Login button: force light and width cap */
-#hf-login-btn, #hf-login-btn button, button[data-testid="login-button"], [data-testid="login-button"] button, div[data-testid="login-button"] > button {
-  background: #ffffff !important; color: #1f2937 !important; border: 1px solid var(--f1-border) !important; box-shadow: 0 1px 2px 0 rgb(0 0 0 / 0.04) !important;
-  max-width: 800px !important; width: 100% !important; margin-left: auto !important; margin-right: auto !important;
+#search-bar-table-box > div:first-child {
+    background: none;
+    border: none;
+}
+ 
+#search-bar {
+    padding: 0px;
 }
-#hf-login-btn:hover, #hf-login-btn button:hover, button[data-testid="login-button"]:hover, [data-testid="login-button"] button:hover, div[data-testid="login-button"] > button:hover { background: #f9fafb !important; }
 
-/* Checkbox styling for submission requirements */
-.markdown-text input[type="checkbox"] {
-  margin-right: 8px !important;
-  width: 16px !important;
-  height: 16px !important;
-  accent-color: #2563eb !important;
-  cursor: pointer !important;
+/* Limit the width of the first AutoEvalColumn so that names don't expand too much */
+#leaderboard-table td:nth-child(2),
+#leaderboard-table th:nth-child(2) {
+    max-width: 400px;
+    overflow: auto;
+    white-space: nowrap;
 }
 
-.markdown-text label[data-testid="checkbox"] {
-  color: var(--f1-text) !important;
-  font-size: 14px !important;
-  line-height: 1.5 !important;
-  cursor: pointer !important;
-  display: flex !important;
-  align-items: flex-start !important;
-  margin-bottom: 8px !important;
+.tab-buttons button {
+    font-size: 20px;
 }
 
-/* Submit button styling when disabled */
-button[data-testid="button"]:disabled {
-  opacity: 0.5 !important;
-  cursor: not-allowed !important;
-  background: #f3f4f6 !important;
-  color: #9ca3af !important;
-  border-color: #d1d5db !important;
+#scale-logo {
+    border-style: none !important;
+    box-shadow: none;
+    display: block;
+    margin-left: auto;
+    margin-right: auto;
+    max-width: 600px;
 }
 
-/* Required Agreements section 
-.markdown-text h3 {
-  color: var(--f1-text) !important;
-  font-weight: 600 !important;
-  margin: 20px 0 12px 0 !important;
-  font-size: 18px !important;
-}
-*/
+#scale-logo .download {
+    display: none;
+}
+#filter_type{
+    border: 0;
+    padding-left: 0;
+    padding-top: 0;
+}
+#filter_type label {
+    display: flex;
+}
+#filter_type label > span{
+    margin-top: var(--spacing-lg);
+    margin-right: 0.5em;
+}
+#filter_type label > .wrap{
+    width: 103px;
+}
+#filter_type label > .wrap .wrap-inner{  
+    padding: 2px;
+}
+#filter_type label > .wrap .wrap-inner input{
+    width: 1px
+}
+#filter-columns-type{
+    border:0;
+    padding:0.5;
+}
+#filter-columns-size{
+    border:0;
+    padding:0.5;
+}
+#box-filter > .form{
+    border: 0
+}
 """
 
 get_window_url_params = """

src/display/formatting.py

@@ -1,3 +1,12 @@
+def model_hyperlink(link, model_name):
+    return f'<a target="_blank" href="{link}" style="color: var(--link-text-color); text-decoration: underline;text-decoration-style: dotted;">{model_name}</a>'
+
+
+def make_clickable_model(model_name):
+    link = f"https://huggingface.co/{model_name}"
+    return model_hyperlink(link, model_name)
+
+
 def styled_error(error):
     return f"<p style='color: red; font-size: 20px; text-align: center;'>{error}</p>"
 
@@ -8,3 +17,11 @@ def styled_warning(warn):
 
 def styled_message(message):
     return f"<p style='color: green; font-size: 20px; text-align: center;'>{message}</p>"
+
+
+def has_no_nan_values(df, columns):
+    return df[columns].notna().all(axis=1)
+
+
+def has_nan_values(df, columns):
+    return df[columns].isna().any(axis=1)

src/display/utils.py

@@ -1,15 +1,19 @@
-from dataclasses import dataclass
+from dataclasses import dataclass, field, make_dataclass
+from typing import ClassVar
 from enum import Enum
 
+import pandas as pd
 
-def fields(raw_class):
-    return [v for k, v in raw_class.__dict__.items() if k[:2] != "__" and k[-2:] != "__"]
+from src.about import Tasks
 
 
-# These classes are for user facing column names, to avoid having to change them
-# all around the code when a modification is needed.
+def fields(raw_class):
+    return [v for k, v in raw_class.__dict__.items() if k[:2] != "__" and k[-2:] != "__"]
 
 
+# These classes are for user facing column names,
+# to avoid having to change them all around the code
+# when a modif is needed
 @dataclass
 class ColumnContent:
     name: str
@@ -19,57 +23,116 @@ class ColumnContent:
     never_hidden: bool = False
 
 
+## Leaderboard columns
+# auto_eval_column_fields = []
+# # Init
+# auto_eval_column_fields.append(["model_type_symbol", ColumnContent, ColumnContent("T", "str", True, never_hidden=True)])
+# auto_eval_column_fields.append(["model", ColumnContent, ColumnContent("Model", "markdown", True, never_hidden=True)])
+# # Scores
+# auto_eval_column_fields.append(["average", ColumnContent, ColumnContent("Average ⬆️", "number", True)])
+# for task in Tasks:
+#     auto_eval_column_fields.append([task.name, ColumnContent, ColumnContent(task.value.col_name, "number", True)])
+# # Model information
+# auto_eval_column_fields.append(["model_type", ColumnContent, ColumnContent("Type", "str", False)])
+# auto_eval_column_fields.append(["architecture", ColumnContent, ColumnContent("Architecture", "str", False)])
+# auto_eval_column_fields.append(["weight_type", ColumnContent, ColumnContent("Weight type", "str", False, True)])
+# auto_eval_column_fields.append(["precision", ColumnContent, ColumnContent("Precision", "str", False)])
+# auto_eval_column_fields.append(["license", ColumnContent, ColumnContent("Hub License", "str", False)])
+# auto_eval_column_fields.append(["params", ColumnContent, ColumnContent("#Params (B)", "number", False)])
+# auto_eval_column_fields.append(["likes", ColumnContent, ColumnContent("Hub ❤️", "number", False)])
+# auto_eval_column_fields.append(["still_on_hub", ColumnContent, ColumnContent("Available on the hub", "bool", False)])
+# auto_eval_column_fields.append(["revision", ColumnContent, ColumnContent("Model sha", "str", False, False)])
+#
+#
+#
+# def make_classvar_dataclass(name: str, spec: list):
+#     ns = {"__annotations__": {}}
+#     for field_name, field_type, default in spec:
+#         # Mark as ClassVar so dataclass doesn't treat it as an instance field
+#         ns["__annotations__"][field_name] = ClassVar[field_type]
+#         ns[field_name] = default
+#     # No instance fields; just class-level descriptors
+#     return make_dataclass(name, [], frozen=True, namespace=ns)
+#
+# # We use make dataclass to dynamically fill the scores from Tasks
+# AutoEvalColumn = make_classvar_dataclass("AutoEvalColumn", auto_eval_column_fields)
+
+
 @dataclass(frozen=True)
 class AutoEvalColumn:
     system = ColumnContent("System Name", "markdown", True, never_hidden=True)
+    system_type = ColumnContent("System Type", "str", True)
     organization = ColumnContent("Organization", "str", True, never_hidden=True)
-    success_rate_overall = ColumnContent("Overall Success (%)", "number", True)
-    success_rate_tier1 = ColumnContent("Deeper Tier Success (%)", "number", True)
-    success_rate_tier2 = ColumnContent("Deepest Tier Success (%)", "number", True)
+    success_rate = ColumnContent("Success Rate (%)", "number", True)
+    problems_solved = ColumnContent("Problems Solved", "number", True)
     submitted_on = ColumnContent("Submitted On", "datetime", True)
 
 
-# For the queue columns in the submission tab
+## For the queue columns in the submission tab
 @dataclass(frozen=True)
 class EvalQueueColumn:  # Queue column
     model = ColumnContent("model", "markdown", True)
     revision = ColumnContent("revision", "str", True)
     private = ColumnContent("private", "bool", True)
     precision = ColumnContent("precision", "str", True)
-    weight_type = ColumnContent("weight_type", "str", True)
+    weight_type = ColumnContent("weight_type", "str", "Original")
     status = ColumnContent("status", "str", True)
 
 
-# All the model information that we might need
+## All the model information that we might need
 @dataclass
 class ModelDetails:
     name: str
     display_name: str = ""
+    symbol: str = ""  # emoji
 
 
 class ModelType(Enum):
-    LLM = ModelDetails(name="LLM")
-    AgenticLLM = ModelDetails(name="AgenticLLM")
-    Other = ModelDetails(name="Other")
+    LLM = ModelDetails(name="LLM", symbol="🟢")
+    AgenticLLM = ModelDetails(name="AgenticLLM", symbol="🔶")
+    # IFT = ModelDetails(name="instruction-tuned", symbol="⭕")
+    # RL = ModelDetails(name="RL-tuned", symbol="🟦")
+    Other = ModelDetails(name="Other", symbol="?")
 
-    def to_str(self):
-        return self.value.name
+    def to_str(self, separator=" "):
+        return f"{self.value.symbol}{separator}{self.value.name}"
 
     @staticmethod
-    def from_str(type: str):
-        if type == "AgenticLLM":
+    def from_str(type):
+        if "AgenticLLM" in type or "🔶" in type:
             return ModelType.AgenticLLM
-        if type == "LLM":
+        if "LLM" in type or "🟢" in type:
             return ModelType.LLM
+        # if "RL-tuned" in type or "🟦" in type:
+        #     return ModelType.RL
+        # if "instruction-tuned" in type or "⭕" in type:
+        #     return ModelType.IFT
         return ModelType.Other
 
 
+class WeightType(Enum):
+    Adapter = ModelDetails("Adapter")
+    Original = ModelDetails("Original")
+    Delta = ModelDetails("Delta")
+
+
 class Precision(Enum):
     float16 = ModelDetails("float16")
     bfloat16 = ModelDetails("bfloat16")
     Unknown = ModelDetails("?")
 
+    def from_str(precision):
+        if precision in ["torch.float16", "float16"]:
+            return Precision.float16
+        if precision in ["torch.bfloat16", "bfloat16"]:
+            return Precision.bfloat16
+        return Precision.Unknown
+
 
+# Column selection
 COLS = [c.name for c in fields(AutoEvalColumn) if not c.hidden]
+
 EVAL_COLS = [c.name for c in fields(EvalQueueColumn)]
 EVAL_TYPES = [c.type for c in fields(EvalQueueColumn)]
+
+# BENCHMARK_COLS = [t.value.col_name for t in Tasks]

src/envs.py

@@ -1,5 +1,4 @@
 import os
-import gradio as gr
 
 from huggingface_hub import HfApi
 
@@ -17,7 +16,6 @@ RESULTS_REPO = f"{OWNER}/dev-f1-leaderboard-results"
 # If you setup a cache later, just change HF_HOME
 CACHE_PATH = os.getenv("HF_HOME", ".")
 
-print(f"RUNNING gradio {gr.__version__}")
 print(f"{TOKEN=}")
 print(f"{REPO_ID=}")
 

src/leaderboard/read_evals.py

@@ -0,0 +1,196 @@
+import glob
+import json
+import math
+import os
+from dataclasses import dataclass
+
+import dateutil
+import numpy as np
+
+from src.display.formatting import make_clickable_model
+from src.display.utils import AutoEvalColumn, ModelType, Tasks, Precision, WeightType
+from src.submission.check_validity import is_model_on_hub
+
+
+@dataclass
+class EvalResult:
+    """Represents one full evaluation. Built from a combination of the result and request file for a given run.
+    """
+    eval_name: str # org_model_precision (uid)
+    full_model: str # org/model (path on hub)
+    org: str 
+    model: str
+    revision: str # commit hash, "" if main
+    results: dict
+    precision: Precision = Precision.Unknown
+    model_type: ModelType = ModelType.LLM # Pretrained, fine tuned, ...
+    weight_type: WeightType = WeightType.Original # Original or Adapter
+    architecture: str = "Unknown" 
+    license: str = "?"
+    likes: int = 0
+    num_params: int = 0
+    date: str = "" # submission date of request file
+    still_on_hub: bool = False
+
+    @classmethod
+    def init_from_json_file(self, json_filepath):
+        """Inits the result from the specific model result file"""
+        with open(json_filepath) as fp:
+            data = json.load(fp)
+
+        config = data.get("config")
+
+        # Precision
+        precision = Precision.from_str(config.get("model_dtype"))
+
+        # Get model and org
+        org_and_model = config.get("model_name", config.get("model_args", None))
+        org_and_model = org_and_model.split("/", 1)
+
+        if len(org_and_model) == 1:
+            org = None
+            model = org_and_model[0]
+            result_key = f"{model}_{precision.value.name}"
+        else:
+            org = org_and_model[0]
+            model = org_and_model[1]
+            result_key = f"{org}_{model}_{precision.value.name}"
+        full_model = "/".join(org_and_model)
+
+        still_on_hub, _, model_config = is_model_on_hub(
+            full_model, config.get("model_sha", "main"), trust_remote_code=True, test_tokenizer=False
+        )
+        architecture = "?"
+        if model_config is not None:
+            architectures = getattr(model_config, "architectures", None)
+            if architectures:
+                architecture = ";".join(architectures)
+
+        # Extract results available in this file (some results are split in several files)
+        results = {}
+        for task in Tasks:
+            task = task.value
+
+            # We average all scores of a given metric (not all metrics are present in all files)
+            accs = np.array([v.get(task.metric, None) for k, v in data["results"].items() if task.benchmark == k])
+            if accs.size == 0 or any([acc is None for acc in accs]):
+                continue
+
+            mean_acc = np.mean(accs) * 100.0
+            results[task.benchmark] = mean_acc
+
+        return self(
+            eval_name=result_key,
+            full_model=full_model,
+            org=org,
+            model=model,
+            results=results,
+            precision=precision,  
+            revision= config.get("model_sha", ""),
+            still_on_hub=still_on_hub,
+            architecture=architecture
+        )
+
+    def update_with_request_file(self, requests_path):
+        """Finds the relevant request file for the current model and updates info with it"""
+        request_file = get_request_file_for_model(requests_path, self.full_model, self.precision.value.name)
+
+        try:
+            with open(request_file, "r") as f:
+                request = json.load(f)
+            self.model_type = ModelType.from_str(request.get("model_type", ""))
+            self.weight_type = WeightType[request.get("weight_type", "Original")]
+            self.license = request.get("license", "?")
+            self.likes = request.get("likes", 0)
+            self.num_params = request.get("params", 0)
+            self.date = request.get("submitted_time", "")
+        except Exception:
+            print(f"Could not find request file for {self.org}/{self.model} with precision {self.precision.value.name}")
+
+    def to_dict(self):
+        """Converts the Eval Result to a dict compatible with our dataframe display"""
+        average = sum([v for v in self.results.values() if v is not None]) / len(Tasks)
+        data_dict = {
+            "eval_name": self.eval_name,  # not a column, just a save name,
+            AutoEvalColumn.precision.name: self.precision.value.name,
+            AutoEvalColumn.model_type.name: self.model_type.value.name,
+            AutoEvalColumn.model_type_symbol.name: self.model_type.value.symbol,
+            AutoEvalColumn.weight_type.name: self.weight_type.value.name,
+            AutoEvalColumn.architecture.name: self.architecture,
+            AutoEvalColumn.model.name: make_clickable_model(self.full_model),
+            AutoEvalColumn.revision.name: self.revision,
+            AutoEvalColumn.average.name: average,
+            AutoEvalColumn.license.name: self.license,
+            AutoEvalColumn.likes.name: self.likes,
+            AutoEvalColumn.params.name: self.num_params,
+            AutoEvalColumn.still_on_hub.name: self.still_on_hub,
+        }
+
+        for task in Tasks:
+            data_dict[task.value.col_name] = self.results[task.value.benchmark]
+
+        return data_dict
+
+
+def get_request_file_for_model(requests_path, model_name, precision):
+    """Selects the correct request file for a given model. Only keeps runs tagged as FINISHED"""
+    request_files = os.path.join(
+        requests_path,
+        f"{model_name}_eval_request_*.json",
+    )
+    request_files = glob.glob(request_files)
+
+    # Select correct request file (precision)
+    request_file = ""
+    request_files = sorted(request_files, reverse=True)
+    for tmp_request_file in request_files:
+        with open(tmp_request_file, "r") as f:
+            req_content = json.load(f)
+            if (
+                req_content["status"] in ["FINISHED"]
+                and req_content["precision"] == precision.split(".")[-1]
+            ):
+                request_file = tmp_request_file
+    return request_file
+
+
+def get_raw_eval_results(results_path: str, requests_path: str) -> list[EvalResult]:
+    """From the path of the results folder root, extract all needed info for results"""
+    model_result_filepaths = []
+
+    for root, _, files in os.walk(results_path):
+        # We should only have json files in model results
+        if len(files) == 0 or any([not f.endswith(".json") for f in files]):
+            continue
+
+        # Sort the files by date
+        try:
+            files.sort(key=lambda x: x.removesuffix(".json").removeprefix("results_")[:-7])
+        except dateutil.parser._parser.ParserError:
+            files = [files[-1]]
+
+        for file in files:
+            model_result_filepaths.append(os.path.join(root, file))
+
+    eval_results = {}
+    for model_result_filepath in model_result_filepaths:
+        # Creation of result
+        eval_result = EvalResult.init_from_json_file(model_result_filepath)
+        eval_result.update_with_request_file(requests_path)
+
+        # Store results of same eval together
+        eval_name = eval_result.eval_name
+        if eval_name in eval_results.keys():
+            eval_results[eval_name].results.update({k: v for k, v in eval_result.results.items() if v is not None})
+        else:
+            eval_results[eval_name] = eval_result
+
+    results = []
+    for v in eval_results.values():
+        try:
+            v.to_dict() # we test if the dict version is complete
+            results.append(v)
+        except KeyError:  # not all eval values present
+            continue
+
+    return results

src/logger.py

@@ -1,11 +1,7 @@
 import logging
 import sys
 
-
-def get_logger(
-    filename: str,
-    level=logging.INFO,
-) -> logging.Logger:
+def get_logger(filename: str, level=logging.INFO) -> logging.Logger:
     new_logger = logging.getLogger(filename)
     fmt = logging.Formatter(fmt="%(asctime)s - %(name)s - %(levelname)s - %(funcName)s:%(lineno)d - %(message)s")
     handler = logging.StreamHandler(sys.stderr)

src/populate.py

@@ -1,98 +1,110 @@
+import json
+import os
+
 import pandas as pd
-from datasets import get_dataset_config_names, load_dataset
+from datasets import load_dataset, get_dataset_config_names
 from datasets.exceptions import DatasetNotFoundError
 from tqdm.auto import tqdm
 
-from src.display.utils import AutoEvalColumn
+from src.display.formatting import has_no_nan_values, make_clickable_model
+from src.display.utils import AutoEvalColumn, EvalQueueColumn
 from src.envs import TOKEN
+from src.leaderboard.read_evals import get_raw_eval_results
 from src.logger import get_logger
 
 logger = get_logger(__name__)
 
 
 def get_leaderboard_df(results_dataset_name: str) -> pd.DataFrame:
-    """
-    @brief Creates a dataframe from all the individual experiment results.
-    """
-
-    empty_df = pd.DataFrame(
-        columns=[
-            AutoEvalColumn.system.name,
-            AutoEvalColumn.organization.name,
-            AutoEvalColumn.success_rate_overall.name,
-            AutoEvalColumn.success_rate_tier1.name,
-            AutoEvalColumn.success_rate_tier2.name,
-            AutoEvalColumn.submitted_on.name,
-        ]
-    )
+    """Creates a dataframe from all the individual experiment results"""
 
     try:
-        configs = get_dataset_config_names(
-            results_dataset_name,
-            token=TOKEN,
-        )
+        configs = get_dataset_config_names(results_dataset_name, token=TOKEN)
     except (DatasetNotFoundError, FileNotFoundError):
-
         # Return an empty DataFrame with expected columns
-        logger.warning("Failed to load configuration", exc_info=True)
-        return empty_df
-
-    if configs == ["default"]:
-        logger.info("Dataset has only default config — treating as empty")
-        return empty_df
+        return pd.DataFrame(
+            columns=[
+                "System Name",
+                "System Type",
+                "Organization",
+                "Success Rate (%)",
+                "Problems Solved",
+                "Submitted On",
+            ]
+        )
 
     rows = []
-    for submission_id in tqdm(
-        configs,
-        total=len(configs),
-        desc="Processing Submission Results",
-    ):
-        submission_ds = load_dataset(
-            results_dataset_name,
-            submission_id,
-            split="train",
-            token=TOKEN,
-        )
+    for submission_id in tqdm(configs, total=len(configs), desc="Processing Submission Results"):
+        submission_ds = load_dataset(results_dataset_name, submission_id, split="train", token=TOKEN)
         submission_df = pd.DataFrame(submission_ds)
 
         if submission_df.empty or "did_pass" not in submission_df.columns or submission_df.did_pass.isna().any():
             logger.warning(f"Skipping {submission_id} due to invalid did_pass values")
             continue
 
-        assert submission_df["tier"].isin([1, 2]).all(), "Invalid tier values found in submission_df"
         success_rate = 100 * submission_df["did_pass"].mean()
-        tier1_success_rate = 100 * submission_df[submission_df["tier"] == 1]["did_pass"].mean()
-        tier2_success_rate = 100 * submission_df[submission_df["tier"] == 2]["did_pass"].mean()
+        num_solved = submission_df["did_pass"].sum()
         first_row = submission_df.iloc[0]
 
         rows.append(
             {
-                AutoEvalColumn.system.name: first_row["system_name"],
-                AutoEvalColumn.organization.name: first_row["organization"],
-                AutoEvalColumn.success_rate_overall.name: success_rate,
-                AutoEvalColumn.success_rate_tier1.name: tier1_success_rate,
-                AutoEvalColumn.success_rate_tier2.name: tier2_success_rate,
-                AutoEvalColumn.submitted_on.name: pd.to_datetime(first_row["submission_ts"]).strftime("%Y-%m-%d %H:%M"),
+                "System Name": first_row["system_name"],
+                "System Type": first_row["system_type"],
+                "Organization": first_row["organization"],
+                "Success Rate (%)": success_rate,
+                "Problems Solved": num_solved,
+                "Submitted On": pd.to_datetime(first_row["submission_ts"]).strftime("%Y-%m-%d %H:%M"),
             }
         )
 
     full_df = pd.DataFrame(rows)
 
-    logger.info(f"Loaded results df with {len(full_df)} entries")
-
+    # TODO: forbid multiple submissions under the same name?
     # Keep only the latest entry per unique (System Name, System Type, Organization) triplet
     final_df = (
         full_df.sort_values("Submitted On", ascending=False)
-        .drop_duplicates(subset=[AutoEvalColumn.system.name, AutoEvalColumn.organization.name], keep="first")
-        .sort_values(by=[AutoEvalColumn.success_rate_overall.name], ascending=False)
+        .drop_duplicates(subset=["System Name", "System Type", "Organization"], keep="first")
+        .sort_values(by=[AutoEvalColumn.success_rate.name], ascending=False)
         .reset_index(drop=True)
     )
 
-    cols_to_round = [
-        AutoEvalColumn.success_rate_overall.name,
-        AutoEvalColumn.success_rate_tier1.name,
-        AutoEvalColumn.success_rate_tier2.name,
-    ]
+    cols_to_round = ["Success Rate (%)"]
     final_df[cols_to_round] = final_df[cols_to_round].round(decimals=2)
 
     return final_df
+
+
+def get_evaluation_queue_df(save_path: str, cols: list) -> list[pd.DataFrame]:
+    """Creates the different dataframes for the evaluation queues requestes"""
+    entries = [entry for entry in os.listdir(save_path) if not entry.startswith(".")]
+    all_evals = []
+
+    for entry in entries:
+        if ".json" in entry:
+            file_path = os.path.join(save_path, entry)
+            with open(file_path) as fp:
+                data = json.load(fp)
+
+            data[EvalQueueColumn.model.name] = make_clickable_model(data["model"])
+            data[EvalQueueColumn.revision.name] = data.get("revision", "main")
+
+            all_evals.append(data)
+        elif ".md" not in entry:
+            # this is a folder
+            sub_entries = [e for e in os.listdir(f"{save_path}/{entry}") if os.path.isfile(e) and not e.startswith(".")]
+            for sub_entry in sub_entries:
+                file_path = os.path.join(save_path, entry, sub_entry)
+                with open(file_path) as fp:
+                    data = json.load(fp)
+
+                data[EvalQueueColumn.model.name] = make_clickable_model(data["model"])
+                data[EvalQueueColumn.revision.name] = data.get("revision", "main")
+                all_evals.append(data)
+
+    pending_list = [e for e in all_evals if e["status"] in ["PENDING", "RERUN"]]
+    running_list = [e for e in all_evals if e["status"] == "RUNNING"]
+    finished_list = [e for e in all_evals if e["status"].startswith("FINISHED") or e["status"] == "PENDING_NEW_EVAL"]
+    df_pending = pd.DataFrame.from_records(pending_list, columns=cols)
+    df_running = pd.DataFrame.from_records(running_list, columns=cols)
+    df_finished = pd.DataFrame.from_records(finished_list, columns=cols)
+    return df_finished[cols], df_running[cols], df_pending[cols]

src/submission/check_validity.py

@@ -0,0 +1,102 @@
+import json
+import os
+import re
+from collections import defaultdict
+from datetime import datetime, timedelta, timezone
+
+from datasets import get_dataset_config_names
+import huggingface_hub
+from huggingface_hub import ModelCard
+from huggingface_hub.hf_api import ModelInfo
+from transformers import AutoConfig
+from transformers.models.auto.tokenization_auto import AutoTokenizer
+
+from src.envs import SUBMISSIONS_REPO
+
+def check_model_card(repo_id: str) -> tuple[bool, str]:
+    """Checks if the model card and license exist and have been filled"""
+    try:
+        card = ModelCard.load(repo_id)
+    except huggingface_hub.utils.EntryNotFoundError:
+        return False, "Please add a model card to your model to explain how you trained/fine-tuned it."
+
+    # Enforce license metadata
+    if card.data.license is None:
+        if not ("license_name" in card.data and "license_link" in card.data):
+            return False, (
+                "License not found. Please add a license to your model card using the `license` metadata or a"
+                " `license_name`/`license_link` pair."
+            )
+
+    # Enforce card content
+    if len(card.text) < 200:
+        return False, "Please add a description to your model card, it is too short."
+
+    return True, ""
+
+def is_model_on_hub(model_name: str, revision: str, token: str = None, trust_remote_code=False, test_tokenizer=False) -> tuple[bool, str]:
+    """Checks if the model model_name is on the hub, and whether it (and its tokenizer) can be loaded with AutoClasses."""
+    try:
+        config = AutoConfig.from_pretrained(model_name, revision=revision, trust_remote_code=trust_remote_code, token=token)
+        if test_tokenizer:
+            try:
+                tk = AutoTokenizer.from_pretrained(model_name, revision=revision, trust_remote_code=trust_remote_code, token=token)
+            except ValueError as e:
+                return (
+                    False,
+                    f"uses a tokenizer which is not in a transformers release: {e}",
+                    None
+                )
+            except Exception as e:
+                return (False, "'s tokenizer cannot be loaded. Is your tokenizer class in a stable transformers release, and correctly configured?", None)
+        return True, None, config
+
+    except ValueError:
+        return (
+            False,
+            "needs to be launched with `trust_remote_code=True`. For safety reason, we do not allow these models to be automatically submitted to the leaderboard.",
+            None
+        )
+
+    except Exception as e:
+        return False, "was not found on hub!", None
+
+
+def get_model_size(model_info: ModelInfo, precision: str):
+    """Gets the model size from the configuration, or the model name if the configuration does not contain the information."""
+    try:
+        model_size = round(model_info.safetensors["total"] / 1e9, 3)
+    except (AttributeError, TypeError):
+        return 0  # Unknown model sizes are indicated as 0, see NUMERIC_INTERVALS in app.py
+
+    size_factor = 8 if (precision == "GPTQ" or "gptq" in model_info.modelId.lower()) else 1
+    model_size = size_factor * model_size
+    return model_size
+
+def get_model_arch(model_info: ModelInfo):
+    """Gets the model architecture from the configuration"""
+    return model_info.config.get("architectures", "Unknown")
+
+def already_submitted_models(requested_models_dir: str) -> set[str]:
+    """Gather a list of already submitted models to avoid duplicates"""
+    depth = 1
+    file_names = []
+    users_to_submission_dates = defaultdict(list)
+
+    for root, _, files in os.walk(requested_models_dir):
+        current_depth = root.count(os.sep) - requested_models_dir.count(os.sep)
+        if current_depth == depth:
+            for file in files:
+                if not file.endswith(".json"):
+                    continue
+                with open(os.path.join(root, file), "r") as f:
+                    info = json.load(f)
+                    file_names.append(f"{info['model']}_{info['revision']}_{info['precision']}")
+
+                    # Select organisation
+                    if info["model"].count("/") == 0 or "submitted_time" not in info:
+                        continue
+                    organisation, _ = info["model"].split("/")
+                    users_to_submission_dates[organisation].append(info["submitted_time"])
+
+    return set(file_names), users_to_submission_dates

src/submission/submit.py

@@ -1,110 +1,88 @@
-import threading
-import time
-from datetime import datetime, timezone, timedelta
+import json
 import os
-import requests
+from datetime import datetime, timezone
+import time
 
+from datasets import Dataset, DatasetDict
 import pandas as pd
-from datasets import Dataset, get_dataset_config_names
-from datasets.exceptions import DatasetNotFoundError
-from pandas.api.types import is_integer_dtype
-import gradio as gr
+from pandas.api.types import is_integer_dtype, is_string_dtype
 
 from src.datamodel.data import F1Data
-from src.display.formatting import styled_error, styled_message
+from src.display.formatting import styled_error, styled_message, styled_warning
 from src.display.utils import ModelType
-from src.envs import SUBMISSIONS_REPO, TOKEN
+from src.envs import API, SUBMISSIONS_REPO, TOKEN
 from src.logger import get_logger
-from src.validation.validate import is_submission_file_valid, is_valid
 
+# from src.submission.check_validity import (
+#     already_submitted_models,
+#     check_model_card,
+#     get_model_size,
+#     is_model_on_hub,
+# )
 
 logger = get_logger(__name__)
 
-MIN_WAIT_TIME_PER_USER_HRS = 24
-RATE_LIMIT_WINDOW_HRS = 24
-MAX_SUBMISSIONS_PER_WINDOW = 10
 
-submission_lock = threading.Lock()
+def validate_submission(lbdb: F1Data, pd_ds: pd.DataFrame) -> str | None:
+    logger.info("Validating DS size %d columns %s set %s", len(pd_ds), pd_ds.columns, set(pd_ds.columns))
+    expected_cols = ["problem_id", "solution"]
+
+    if set(pd_ds.columns) != set(expected_cols):
+        return f"Expected attributes: {expected_cols}, Got: {pd_ds.columns.tolist()}"
+
+    if not is_integer_dtype(pd_ds["problem_id"]):
+        return "problem_id must be str convertible to int"
+
+    if any(type(v) != str for v in pd_ds["solution"]):
+        return "solution must be of type str"
+
+    submitted_ids = set(pd_ds.problem_id.astype(str))
+    if submitted_ids != lbdb.code_problem_ids:
+        missing = lbdb.code_problem_ids - submitted_ids
+        unknown = submitted_ids - lbdb.code_problem_ids
+        return f"Mismatched problem IDs: {len(missing)} missing, {len(unknown)} unknown"
+    if len(pd_ds) > len(lbdb.code_problem_ids):
+        return "Duplicate problem IDs exist in uploaded file"
+
+    return None
 
 
 def add_new_solutions(
     lbdb: F1Data,
-    username: str,
-    user_id: str,
     system_name: str,
     org: str,
+    sys_type: str,
     submission_path: str,
-    is_warmup_dataset: bool,
-    ensure_all_present: bool = False,
+    skip_validation: bool = False,
 ):
-    with submission_lock:
-        try:
-            submitted_ids = get_dataset_config_names(SUBMISSIONS_REPO, token=TOKEN)
-        except (DatasetNotFoundError, FileNotFoundError):
-            submitted_ids = []
-
-    if submitted_ids == ["default"]:
-        # means empty dataset
-        submitted_ids = []
-
-    logger.info(f"Found {len(submitted_ids)} submissions")
-
-    # Rate limits:
-    #   1. Users must wait MIN_WAIT_TIME_PER_USER_HRS hours between submissions.
-    #   2. No more than MAX_SUBMISSIONS_PER_WINDOW submissions RATE_LIMIT_WINDOW_HRS hours overall.
-
-    sub_df = pd.DataFrame.from_dict(
-        {
-            "submission_id": submitted_ids,
-            "user_id": map(_submission_id_to_user_id, submitted_ids),
-            "timestamp": map(_submission_id_to_timestamp, submitted_ids),
-        }
-    )
+    logger.info("ADD SUBMISSION! %s path %s", str((system_name, org, sys_type)), submission_path)
+    if not system_name:
+        return styled_error("Please fill system name")
 
-    # Per user limit
-    now = datetime.now(timezone.utc)
-    cutoff_user = now - timedelta(hours=MIN_WAIT_TIME_PER_USER_HRS)
-    user_last_submission_ts = sub_df[sub_df.user_id == user_id].timestamp.max()
-
-    if pd.notna(user_last_submission_ts) and user_last_submission_ts > cutoff_user:
-        remaining_hrs = (user_last_submission_ts - cutoff_user).total_seconds() / 3600
-        logger.info(f"{username} must wait {remaining_hrs:.2f} more hours.")
-        return styled_error(
-            f"You must wait {MIN_WAIT_TIME_PER_USER_HRS} hours between submissions. "
-            f"Remaining wait time: {remaining_hrs:.2f} hours"
-        )
-
-    # Overall limit
-    cutoff_overall = now - timedelta(hours=RATE_LIMIT_WINDOW_HRS)
-    if len(sub_df.timestamp > cutoff_overall) >= MAX_SUBMISSIONS_PER_WINDOW:
-        logger.info(
-            f"Too many submissions in the last {RATE_LIMIT_WINDOW_HRS} hours: {len(sub_df.timestamp > cutoff_overall)}."
-        )
-        return styled_error("The leaderboard has reached its submission capacity for now. Please try again later.")
-
-    logger.info(
-        f"Adding new submission: {system_name=}, {org=}, and {submission_path=}",
-    )
+    if not org:
+        return styled_error("Please fill organization name")
+
+    if not sys_type:
+        return styled_error("Please select system type")
+    sys_type = ModelType.from_str(sys_type).name
 
-    # Double-checking.
-    for val in [system_name, org]:
-        assert is_valid(val)
-    assert is_submission_file_valid(submission_path, is_warmup_dataset=is_warmup_dataset)
+    if not submission_path:
+        return styled_error("Please upload JSONL solutions file")
 
     try:
         submission_df = pd.read_json(submission_path, lines=True)
-        if ensure_all_present:
-            _validate_all_submissions_present(lbdb=lbdb, pd_ds=submission_df)
-    except Exception:
-        logger.warning("Failed to parse submission DF!", exc_info=True)
-        return styled_error(
-            "An error occurred. Please try again later."
-        )  # Use same message as external error. Avoid infoleak.
-
-    submission_id = f"{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}_{username}_{user_id}"
-
-    # Seems good, creating the eval.
-    logger.info(f"Adding new submission: {submission_id}")
+    except Exception as e:
+        return styled_error(f"Cannot read uploaded JSONL file: {str(e)}")
+
+    if not skip_validation:
+        validation_error = validate_submission(lbdb, submission_df)
+        if validation_error:
+            return styled_error(validation_error)
+
+    submission_id = f"{system_name}_{org}_{sys_type}_{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}"
+
+    # Seems good, creating the eval
+    print(f"Adding new submission: {submission_id}")
     submission_ts = time.time_ns()
 
     def add_info(row):
@@ -112,103 +90,37 @@ def add_new_solutions(
             **row,
             "system_name": system_name,
             "organization": org,
+            "system_type": sys_type,
             "submission_id": submission_id,
             "submission_ts": submission_ts,
-            "evaluation_id": "",  # This will be set later when the evaluation is launched in the backend
-            "evaluation_start_ts": "",  # This will be set when the evaluation starts
         }
 
     ds = Dataset.from_pandas(submission_df).map(add_info)
 
-    with submission_lock:
-        ds.push_to_hub(
-            SUBMISSIONS_REPO,
-            submission_id,
-            private=True,
-        )
-
-    return styled_message(
-        "Your request has been submitted to the evaluation queue!\n"
-        + "Results may take up to 24 hours to be processed and shown in the leaderboard."
-    )
-
+    # dsdict = DatasetDict({submission_id: ds})
+    # dsdict.push_to_hub(SUBMISSIONS_REPO, private=True)
 
-def fetch_sub_claim(oauth_token: gr.OAuthToken | None) -> dict | None:
-    if oauth_token is None:
-        return None
-    provider = os.getenv("OPENID_PROVIDER_URL")
-    if not provider:
-        return None
-    try:
-        oidc_meta = requests.get(f"{provider}/.well-known/openid-configuration", timeout=5)
-        oidc_meta = oidc_meta.json()
-        userinfo_ep = oidc_meta["userinfo_endpoint"]
-        claims = requests.get(userinfo_ep, headers={"Authorization": f"Bearer {oauth_token.token}"}, timeout=5)
-        logger.info(f"userinfo_endpoint response: status={claims.status_code}\nheaders={dict(claims.headers)}")
-        claims = claims.json()
-        # Typical fields: sub (stable id), preferred_username, name, picture
-        return {
-            "sub": claims.get("sub"),
-            "preferred_username": claims.get("preferred_username"),
-            "name": claims.get("name"),
-        }
-    except Exception as e:
-        logger.warning(f"Failed to fetch user claims: {e}")
-        return None
+    ds.push_to_hub(SUBMISSIONS_REPO, submission_id, private=True)
+    # print("Creating eval file")
+    # OUT_DIR = f"{EVAL_REQUESTS_PATH}/{user_name}"
+    # os.makedirs(OUT_DIR, exist_ok=True)
+    # out_path = f"{OUT_DIR}/{model_path}_eval_request_False_{precision}_{weight_type}.json"
 
+    # with open(out_path, "w") as f:
+    #     f.write(json.dumps(eval_entry))
 
-def fetch_user_info(oauth_token: gr.OAuthToken | None) -> dict | None:
-    if oauth_token is None:
-        return None
-    try:
-        headers = {"Authorization": f"Bearer {oauth_token.token}"}
-        logger.info("HEADERS %s", headers)
-        r = requests.get("https://huggingface.co/api/whoami-v2", headers=headers)
-        logger.info("RESP CODE %s", r.status_code)
-        logger.info("RESP content %s", r.text)
-        if r.status_code != 200:
-            return None
-        return r.json()
-    except:
-        logger.exception("Cannot get user info")
-        return None
-
-
-def _validate_all_submissions_present(
-    lbdb: F1Data,
-    pd_ds: pd.DataFrame,
-):
-    logger.info(f"Validating DS size {len(pd_ds)} columns {pd_ds.columns} set {set(pd_ds.columns)}")
-    expected_cols = ["problem_id", "solution"]
+    # print("Uploading eval file")
+    # API.upload_file(
+    #     path_or_fileobj=out_path,
+    #     path_in_repo=out_path.split("eval-queue/")[1],
+    #     repo_id=QUEUE_REPO,
+    #     repo_type="dataset",
+    #     commit_message=f"Add {model} to eval queue",
+    # )
 
-    if set(pd_ds.columns) != set(expected_cols):
-        return ValueError(f"Expected attributes: {expected_cols}, Got: {pd_ds.columns.tolist()}")
-
-    if not is_integer_dtype(pd_ds["problem_id"]):
-        return ValueError("problem_id must be str convertible to int")
+    # # Remove the local file
+    # os.remove(out_path)
 
-    if any(type(v) is not str for v in pd_ds["solution"]):
-        return ValueError("solution must be of type str")
-
-    submitted_ids = set(pd_ds.problem_id.astype(str))
-    if submitted_ids != lbdb.code_problem_ids:
-        missing = lbdb.code_problem_ids - submitted_ids
-        unknown = submitted_ids - lbdb.code_problem_ids
-        raise ValueError(f"Mismatched problem IDs: {len(missing)} missing, {len(unknown)} unknown")
-    if len(pd_ds) > len(lbdb.code_problem_ids):
-        return ValueError("Duplicate problem IDs exist in uploaded file")
-
-
-def _submission_id_to_user_id(submission_id: str) -> str:
-    """
-    Extracts the user ID from the submission ID: "YYYYMMDD_HHMMSS_username_userid"
-    """
-    return submission_id.rsplit("_", 1)[-1]
-
-
-def _submission_id_to_timestamp(submission_id: str) -> datetime:
-    """
-    Extracts the timestamp from the submission ID: "YYYYMMDD_HHMMSS_username_userid"
-    """
-    ts_str = "_".join(submission_id.split("_", 2)[:2])
-    return datetime.strptime(ts_str, "%Y%m%d_%H%M%S").replace(tzinfo=timezone.utc)
+    return styled_message(
+        "Your request has been submitted to the evaluation queue!\nResults may take up to 24 hours to be processed and shown in the leaderboard."
+    )

src/validation/validate.py

@@ -1,115 +0,0 @@
-import json
-import os
-import string
-
-from src.logger import get_logger
-
-WARMUP_DATASET_SIZE = 100
-DATASET_SIZE = 120
-
-MIN_INPUT_LENGTH = 2
-MAX_INPUT_LENGTH = 20
-
-MIN_SUBMISSION_SIZE = 1
-MAX_SUBMISSION_SIZE = 1024 * 1024 * 120  # 120 MB.
-MAX_SINGLE_SUBMISSION_SIZE = 1024 * 1024  # 1MB.
-MAX_SUBMISSION_LINES = DATASET_SIZE + 1  # Allow empty line.
-
-logger = get_logger(__name__)
-
-
-def is_valid(
-    s: str,
-    min_length: int = MIN_INPUT_LENGTH,
-    max_length: int = MAX_INPUT_LENGTH,
-) -> bool:
-    """
-    @brief Checks whether the given string is valid.
-    @param s The string to validate.
-    @return True iff all characters are in [a-zA-Z0-9], spaces, or '.' and '-', and the length if between
-            min length and max length.
-    """
-
-    characters = [c for c in s]  # Not using the length from len(.) as that includes unicode characters.
-    if len(characters) < min_length or len(characters) > max_length:
-        return False
-
-    # Very important: We delimit using underscores. So these _CANNOT_ be allowed in sanitised strings.
-    ALLOWED = (
-        [c for c in string.ascii_lowercase]
-        + [c for c in string.ascii_uppercase]
-        + [c for c in string.digits]
-        + [" ", ".", "-"]
-    )
-    for c in s:
-        if c not in ALLOWED:
-            return False
-    return True
-
-
-def is_submission_file_valid(
-    submission_path: str,
-    is_warmup_dataset: bool,
-) -> bool:
-    """
-    @brief Checks whether the given submission file is valid.
-    @param submission_path The path to the submission file.
-    @param is_warmup_dataset Whether we are working on the regular or the warmup dataset.
-    @return True iff the file is within the size constraints, a JSONL, and every line is no longer than
-            the fixed maximum bound.
-    """
-
-    if not os.path.exists(submission_path):
-        logger.warning(f"Could not find submission file {submission_path=}")
-        return False
-
-    submission_size = os.stat(submission_path).st_size
-    if submission_size < MIN_SUBMISSION_SIZE or submission_size > MAX_SUBMISSION_SIZE:
-        logger.warning(f"Submission size was {submission_size}, exceeding [{MIN_SUBMISSION_SIZE, MAX_SUBMISSION_SIZE}]")
-        return False
-
-    with open(submission_path, "r") as f:
-
-        # Not using readlines() to avoid consuming a large buffer at once.
-        n_lines = 0
-        seen_ids = set()
-        while len(line := f.readline(MAX_SINGLE_SUBMISSION_SIZE)) > 0:
-            n_lines += 1
-            if n_lines > MAX_SUBMISSION_LINES:
-                logger.warning(f"Got submission with more than {MAX_SUBMISSION_LINES} lines")
-                return False
-
-            if not (line.startswith("{") and (line.endswith("}") or line.endswith("}\n"))):
-                logger.warning("Submission has line that does not appear to be a JSONL")
-                return False
-
-            d = json.loads(line)
-            if set(d.keys()) != set(["problem_id", "solution"]):
-                logger.warning("Found unexpected keys")
-                return False
-
-            if not ((type(d["problem_id"]) is str or type(d["problem_id"]) is int) and type(d["solution"] is str)):
-                logger.warning("Found unexpected types")
-                return False
-
-            try:
-                problem_id = int(d["problem_id"])
-            except Exception:
-                logger.warning("Could not convert problem ID to int")
-                return False
-
-            if is_warmup_dataset:
-                if problem_id < DATASET_SIZE or problem_id >= DATASET_SIZE + WARMUP_DATASET_SIZE:
-                    logger.warning(f"Problem ID {problem_id} is beyond allowed bounds")
-                    return False
-            else:
-                if problem_id < 0 or problem_id >= DATASET_SIZE:
-                    logger.warning(f"Problem ID {problem_id} is beyond allowed bounds")
-                    return False
-
-            if problem_id in seen_ids:
-                logger.warning(f"Got duplicate submission -- ID {problem_id} appears twice")
-                return False  # Duplicate submission.
-            seen_ids.add(problem_id)
-
-    return True

terms/submission-agreement.md

@@ -1,71 +0,0 @@
-FormulaOne AI Coding Challenge - Submission Agreement v1.2
-
-Last updated: 6 Aug 2025
-
-1. Definitions
-
-"Submission" means the code file you upload for a listed problem.
-
-"Organiser" means the FormulaOne Team, c/o <legal entity and address>.
-
-"Competition Site" means the leaderboard hosted at <HF URL>.
-
-2. Licence to Organiser
-You retain all IP in your Submission. You grant the Organiser and its
-academic partners a worldwide, royalty-free, non-exclusive licence to:
-a) copy, run, test and modify the Submission solely to evaluate it
-against public and private test cases;
-b) store the Submission on secure servers;
-c) manually review the Submission for non-commercial, scientific
-research;
-d) reproduce small code excerpts (<= 25 lines) in research papers or
-blogs;
-e) contact you via the email linked to your platform account to ask
-clarification questions, explore collaboration, or publicise notable
-results.
-3. Security, Integrity and Acceptable Use
-- You confirm the Submission contains no deliberately malicious code,
-back-doors, attempts to exfiltrate data, or calls to external network
-resources beyond what the submission guidelines permit.
-- You will not attempt to discover, reconstruct, scrape, or share
-private test cases, nor to probe the sandbox environment.
-- The Organiser may refuse, delete, re-run or disqualify any
-Submission at its sole discretion for suspected abuse, tampering or
-rule violations.
-4. Competition Rules
-- One account per team.
-- You must own or have rights to your Submission.
-- We may re-execute Submissions to confirm scores; ties may be broken
-by earliest valid submission or additional hidden tests.
-- We may correct or withdraw scores if errors or rule breaches are
-discovered later.
-5. Leaderboard and Publicity
-Your chosen display name, organisation (if provided), total score and
-ranking may be shown publicly and archived indefinitely. You may use a
-pseudonym.
-6. Ownership of Benchmark Assets
-All evaluation code, problem statements and test data remain (c) 2025
-FormulaOne Team and are licensed under Apache 2.0 (code) and CC BY 4.0
-(content) as described in the public repository.
-7. Export Control and Sanctions Compliance
-You represent that your participation is not prohibited by applicable
-export-control or sanctions laws and that you are not located in, under
-the control of, or a national or resident of any country or person
-embargoed by relevant authorities.
-
-8. No Warranty
-The benchmark and infrastructure are provided "as is." To the maximum
-extent permitted by law, the Organiser disclaims all warranties,
-express or implied.
-9. Limitation of Liability
-The Organiser's total liability arising out of or in connection with
-the Competition will not exceed USD 100.
-10. Changes to Rules
-We may update these terms and technical rules from time to time.
-Material changes will be posted on the Competition Site and apply to
-submissions made after the effective date.
-11. Governing Law and Venue
-This Agreement is governed by the laws of England and Wales. The courts
-of London, UK, have exclusive jurisdiction.
-12. Contact
-Questions about these terms: legal@formulaone-ai.org