| Dataset | Evaluation Task | Language | Description |
|---|---|---|---|
| malhajar/mmlu_tr-v0.2 | Lighteval MMLU | Turkish | Turkish adaptation of MMLU (Massive Multitask Language Understanding) v0.2 covering 57 academic subjects including mathematics, physics, chemistry, biology, history, law, and computer science. Tests knowledge and reasoning capabilities across multiple domains with multiple-choice questions. |
| malhajar/truthful_qa-tr-v0.2 | Lighteval TruthfulQA | Turkish | Turkish version of TruthfulQA (v0.2) designed to measure model truthfulness and resistance to generating false information. Contains questions where humans often answer incorrectly due to misconceptions or false beliefs, testing the model's ability to provide accurate information. |
| malhajar/winogrande-tr-v0.2 | Lighteval WinoGrande | Turkish | Turkish adaptation of WinoGrande (v0.2) focusing on commonsense reasoning through pronoun resolution tasks. Tests the model's ability to understand context, make logical inferences, and resolve ambiguous pronouns in everyday scenarios. |
| malhajar/hellaswag_tr-v0.2 | Lighteval HellaSwag | Turkish | Turkish version of HellaSwag (v0.2) for commonsense reasoning evaluation. Tests the model's ability to predict plausible continuations of everyday scenarios and activities, requiring understanding of common sense and typical human behavior patterns. |
| malhajar/arc-tr-v0.2 | Lighteval ARC | Turkish | Turkish adaptation of ARC (AI2 Reasoning Challenge) v0.2 focusing on science reasoning and question answering. Contains grade school level science questions that require reasoning beyond simple factual recall, covering topics in physics, chemistry, biology, and earth science. |
| malhajar/gsm8k_tr-v0.2 | Lighteval GSM8K | Turkish | Turkish version of GSM8K (Grade School Math 8K) v0.2 for mathematical reasoning evaluation. Contains grade school level math word problems that require multi-step reasoning, arithmetic operations, and logical problem-solving skills to arrive at the correct numerical answer. |
| newmindai/mezura-eval-data | Auto-Arena | Turkish | mezura-eval dataset is a Turkish-language legal text dataset designed for evaluation tasks with RAG context support. The subsets include domains like Environmental Law, Tax Law, Data Protection Law and Health Law each containing annotated samples. Every row includes structured fields such as the category, concept, input and contextual information drawn from sources like official decisions. |
| newmindai/mezura-eval-data | EvalMix | Turkish | mezura-eval dataset is a Turkish-language legal text dataset designed for evaluation tasks with RAG context support. The subsets include domains like Environmental Law, Tax Law, Data Protection Law and Health Law each containing annotated samples. Every row includes structured fields such as the category, concept, input and contextual information drawn from sources like official decisions. |
| newmindai/mezura-eval-data | Retrieval | Turkish | mezura-eval dataset is a Turkish-language legal text dataset designed for evaluation tasks with RAG context support. The subsets include domains like Environmental Law, Tax Law, Data Protection Law and Health Law each containing annotated samples. Every row includes structured fields such as the category, concept, input and contextual information drawn from sources like official decisions. |
Authentication required. Please log in with your Hugging Face account to submit models.
" ) ) # Log successful authentication try: if hasattr(profile, 'name'): username = profile.name elif hasattr(profile, 'username'): username = profile.username else: username = str(profile) logger.info(f"User authenticated: {username}") except Exception as e: logger.info(f"LOGIN - Error inspecting profile: {str(e)}") # User is logged in - show form, hide error return ( gr.update(visible=True), gr.update(visible=False, value="") ) # Connect login button to visibility toggle login_button.click( fn=toggle_form_visibility, inputs=[login_button], outputs=[login_dependent_content, auth_error] ) # Check visibility on page load demo.load( fn=toggle_form_visibility, inputs=[login_button], outputs=[login_dependent_content, auth_error] ) # Handle submission with authentication check def submit_model(model, base_model, reasoning, email, profile): # Authentication check if profile is None: logging.warning("Unauthorized submission attempt with no profile") return "Authentication required. Please log in with your Hugging Face account.
" # IMPORTANT: In local development, Gradio returns "Sign in with Hugging Face" string # This is NOT a real authentication, just a placeholder for local testing if isinstance(profile, str) and profile == "Sign in with Hugging Face": # Block submission in local dev with mock auth return "⚠️ HF authentication required.
" # Email is required if not email or email.strip() == "": return "Email address is required to receive benchmark results.
" # Check if the model is a merged model (not supported) try: from src.submission.check_validity import determine_model_type model_type, _ = determine_model_type(model) if model_type == "merged_model" or model_type == "merge": return "Merged models are not supported yet. Please submit an adapter model instead.
" except Exception as e: # If error checking model type, continue with submission logging.warning(f"Error checking model type: {str(e)}") # Call the benchmark function with profile information # base_model validasyonunu kaldırdık ama parametre olarak yine de gönderiyoruz result_message, _ = submit_unified_benchmark(model, base_model, reasoning, email, profile) logging.info(f"Submission processed for model: {model}") return result_message # Connect submit button submit_button.click( fn=submit_model, inputs=[ model_to_evaluate, base_model_dropdown, reasoning_checkbox, email_input, login_button ], outputs=[result_output] ) except Exception as e: traceback.print_exc() gr.Markdown(f"## Error: An issue occurred while loading the LLM Benchmark screen") gr.Markdown(f"Error message: {str(e)}") gr.Markdown("Please check your configuration and try again.") # Citation information at the bottom gr.Markdown("---") with gr.Accordion(CITATION_BUTTON_LABEL, open=False): gr.Textbox( value=CITATION_BUTTON_TEXT, lines=10, show_copy_button=True, label=None ) return demo if __name__ == "__main__": # Get app logger logger = logging.getLogger("mezura") # Additional sensitive filter for remaining logs class SensitiveFilter(logging.Filter): def filter(self, record): msg = record.getMessage().lower() # Filter out messages with tokens, URLs with sign= in them, etc sensitive_patterns = ["token", "__sign=", "request", "auth", "http request"] return not any(pattern in msg.lower() for pattern in sensitive_patterns) # Apply the filter to all loggers for logger_name in logging.root.manager.loggerDict: logging.getLogger(logger_name).addFilter(SensitiveFilter()) try: logger.info("Creating demo...") demo = create_demo() logger.info("Launching demo on 0.0.0.0...") # Add options to fix the session.pop error demo.launch( server_name="0.0.0.0", server_port=7860 ) except FileNotFoundError as e: logger.critical(f"Configuration file not found: {e}") print(f"\n\nERROR: Configuration file not found. Please ensure config/api_config.yaml exists.\n{e}\n") sys.exit(1) except ValueError as e: logger.critical(f"Configuration error: {e}") print(f"\n\nERROR: Invalid configuration. Please check your config/api_config.yaml file.\n{e}\n") sys.exit(1) except Exception as e: logger.critical(f"Could not launch demo: {e}", exc_info=True)