Spaces:
Running
on
CPU Upgrade
chat template didn't get applied to the model
Is the chat template only applied to alrage, as i see that the chat template isn't applied to the model on any dataset except alrage , which degrades what we expected , we ran this benchmark locally with lighteval before submission and this degraded much on the performance. can you tell me exactly what happened, i turned on chat_template flag on submission, so what happened
Hi
@AIC-MCIT
You will notice that upon submission, we ask you whether you wish to use chat_template or not, and based on the submitter's input we will enable/disable chat template.
In case you have submitted a model with the incorrect preference in chat_template, we are happy to rerun the evaluation with your desired setup
the problem is that we submitted with "Apply_chat_template" and i saw the results in our model on the details_our_model_name , the full prompt field doesn't contain the template, although the alrage one has the chat_template in the full prompt field , that what i am talking about
Applied-Innovation-Center/AIC-1
@AIC-MCIT
I looked at how your model was evaluated, and I agree that your model didn't have the chat template enabled when evaluating some of the benchmarks.
After rerunning the eval, I noticed the only noticeable discrepancy occurred in the Madinah benchmark, which moved from the 50s to the high 70s.
I'm about to update the scores of your model, so please confirm that our findings are correct?