Chat Template Delimiter Handling Update

Overview

This change modifies how delimiters are handled when applying chat templates in the request construction process for likelihood and multiple-choice based tasks. When apply_chat_template is set to True, the target delimiter is now set to an empty string instead of using the configured delimiter.

Background

By default, the system uses a target delimiter (typically a whitespace " ") between the context and target text when constructing prompts. The full string is constructed as:

doc_to_text(doc) + target_delimiter + doc_to_target(doc)

While this worked well for base models where we wanted the model to predict a single whitespace followed by the answer, chat models have their own formatting conventions that handle spacing differently.

The Change

When apply_chat_template=True, the target delimiter is now empty ("") instead of the default whitespace
This prevents interference between chat template formatting and the default delimiter system
Particularly important for multiple choice tasks where the template itself handles spacing

Example

# Before (with default delimiter " ")
<user>Question: What color is the sky?\nAnswer:<assistant> blue

# After
<user>Question: What color is the sky?\nAnswer:<assistant>blue