ibm-granite

/

rag-intrinsics-lib

like 6

Follow

IBM Granite 3.28k

Review comments:

• In general it would be a good idea to word-wrap this file so that lines are less than 80 characters long. I needed to copy the diff into a text editor to read it.
• Intro needs to be updated to reflect the fact that this model is not part of the granite-experiments collection.
• Description of which adapters are available should reflect the full set of adapters we have trained and evaluated models. According to my notes there are six adapters. The evaluation section specifically references aLoRA adapters, so it is very important that the list of available intrinsic models include aLoRA adapters.
• The "Model Output" section should explain that the raw output of the model is a single token, but the processed output that the model sees is an assistant message containing a JSON record. This section should explain that this JSON record contains a real-valued answerability likelihood score, not a categorical true/false value.
• The first example of usage should use vLLM's OpenAI-compatible inference API as shown in this notebook. This example should be preceded with instructions for using the shell script here to start a local copy of vLLM with all of the intrinsics in this library.
• The second example of usage should use Transformers to run the LoRA adapter, as shown in this notebook. This second example should start with its input as a chat completion request. Unlike the notebook, which reads this request from a file, the example should contain the request inline as parsed JSON data represented by Python dictionaries, lists, etc. This second example should end with its result as a chat completion result, as shown in the notebook. This second example should use the helper functions provided in granite-common to translate between OpenAI APIs and Transformers APIs. It is VERY IMPORTANT that we not include example code that manipulates prompt strings and low-level output strings directly, because that is NOT how we want our users to use any of our intrinsics.
• In the "Evaluation/Answerability Classification" section, the detailed table of MT-RAG results do not contain the same set of models as the final summary table immediately afterwards. Please add the missing rows to the detailed table or explain why they are not present.

Hi @frreiss Thank you for your comments. I have addressed them. pls lmk if I missed anything. Thank you.