`ibm-granite/granite-3.3-8b-math-prm-v2` powered by vLLM - [Original Model Card](https://huggingface.co/ibm-granite/granite-3.3-8b-math-prm-v2) - [vLLM Documentation](https://docs.vllm.ai) Granite 3.…
ibm-granite/granite-3.3-8b-math-prm-v2 powered by vLLM
Granite 3.3 8B Math PRM v2 is finetuned version of Granite-3.3-8B-Instruct, which gives the language model the ability of process supervision on mathematical reasoning steps by assessing the correctness of each step of a reasoning chain.
At inference, the model takes a question and a response which can be broken down into generated steps, and for each step it determines whether the reasoning chain so far is correct (indicated by generating a single token, Y) or incorrect (indicated by generating N).
Before obtaining a response, the model expects the user generated prompt "Is this response correct so far (Y/N)?", which should be added at the end of every step of the reasoning chain. The model will then generate either "Y" or "N" as the next token in the response, indicating whether the reasoning chain so far is correct or not
Therefore, each user message should be formatted as follows:
{
"role": "user",
"content": <query> + " " + <response_step> + "Is this response correct so far (Y/N)?"
}
Additionally, you can add logprobs: true parameter in the payload to obtain the log probability of the generated token, which can be used as an outcome reward score after normalization.
Check the Usage section on the model card for more details.
NOTE: Keep in mind that this deployment doesn't support returning logprobs for multiple assistant messages in the same request, as it only returns logprobs for the last generated token. Using this deployment as a PRM would require multiple requests and extra logic for incrementally add the reasoning steps in each request.
Chat Completions API
Send Request
You can use cURL or any REST Client to send a request to the Azure ML endpoint with your Azure ML token.
curl <AZUREML_ENDPOINT_URL> \
-X POST \
-H "Authorization: Bearer <AZUREML_TOKEN>" \
-H "Content-Type: application/json" \
-d '{"model":"ibm-granite/granite-3.3-8b-math-prm-v2","messages":[{"role":"user","content":"What is the result of 2 + 2? The answer of 2 + 2 is 4. Is this response correct so far (Y/N)?"}],"logprobs": true}'
Supported Parameters
The following are the only mandatory parameters to send in the HTTP POST request to v1/chat/completions.
- model (string): Model ID used to generate the response, in this case since only a single model is deployed within the same endpoint you can either set it to ibm-granite/granite-3.3-8b-math-prm-v2 or leave it blank instead.
- messages (array): A list of messages comprising the conversation so far. Depending on the model you use, different message types (modalities) are supported, like text, images, and audio.
The rest of the parameters are optional, and since this model is powered by vLLM with an OpenAI compatible interface on top for the Chat Completions API for text generation, then the I/O interfaces for both generation and streaming are the same as in OpenAI Chat Completions API.
You can find the whole specification of the allowed parameters in the OpenAI Chat Completion API Specification, or alternatively in the endpoint /openapi.json for the current Azure ML Endpoint.
Example payload
{
"model": "ibm-granite/granite-3.3-8b-math-prm-v2",
"messages": [
{"role":"user","content":"What is the result of 2 + 2? The answer of 2 + 2 is 4. Is this response correct so far (Y/N)?"}
],
"logprobs": true
}