I am currently working on a project where I have to use the Qwen/Qwen3-0.6B LLM to ensure JSON -compatible answers via constrained decoding. The LLM's job is to receive a user prompt and to specify what function should it use out of a predefined set of functions and what parameters should it pass to this function in such format:
{
"prompt": "Calculate compound interest on 1234567.89 at 0.0375 rate for 23 years",
"name": "fn_calculate_compound_interest",
"parameters": {
"principal": 1234567.89,
"rate": 0.0375,
"years": 23}
}
The main problem is that I can't really know when the model finishes predicting a parameter name for example; so it keeps generating ample tokens. I tried naively to tell the model to end its answers with a <|im_end|> but that didn't work.
I tried to end generating new tokens when the model predicts a comma or a curly brace (indicating either a new parameter or the end of the list of parameters"; yet parameters may contain those characters sometimes and then the answer will be truncated. Therefore, a special token like is the most reasonable option because it would never exist in text.
So is there any other way I can deduce the completion of a parameter prediction ? And if was the best solution, how to force the model to end its answers with it ?
If you're already using constrained decoding, the cleanest approach is to define your JSON schema with precision and allow the grammar itself to indicate when you're finished once the closing brace of the parameters object becomes the sole valid next token based on your schema, the model must emit it, and you'll know the response is complete without requiring any special token. Both Outlines and lm-format-enforcer work great for this with Qwen models 😊
If you’re already using constrained decoding, i’d stop based on the JSON grammar reaching a valid terminal state rather than waiting for punctuation or EOS
