I am using a code-completion model for (will be open sourced very soon).
Qwen2.5-coder 1.5b though tends to repeat what has already been written, or change it slightly. (See the video)
Is this intentional? I am passing the prefix and suffix correctly to ollama, so it knows where it currently is. I’m also trimming the amount of lines it can see, so the time-to-first-token isn’t too long.
Do you have a recommendation for a better code model, better suited for this?
Fair enough. I haven’t had repetition being an issue for quite some time now. Usually that happened when I manually messed with the context, or had the parameters set incorrectly. Are you sure the fill-in-the-middle support is set up correctly and the framework inserts the correct tokens for that? I mean if it does other things properly, but isn’t able to fill in, maybe it’s that.
Your assumption is very good, but I am sure that the completion is setup correctly. Sometimes it does fill in correctly, like suggesting variable types and adding comments to functions.
So sometimes completion works fieny but other times it doesn’t. I use the ollama REST API for the completion, so the token handling isn’t on my side.
I hope someone else chimes in and can offer some advice. You could have a look at the ollama log / debug output and see if the
<|fim_prefix|>,<|fim_suffix|>and<|fim_middle|>tokens are at the correct spots when fed into the LLM. (as per https://github.com/QwenLM/Qwen2.5-Coder?tab=readme-ov-file#3-file-level-code-completion-fill-in-the-middle ) Other than that, I don’t have a clue. You could also try a different model. But I guess there is something wrong somewhere. I mean coding sometimes is repetetive. But it shouldn’t do it like that.