There is a reason there is sometimes a notable decrease in quality of the same AI model a while after it’s released.
Hosters of the models (like OpenAI or Microsoft) may have switched to a quantized version of their model. Quantization is a common practice to increase power efficiency and make the model easier to run, by essentially rounding the weights of the model to a lower precision. This decreases VRAM and storage usage significantly, at the cost of a bit of quality, where higher quantization results in worse quality.
For example, the base model will likely be in FP16, full floating point precision. They may switch to a Q8 version, which nearly halves the size of the model, with about a 3-7% decrease in quality.
Is it just me, or are Github Copilot and ChatGPT getting dumber? I’m quite underwhelmed lately.
There is a reason there is sometimes a notable decrease in quality of the same AI model a while after it’s released.
Hosters of the models (like OpenAI or Microsoft) may have switched to a quantized version of their model. Quantization is a common practice to increase power efficiency and make the model easier to run, by essentially rounding the weights of the model to a lower precision. This decreases VRAM and storage usage significantly, at the cost of a bit of quality, where higher quantization results in worse quality.
For example, the base model will likely be in FP16, full floating point precision. They may switch to a Q8 version, which nearly halves the size of the model, with about a 3-7% decrease in quality.
Maybe the more copilot is used, the more code on github is ai garbage, ths more copilot trains on github, the worse it gets.
Probably quite a lot of other things too, but I haven’t used it so I don’t know if it has got worse.
garbage in garbage out - sounds like a reasonable take
Novelty weared off
could be, maybe expectations went up too
ChatGPT has its moments but generally it creates more problems than it solves.
Dont knows for these two but the enterprise version of copilot for office workers is so damn awefull its embarrassing.
It literally suggests to do stuff that it is incapable of doing.
I tried to get some doc as a flowchart using mermaid script, i know for a fact other llms have no problems with that:
After 3 attempts of it trying to code but resulting in an actual error it gives me,
Do you want me to provide this flowchart in visio format instead.
Generates a powerpoint about the topic.
Tries to generate an image of a flowchart with not a single word spelled correctly and arrows going nowhere.
Sorry, if you want i can provide a flowchart in viso file format instead.
Generates a brand new bad PowerPoint.
I copy pasted the entire conversation to claude and it instantly gave me what i asked.
On par for Microsoft software in general. Seems like every week I discover new bugs in outlook.com…