Also saw some comments on Twitter, that ChatGPT was getting worse in generating code compared to a few months ago.

I suppose it’s the same, but gives inconsistent answers (by design)

Since it’s just a complex algorithm, and not what many people think true AI, it does not change and definitely it’s not self-rewritting AI in 5 min. It’s important to note what the “design” is. The AI is a … how to call it best (?) - a “smart” data aggregation? Especially AI run locally is too fast to browser whole network for the best answers, so what the algorithm could do then?

1. Use fixed number of sources - this obviously drastically limits quality, but standalone it’s not so big deal. What happens if said source is used less and less times? Some time ago we talked about StackOverflow’s decrease of popularity. In that case some topics, even if they are popular in specific community could, may be completely not available to AI. What it should do then? Since many errors and warnings are using common naming AI may give a solution for a “similar” problem or propose something that’s outdated.

2. Use search engine - at first it looks a better approach, but what if the search engine works as same as for the user?

• If “bad” algorithm is instructing “better” AI which in best case means just a research speed improvement i.e. the AI would not find what human could not too. In worst case it means that “AI” would produce worse results, but much faster than user would do. If we talk about float type it’s fine. If we talk about professional work (like when decimal is used for operating on money) it’s a terrible solution.

• However the search engine also could have (and most probably it’s how it works) it’s own “AI” model. Then no matter which AI model is worse the results would always be far away from perfection.

3. The data to train a model is selected by AI developer - at one side it’s awesome since we should be safe in terms of quality, but on the other side there are similar limitations like those above (either not enough source data or there’s an algorithm/developer deciding which source is trusted and which not)

So what should good AI do? I guess same thing as on chess sites. It quickly gives “some” results and if the user waits a bit more then after processing more data the results may be modified. Secondly if AI suggest something it should be able to try it. What’s a problem with generating a simple script or project with specific dependencies (if any), compile and run code? We already have lots of automation tools for that and we do not need a “true” AI to able to automatically test if specific scenario is good. The best part however is said script/app as since it’s small it’s really easy to check what has been changed, so similar changes could be applied to desired project much smoothly.