Definitely depends on the type of question. I find for documentation type questions I get the 90% good answers, like how do I do something with this library, it’s good, which makes sense because that libraries documentation is probably in the training data. But for more open ended questions, like how do I solve this problem, I see similar performance to what you’re saying. I think it’s a good retrieval and synthesises tool which can really save a ton of time if you already have a high level plan of action and just use it to fill in some specific details.
Definitely depends on the type of question. I find for documentation type questions I get the 90% good answers, like how do I do something with this library, it’s good, which makes sense because that libraries documentation is probably in the training data. But for more open ended questions, like how do I solve this problem, I see similar performance to what you’re saying. I think it’s a good retrieval and synthesises tool which can really save a ton of time if you already have a high level plan of action and just use it to fill in some specific details.