The mantra of ‘more compute, more data’ is still strong with AI companies.
I agree that the latest models haven’t had the massive gains that e.g. GPT-3 to GPT-4 had and people say that data is mostly used up. I also don’t see a big reduction in hallucinations.
Still, none of the big players have admitted that they hit a wall, either with scaling or data. Getting reliable information from such secretive companies is hard.
And their actions (building new data centers, new chip designs) seem to indicate confidence in their path.
if? They have shown their limits, and we don’t have a way forward.
The mantra of ‘more compute, more data’ is still strong with AI companies.
I agree that the latest models haven’t had the massive gains that e.g. GPT-3 to GPT-4 had and people say that data is mostly used up. I also don’t see a big reduction in hallucinations.
Still, none of the big players have admitted that they hit a wall, either with scaling or data. Getting reliable information from such secretive companies is hard.
And their actions (building new data centers, new chip designs) seem to indicate confidence in their path.
If that confidence is warranted… who knows?