If they’re using stories from Am I the Asshole, there may be some issues with recursion.
I strongly suspect a lot of the recent stories on that subreddit are AI generated. As are the responses. So one LLM is prompted to describe (generic AITA scenario, probably about a racial or sexual minority, a fat person, or a feminist behaving badly, because last I checked that sub was crawling with crypto-fascist propaganda), as if they are in the right. Then the second LLM is asked to determine whether or not the first LLM was actually in the right. And the second LLM will probably agree with the first just because they’re both running on similar algorithms.
And if the AITA data set was pulled from pre-LLM posts, the LLM they tested was probably trained on those AITA posts, which could likewise cause different results than stories it hadn’t been trained on.
Wow, I haven’t thought about AITA in years. I didn’t realize how much better my life was without it 😆
If they’re using stories from Am I the Asshole, there may be some issues with recursion.
I strongly suspect a lot of the recent stories on that subreddit are AI generated. As are the responses. So one LLM is prompted to describe (generic AITA scenario, probably about a racial or sexual minority, a fat person, or a feminist behaving badly, because last I checked that sub was crawling with crypto-fascist propaganda), as if they are in the right. Then the second LLM is asked to determine whether or not the first LLM was actually in the right. And the second LLM will probably agree with the first just because they’re both running on similar algorithms.
And if the AITA data set was pulled from pre-LLM posts, the LLM they tested was probably trained on those AITA posts, which could likewise cause different results than stories it hadn’t been trained on.
Wow, I haven’t thought about AITA in years. I didn’t realize how much better my life was without it 😆