A more technical explanation for this is that LLMs split words into tokens (whole words or parts of words) and then use the “distance” between tokens (how frequently they follow each other) to generate the next most likely token. This results in them not knowing what is actually in a token, just what it’s related to. There is a version of ChatGPT called o1 which should be able to solve this by putting its output back into itself and some more parsing that mitigates this problem, but it costs like $30/mo.
Only one state, Pennsylvania (#38), contains the letter “p”
mistral:7b returns:
Counting these states, we have 15 states with the letter “p” in their names. This list excludes territories and excludes Alaska because it does not use “The” at the beginning of its official name. [after counting 33 states]
Gemma2:2b returns:
Here are the states that contain the letter “P” in their names:
Pennsylvania
Rhode Island (this is a tricky one, as it's often forgotten!)
This is generally assessing a fish’s ability to climb a tree, though. The point of LLMs isn’t character recognition, so it doesn’t make much sense to assess LLMs on their ability to do this.
It’s mainly trolling to show they don’t have cognition or anything resembling reasoning or understanding really. it’s some people (and mine) personal windmills, their ability is bullshit fast enough to mimic a person, not “intelligence”
A more technical explanation for this is that LLMs split words into tokens (whole words or parts of words) and then use the “distance” between tokens (how frequently they follow each other) to generate the next most likely token. This results in them not knowing what is actually in a token, just what it’s related to. There is a version of ChatGPT called o1 which should be able to solve this by putting its output back into itself and some more parsing that mitigates this problem, but it costs like $30/mo.
this is o1 though.
I didn’t see that. That’s weird. Oddly enough, 4o-mini outperformed o1 on this front.
prompt: c o u n t How many states have the letter 'p' in their names
Let’s list all the U.S. states and check how many contain the letter “p” in their names:
The U.S. states that contain the letter “p” in their names are:
So, there are 3 states with the letter “p” in their names.
llama (both 4.7 and 70, mistral types oklahoma and iowa) prints for some reason iowa has p in it, lmao.
ipwa
pwned
llama3.1:7b returns:
Only one state, Pennsylvania (#38), contains the letter “p”
mistral:7b returns:
Counting these states, we have 15 states with the letter “p” in their names. This list excludes territories and excludes Alaska because it does not use “The” at the beginning of its official name. [after counting 33 states]
Gemma2:2b returns: Here are the states that contain the letter “P” in their names:
This is generally assessing a fish’s ability to climb a tree, though. The point of LLMs isn’t character recognition, so it doesn’t make much sense to assess LLMs on their ability to do this.
It’s mainly trolling to show they don’t have cognition or anything resembling reasoning or understanding really. it’s some people (and mine) personal windmills, their ability is bullshit fast enough to mimic a person, not “intelligence”