The AI bros (who clearly don't understand AI) in the comments are hilarious. "ask stupid questions, get stupid answers". LLMs have no concept of how intelligent a question is, and they certainly were not designed to deliberately give stupid responses if they deem your question too stupid. 1/?
It’s so funny to think about the fact that there’s people out there deferring every life decision to this
What's most likely happening is easier to understand if you mentally model LLMs as glorified search engines. They are matching your search query (prompt) with some text they found on the internet. Internally it's a bit more complex than that, but much closer to Google than to a thinking machine. 2/? Essentially, it doesn't have an answer for whether you should walk or drive to a car wash, as it's not a commonly asked question (because it's stupid). Just like if you Google searched the question, you wouldn't find a good result because it's not a common question. 3/? But the question of whether you should walk or drive a given distance is common, so it basically just ignores the car wash part and answers the question of walking or driving in a more generic sense. The problem isn't that someone is going to ask the LLM this exact question, it's the handling. 4/? It clearly can't infer from a simple one sentence question that I might need my car to wash my car, because it isn't intelligent, doesn't possess logic, and can't reason. When you ask it anything complex, it's coming across 10s, 100s, or 1000s of points where it has to infer context. 5/? The reason it appears to do well at much more complex problems is because there are "search results" for that "query". It can explain general relativity, because it has data from millions of people explaining general relativity, not because it understands any of the underlying concepts. 6/? So, when you use LLMs as thinking machines, you will run into variants of a core problem: It isn't reasoning. It's just a fancy bot that copy & pastes then rewords data written by humans. Which means: 1. It's implicitly capped by its human produced-training data, which is largely garbage. 7/? 2. if you try to use it for a task that diverges too far from something it has in its training data, it will shit the bed (as is evident by the car wash question, which any intelligent being could easily answer). 8/? 3. Even when working on tasks for which the LLM has comprehensive data on, it's having to infer context constantly, and every time it mischaracterizes or drops some context, it goes further off the rails. 9/? Just as I would not think to explain to someone that they need to go to the car wash WITH THEIR CAR in order to wash it (because most people aren't that stupid). Most human produced text forgoes explaining a lot of obvious context, as it's not needed. Which is a problem if LLMs can't infer it. 10/? So the OP is just a really obvious example of how LLMs fundamentally do not understand things, thus miss even the most obvious context. Yet people are happy to pretend that when doing complex tasks, they just magically perfectly understand every bit of implied context encountered along the way. 11/? LLMs do not understand, will not understand, and cannot ever understand information. It's a glorified search engine combined with auto-compete, and you should treat it as such. No matter how much LLM companies try to tell you their machines can think or reason, they cannot and will not. 12/12