297

M.J. CrockettFebruary 26, 2026 at 8:18 AM EST

I'm a cognitive scientist with an interest in epistemic vigilance, and this essay that's been going around gave me pause. I don't think it's straightforward to apply the concept of epistemic vigilance to interactions with LLMs, as this essay does. 🧵/ sbgeoaiphd.github.io

Epistemic vigilance is a concept developed by Sperber et al. describing how we guard against the possibility that other people might misinform us. Two key points in this literature: 1. We need to scrutinize the content of messages 2. We need to scrutinize the source of messages 2/

The essay argues that, as a default, LLMs erode epistemic vigilance, which it defines exclusively with regards to content. 3/

The essay claims users can eliminate risk from LLMs as long as they exercise (content-based) epistemic vigilance + active participation. But this argument misses half the story of epistemic vigilance - arguably the more important half. Let's take a look at source-based epistemic vigilance. 4/

Whenever I encounter a new piece of writing the first thing I look at is the source. In this case, Sam Barrett works in AI (so stands to benefit financially from pro-AI arguments). His background seems to be in geosciences (so not necessarily well informed about epistemic vigilance) 5/

Perusing some of the other posts on Sam's blog, I see that he likes to use Claude when writing essays. So probably he used Claude to write this essay as well, even though this is not disclosed explicitly (a bit odd for an essay on epistemic vigilance) sbgeoaiphd.github.io 6/

But even if it were disclosed that this essay was Claude-generated, we'd still have a problem with respect to epistemic vigilance. This was a concept developed to explain how we guard against misinformation *by other people*. LLMs (despite what Anthropic would have us believe) are *not people*. 7/ Specifically, the idea of source credibility just doesn't apply straightforwardly to LLM interactions. When I get info from another person I can ask: how does the content relate to their expertise? Their lived experience? Their interests and biases? 8/ For example if someone wants to tell me about space travel, I should treat that information differently if they are a friend I trust who has a PhD in astrophysics, versus some random person on the internet, versus a psychotic billionaire trying to sell a space travel company 9/ We can't exercise this kind of source-based epistemic vigilance with LLMs. For proprietary products like Claude or ChatGPT, the companies that own them haven't disclosed what's in their training data. The sources of the content produced by LLMs are hidden. 10/ Moreover, even if the training data were fully disclosed, it's not straightforward to know exactly where LLMs outputs come from. A colleague can say "I got this idea from reading Sperber et al. 2010". We cannot get this kind of info from Claude, that's not how LLMs work. 11/ In human communication we can check the credibility of our sources, and our sources' sources, and so on, down the chain (Wikipedia does this too!). We cannot know the source of the LLM's outputs in the same way we can query our human communication partners. 12/ Epistemic vigilance towards content alone is clearly not enough to guard against misinformation, as this essay demonstrates. The human author here is unfamiliar with cognitive science so misses that his definition of epistemic vigilance is incomplete, in a way that misleads readers 13/ Because the core argument of the essay is that (content based) epistemic vigilance is enough to ensure the accuracy of co-writing with LLMs. But the essay is inaccurate because it fails to consider the importance of source credibility. 14/ If the author had appropriately exercised epistemic vigilance here, he would have recognized that (a) Claude is not a credible source, that (b) he lacks the expertise to apply content-based epistemic vigilance on this topic, and so 15/ (c) maybe should have asked a cognitive scientist or read some cognitive science literature to see whether the concept of epistemic vigilance can do the work this author wants it to do. (Would welcome more conversation about how epistemic vigilance *can* apply to LLMs!) 16/ More broadly, the essay is a great example of the "illusion of understanding" that @lmesseri.bsky.social and I warned in 2024 would be a risk of using LLMs, especially as they become more capable 17/ nature.com

(non-paywalled version here) 18/

Neither Claude nor Sam understands epistemic vigilance, because if either one did, the argument in the essay would have been very different. Instead, Claude simulates understanding, and Sam experiences an illusion of understanding because he has not exercised (source based) epistemic vigilance. 19/ The newest models can be pretty damn convincing unless you are a huge nerd on the topic you're reading about. I almost fell for a feminist defense of Claude the other day (the essay cited Silvia Federici!!) Be careful out there, friends. end/

1 / 3