What Kinds of Conversations Can You Have With a Parrot?

This post is one of my advice & arguments pages about the harms and hazards of the AI Hype Movement.

TL;DR

You can have a real conversation with a parrot about food, in which both you and the parrot understand each other, in the sense that you’re both forming mental states as a result of interpreting vocalizations and those mental states are similar enough to support the functional goals of the conversation: you say “Polly want a cracker?” and Polly understands the word “cracker” in the sense that they know what it refers to, their understanding of “cracker” is similar enough to yours that you’re referring to the same thing, and when they say “yes” or “no” it can be because they actually do or don’t want the cracker you’re offering.

You can also have a fake conversation with a parrot about investment portfolios. With some luck and just the right training, it’s possible to have a conversation with a parrot where the utterances of the parrot are a perfect match to the responses of a real human investment banker, where the parrot effectively gives good financial advice to you about investment decisions, and where there’s even some utility to the whole conversation on your side of it. But the parrot does not have mental states about the stock market that have any bearing to the conversation you just had. The parrot is just giving memorized responses to specific cues that you luckily happened to provide, with no understanding of what it is saying. To get more specific, it has “no understanding” in the sense that the mental states that it uses to shape its responses have no correlation to the referents that you’re trying to invoke with your language. You mention stocks and bank accounts because you’re hoping to get investment advice from someone who understands what real-world things those words refer to and can give calculated advice based on that real-world understanding, but the parrot is merely keeping track of what part of it’s “investment banking advice” script it’s at with no correspondence between its mental states and actual stocks or bank accounts, and the logic behind its response is one of rote selection, not any reasoning based on actual finances.

Asking parrots for investment advice is a bad idea, because the mechanism behind their responses cannot on average lead to good recommendations. It could lead to plausible recommendations, potentially across a wide variety of things you might say, depending on how many responses the parrot has memorized and what words it’s listening for to pick one. The parrot’s advice in one particular example conversation might not be nonsense; it might even be great advice. But because the mechanism it’s using to decide what to say doesn’t bear any relation to actual finances, it’s impossible for the parrot to generate advice that’s good on average (unless it merely responds with useful platitudes that enhance your own decisions, in which case you could get those from a book instead of a parrot).

Note that there’s a very real risk that the parrot will run out of relevant responses across many investment conversations, and its mechanisms will be exposed as it gives a nonsensical or clearly wrong answer. However, the fact that its advice is unrelated to the actual real-world situation is a real problem even if through luck, it never slips up. The parrot’s mechanism produces both a risk-of-conversation-breakdown (because its mechanism of memorized responses isn’t actually suitable for the range of topics possible) and a risk-of-bad-advice (because its mechanism doesn’t depend in any useful way on the relevant facts) and these are two separate risks.

Now imagine a cybernetic parrot[1]. Rather than a few dozen memorized responses about investment topics it’s got millions or even billions. It’s method of deciding how to respond to a given question is also necessarily more complicated: it can instantly consult a vast database of prior conversations to come up with the most likely response to any given question based on what it’s seen before, and can even adapt to some degree the response to the particulars of your question. It can still have a real conversation with you about food, of course: one in which both you and it have convergent mental structures that share referents such that actual communication is taking place. And it can have much more convincing conversations with you about investment decisions. The risk-of-conversation-breakdown has been all but eliminated, assuming you stay somewhat on-topic (don’t ask it how many ’r’s are in the word “strawberry”).

[1] Indeed this is a famous metaphor I’m borrowing from Bender, Gebru, McMillan-Major, and Mitchell; see On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜.

Can you have a “real” conversation with the cybernetic parrot about your investment portfolio?

It’s tempting to quickly say either ‘yes’ or ‘no’ depending on your predilection:

“Yes” because even though it imitates past conversations, surely the wisdom of the past is at least somewhat applicable to the present, and with billions of memorized conversations, the most-relevant past conversation response might actually be pretty good advice a lot of the time. We could even devise some standardized test to measure how often the parrot gives “good” or “bad” advice and it might score pretty highly!

“No” because the mechanism has changed, but not enough. It’s still fundamentally a system based on memorization, and so the mechanism of the response still does not really take into account the real-world situation you’re trying to explain to it. The risk-of-bad-advice might not correlate at all with the risk-of-conversation-breakdown, and even if we say the parrot gives “good advice 95% of the time”, if the 5% bad advice is bad enough, you can easily lose more than you gain by following its advice in the long run.

I believe that the answer is more complicated (and not just because complicated answers sound more sophisticated). If the parrot really has billions of memorized conversations to sift through, the mechanism for selecting one is quite likely to depend on the details of what you’re saying, including relevant details about the portfolio you’re talking about. With the original parrot’s limited memory capacity, it might be selecting responses randomly or based on some superficial aspects of what you say, such as your tone of voice. The cybernetic parrot absolutely could still be doing that, but it could also be the case that its choice among billions of potential responses does depend on the specific meanings of the investment-related words you use, in which case we’ve started to build a mechanism that actually does relate to the real-world situation at hand, and some form of real communication is happening in the sense that your mental states about real-world stocks and accounts which shaped what you said caused similar mental states that still have some relation to those same real-world things when they’re constructed by the cybernetic parrot as it hears what you say. However, there’s a potential fuzziness there: the parrot’s selection mechanism might trigger based on both superficial and meaningful aspects of what you’re saying, and the sense in which its mental states[2] (internal to that selection mechanism or not) actually correspond to the things you’re trying to get them to correspond to might be loose.

[2] Some will probably object to the term “mental state” to refer to the cybernetic parrot’s internal representations, or at least object when such a term is used with a non-biological example like a large language model. I think it’s a fair objection, but for the purposes of this argument, it doesn’t matter whether the state is “mental” or merely “internal” and I use “mental” because I’m talking about correspondences with human mental states and it’s easier to use the same language on both sides (plus real parrots and even hypothetical cybernetic ones, do have real mental states).

Can you have a “real” conversation with the cybernetic parrot about your investment portfolio? As I’ve defined “real conversation” I think the answer is: “maybe”. I think it’s best to view the parrot as an alien intelligence whose understanding of concepts works radically differently from ours and whose reasoning processes might also be totally alien. Because of these differences, there will always be some measure of slippage between the concepts we try to communicate to it and the internal structures it comes up with based on them, as well as slippage between the conclusions we’d draw given the same knowledge base and the utterances it chooses (whether it even “draws conclusions” in any meaningful sense is up for debate). We also know that memorization and imitation play a much larger role in the responses we get than they would for a human interlocutor. In cases where the cybernetic parrot effectively “doesn’t understand” what we’re saying, it falls back on simpler selection processes based on less-relevant factors in our speech, still trying to imitate a knowledgeable response. Humans also do this when we’re bullshitting, but we much more often communicate our confusion, which is a safer and healthier conversation move.

Because this answer means that some parts of any given conversation could be the result of “real communication” while others are almost certainly the result of “mere imitation”, I think it’s quite dangerous to have a conversation with the cybernetic parrot, and if we ask “should you attempt to have a conversation with the cybernetic parrot?” the answer is: “No, unless you have serious extenuating circumstances and no other options.” Humans just fundamentally aren’t good at judging the real from the fake parts of such a conversation (the fake parts are engineered to be as real-sounding as possible after all) and as mentioned above, even if the parrot produces “good” output a lot of the time, its systematic failures can easily have awful consequences that undo all of the utility of its good responses. Especially if talking to the cybernetic parrot involves suffering for it, thousands of strangers, and your own children, it’s definitely best to just have a conversation with a real parrot instead (where you can easily understand which parts are fake) and get your investment advice from a trustworthy humans (sometimes hard to find, but if you can’t find one going without advice is probably better than taking the advice of the cybernetic parrot).

Transparently (I hope) this is really a metaphor for large language models. They’re pretty close to the cybernetic parrot example, even if they don’t actually have “mental” states. Per my other pieces (and links) here, their existence and operation causes a ton of harm, which alone is a good reason not to deal with them unless you absolutely have to (and in that case better if you can mitigate some of the harms, even though that’s not easy or really possible in all cases). But the point of this piece is to demonstrate that even if they didn’t have all those harms attached, they’d be dangerous and unreliable tools at best. They give a convincing illusion of “understanding” and in many cases it’s easily to construct a conversation that shows that they “must understand” a certain concept. But even in those cases, they don’t necessarily “understand” it the same way a human does (their internal states and what passes for their “reasoning” are completely alien), and in cases where things are slipping or unclear, they almost always “bluff competence” rather than “asking for clarification” or “admitting confusion”.[3] This makes for a dangerously untrustworthy interlocutor, and predictably leads to both small failures that create constant friction and occasional spectacular failures that result in net harm across many interactions.

[3] This is anthropomorphizing language in an attempt to be succinct; really what they’re doing is not a “bluff” and they can’t really “ask for clarification” or “admit confusion” in the sense that a human can. They “bluff” because they’re trained to predict the most-likely response on data where most confused and unconfident responses have been structurally filtered out, and at best they could imitate a confused response or an ask for clarification, but they can’t actually experience confusion nor can a clarifying response actually make things clear to them: they don’t have the proper internal states for that to happen in the same way it would or a human.

To summarize my point here, a “real conversation” depends on being able to share concepts across a language interface, but even in the most rosy interpretation of how large language models work, their internal states are so alien to ours that concept-sharing frequently breaks down. Where a (competent, well-meaning) human would respond to this breakdown by asking for clarification, a large language model instead bullshits (not intentionally but as a structural product of its training process). Given the highly deceptive level of bullshit an LLM can produce, this makes them fundamentally untrustworthy as a conversation partner, as you’ll be oscillating wildly between “real conversation” (if you’re lucky) and “fake conversation” with little way to tell what’s what.

If you’re interested in further reading about the concepts here:

1. Gricean Maxims are a great place to start thinking more formally about how conversations work (LLMs subtly violate these all the time which is an additional vector of misinformation in exchanges with them).
2. Emily M. Bender’s thought experiment about the Thai National Library is an interesting one and makes a similar point about what an LLM can “understand”, although I don’t fully agree with her conclusions (and wish that English-speaking thought experiments positioning Asian languages as prototypically alien would just stop). My argument here allows for the possibility that processing rules derived purely from linguistic form could effectively have real referents, but shows that even if they could, the degree to which those references are functionally similar to or distinct from humans’ references matters a lot, and the behavior of the system when references slip or are loose, and the degree to which it also sometimes operates without referents matters too.
3. Dennett’s concept of the “intentional stance” is quite relevant here. Another summary of my argument might be: taking the intentional stance towards LLMs can often succeed spuriously, and thinking about them from the design stance can help us understand where the intentional stance might work or not. Because of their ability to seem intentional even when they functionally aren’t, they are dangerous and untrustworthy.
4. Reddy’s Conduit Metaphor is also quite relevant here. In my example, the original parrot is simply unable to build any meaning about investment portfolios no matter what words you use, due to lack of any remotely relevant life experience, but it can still mimic relevant conversational responses if it’s lucky. The cybernetic parrot might sometimes rely on the same pure mimicry (just turbocharged by its expanded memory) or might in fact build some kind of internal state from what you say, but of a form that’s alien to your original meaning, sometimes in important ways, because its process for building internal states is totally alien to human processes.