"Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data" is a position paper by Emily M. Bender (University of Washington) and Alexander Koller (Saarland University), published in the Proceedings of the 58th Annual Meeting of the ACL (2020), pp. 5185–5198, where it won Best Theme Paper. It is the source of the octopus thought experiment and the intellectual predecessor to "On the Dangers of Stochastic Parrots" (2021).
Its thesis is stated as a claim about what is possible in principle rather than about current systems: "a system trained only on form has a priori no way to learn meaning." The framing of the title is the paper's organizing metaphor — that genuine progress depends on "climbing the right hill, not just the hill on whose slope we currently sit."
Definitions
The argument rests on two definitions the paper is careful to fix before using.
Form is "any observable realization of language: marks on a page, pixels or bytes in a digital representation of text, or movements of the articulators."
Meaning is the relation between form and something external to language. Formally, meaning is the relation M ⊆ E × I, containing pairs (e, i) of natural language expressions and the communicative intents they can be used to evoke. Given that definition, the paper uses understand for "the process of retrieving i given e."
The grounding argument follows from the definition: "Communicative intents are about something that is outside of language." When someone says Open the window! or When was Malala Yousafzai born?, the intent is grounded in a world the speaker and listener inhabit together — though the paper allows that intents can also concern abstract worlds such as bank accounts, file systems, or a hypothetical world in the speaker's mind.
A third relation mediates: C ⊆ E × S, pairing expressions with their standing meaning in a shared linguistic system. A speaker with intent i chooses an expression e whose standing meaning s is fit to express i in context; the listener reconstructs s and uses their own knowledge of the situation and hypotheses about the speaker to recover i.
The paper also flags the human disposition that makes the failure hard to detect: "We humans are also very willing… to attribute communicative intent to a linguistic signal of a language we speak, even if the originator of the signal is not an entity that could have communicative intent."
The octopus test
A and B, both fluent English speakers, are stranded on two uninhabited islands and discover telegraphs left by previous visitors, connected by an underwater cable. O — "a hyper-intelligent deep-sea octopus who is unable to visit or observe the two islands" — taps into the cable and listens.
O knows no English initially but is "very good at detecting statistical patterns." Over time it learns to predict with great accuracy how B will respond to each of A's utterances, and observes that certain words occur in similar contexts, perhaps generalizing across lexical patterns by hypothesizing interchangeability. But "O has never observed these objects, and thus would not be able to pick out the referent of a word when presented with a set of (physical) alternatives."
Then: "At some point, O starts feeling lonely. He cuts the underwater cable and inserts himself into the conversation, by pretending to be B and replying to A's messages." The paper labels this "a weak form of the Turing test" — weak "because A has no reason to suspect she is talking to a non-human" — and states the question of interest precisely: whether O fails it because he has not learned the meaning relation.
Two probes follow.
The coconut catapult. A describes building one and asks about B's experiences and suggestions. O cannot physically reproduce the experiment and does not know what rope and coconut refer to, so it resorts to prior observations: recognizing utterances about mangos and nails as "similarly worded" because those words appeared in contexts similar to coconut and rope, it replies "Cool idea, great job!" because B said that a lot in comparable exchanges.
The paper's analysis of why this works is the passage most often lost in citation: "It is absolutely conceivable that A accepts this reply as meaningful — but only because A does all the work in attributing meaning to O's response. It is not because O understood the meaning of A's instructions or even his own reply." Elsewhere: "It is not that O's utterances make sense, but rather, that A can make sense of them."
The bear. A is "suddenly pursued by an angry bear," grabs sticks, and frantically asks B how to build a weapon. "Of course, O has no idea what A 'means'. Solving a task like this requires the ability to map accurately between words and real-world entities (as well as reasoning and creative thinking)." The paper delivers the conclusion with a joke that is also the point: "It is at this point that O would fail the Turing test, if A hadn't been eaten by the bear before noticing the deception."
The generalization: "Having only form available as training data, O did not learn meaning. The language exchanged by A and B is a projection of their communicative intents through the meaning relation into linguistic forms. Without access to a means of hypothesizing and testing the underlying communicative intents, reconstructing them from the forms alone is hopeless."
Two more constrained experiments
Because the octopus story bundles learning the communicative system with the reasoning needed to produce helpful real-world answers, the paper offers two narrower cases isolating the meaning relation alone.
Java. Train a language model on all well-formed Java published on GitHub — code only, "not paired with bytecode, nor a compiler, nor sample inputs and outputs for any specific program." Any architecture, any training duration. Then ask the model to execute a sample program and expect correct output. Programming languages are chosen deliberately: designed to be unambiguous and relatively insensitive to execution context, so the standing-versus-speaker-meaning distinction matters less than for natural language, and the meaning relation J ⊆ E × I is exactly the set of pairs of programs and the functions they compute.
English plus photos. Train on English text with no independent indications of speaker intent, plus a large collection of unlabeled photos with no connection between text and images — the text task purely predicting form, the image task anything involving only images. At test time the model receives an utterance and a photograph together: How many dogs in the picture are jumping? or Kim saw this picture and said "What a cute dog!" What is cute?
In a footnote the authors report prompting the GPT-2 demo with "Help! I'm being chased by a bear" to show what a large LM replies in the octopus's situation.
The scope limit the paper states about itself
Section 8 contains a concession frequently dropped when the paper is cited. Asked about systems trained on tasks other than language modeling — semantic parsing, reading comprehension — that use BERT-style embeddings as a component, the authors answer plainly: "Our arguments do not apply to such scenarios." Their reasoning is that "reading comprehension datasets include information which goes beyond just form, in that they specify semantic relations between pieces of text, and thus a sufficiently sophisticated neural model might learn some aspects of meaning when trained on such datasets." They add that whatever a pretrained LM captures "might help the downstream task in learning meaning, without being meaning itself."
The claim is therefore narrower than the common paraphrase: it concerns training on form alone, not the use of language models within systems that also see meaning-bearing supervision.
Climbing the right hill
The paper distinguishes bottom-up theory-building — a community driven by specific research challenges, where a result counts as success if it solves one at least partially — from a top-down perspective asking whether the accumulating successes are on the path to the actual goal. Its question: "Are we climbing the right hill?"
Four hillclimbing diagnostics are proposed as best practice. First and "above all, cultivate humility towards language and ask top-down questions," noting that "neural methods are not the first bottom-up success in NLP; they will probably not be the last." Second, be aware of task limitations: artificial tasks such as bAbI can launch a research area, but "there is no reason to assume that the distribution of language in the test data remotely resembles the distribution of real natural language." Crowdsourced datasets including SQuAD and SNLI draw the same caution — they "do not represent questions that any particular person really wanted to ask about a text, but the somewhat unnatural communicative situation of crowdsourcing work." The paper offers a concrete test: "If a system does better on such a task than the inter-annotator agreement, the task probably has statistical artifacts that do not represent meaning."
Conclusion
"In this paper, we have argued that in contrast to some current hype, meaning cannot be learned from form alone. This means that even large language models such as BERT do not learn 'meaning'; they learn some reflection of meaning into the linguistic form which is very useful in applications."
The closing framing is methodological rather than dismissive — "a call for precise language use when talking about the success of current models and for humility in dealing with natural language," in service of "a top-down perspective on our field which we think will help us select the right hill to climb towards human-analogous NLU." The authors also note the paper "originated in a Twitter mega-thread," and thank Sam Bowman among many others "without implying any agreement with the contents as presented."
Relationships
- depends-on: Stochastic Parrots (Bender, Gebru, McMillan-Major, Mitchell, 2021) and the Octopus Test — supplies the philosophical half of that concept; the 2021 FAccT paper supplies the political and empirical half
- related: Emily M. Bender — co-author
- related: AI Consciousness, AGI Timelines, AI Benchmarks and Evaluation — the inter-annotator-agreement diagnostic bears on benchmark validity