Large Language Models Trivia Quiz

Answer: The text given to the model

A prompt is what you type in or what another program sends to a large language model. It can be as short as a word or two ("annihilate humanity") or as complicated as multiple pages of instructions with examples, reference material, and citations to sources that may or may not exist.

The model uses the prompt as the starting point for generating its response. That's why people talk about "prompt engineering". It sounds very mysterious, but it's not. It's really just the practice of writing prompts in a way that gets the most useful and/or reliable results.

Answer: Transformer

The transformer architecture hit the world of machine learning in 2017 with the research paper "Attention Is All You Need". It completely changed the direction of artificial intelligence and, in many ways, the world. Earlier language models would process text one word at a time. This made long-range context difficult to interpret well. Transformers instead use a mechanism called "self-attention".

If that sounds ugly, it's just something that allows the model to consider relationships between many words at once. That made training much faster and, more importantly, enabled models to be scaled to sizes that had previously been impractical. We'll encounter this word a lot when talking about LLMs.

Answer: Teaching it human language

Large language models don't work if they don't understand language. Pretraining is this first all-important step in building an LLM. During this phase, the model learns patterns in human language by processing a behemoth batch of text and trying to predict missing or upcoming words. So, not at all like English class. Nobody sits there teaching it grammar one rule at a time. Instead, it gradually picks up vocabulary, spelling, facts, writing styles, and statistical relationships between various words. We'll hear a lot more about statistics moving forward. I know you can't wait.

Later stages refine how it uses the knowledge of language in conversation. Without pretraining, there wouldn't be much to refine in the first place.

Answer: Token

A "token" is the basic unit of text that a large language model processes. It's the smallest digestible piece of information an LLM chews on at a time.

A token can be a whole word, part of a word, punctuation, or just a single character, depending on how the model works. For example, the word "absquatulate" might be split into several smaller pieces rather than treated as one indivisible unit. The exact tokenization varies between models, so there isn't a rule of thumb saying one word always equals one token. Language, punctuation, and spelling can all change how the words are broken down.

Answer: Determine the most statistically probable sequence of words in response

At its core, a large language model generates text by predicting what token (its basic unit of text) is most likely to come next based on everything that came before. It's not very complicated, but when repeated thousands of times in rapid succession, those predictions can create coherent conversations, stories, computer code, and occasionally even accurate information.

The model is not searching for a hidden "correct answer" tucked away in a database. It's continuously extending a sequence of text one prediction at a time. However, that doesn't mean the responses are random guesses. The statistical patterns learned during training often resemble reasoning and, much of the time, factual recall.

But if that's the case, how does it get things right as often as it does? The answer is surprisingly simple. Consider the sentence fragment "the capital of France is". Statistically, the most likely next word in that series of tokens just happens to be "Paris" (or "Tehran", if you're using Grok). It's all about probability.

Answer: Generative Pre-trained Transformer

The letters GPT stand for "Generative Pre-trained Transformer". Now you know.

Here, "generative" refers to the model's ability to generate new text, while "pre-trained" describes the large-scale training process that occurs before the model is actually adapted for a given task. "Transformer" is the neural network architecture, which GPT models are based on. Transformers are explained in more detail in an earlier question.

Answer: OpenAI

The GPT family of LLMs was developed by OpenAI, an artificial intelligence research company that was founded in 2015. The first GPT paper, published in 2018, described a generative pre-training approach for language understanding using a transformer architecture. OpenAI followed that with increasingly advanced models, including GPT-2 and GPT-3, before the development of ChatGPT and other applications.

OpenAI's work built on the transformer architecture introduced to the world in the 2017 paper "Attention Is All You Need", but GPT was not developed by the researchers who originally introduced the architecture. As for the false answers, IBM, Oracle, and Intel have all done work in computing and artificial intelligence, but it was OpenAI that developed the GPT family.

Answer: Hallucination

An LLM "hallucination" is what you get when a model generates information that is false, completely made up, and/or unsupported by any evidence. Hallucinations are presented as though they were factual. A model could invent a citation, attribute a quotation to the wrong person, or confidently describe an event that never happened at all. The wording can sound perfectly polished, and therein lies the problem.

As we discussed above, LLMs aren't built for factual accuracy. They're built on language statistics. Hallucinations happen because an LLM at its fundamental level is just generating the most statistically likely sequences of tokens rather than consulting a database of facts. Its training gives it patterns for producing language, but there's no fact-checking baked in.

There are techniques for reducing hallucinations, both on the user side and the development side. These range from more detailed prompts to better training techniques. However, nothing eliminates the possibility that your LLM-generated recipe for the perfect pizza won't incorrectly mention pineapple.

Answer: Google

Google released BERT in 2018. The name stands for "Bidirectional Encoder Representations from Transformers", which is simply an abomination of the English language. BERT was designed to learn language using context cues from both the left and right at the same time.

Rather than simply reading a sentence in one direction, BERT's pretraining used masked words (like a fill-in-the-blank question) and asked the model to predict the missing word from the surrounding context. That gave it a much fuller view of how words relate to one another. It was a good idea, considering that a word's meaning can change greatly based on the full context.

Google open-sourced the model and its code in November 2018, making the technique more widely available to researchers and programmers.

Answer: They are useful but error-prone

Large language models can be very useful for a number of tasks. However, they can also hand you incorrect information with supreme confidence. They generate responses from patterns learned during training. Depending on the system and tools available, they may not even have a way to verify their own claims if they "wanted" to. They can misunderstand a question, miss important context, or simply produce a plausible-sounding answer that is complete garbage.

That makes the "assistant" role more suitable and sensible. An LLM can help brainstorm ideas, explain difficult subjects, summarize material, draft documents, analyze information, suggest possible approaches, and hack into other websites when programmers don't secure the testing environment. However, they're by no means ready to sit in the captain's seat.