For AI Beginners:
How Does AI Generate Text?
Explaining the magic with simple examples!
Article Summary 📝
“How can an AI like ChatGPT write such natural, human-like sentences?”
This article answers that question by exploring the history of how text-generating AIs have evolved and explaining their basic mechanics in a way that’s easy for beginners to understand. Let’s embark on this AI journey together! 🚀
1. The First Steps of AI Text Generation 🐾
As smart as today’s AI is, its journey began with a very simple idea: “predict the next word.”
1-1. Predicting with Statistics: “Today’s weather is” … “sunny”? (N-gram Model)
The earliest AIs would read a vast amount of text and simply calculate probabilities, like “after this word, that word is most likely to appear.” For example, given the phrase “Today’s weather is,” it would choose “sunny” if “sunny” was the most frequent word to follow that phrase in its data.
While very simple, this method struggled to understand the context of longer sentences.
1-2. Remembering the Recent Past to Predict the Next Word (RNN)
Next came a technique called RNN. This was an attempt to predict the next word by not just looking at the previous word, but by “remembering” the flow of words that came before it.
However, RNNs had a critical weakness: they were “forgetful.” As sentences got longer, they would forget what was written at the beginning, leading to incoherent or contradictory text.
2. A Revolution in AI: The “Transformer” Arrives ✨
The technology that solved the “forgetfulness” of RNNs and revolutionized the world of AI was the Transformer, introduced in 2017. It was announced in a paper with the catchy title, “Attention Is All You Need.”
The Key is “Attention”!
The Transformer pays “attention” to which words in a sentence are related to each other. This allows it to understand context without losing track of important information, even in long passages.
Furthermore, Transformers have the major advantage of being able to process information in parallel, allowing them to learn much faster than ever before. This revolutionary technology is the foundation of today’s most advanced AIs.
3. How to Build a Giant Brain 🧠
3-1. Just Keep Predicting the “Next Word”: Pre-training
With the powerful Transformer engine, AIs began a new training regimen called “Pre-training.” This is a process where the AI is fed a massive amount of text from the internet (books, news, blogs, etc.) to learn the “rules of language” and “common knowledge” on its own.
AI’s Self-Study Method: A “Guess the Next Word” Game!
What the AI does during pre-training is actually quite simple. It’s essentially a continuous quiz where it’s shown part of a sentence and asked, “What’s the next word?” By repeating this billions or trillions of times, the AI masters grammar, word meanings, and contextual understanding.
[Question for AI] “To be or not to be, that is the ___”
[AI’s Prediction] “question”
This method is called Causal Language Modeling (CLM) and is the fundamental training method for many AIs, including the GPT series.
4. “Tuning” the AI to be Smarter and Safer 🤖
An AI that has finished pre-training has a vast amount of knowledge, but that alone doesn’t guarantee it can answer our questions correctly or follow instructions. That’s where additional “tuning” becomes necessary.
🎓 Fine-tuning
This is training to make the AI an “expert” in a specific field. For example, by having it study a large corpus of medical literature, it can become an “AI specializing in medicine,” or by learning legal precedents, it can specialize in law.
🤝 Alignment
This involves adjusting the AI’s responses to better align with human values and preferences. A key technique is RLHF, where humans rate the AI’s answers, providing feedback like “this answer is better” to guide it toward generating safer and more helpful responses.
5. Techniques to Make AI “Think” 💡
Modern AIs are becoming adept not just at generating text, but also at “reasoning” to solve complex problems. There are fascinating techniques to draw out this ability.
Making AI Verbalize Its Thought Process: “Chain-of-Thought“
When humans solve difficult problems, we don’t just jump to the answer; we write down our intermediate steps. The “Chain-of-Thought” technique makes AI do the same thing.
By simply adding “Think step-by-step” to the end of a question, the AI is prompted to lay out its reasoning process. Mysteriously, doing this has been shown to dramatically increase the accuracy of the final answer.
[User’s Question]
If you have 3 red king crabs and 2 snow crabs, how many legs are there in total?
Please think step-by-step.
[AI’s Answer]
Of course. I will think step-by-step.
1. First, I need to determine the number of legs for each type of crab.
– Red king crabs are technically not “true crabs” and belong to the hermit crab family. They have 8 walking legs.
– Snow crabs are true crabs and have 10 legs.
2. Next, I will calculate the total number of legs for each group.
– Red king crabs: 8 legs/crab × 3 crabs = 24 legs
– Snow crabs: 10 legs/crab × 2 crabs = 20 legs
3. Finally, I will sum the totals.
– 24 legs + 20 legs = 44 legs
Therefore, there are a total of 44 legs.
6. Meet the All-Stars: Today’s Leading AI Models 🌟
Today, companies around the world are developing their own unique AIs, each with its own strengths and characteristics.
GPT Series (OpenAI)
The model that ignited the AI boom. An all-rounder that boasts high performance across a wide range of tasks, from natural language generation to complex instruction following.
Gemini (Google)
Designed from the ground up as a “multimodal AI” that can handle images and video. It excels at understanding and integrating diverse types of information beyond just text.
Claude (Anthropic)
Built with a strong emphasis on AI safety and ethics. Its standout feature is the ability to read and comprehend extremely long documents (tens to hundreds of thousands of words) at once.
Llama Series (Meta)
Characterized by its open-source model, allowing anyone to participate in research and improvement. It aims to democratize AI technology.
Conclusion: So, How Does AI Generate Text?
In this article, we’ve journeyed through the evolution of text-generating AI. Let’s recap the key points!
- AI text generation began with the simple mechanism of “predicting the next word.”
- The invention of the “Transformer” allowed AI to understand the context of an entire passage, leading to a dramatic leap in performance.
- Modern AI undergoes “pre-training” (self-study) on vast amounts of text to learn the rules of language and common-sense knowledge.
- It is then “tuned” through “fine-tuning” and “alignment” to become more specialized and helpful to humans.
In short, modern AI uses its enormous knowledge base to read context, probabilistically select the most fitting words, and string them together into natural sentences to answer our questions.
We hope this has made the inner workings of AI feel a little more accessible! 🎉
