LLM is ...

Here is a textbook about large language models: their history, anatomy, and their application to solve practical problems. The materials will help you quickly understand the basic concepts, prepare for an interview, and assist in structuring information. For more in-depth study, the articles contain links to original scientific works. Additionally, at the end of each article, you will find questions and tests for self-assessment.

The field is moving so rapidly that by the time educational materials are developed, they often become outdated. In this regard, most of the text was generated and then edited by a human. Furthermore, the materials are maintained by the community, so if you find any inaccuracies, highlight the text, right-click, and select "Report an inaccuracy." By doing so, you will help keep the materials up-to-date and accurate.

Part I — Foundations of Language Modeling

1.1What Is a Language Model, Really 1.2Statistical and Count-Based Language Models 1.3Neural Language Models Before RNNs 1.4Recurrent Networks as Language Models 1.5Attention Before Transformers

Part II — The Transformer Architecture

2.1Attention Is All You Need 2.2Positional Information 2.3Architectural Variants 2.4Normalization and Stabilization 2.5The Feed-Forward Layer

Part III — Scaling the Architecture

3.1The Cost of Attention 3.2Long Context 3.3Mixture of Experts 3.4Inference-Time Efficiency 3.5Attention Variants for Inference 3.6Beyond Attention

Part IV — Training, Scaling, and Alignment

4.1Tokenization 4.2Pretraining Objectives and Data 4.3Scaling Laws 4.4Distributed Training 4.5From Base Model to Assistant 4.6Alignment and Safety Basics 4.7Parameter-Efficient Fine-Tuning: LoRA and QLoRA

Part V — Capabilities and Evaluation

5.1Reasoning and Test-Time Compute 5.2Retrieval-Augmented Generation 5.3Multimodal LLMs 5.4Agents and Tool Use 5.5Evaluation 5.6Structured Output: From Grammars to Constrained Decoding

Part VI — System Design and the Road Ahead

6.1System Design for LLM Products 6.2Where the Field Is Heading