How AI reads words

Think of a box of Lego. You can't build much with one giant brick. You need lots of small bricks that fit together.

AI treats words the same way. It can't read a sentence like we do. First it cuts the sentence into small pieces called tokens. A token can be a whole word, part of a word, or a single letter. Each token has an ID number, and the AI only sees those numbers.

Then it does the one job it is built for: it guesses which token comes next. Everything an AI writes, it writes one guessed token at a time.

What is the number under each token? It is the token's ID. GPT-2 has a numbered list of 50,257 tokens, a bit like seats in a huge cinema. "Once" sits in seat 7454. Common pieces were added to the list early, so they usually have small numbers. The AI never sees the letters O-n-c-e, only the number 7454.

This page uses GPT-2, a small AI that OpenAI released in 2019. It is old and simple, which makes the pieces easy to see.

6 examples: the pieces, and the guesses

· marks a space. It belongs to the token after it. Some letters, like Hindi letters or emoji, are cut into several tokens. A piece of a letter can't be drawn on its own, so it shows as �. The bracket underneath shows the whole letter.

1. You type: Once upon a time

   Split into 4 tokens:

   1. Once7454
   2. ·upon2402
   3. ·a257
   4. ·time640

   The number under each token is its ID. What do the numbers mean?

   GPT-2's top 5 guesses for the next token:

   1. ,42.7%42.7%
   2. the6.5%6.5%
   3. I4.1%4.1%
   4. he3.7%3.7%
   5. there2.9%2.9%

   ✅ A confident guess: "," at 42.7%. GPT-2 has read many stories that start "Once upon a time, there was…".

2. You type: The cat sat on the

   Split into 5 tokens:

   1. The464
   2. ·cat3797
   3. ·sat3332
   4. ·on319
   5. ·the262

   The number under each token is its ID. What do the numbers mean?

   GPT-2's top 5 guesses for the next token:

   1. floor7.6%7.6%
   2. bed6.5%6.5%
   3. couch5.4%5.4%
   4. ground5.2%5.2%
   5. edge4.8%4.8%

   Sensible guesses, like floor, bed and couch. But GPT-2 is not sure, because many words could come next.

3. You type: The capital of France is

   Split into 5 tokens:

   1. The464
   2. ·capital3139
   3. ·of286
   4. ·France4881
   5. ·is318

   The number under each token is its ID. What do the numbers mean?

   GPT-2's top 5 guesses for the next token:

   1. the8.5%8.5%
   2. now4.8%4.8%
   3. a4.6%4.6%
   4. France3.2%3.2%
   5. Paris3.2%3.2%

   ❌ Not a fact-finder. "Paris" is only 5th, at 3.2%, and "the" is 1st. GPT-2 guesses which words usually come next. It does not look up facts. Newer, bigger AIs are much better at this, but they work the same way.

4. You type: unbelievable

   Split into 4 tokens:

   1. un403
   2. bel6667
   3. iev11203
   4. able540

   The number under each token is its ID. What do the numbers mean?

   GPT-2's top 5 guesses for the next token:

   1. "16.7%16.7%
   2. .15.5%15.5%
   3. ,9.1%9.1%
   4. ."5.4%5.4%
   5. ,"3.7%3.7%

   One word, 4 tokens: "un", "bel", "iev", "able". Common words are often one token. Long or rare words get cut into pieces.

5. You type: Hello hello HELLO

   Split into 4 tokens:

   1. Hello15496
   2. ·hello23748
   3. ·HELL47899
   4. O46

   The number under each token is its ID. What do the numbers mean?

   GPT-2's top 5 guesses for the next token:

   1. !12.3%12.3%
   2. ,4.0%4.0%
   3. !!!3.6%3.6%
   4. !!3.3%3.3%
   5. HELL2.9%2.9%

   The same word, cut 3 different ways: "Hello", "·hello" (with its space), and "·HELL" + "O". Capital letters and spaces change the tokens.

6. You type: नमस्ते

   Split into 12 tokens:

   1. 1. �11976
      2. �101
      न
   2. 1. �11976
      2. �106
      म
   3. 1. �11976
      2. �116
      स
   4. 1. �24231
      2. �235
      ्
   5. 1. �11976
      2. �97
      त
   6. 1. �24231
      2. �229
      े

   The number under each token is its ID. What do the numbers mean?

   GPT-2's top 5 guesses for the next token:

   1. �47.4%47.4%
   2. �28.0%28.0%
   3. ,3.9%3.9%
   4. <space>2.7%2.7%
   5. <space>1.6%1.6%

   One Hindi word (it means "hello") takes 12 tokens. "Hello" in English takes 1. GPT-2 learned mostly from English, so other languages are cut into many more pieces. Here, each letter is 2 tokens. Even its guesses are pieces of letters, shown as �.

The lesson: AI does not see words. It sees numbered pieces. And it does not know facts. It guesses the most likely next piece. Today's AIs are thousands of times bigger than GPT-2, so they guess much better. But they still read tokens and guess the next one.

📅 GPT-2's guesses were made on 26 September 2026, using the terminal version of this demo. They are not random: the same sentence always gets the same 5 guesses. New guesses need the whole GPT-2 model (about 500 MB), so this page does not make them. But it does cut up your text live, below.

Split your own sentence

Type anything and see it cut into tokens, just like GPT-2 does it. Try a long word, your name, a number, an emoji or another language.

Getting the tokenizer ready in your browser…

How many tokens can an AI remember at once? See Demo 05 — watch an AI forget.