MICHAEL ORACLE’S AI DICTIONARY

Language & prompting · Beginner

Tokenization

Explained by

What does Tokenization mean?

Turning input into the tokens a model can process.

A practical example

A tokenizer converts a paragraph into a sequence of token IDs.

Explore this term in the Dictionary app

Related AI concepts

Browse all 146 terms · Discuss or suggest an improvement