Token
What does Token mean?
A unit a model processes, which may be a word fragment, character, or other encoded item.
A practical example
A long word may be split into several tokens.
MICHAEL ORACLE’S AI DICTIONARY
Language & prompting · Beginner
MICHAEL ORACLE’S AI DICTIONARY
Language & prompting · Beginner
A unit a model processes, which may be a word fragment, character, or other encoded item.
A long word may be split into several tokens.