What is Token?
A token is a small piece of text, often a word or part of a word, that a language model breaks text into.
A language model does not read letters or whole sentences, but tokens: bite-sized pieces. The word 'office' can be one token, while a long word is made of several tokens. The model counts in tokens. If you pay for the tool, the company behind the model bills per token too. A rule of thumb: 1,000 tokens is about 750 English words.
Paid AI tools often charge per token. And a model has a maximum number of tokens it can handle at once, so very long texts sometimes will not fit.
Put precisely
A token is not a word. Dutch text usually costs more tokens than the same text in English, because the models are trained mainly on English.
Common misconception
That per token is the same as per word. That is why a bill for Dutch text comes out higher than you expect.
Related words
Understand AI, calmly
This word is a start. The inklaretaal Learning curve explains AI step by step in plain language, tuned to your field. The first month is free.