LLM token counter

What a token actually is

A language model does not read your text as letters or as words. Before anything else happens, the text is cut into tokens, and a token is a sequence of bytes that the model has a single number for. Some tokens really are whole words. Many are a word plus the space in front of it, which is why the same word can cost a different amount depending on what comes before it. Others are a suffix, a punctuation mark, an indent, or a fragment that is not a piece of anything a reader would name. The cutting is done by a table that was built by counting which sequences occur together most often in a large body of text, and nothing about it was designed to match the way anybody writes.

• A token is a sequence of bytes, not a sequence of characters. That distinction sounds pedantic until you write outside the Latin alphabet, where it is the whole story.
• A common English word is usually one token. A rare one, a name, a product code or a typo is several, because the table has no entry for the whole of it and has to build it out of shorter pieces.
• Whitespace is not free. Indentation, blank lines and the spaces inside a pasted table are all bytes, and they are all counted. A prompt that looks tidy can cost noticeably more than the same prompt written flat.
• The count depends on the exact bytes and nothing else. Two texts that look identical on screen but differ in an invisible character — a non-breaking space, a byte order mark, a different kind of apostrophe — are different texts and can come to different numbers.

This is why the rule of thumb everybody repeats — divide the characters by four — is not a rule at all. It is one average over one kind of text in one language, and it is wrong in the direction that costs money the moment your text is code, a table, a list of identifiers, or written in anything but English. The number on this page is not that average. It is the actual cut, made by the actual table, in your browser.

The encoding is what does the counting

The table and the merge rules together are called an encoding, and it has a name of its own. That name is the thing worth learning, because the count belongs to it rather than to any model: several models share one encoding, and a model that is replaced by a newer one very often keeps the same encoding as its predecessor. When this page shows you a number, it shows you the encoding it was computed under, right beside the model you picked.

• A model uses an encoding. The mapping is many-to-one, so knowing which encoding your model uses tells you more than knowing the model does.
• An encoding has a published name of its own, such as o200k_base, and that name means the same table wherever you meet it. A count taken under one encoding on this page and a count taken under the same encoding anywhere else agree, which is what makes a count with its encoding attached checkable at all.
• A count is only meaningful with its encoding attached. A number quoted with a model name and no encoding is a number you cannot check, and it is what most pages on this subject give you.

The practical consequence is one that surprises people: switching between two models that share an encoding will not move the number at all, and that is the tool being right rather than the tool being broken. If you want to see a count change, you have to change the encoding, not the model name.

Why text that is not English costs more

Here is the part that no page written English-first will tell you, and it is the reason this one exists. The encodings in use are byte-level: they work on the UTF-8 bytes of your text rather than on its characters. A Latin letter is one byte. A Hebrew, Arabic, Greek or Cyrillic letter is two. A Chinese, Japanese or Korean character is three. An emoji is four, and a flag or a family joined together is several of those in a row. So before any table is consulted at all, the same sentence is already two or three times as many bytes in one language as in another.

• The tables were built by counting sequences in a corpus that is overwhelmingly English, so English words got entries and words in other languages often did not. A language with no entry for its own common words is spelled out in fragments.
• That compounds with the byte cost rather than replacing it. A word that has to be assembled from three fragments, each of which is two or three bytes, is expensive twice over.
• The effect is not uniform across the languages that are not English. A script with its own frequent sequences in the table does much better than one that was barely represented, and the only way to know where yours sits is to paste your own text and look.
• It is why a budget worked out in English does not transfer. If you were told a context window holds a certain number of pages, that figure was almost certainly measured in English, and your own language will not reach it.

None of this is a defect you can work around by writing differently, and it is not a reason to write in English. It is a cost you should be able to see before you commit to it, which is the whole job of the number above.

A character can be split across tokens

Because a token is a sequence of bytes and a character outside the Latin alphabet is several bytes, a single character is regularly cut in half: its first bytes fall in one token and its last bytes in the next. Neither of those tokens is text on its own. Printing them as if they were is what produces the row of replacement characters you may have seen on other pages that promise to show you your tokens, and it destroys exactly the insight the picture was supposed to give.

So this site draws a span instead: the shortest sequence of consecutive tokens whose bytes come out as complete text. Usually that is one token. Where a character was split it is two or more — as many as four, for a character that is four bytes and was cut a byte at a time — and the span is drawn whole with the number of tokens inside it stated beside it, so you see the split rather than losing the character to it. The count is never affected by any of this. Splitting is a fact about how the bytes were cut, and the total is the total.

Reading the picture of your own text

Under the count, your text is drawn a second time with its boundaries showing. Every block in that picture is one span, the blocks run in the order you wrote them, and nothing is left out or shortened, so what you are looking at is your own text rather than a list of fragments taken out of it. That is the question the number cannot answer: not how much the whole thing costs, but which parts of it are the expensive ones.

• A block with no number on it is a single token, which is the ordinary case. It is left unmarked deliberately: a numeral on every block would be a wall of digits over your own sentences and would bury the blocks that are worth looking at. The boundary is the reading, and the shading is what makes one block end and the next begin.
• A block carrying a small number is a span of more than one token, and that is where a character was cut across a boundary. The character is still there, drawn whole, with the number of tokens it took beside it — so a split shows up as a cost rather than as damage to your text. An emoji or a character from an older script can reach four.
• Spaces, line breaks and indentation are drawn like everything else, because they are counted like everything else. A stretch of blocks over what looks like nothing at all is the cheapest part of a long prompt to fix, and it is invisible until you can see it.
• Switching the model reshapes the picture whenever it changes the encoding, and leaves it exactly as it was whenever it does not. That is the same fact the number shows, in a form where you can see where the difference falls rather than only that there is one.

A very long text is drawn only so far, and where the drawing stops the page says how many blocks it drew out of how many. The count is never limited in the same way: it is over everything you pasted, at any length, and nothing about the picture reaches it. The limit exists because a browser lays out every block one at a time and a whole document would be more of them than any page could usefully show — so what you lose past that point is the picture of the rest, never the answer about it.

Why some models carry no number here

Not every model can be counted in your browser, and the honest thing to do about that is to say which and why rather than to show you a number that was guessed. There are three states a model can be in on this page, and they are told apart on screen rather than blurred together.

• Its encoding is published and this site serves it. You get an exact count, computed here, against the real table.
• Its encoding is published and this site does not serve it. The row names the encoding and says why — most often because the file is published by nobody who states a licence permitting anyone else to republish it. The table exists; the refusal is ours, and the row says so.
• Its encoding was never published at all. There is no table to serve and none to refuse, so no page anywhere can count that model in your browser, whatever it claims. The row says that, and carries no number, no estimate and no range.

That last row is where most pages on this subject quietly cheat: they run your text through a different model’s table and present the result as though it were a measurement. It is not merely imprecise, it is biased, and it is biased worst on exactly the text that is hardest to budget for — code, and anything not written in English. The exact count for such a model comes from its maker’s own token-counting service, which needs an API key that a page running entirely in your browser has no way to hold and should never ask you for. Reaching for that service is the honest answer, and it is the one this page gives you instead of a number.

Turning a count into a cost yourself

This page states no prices, and that is deliberate: a price goes stale faster than anything else on this subject, and a stale price does not leave a page incomplete, it leaves it false. What does not go stale is the arithmetic, so here it is, and you can apply it to whatever the provider’s own page says today.

• Providers quote a rate per million tokens. Take the count from this page, divide by a million, and multiply by that rate. That is the cost of sending the text once.
• Input and output are priced separately, and output is usually the dearer of the two. This page counts what you send. It cannot count a reply that does not exist yet, so any cost you work out from it is a floor and never a total.
• A conversation resends its history. If your text is a system prompt that goes with every turn, multiply by the number of turns you expect, and again by the number of users, before deciding whether shortening it is worth an afternoon.

The same count answers the other question people bring here, which is whether a text will fit. This page will not tell you that either, because a context window is a figure that moves and a promise about fitting is one no tool that is not the model itself can keep. What it gives you is the number to compare against whatever figure the provider publishes.

Nothing you paste leaves your browser

The counting happens in the page you already loaded. The table for the encoding you picked is fetched once from this site — that is a fetch of a file, not an upload of your text — and everything after it runs on your own machine, in a background thread so that a long document does not freeze the tab. Open your browser’s network panel and watch while you type: no request goes out carrying what you wrote.

That is what makes it safe to paste the thing you actually want counted — a customer’s prompt, an internal system message, a chunk of a private document — rather than a sanitised version of it that would have come to a different number anyway. Nothing is stored either: reload the page and the field is empty, and there is no history and nothing to delete afterwards.