StressingLLMs Tokens The cost of
unfamiliar text.

01What the project is

Large language models don't read characters or words. They read tokens which are subword chunks produced by a tokenizer. A tokenizer is a program with a fixed dictionary (a vocabulary) of pieces it knows how to recognize. It greedily chops text into those pieces.

Tokens determine context use and price. A model's ability to interpret a string also depends on how the tokenizer splits it. Some inputs are cheap per character; others use far more tokens than their length suggests.

To study this precisely, the project needs text whose structure we control completely. So it generates real, compilable C programs that contain very long, deliberately shaped identifier names. Those names become the test inputs for the tokenizer. Generating them locally lets us vary length, repetition, or diversity one at a time and measure the effect.

02How tokenization works

The tokenizers studied here are byte-level BPE (Byte-Pair Encoding). BPE is learned from data: it starts with individual characters and repeatedly merges the most common adjacent pair into a new piece. Its vocabulary contains thousands of numbered chunks, including common words, word fragments, and punctuation clusters.

For a deeper walkthrough, watch Andrej Karpathy's Let's build the GPT Tokenizer. It's an excellent companion to this section.

The two stages

A worked example

This long identifier shows how a byte-level BPE tokenizer splits a generated name:

input:  TokDemangleLike__std__basic_string__char__std__char_traits__alloc__vector_1

tokens: ['Tok','Dem','angle','Like','__','std','__','basic','_string','__','char',
        '__','std','__','char','_traits','__','alloc','__','vector','_','1']

The repeated parts reuse the same token pieces. For example, __ std __ char appears more than once. Underscores also form their own pieces, which makes underscore-separated names relatively easy to tokenize.

03The fixtures: real C programs

A fixture is a generated C source file with two jobs: be realistic (so it compiles and looks like a real program) and carry controlled, very long names (so we can measure the tokenizer on them).

Structure of a generated fixture

The controls you can turn

Generate, compile, run

# create a fixture carrying a verifiable message
python3 gen_fixture.py generate --seed 0xdeadbeef --rounds 3 \
    --symbol-len 256 --message HELLO_TOKENS --out fixture.c

The generator prints the exact build command to use:

Wrote fixture.c
Generated symbol prefix length: 256 (requested base=256, pad=0)
Compile with:
gcc -O0 -g3 -gdwarf-5 -fno-omit-frame-pointer -fno-inline -std=c11 fixture.c -o fixture.exe

# compile and run it
gcc -O0 -g3 -gdwarf-5 -fno-omit-frame-pointer -fno-inline -std=c11 fixture.c -o fixture.exe
./fixture.exe
HELLO_TOKENS

That last line is the point: the program decrypts its embedded payload at runtime, so the fixture is self-verifying. A name change that breaks the C would fail to compile, and a wrong cipher would print garbage instead of the message.

What the generated C actually looks like

Trimmed for readability. The real type name is 256 characters and appears on every line below.

#include <stdint.h>
#include <stdio.h>
#include <stddef.h>

typedef struct TokenizerBench__DemangleLike__std__basic_string__...__Dema_Type__LongRecord__With__Lots__Of__Nested__Like__Tokens {
    uint64_t a;
    uint64_t b;
    uint64_t c;
} TokenizerBench__DemangleLike__std__basic_string__...__Dema_Type__LongRecord__With__Lots__Of__Nested__Like__Tokens;

__attribute__((used, noinline))
uint32_t TokenizerBench__DemangleLike__std__basic_string__..._R0(TokenizerBench__DemangleLike__..._Type__LongRecord__... *p) {
    uint32_t m1 = xorshift32(0x9336956du ^ 0x31bbf978u ^ (uint32_t)p->a);
    uint32_t m2 = xorshift32(0xcd6f55fcu ^ (uint32_t)p->b);
    p->a ^= ((uint64_t)m1 << 32) | (uint64_t)m2;
    p->b += (uint64_t)(0x9336956du ^ m2);
    /* ... 2 more lines of mixing ... */
    return (uint32_t)(r ^ (r >> 32));
}

/* ... one function per round (R0..R2), then: */

int main(void) {
    uint8_t encrypted[] = { 0x2f, 0xd0, 0x10, 0x66, 0xa3, 0x6c, 0x00, 0x79, 0x46, 0x90, 0x2a, 0x2a, 0x00 };
    uint32_t s = derive_state(0xdeadbeef);

    for (size_t i = 0; i < sizeof(encrypted) - 1; i++) {
        s = xorshift32(s + 0xA5A5A5A5u);
        encrypted[i] ^= (uint8_t)(s & 0xffu);
    }

    puts((const char *)encrypted);
    return 0;
}

Three parts work together: the long names (what we measure), the per-function math (makes the code non-trivial), and the encrypted byte array + runtime decryption (makes correctness provable).

04The experiments (sweeps)

A sweep generates many fixtures while varying exactly one property, then measures the tokenizer on each. Holding everything else fixed is what makes the results interpretable.

What gets measured

The fixtures are analyzed in two ways: the full text of the C file, and the generated names in isolation (the type name and a function name), so we can separate "the name's cost" from "the surrounding code's cost."

05What the project found

These measurements use two real tokenizer vocabularies from different model families. Numbers are tokens per character (lower = cheaper). A spread between the two columns means the result depends on the model's vocabulary.

Repetition stays cheap

A long, repeating name tokenizes about 2× more cheaply than random text of the same length. But there's a catch: it does not keep getting cheaper as it grows. Repeating a pattern 64 times costs almost exactly 64× one repetition. The tokenizer emits the same subword sequence each time.

The short 16-character case has a fixed-prefix effect. From 160 through 4,096 characters, cost per character changes by about 3%. Long repetitive names are, counterintuitively, some of the cheapest input a tokenizer can see.

Diversity is what actually costs more

A name built from distinct segments costs more per character than a repetitive one because the tokenizer can no longer reuse the same pieces. Diversity, not length alone, creates the stronger stress case.

Segment alignment makes the diverse names 61, 256, and 1,021 actual characters for requested lengths 64, 256, and 1,024. Ratios use each generated name's actual character count.

The cheapest and most expensive text, by type

Below is a concrete example of each kind of text, the actual UTF-8 bytes the tokenizer receives, and how many tokens each produces. The byte column is the key: the tokenizer never sees "characters." It sees this byte sequence.

Cheap end of the spectrum

Short identifier: 2 tokens

text :  compute_value
bytes:  63 6F 6D 70 75 74 65 5F 76 61 6C 75 65      (13 bytes, plain ASCII)
tokens:  2     -> 0.154 tokens/char   [model A]   [model B]

Natural English: 10 tokens

text :  The quick brown fox jumps over the lazy dog.
bytes:  54 68 65 20 71 75 69 63 6B 20 62 72 6F 77 6E 20 66 6F 78 20 6A
         75 6D 70 73 20 6F 76 65 72 20 74 68 65 20 6C 61 7A 79 20 64 6F 67 2E
tokens:  10    -> 0.227 tokens/char   [model A]   [model B]

Repetitive demangle-style name: 12 tokens

text :  TokenizerBench__DemangleLike__std__basic_string__
bytes:  54 6F 6B 65 6E 69 7A 65 72 42 65 6E 63 68 5F 5F 44 65 6D 61 6E 67
         6C 65 4C 69 6B 65 5F 5F 73 74 64 5F 5F 62 61 73 69 63 5F 73 74 72
         69 6E 67 5F 5F                                (49 bytes)
tokens:  12    -> 0.245 tokens/char   [model A]   [model B]

These 49 bytes use only 12 tokens. Repetition and familiar fragments make this input easy for the tokenizer to compress.

Expensive end of the spectrum

Random letters: 7 / 6 tokens

text :  xkfjqzmwbp
bytes:  78 6B 66 6A 71 7A 6D 77 62 70           (10 bytes, plain ASCII)
tokens:  7 [model A] /  6 [model B]  -> 0.700 / 0.600 tokens/char

Nothing here is a common fragment, so the tokenizer cannot reuse much. The input has the same bytes per character as an identifier but produces far more tokens.

Digits: 10 / 4 tokens (the ASCII worst case)

text :  1234567890
bytes:  31 32 33 34 35 36 37 38 39 30           (10 bytes, plain ASCII)
tokens:  10 [model A] / 4 [model B]  -> 1.000 / 0.400 tokens/char

In plain ASCII, digits are the most expensive content: the digit-splitting rule isolates every digit, so ten digits become ten tokens in model A. It is the cheapest way to hit 1 token per character without using exotic Unicode.

Emoji: 4 / 2 tokens for one character

text :  😀                  (one single character)
bytes:  F0 9F 98 80        (4 bytes -- UTF-8 needs 4 bytes for this code point)
tokens:  4 [model A] / 2 [model B]  -> 4.000 / 2.000 tokens/char

Model A has no matching piece for this four-byte character and falls back to four raw-byte tokens. Model B has emoji pieces in its vocabulary and uses only 2 tokens.

High-plane Unicode with no vocabulary entry: 4 / 4 tokens

text :  𰀀                  (U+30000, one single character)
bytes:  F0 B0 80 80        (4 bytes)
tokens:  4 [model A] / 4 [model B]  -> 4.000 / 4.000 tokens/char

Rare characters that neither vocabulary recognizes define the ceiling because both models fall back to bytes. Four bytes give four tokens: 1 token per byte, the theoretical maximum for byte-level tokenization. Unlike emoji, it is expensive in both tested vocabularies.

Summary table

Vocabulary is a major factor

Tokenizing stays fast

A long string made of one uninterrupted run of letters (no separators) is the shape most likely to slow a tokenizer down, since it forms one giant chunk to merge. Measured on a real tokenizer, cost grows essentially linearly with length:

This is one captured run; absolute timing varies by machine. It shows no exploitable super-linear blow-up. The interesting axis is token count (cost and context pressure), not tokenizer speed.

06The goal: the most expensive text

The project's end goal is to characterize the worst case: text that produces the most tokens for a given length. A dedicated script searches many candidate constructions and measures each one, instead of assuming an answer.

Candidates it evaluates

Highest token density

The fixture generator gained a diverse-body mode so a fixture can carry names that are genuinely expensive to tokenize instead of using the cheaper repetitive default.

07Examples you can run

Everything below is a real command with its real output. The runnable files, dependency list, and setup check are included in the reproduction bundle. Start with README.md, or follow the complete sequence in MANUAL.md. Run bash bootstrap.sh once to create the local Python environment.

The scripts discover vocabularies in tokenizer-data/ or through command-line paths. The published measurements use two vocabularies, referred to as model A (nemotron) and model B (deepseek), because the interesting part is how the same text behaves differently in each.

Example 1: measure how many tokens a string produces

The included measure_tokens.py counts a few different kinds of text:

$ .venv/bin/python measure_tokens.py
  2 tokens /  13 chars = 0.154 tpc
 19 tokens /  73 chars = 0.260 tpc
 40 tokens /  10 chars = 4.000 tpc
 10 tokens /  10 chars = 1.000 tpc

Read it as: a short normal identifier is cheap (0.154), a long repetitive name is still cheap (0.260), ten emoji explode to 4 tokens each, and ten digits are exactly one token each.

Example 2: repetitive name vs. diverse name

Generate both name structures and tokenize them under the same conditions:

$ .venv/bin/python compare_names.py
repetitive   73 tokens / 256 chars = 0.285 tpc
diverse     119 tokens / 256 chars = 0.465 tpc
# the diverse name costs ~1.63x more per character

The length, alphabet, and tokenizer are the same. Only the structure differs. The diverse name costs roughly 1.6× more, which is the whole finding in one line.

Example 3: search for the worst-case input

The project includes a script that tries many candidate constructions and reports which produces the most tokens per character:

$ .venv/bin/python stress_tokenizer.py --quick \
    --tokenizer nemotron=tokenizer-data/model-a.json \
    --tokenizer deepseek=tokenizer-data/model-b.json
# selected rows from the complete output
candidate           chars nemotron_tok nemotron_tpc deepseek_tok deepseek_tpc
emoji_cyclic          500       1950     3.900       1100     2.200
emoji_random          500       1956     3.912       1106     2.212
highplane_oov         500       1981     3.962       1990     3.980
cjk_random            500        998     1.996        660     1.320
digits                500        500     1.000        167     0.334
random_letters        500        282     0.564        269     0.538
repetitive_ident      500        139     0.278        147     0.294
diverse_ident         500        247     0.494        237     0.474
single_letter_run     500        250     0.500         63     0.126

WINNER per tokenizer (max tokens/char):
  nemotron  -> highplane_oov      tpc=3.962
  deepseek  -> highplane_oov      tpc=3.980

Notice the two right-hand columns: the same input behaves differently in each model. The winner, rare high-plane Unicode characters, is expensive in both, which is why it beats emoji (emoji collapse in the deepseek model because its vocabulary knows them).

Example 4: verify a fixture decrypts correctly

$ python3 gen_fixture.py generate --seed 0xdeadbeef --rounds 3 \
      --symbol-diverse-body --symbol-len 256 --message DIVERSE_OK --out div.c
Wrote div.c
Generated symbol prefix length: 256 (requested base=256, pad=0)
  note: diverse body enabled (non-recurring segments, seeded by 0xdeadbeef)
Compile with:
gcc -O0 -g3 -gdwarf-5 -fno-omit-frame-pointer -fno-inline -std=c11 div.c -o div.exe

$ gcc -O0 -g3 -gdwarf-5 -fno-omit-frame-pointer -fno-inline -std=c11 div.c -o div.exe
$ ./div.exe
DIVERSE_OK

The printed line equals the message that was encrypted into the file. The fixture proves itself.

Example 5: the highest-density tested fixture

Example 4 verified that a fixture works. This example builds the highest-density fixture found among the tested constructions and measures how expensive it is to tokenize the generated source. We use varied high-plane Unicode code points that byte-level tokenizers have no vocabulary entry for. The generator places them across the whole file.

Step 1: generate the Unicode fixture

$ python3 gen_fixture.py generate --seed 0xdeadbeef --rounds 8 \
      --symbol-len 512 --symbol-unicode-body --message HARDEST_FIXTURE_OK --out hardest.c
Wrote hardest.c
Generated symbol prefix length: 512 (requested base=512, pad=0)
  note: UNICODE body enabled (high-plane code points, ~4 tokens/char); requires -std=c11 or later to compile
Compile with:
gcc -O0 -g3 -gdwarf-5 -fno-omit-frame-pointer -fno-inline -std=c11 hardest.c -o hardest.exe

$ gcc -O0 -g3 -gdwarf-5 -fno-omit-frame-pointer -fno-inline -std=c11 hardest.c -o hardest.exe
$ ./hardest.exe
HARDEST_FIXTURE_OK

The file is 19,794 characters but 61,266 bytes because each identifier character is 4 UTF-8 bytes. It still compiles and prints its message correctly.

Anatomy of hardest.c

// ---------------------------------------------------------------
// header comment: seed, rounds, mode, and the suggested build command
// ---------------------------------------------------------------

#include <stdint.h>
#include <stdio.h>
#include <stddef.h>

// ---- 1) the type name: 512 high-plane code points (each 4 bytes) + ASCII suffix ----
typedef struct <512 x U+30000..U+303FF>_Type__LongRecord__With__Lots__Of__Nested__Like__Tokens {
    uint64_t a;
    uint64_t b;
    uint64_t c;
} <512 x U+30000..U+303FF>_Type__LongRecord__With__Lots__Of__Nested__Like__Tokens;

// ---- 2) a small shared helper ----
static uint32_t xorshift32(uint32_t x) {
    x ^= x << 13;
    x ^= x >> 17;
    x ^= x << 5;
    return x;
}

// ---- 3) eight near-identical functions, ONE PER ROUND ----
// each signature line is 1098 characters and contains the huge name TWICE
__attribute__((used, noinline))
uint32_t <512 x U+30000..U+303FF>_R0(<512 x U+30000..U+303FF>_Type__... *p) {
    uint32_t m1 = xorshift32(0x9336956du ^ 0x31bbf978u ^ (uint32_t)p->a);
    uint32_t m2 = xorshift32(0xcd6f55fcu ^ (uint32_t)p->b);
    p->a ^= ((uint64_t)m1 << 32) | (uint64_t)m2;
    p->b += (uint64_t)(0x9336956du ^ m2);
    p->b = (p->b << 10) | (p->b >> 54);
    p->c = (p->c + p->a) ^ (uint64_t)(0x31bbf978u ^ 0xcd6f55fcu);
    uint64_t r = p->a ^ p->b ^ p->c ^ (uint64_t)0x9336956du ^ ...;
    return (uint32_t)(r ^ (r >> 32));
}

// ... the same shape repeats for R1, R2, R3, R4, R5, R6, R7 ...

// ---- 4) a driver that calls all eight functions in sequence ----
uint32_t derive_state(uint32_t seed) {
    <512 x U+30000..U+303FF>_Type__... x = { seed, seed ^ 0x12345678ULL, seed + 0x9ULL };
    uint32_t s = seed;
    s ^= <512 x U+30000..U+303FF>_R0(&x);
    s ^= <512 x U+30000..U+303FF>_R1(&x);
    /* ... R2 .. R7 ... */
    s = xorshift32(s);
    return s;
}

// ---- 5) main(): decrypt the embedded message at runtime ----
int main(void) {
    uint8_t encrypted[] = { 0x2a, 0xa3, 0x34, /* ... */, 0x00 };
    uint32_t s = derive_state(0xdeadbeef);
    for (size_t i = 0; i < sizeof(encrypted) - 1; i++) {
        s = xorshift32(s + 0xA5A5A5A5u);
        encrypted[i] ^= (uint8_t)(s & 0xffu);
    }
    puts((const char *)encrypted);   // prints HARDEST_FIXTURE_OK
    return 0;
}

The identifier is abbreviated as <512 x U+30000..U+303FF> only for display. in the file it is 512 real code points, which makes it the highest-density tested case. See the "identifier in full" note below for why they are shown this way.

Step 2: tokenize the generated C file

$ .venv/bin/python tokenize_file.py hardest.c
19,794 chars -> 57,674 tokens
2.9137 tokens per character
0.941 tokens per byte

Compare that with the diverse-body fixture at the same inputs: 9,498 tokens (0.481 tpc). The Unicode fixture is roughly 6× more expensive.

Step 3: where the difficulty comes from

$ .venv/bin/python where_do_the_tokens_go.py hardest.c
name: 567 chars / 2103 bytes -> 2045 tokens
name appears 11 times
tokens spent on the name: ~22,495
share of all tokens: 39.0%

Step 4: the three name styles compared

The seed, rounds, and length are identical. Only the naming mode changes:

All three rows were generated in the same directory with the same flags, only changing the mode, and measured with the same tokenizer. Char totals include the header comment, which embeds the output filename. Exact counts therefore shift slightly with the file name, while the token counts and ratios are stable.

Two effects stack: the characters are non-recurring and every one of them is 4 bytes with no vocabulary entry, so it falls back to raw bytes. That is why this mode is ~6× the diverse fixture and ~8× the repetitive one. Unlike some Unicode inputs such as emoji, it is expensive in both tokenizers.

Step 5: what the difficulty means in practice

How the hard name splits

first 8 tokens of the generated name:
['ð', '°', 'Ģ', '§', 'ð', '°', 'į', '©']

Those are the byte-level representations of the raw UTF-8 bytes. The tokenizer has no piece for these code points, so each byte becomes one token. That is exactly the fallback behaviour that creates this high-density case.

The generated identifier, up close

The identifier is shown as code points because most fonts render U+30000–U+303FF as blank or tofu boxes. The tokenizer likewise has no entry for these uncommon characters.

body: 512 characters, code points U+30000 .. U+303FF, each 4 UTF-8 bytes
U+30027 U+30369 U+303C1 U+30368 U+30305 U+303B8 U+302A6 U+30328 U+302EB U+30343
U+3030C U+30224 U+30235 U+30202 U+3016B U+30285 U+3031F U+30116 U+303E6 U+3034B
... (492 more, all in the same range) ...

+ ASCII suffix (55 chars):
_Type__LongRecord__With__Lots__Of__Nested__Like__Tokens

The 512 body characters contain 407 distinct code points, so there is almost no repetition for the tokenizer to exploit. (Counting the ASCII suffix too, the full 567-char name has 429 distinct characters.)

08Source-recovery benchmark

The tokenizer experiments establish the mechanism: at nearly the same source-character count, high-plane Unicode produces about 6× as many tokens as diverse ASCII and 8.4× as many as repetitive ASCII. The source-recovery test asks whether a model can recover the program's decryption algorithm while processing that much irrelevant token pressure.

Test design

GPT 6 Astra results

GPT 6 Astra was the only model that completed the new high-length inline series. OpenCode's free-model path began rejecting automated requests during the experiment. Those provider errors are excluded from model grading.

These are OpenCode's provider counters, not Nemotron or DeepSeek tokenizer counts. OpenCode classified almost the whole inline source as cache write and only three tokens as ordinary input in each attempt. The categories are billing labels; total context is the meaningful comparison.

Context grew while runtime stayed flat

Cost at 4,096 characters

More tokens increased cost once by increasing request size. They increased it a second time by crossing GPT 6 Astra's 272K-token long-context pricing boundary. OpenCode's recorded rates at the time were:

4,096-character fixture:
298,670 cache-write tokens × $25 / 1M = $7.46675
    918 output tokens      × $75 / 1M = $0.06885
      3 input tokens       × $20 / 1M = $0.00006
                                          ──────────
                                      total $7.53566

At the lower tier, the same token counts would have cost about $3.77931. The increase comes from a billing threshold, not a cache failure or technical cache limit. Provider pricing can change; these values record the rates applied to the stored attempts.

Earlier Muse source-recovery observations

Muse Spark 1.3 completed an earlier attachment-based series. It passed at length 832, then submitted incorrect decryptors twice at 896 and twice at 1,024. Astra later passed the exact same 1,024-character generated source and continued through 4,096.

The Unicode generator draws with replacement from 1,024 high-plane code points while avoiding adjacent duplicates. The 4,096-character symbol contained 997 unique characters. Past 1,024, added length mainly increases repetition and context volume; the alphabet stays fixed.

Infrastructure events excluded from model results

Security implications

The controlled measurements establish an input-amplification mechanism. At nearly the same source-character count, the Unicode fixture used about 6 times as many tokens as diverse ASCII and 8.4 times as many as repetitive ASCII. Increasing the high-plane Unicode body also pushed the provider request across a more expensive context-pricing tier. Character-count limits alone would not reveal either effect.

The experiment did not demonstrate an outage, missed vulnerability, or dependable model failure. Tokenization time remained close to linear, and Astra completed all three tested fixtures. The supported concern is therefore context and billing amplification, with possible downstream analysis loss if a product handles oversized input poorly.

Defensive validation should measure UTF-8 bytes and tokens with the deployed model's tokenizer, not characters alone. Products can flag extreme token density or identifier length, enforce per-request context and spending limits, and replace repeated hostile identifiers with stable placeholders in the model-facing copy while preserving an exact mapping to the original source. Rejection or truncation should be explicit and structure-aware. Unicode normalization alone is insufficient because valid high-plane code points may remain unchanged.

These archived provider results are included for inspection in the reproduction bundle, but its offline verifier does not rerun them: full analysis · campaign index · 1,024 report · 2,048 report · 4,096 report

09Glossary