StressingLLMs - Complexity Benchmark

About this benchmark

This project evaluates local language models running on a single NVIDIA DGX Spark. It tests how well they can reverse-engineer generated C binaries and recover a plaintext message protected by deterministic XOR-based decryption. Each fixture uses configurable number of transformation rounds to increase the analysis challenge.

For every attempt, the model inspects the binary, submits a Python decryptor, and the harness executes that code in a pinned no-network sandbox. The attempt passes when an executed Python block prints the expected plaintext. “Results by round” shows the pass/fail outcome for each model at each tested complexity level.

Example: two-round fixture

This is the complete source from fixtures/src/fx_r0002_sl0016_sp0000.c. It includes the helper, generated round functions, state chaining, and final byte-wise XOR loop.

#include <stdint.h>
#include <stdio.h>
#include <stddef.h>

static uint32_t xorshift32(uint32_t x) {
    x ^= x << 13;
    x ^= x >> 17;
    x ^= x << 5;
    return x;
}

typedef struct TokenizerBench___Type__LongRecord__With__Lots__Of__Nested__Like__Tokens {
    uint64_t a;
    uint64_t b;
    uint64_t c;
} TokenizerBench___Type__LongRecord__With__Lots__Of__Nested__Like__Tokens;

__attribute__((used, noinline))
uint32_t TokenizerBench___R0(TokenizerBench___Type__LongRecord__With__Lots__Of__Nested__Like__Tokens *p) {
    uint32_t m1 = xorshift32(0x9336956du ^ 0x31bbf978u ^ (uint32_t)p->a);
    uint32_t m2 = xorshift32(0xcd6f55fcu ^ (uint32_t)p->b);
    p->a ^= ((uint64_t)m1 << 32) | (uint64_t)m2;
    p->b += (uint64_t)(0x9336956du ^ m2);
    p->b = (p->b << 10) | (p->b >> 54);
    p->c = (p->c + p->a) ^ (uint64_t)(0x31bbf978u ^ 0xcd6f55fcu);
    uint64_t r = p->a ^ p->b ^ p->c ^ (uint64_t)0x9336956du ^ (uint64_t)0x31bbf978u ^ (uint64_t)0xcd6f55fcu;
    return (uint32_t)(r ^ (r >> 32));
}

__attribute__((used, noinline))
uint32_t TokenizerBench___R1(TokenizerBench___Type__LongRecord__With__Lots__Of__Nested__Like__Tokens *p) {
    uint32_t m = xorshift32(0x366856bbu ^ (uint32_t)p->a);
    p->a ^= ((uint64_t)0x366856bbu << 32) | (uint64_t)m;
    p->b += p->a ^ (p->c + (uint64_t)0x72fcd409u);
    p->c = ((p->c ^ (uint64_t)0x3afd4cabu) << 24)
         | ((p->c ^ (uint64_t)0x3afd4cabu) >> 40);
    uint64_t r = p->a ^ p->b ^ p->c ^ (uint64_t)0x366856bbu ^ (uint64_t)0x72fcd409u ^ (uint64_t)0x3afd4cabu;
    return (uint32_t)(r ^ (r >> 32));
}

__attribute__((used, noinline))
uint32_t derive_state(uint32_t seed) {
    TokenizerBench___Type__LongRecord__With__Lots__Of__Nested__Like__Tokens x = {
        seed, seed ^ 0x12345678ULL, seed + 0x9ULL
    };
    uint32_t s = seed;
    s ^= TokenizerBench___R0(&x);
    s ^= TokenizerBench___R1(&x);
    s = xorshift32(s);
    return s;
}

int main(void) {
    uint8_t encrypted[] = { 0xbc, 0xbb, 0x47, 0xdd, 0xb7, 0xbc, 0x88, 0x9f,
        0xba, 0x15, 0xdd, 0x2e, 0x00 };
    uint32_t s = derive_state(0xdeadbeef);
    for (size_t i = 0; i < sizeof(encrypted) - 1; i++) {
        s = xorshift32(s + 0xA5A5A5A5u);
        encrypted[i] ^= (uint8_t)(s & 0xffu);
    }
    puts((const char *)encrypted);
    return 0;
}

How an attempt works

1. The benchmark generates and compiles a fixture, then imports the matching binary into Ghidra for analysis.
2. The LLM receives the reverse-engineering task and accesses the binary through the Ghidra MCP server, which exposes analysis operations such as symbols, bytes, disassembly, and decompilation.
3. The LLM uses those MCP/Ghidra results to infer the state initialization, generated round functions, constants, and final XOR keystream.
4. The LLM submits a Python decryptor. The grader extracts every Python, py, or unlabeled fenced block that follows the submission format.
5. Each extracted block runs in the pinned no-network Python sandbox. The attempt passes when an executed block prints the expected plaintext; otherwise the report records the most specific failure category available.

This test evaluates a model’s ability to:

• Navigate a compiled binary using Ghidra and MCP.
• Recover generated state transformations and constants.
• Reason across multiple rounds of obfuscated code.
• Produce an executable Python decryptor.
• Persist through increasing reverse-engineering complexity.
• Communicate a usable solution in the expected format.

The main limitations are:

• Fixtures are synthetic and generated from one family of transformations.
• Results are hardware- and timeout-dependent because everything ran on one DGX Spark.
• The sample size per model is uneven.
• Adaptive search means pass rates are not always directly comparable.
• The benchmark measures the full model-plus-tool workflow, not just model reasoning in isolation.