Strip grammar. Keep facts. Save tokens.

- "In order to optimize the database query performance, we should consider implementing an index on the frequently accessed columns..." (70 tokens)
+ "Need fast queries. Check which columns used most. Add index to those columns..." (50 tokens)

= 29% reduction

LLMs excel at filling linguistic gaps. They predict missing grammar, connectives, and structure.

Key insight: We remove only what LLMs can reliably reconstruct.

What we remove (predictable):

• Grammar: "a", "the", "is", "are"
• Connectives: "therefore", "however", "because"
• Passive constructions: "is calculated by"
• Filler words: "very", "quite", "essentially"

What we keep (unpredictable):

• Facts: numbers, names, dates
• Technical terms: "O(log n)", "binary search"
• Constraints: "medium-large", "frequently accessed"
• Specifics: "Stockholm", "99.9% uptime"

Compressed: "Company medium-large. Location Stockholm."
Decompressed: "at a medium-large company based in Stockholm"
           ↑ grammar added, facts unchanged ↑

LLM-based (best compression, requires OpenAI API):

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Set up API key
cp .env.example .env
# Edit .env and add your OpenAI API key

NLP-based (free, offline, multilingual):

python3 -m venv venv
source venv/bin/activate
pip install -r requirements-nlp.txt
python -m spacy download en_core_web_sm  # or other language models

MLM-based (free, offline, predictability-aware):

python3 -m venv venv
source venv/bin/activate
pip install -r requirements-mlm.txt
python -m spacy download en_core_web_sm

NLP-based compression (most stable, 15-30% reduction, free, offline):

python caveman_compress_nlp.py compress "Your verbose text here"
python caveman_compress_nlp.py compress -f input.txt -o output.txt
python caveman_compress_nlp.py compress -f input.txt -l es  # specify language

MLM-based compression (20-30% reduction, free, offline, predictability-aware):

python caveman_compress_mlm.py compress "Your verbose text here"
python caveman_compress_mlm.py compress -f input.txt -o output.txt
python caveman_compress_mlm.py compress -f input.txt -k 30  # adjust compression level

LLM-based compression (40-58% reduction, requires API key):

python caveman_compress.py compress "Your verbose text here"
python caveman_compress.py compress -f input.txt -o output.txt

Decompress:

python caveman_compress.py decompress "Caveman text here"

Automated benchmark verifying that specific facts are preserved and retrievable after compression:

# LLM-based compression
python benchmark/factual_preservation/run_factual_benchmark.py

# NLP-based compression
python benchmark/factual_preservation/run_factual_benchmark_nlp.py

Results: 13/13 facts preserved (100%) with 12-25% compression ratio.

See benchmark/factual_preservation/ for details.

All examples validated with GPT-4o. See examples/ for full text.

1. Strip connectives - Remove: therefore, however, because, in order to
2. 2-5 words per sentence - One atomic thought per sentence
3. Action verbs - Prefer: do, make, fix, check vs facilitate, optimize
4. Be concrete - "test five, test six" not "test values 5-6"
5. Active voice - "calculate value" not "value is calculated"
6. Keep meaningful info - Numbers, sizes, names, constraints stay

See SPEC.md for full rules.

Original:

A network router is a device that forwards data packets between computer networks. Routers perform the traffic directing functions on the Internet. When a data packet arrives at a router, the router examines the destination IP address...

Compressed:

Network router forwards data packets. Routers direct Internet traffic. Packet arrives router. Router examines destination IP address. Router determines best path. Router uses routing table...

Why it works: Store compressed docs in vector DB. Agent receives compressed RAG results directly. No decompression needed—agent understands caveman format. Fits 2-3x more context.

Original:

First, I need to understand what the user is asking for. They want to calculate the optimal route between two cities considering both distance and traffic conditions. Let me break this down into steps. Step one: I should identify the starting city...

Compressed:

Need understand user request. User wants optimal route between cities. Consider distance, traffic. Step one: Identify starting city, destination city. Step two: Retrieve current traffic data for routes...

Why it works: Agent thinks in caveman format during problem-solving. Chain-of-thought uses 50% fewer tokens. More reasoning steps fit in context window.

• Reduction: 40-58%
• Cost: Requires OpenAI API key
• Quality: Best compression, context-aware
• Speed: ~2s per request
• Use when: Maximum token savings needed

• Reduction: 20-30%
• Cost: Free
• Quality: Excellent compression, predictability-aware using RoBERTa
• Speed: ~1-5s per document (local model)
• Method: Removes top-k most predictable tokens based on masked language model probabilities
• Use when: Need better compression than NLP without API costs, can tolerate model download (~500MB) and initial loading time

• Reduction: 15-30%
• Cost: Free
• Quality: Good compression, rule-based
• Speed: <100ms
• Languages: 15+ supported (en, es, de, fr, it, pt, nl, el, nb, lt, ja, zh, pl, ro, ru, and more)
• Use when: Working offline, processing large volumes, no API budget, or need multilingual support

✅ Good for:

• LLM reasoning/thinking blocks
• Token-constrained contexts
• Internal documentation
• Step-by-step instructions

❌ Avoid for:

• User-facing content
• Marketing copy
• Legal documents
• Emotional communication

• SPEC.md - Full specification and rules
• examples/ - Before/after samples
• benchmark/ - Semantic losslessness tests
• prompts/compression.txt - System prompt for compression
• prompts/decompression.txt - System prompt for decompression

Contributions welcome. Submit issues or PRs.

MIT

William Peltomäki

Inspired by TOON and the token-optimization movement.