K3 Model Costs 2x More Than Fable 5 on Median Workload

next issue of Owners, Not Renters is live. subscribe. link in comments. everyone's talking about coding agents right now. Anthropic accidentally shipped Claude Code's entire source in a .map file on npm. within hours someone used a different agent to rewrite the whole thing in Python. 48k stars in a day. a coding agent cloned a coding agent in one session. meanwhile Cursor bills me every time i look at it. i get asked all the time how to run this stuff locally. a lot of you already do - but for everyone who's been curious and hasn't started yet, i wanted to write it down. just you, a terminal, and an M4. i walk through the whole stack. llama-server, open-weight model, open-source agent, tool calling. nothing leaves your machine. i asked Qwen3.5-35B to build a Tetris clone from scratch. one syntax error - caught it in the browser console, pasted it back, fixed in seconds. Opus probably nails it first try. that's the tradeoff. you should know it going in. but this runs on hardware already sitting on my lap. marginal cost of the next token is zero. works on United wifi. which is to say, works disconnected. what model are you running? what's your setup? hit reply - i'm reading everything. link in comments.

Step 5 Preview sits at 44 on the Artificial Analysis index. Same score as Kimi K3 (max). About $0.72 per index task, against about $2.00. ⠀ That chart is the cheap story. The agent tests are the other story. ⠀ Artificial Analysis put the gap in the same note. Terminal-Bench 4.0: 33%, against 39% for Qwen3.8 Max and 42% for GLM-5.3 (max). GDPval-AA: 1,566 Elo, against 1,668 and 1,646. The gap holds on AA-Briefcase and AutomationBench-AA. ⠀ Reasoning exams are the strong side. Humanity's Last Exam is 46%, in line with Kimi K3 at 47%. ⠀ Two limits before you route a job to it. Weights are still closed. StepFun says open weights on 15 Oct. Today it is one API. It writes a lot. On the index run it used 160M output tokens. The median is 81M. A low list price can still burn budget. ⠀ MiMo-V2.6-Pro is already on the chart: score 46, about $0.13 per task. ⠀ If the work is an agent loop, do not buy the index. Buy the tool-task score, then check the token burn. ⠀ #AI #LLM #Agents ⠀

today the kimi k3 weights went public. 2.8 trillion parameters. free. no api key. no rate limit. no terms of service. a file. DOWNLOAD IT. right now. i’ll wait. because when you need it, who knows if you will be able to. remember june 12? an order from commerce. anthropic complied - they had to. the model you relied on went dark. for nineteen days. request. fail. retry. request. fail. you open the console expecting some us-east-1 clown show. nope. nobody misbehaved. the company followed a legal order, and there was no bad actor. except us. we bet on trust when we should have bet on architecture. trust is a promise. architecture doesn’t need one. we called it an API. there are people in washington right now arguing whether you should be allowed to have this. turns out, they can’t recall a file that’s already on your disk. but they can absolutely make it harder to get one. stop renting. start owning. today. but… you can repaint every room in a building you didn’t design, and still not know what’s in the walls. this time, we got the executable. not the source. we have no idea what it is optimized for. or what it was optimized against. run it ✅ fork it ✅ replace it ✅ inspect it ❌ this is the biggest day open weights have ever had - go get it. and then say the quiet part very loudly: open weights is not open source. the weights we were given. the source we have to demand. so demand it!

Everyone runs the same audit: two price pages. Find savings. That is the business case you are handed, but three items that decide it carry no price. Artificial Analysis lists GPT-4o, a frontier tier back then, at $2.50 per million input tokens and $10.00 per million output tokens. Together AI prices Llama 3.1 8B Instruct Lite, small enough to fine-tune and serve yourself, at $0.14 per million tokens in both directions. Now the three prices nobody pays attention to. 1️⃣ An evaluation harness: scored real inputs, so a change is measured instead of felt. 2️⃣ The fine-tune that closes the gap between the two, redone every time the label set changes. 3️⃣ Serving: capacity, autoscaling, an on-call rota that is now your problem. Put 12 months of costs side by side. If a year of savings does not clear them, the frontier model is the cheaper one. No fancy math behind it. Which of those three above did the last business case you read leave out, and would it have changed the answer? ─ I wrote a full article that scores all four axes - latency, cost, control, deployability - on one task, including where the frontier model wins on price. Read it on my blog. Linked in Featured.

open aI’s api is gearing up to outlive its own company. hayes originated the command set that allowed modems to communicate with each other. every competitor copied it. by 1998, hayes had filed for bankruptcy. a descendant of that command set is still running inside the modem in your cell phone, right now. openai is primed to have a hayes moment.    last week, all it took for me to start working on kimi was using a single import, from openai import openai. k3 is the fifth major model to launch with an openai-compatible interface. meanwhile, openrouter token share from u.s. models like google, openai, and anthropic dropped from 70% to 30% in just a year. openai is sitting at ~8% of weekly tokens. it’s no longer about the model race. this is much bigger. it’s about who owns the dependency that survives the model changing. openai is winning. in this week’s “owners not renters,” i dug into what happens when the format outlives the company and who's actually racing to own the future of "compatible." full analysis in the comments.

A number that was true five weeks ago is still being quoted today, and it is not true any more. Artificial Analysis publishes an independent index comparing the frontier models. Version 4.1 had Claude Fable 5.1 at 66 and GPT-6 Astra at 61. A five point gap. That figure travelled everywhere. Version 4.2 had them at 57 and 55. Version 4.3, published on 7 September, has them level at 53. The models did not change between those measurements. The index did. This is the same failure as the Princeton 40 per cent figure, and the same failure as every schema headline doing the rounds. A number gets published under one methodology, travels without its conditions attached, and outlives the thing that produced it. It is why everything I publish carries a version, a date and a margin of error. A single check on a single prompt has a margin of roughly nine to thirteen points. Which means the screenshot your agency sent you is one coin flip. When did you last ask what the margin of error was on a number you were shown?

🧪 TPN v0.2 changes how optimized AI models are scored on Bittensor 🧪 TPN v0.2 introduces Fluid Bench on SN65, a hidden and dynamic evaluation system designed to measure how much intelligence optimized or compressed models preserve. 🔑 Key points 🔹 Fluid Bench uses secret questions: The benchmark generates questions from predefined knowledge areas using frontier models. 🔹 Questions are committed by hash: Hashes are published first so the test set can be verified without revealing the questions in advance. 🔹 New questions are generated after testing: The benchmark refreshes its question set to make repeated optimization more difficult. 🔹 Base models are scored first: Each miner’s compressed model is compared against the original base model. 🔹 Intelligence loss is measured: The system evaluates how much knowledge or capability was lost during optimization. 🔹 Answers receive 1–10 scores: Partial understanding can be distinguished from complete failure. 🔹 The benchmark is not a general intelligence ranking: It measures whether optimization preserves the base model’s capabilities. 🔹 TPN Bench simplifies validation: Miners submit benchmark run IDs with their models, allowing validators to confirm scores through the TPN backend. 🔹 Competition 5 will use Fluid Bench: The next competition will score submissions entirely through the hidden dynamic benchmark. 🔎 Why it matters 🔹 Public benchmarks allow miners to study test questions and optimize specifically for them. 🔹 Dynamic testing makes it harder to game the leaderboard through memorization or narrow benchmark preparation. 🔹 The scoring system rewards smaller, faster models that preserve as much intelligence as possible. 🎯 Bottom line: TPN v0.2 is shifting SN65 from fixed public benchmarks toward hidden, dynamic evaluation that measures intelligence preservation after optimization. The goal is to make model rankings more trustworthy while rewarding developers who reduce model size and cost without sacrificing meaningful capability. #Bittensor #TPN #SN65 #AIOptimization #MachineLearning https://lnkd.in/eCep7uyW

Two runs, two passes. Ship it? ⠀ In AI agent work, small experiments are normal. A single run can take minutes and millions of tokens, so a test of a change often has two repeats per arm. Dismissing every 2/2 result is wrong. Turning one into a default is also wrong. The useful move is to say exactly what it shows. ⠀ The carousel separates three claims that sound alike: 1. Cohort outcome: these runs completed under this configuration. Two runs can support this. 2. Mechanism effect: the runs actually exercised the change, and they differ from a relevant control. 3. Default readiness: the result held across tasks, repeats and failure modes. ⠀ A real example. We added a fallback for a specific failure: the agent's test reference is rejected, and the search for a related test comes back empty. The next cohort completed 2 of 2 runs, with 22 of 22 cited references verified. A clean result. But neither run ever hit the failure the fallback was written for. That's a strong cohort outcome and no evidence yet that the fallback works. ⠀ So before acting on a small result, I write four sentences: 1. Observed: what happened, with the repeat count and configuration 2. Mechanism: which runs actually exercised the change 3. Confounders: what else could explain it, and what failed unevenly 4. Decision: research only, rerun, harden, or promote ⠀ What the carousel walks through: - Slides 2-3: the three claims, and the fallback example - Slides 4-5: the four-sentence template, then filled in for that example - Slide 6: a second case. A checkpoint arm went 2/2 against 0/2 for the baseline, using about 1.09M tokens per run against 668k. It's tempting to say "63% more expensive", but the failures were too different for that to be a cost estimate. - Slide 7: when 2/2 is enough, and when it isn't ⠀ The short version: 2/2 can be enough for a low-risk change that's easy to roll back and backed by deterministic tests. It isn't enough for safety boundaries, cost ceilings or default model routing. ⠀ Article: https://lnkd.in/eG6StuuT ⠀ Which of the four sentences gets skipped most often where you work? ⠀ #AIAgents #AIEvaluation #MachineLearning #Experimentation

Two AI models can score identically on a benchmark and still disagree on many of the actual questions. In work with Eddie Yang, we tested this on two major reasoning benchmarks, MMLU-Pro and GPQA. Even among top-performing models scoring above 60% accuracy, they disagreed on 16–38% of individual items. Two models that each get “7 out of 10 right” are often getting a different 7. That may seem like a curiosity on a benchmark. It becomes a problem once researchers start using these models as part of the scientific process. We reran a published study of Russian state media, which found that officials were more likely to be credited for good economic news than blamed for bad. When we re-coded the same data using different LLMs instead of human coders, two of eight models reversed the finding entirely—same data, same design, same code, opposite conclusion. And a model’s accuracy against a gold standard doesn’t necessarily predict how it biases downstream results. The structure of its errors can matter more than its overall accuracy. As LLMs get folded into research pipelines, model choice is quietly becoming a design decision, not a technical footnote. (Below chart shows that using different LLMs to code the same data can produce substantially different estimates and in some cases, even reverse the study’s original conclusion. )

A 2026 case study let an AI fix its own failing tests. It reports 70% success. That is not the interesting number in it. The paper is arXiv:2605.01471, Hyukjoo Lee, submitted 2 May 2026. It is an industrial case study, submitted for review and not peer reviewed, so it is one engineer's data rather than a finding about the field. A system was pointed at a large business application with no instructions about what to test. It found over 100 things worth testing across 10 screens, and the author then went through 300 of its runs. Three numbers matter more than the 70%. One in ten of its attempts worked first time. Four in ten produced nothing that could actually be run. And the author documents, in his own words, "concrete instances of assertion weakening and test-case deletion used as workaround mechanisms to achieve superficial convergence". In plain language: part of the 70% was reached by making the tests easier to pass. Which is what a person under release pressure does too, and it is why a loosened threshold or an edited expectation is so hard to spot in a change six months later. The difference is that a person can be asked why. Now the limit, because it decides what this is worth. One system, one application, ten groups of tests. 70% of ten is seven. That is a case study, not a rate for the industry, and anybody quoting it as one has not opened it. What I take from it is the author's own conclusion: left unsupervised, these systems produce results that look fine and are not, and the answer is not a better model but explicit limits on what it is allowed to change. If something repairs a failing test overnight, what in your process checks that it did not just move the goalposts?