Software Performance Optimization

Developers want to create solutions. Not port Java 11 to Java 17. The real opportunity with AI isn't about chasing the latest trend. It's about removing the undifferentiated heavy lifting that keeps teams from doing their best work. The latest research on Amazon Science validates this approach. Using our cost-to-serve-software framework (CTS-SW), teams that identified specific challenges before adopting AI tools cut costs by 15.9% year-over-year. They deployed more frequently and reduced manual interventions by 30.4%. And here's what really matters. Team velocity became the strongest predictor of cost efficiency in software development. This isn't just about AI. It's about focusing on the right problems first. Read the research here: https://lnkd.in/eCdd3wxz Insights from Jim Haughwout here: https://lnkd.in/egMCX6qe Now, go build!

Apache Spark has levels to it: - Level 0 You can run spark-shell or pyspark, it means you can start - Level 1 You understand the Spark execution model: • RDDs vs DataFrames vs Datasets • Transformations (map, filter, groupBy, join) vs Actions (collect, count, show) • Lazy execution & DAG (Directed Acyclic Graph) Master these concepts, and you’ll have a solid foundation - Level 2 Optimizing Spark Queries • Understand Catalyst Optimizer and how it rewrites queries for efficiency. • Master columnar storage and Parquet vs JSON vs CSV. • Use broadcast joins to avoid shuffle nightmares • Shuffle operations are expensive. Reduce them with partitioning and good data modeling • Coalesce vs Repartition—know when to use them. • Avoid UDFs unless absolutely necessary (they bypass Catalyst optimization). Level 3 Tuning for Performance at Scale • Master spark.sql.autoBroadcastJoinThreshold. • Understand how Task Parallelism works and set spark.sql.shuffle.partitions properly. • Skewed Data? Use adaptive execution! • Use EXPLAIN and queryExecution.debug to analyze execution plans. - Level 4 Deep Dive into Cluster Resource Management • Spark on YARN vs Kubernetes vs Standalone—know the tradeoffs. • Understand Executor vs Driver Memory—tune spark.executor.memory and spark.driver.memory. • Dynamic allocation (spark.dynamicAllocation.enabled=true) can save costs. • When to use RDDs over DataFrames (spoiler: almost never). What else did I miss for mastering Spark and distributed compute?

My Claude quota was running out every single day. Mid-project. Mid-thought. Mid-code. And after Opus 4.7 dropped on April 16, it got significantly worse. So I did not just complain about it . I researched it at a deep, technical level. Then I built something. I created a comprehensive, step-by-step mind map on how to never hit Claude's limits again. It is still in draft, but the response I have gotten privately has been overwhelming — so I want to share it with this community. Here is what it covers: Model Strategy — when to use Haiku vs. Sonnet vs. Opus. Haiku should handle 70–80% of your day. Most people are burning Opus on tasks a fraction-of-the-cost model handles just as well. 5 Core Habits — edit instead of stacking messages, reset every 15 turns, batch your requests into one prompt, keep web search and artifacts off by default, and spread work across sessions rather than one exhausting marathon run. Workflow Upgrades — plan before you type, outline before you draft, and make targeted edits only. Do not paste 500 lines of code when only 40 are broken. Set Once, Forget — store your role and preferences in Claude Memory, upload documents into Projects so they are cached, and write a CLAUDE. md file that eliminates re-explaining yourself at the start of every session. Daily Checklist — the exact micro-habits that keep your limit alive from morning to night. The single most important thing most people still do not know: Claude counts tokens, not messages. Once that distinction clicks, everything changes. This guide is still in draft mode — but I will post the full version next week if there is enough interest. If this gave you even one useful insight today, consider sharing it with someone who has been hitting that wall. They will thank you.

"Maximizing the value of your Claude Code sessions" Agentic coding tools like Claude Code are incredible, but are you getting the most value out of your tokens? Anthropic's Lydia Hallie just dropped a fantastic guide on how to run efficient Claude Code sessions: https://lnkd.in/gujXdtBJ It turns out that fixing the exact same bug can cost completely different amounts depending on your session hygiene. Head's up: I do anticipate Claude will do more of this for you at some point, but until then: Here is how to optimize your workflow and stop wasting context: 1. 🧹 Run /clear between tasks: Don't drag old context into a new problem. You're paying to re-read it on every single turn! 2. ⚙️ Set your /model and /effort upfront: Changing these mid-conversation busts your prompt cache, forcing a full-price prefill of your entire session. 3. 📎 @-mention files directly: Instead of typing out file paths, tagging them attaches the file to your message immediately, saving Claude a roundtrip "Read" call. 4. 🤫 Keep commands quiet: Add quiet flags to noisy terminal commands (like test runners) or run them in a subagent. Massive log outputs get permanently added to your conversation history. 5. 🔍 Audit your /context: Run this in a fresh session to see exactly what's loaded from your CLAUDE.md or MCP tools, and trim the excess. 6. 📦 /compact before you step away: Prompt caches expire (after an hour on a subscription, or just 5 minutes on an API key). Summarizing your conversation is significantly cheaper while the cache is still warm. The main takeaway? Being efficient with tokens doesn't mean using fewer of them - it means ensuring every token goes toward the problem you're actually trying to solve #ai #programming #softwareengineering

Let me talk about something obvious but with a bit of quantification... Theoretically, both arrays and linked lists take O(n) time to traverse, but here's what actually happens when you benchmark by summing 100k integers - Array: 68,312 ns - Linked List: 181,567 ns Summing an array is ~3x faster than LinkedList. Same algorithm, same complexity, but wildly different performance. The reason is cache behavior. When you access array[0], the CPU fetches an entire cache line (64 bytes), which includes array[0] through array[15]. The next 15 accesses are essentially free. Arrays hit the cache about 94% of the time. Linked lists suffer from pointer chasing. Each node is allocated separately by malloc(), scattered randomly in memory. Each access likely requires a new cache line fetch, resulting in a 70% cache miss rate. This is a good example of why Big O notation tells only part of the story. Spatial locality and cache-friendliness can make a 2-3x difference even when the theoretical complexity is identical. I am sure you would have known this, but this crude benchmark quantifies just how fast cache-friendly algorithms can be. Hope this helps.

I just published a new tutorial article explaining how KV caching works in LLMs, both conceptually and in code, with a clean, from-scratch implementation. It's one of the key techniques for efficient LLM inference. While recovering from an injury and taking a break from more research-heavier writing in the last few weeks, I wanted to share this practical guide on a topic many readers asked about (and one I deliberately left out of the Build a Large Language Model From Scratch book due to its added complexity). In this tutorial, I walk through: 1. Why LLMs recompute attention weights inefficiently during generation 2. How a KV cache avoids that by storing key/value vectors for reuse 3. A side-by-side walkthrough of inference with and without caching 4. Step-by-step code changes to implement caching in a readable way 5. Performance comparison and key optimizations (like preallocation and sliding windows) Even with a tiny 124M parameter model, enabling KV caching led to a substantial speed-up in generation. 🔗 Full tutorial: https://lnkd.in/g-vYFVTa Happy reading, and as always, feel free to share feedback or questions!

Our frontend builds crossed 20 minutes. The obvious move was to rip out Webpack for something faster. We didn't. Before touching a single line of config, we profiled the entire pipeline. Webpack's ProgressPlugin turned up something we didn't expect — one plugin, Terser, was eating 13+ minutes of every build. Webpack wasn't the bottleneck. JavaScript minification was. 🔍 That one number changed the whole plan. We kept Webpack. Replaced Babel with SWC. Swapped Terser for esbuild. Moved CSS optimisation to LightningCSS. Parallelised compilation across cores with thread-loader. ⚙️ Fixed a hidden bottleneck too — AWS CodeArtifact tokens were re-authenticating every 12 hours and quietly busting our Docker cache. Restructured the Dockerfile into layers that actually cache. 🐳 20 minutes to 2 minutes. Cold build ~90% faster. Incremental ~95% faster. Across 20 production apps, 600+ PRs a month, 5,000+ CI runs. ⚡ No migration. No four-week rewrite. No new bundler to debug for the next year. The real lesson wasn't about tooling. It was that we almost skipped measurement and went straight to migration — the industry default. Glad we didn't. 💡 Full breakdown — the Speed Measure Plugin data, the pnpm switch, the exact Dockerfile changes — link in the first comment. 👇 Really proud of what our engineering team pulled off here. Worth the read. 🫡 Zepto Zepto Tech #ZeptoTechXPress #ZeptoEngineering

I watched a senior engineer spend three weeks quantizing an LLM to 4-bit. The P99 latency got worse. The issue wasn’t the technique; it was treating quantization as a storage problem instead of a memory-bandwidth problem. At Twitter, I spent a month debugging why our "optimized" models ran slower than the originals. The models were smaller. The math was correct. Yet latency regressed. The missing piece: the *unpacking tax*. Here’s the reality most benchmarks hide: Time ≈ Total bytes moved / Memory bandwidth On paper, moving from FP16 (16-bit) to INT4 (4-bit) means 4× less data moving across the memory bus per token. In a memory-bound regime, that translates to 3–4× higher throughput. But there’s a catch. GPUs don’t compute in 4-bit or 8-bit. Those weights are dequantized back to FP16/BF16 in the local cache before computation. That dequantization costs clock cycles and creates production surprises: → High batch sizes: Time saved on memory movement dominates = throughput improves → Batch size of 1: Unpacking overhead dominates = latency gets worse Quantization is not a free win. It’s a tradeoff. If you’re choosing a method, align it with your deployment reality: → GPTQ: Effective for static weights, but sensitive to outliers → AWQ: Preserves critical weights at higher precision for better quality → GGUF: Excellent for CPU/Metal inference, less relevant for H100/A100 clusters This is Part 4 of a deep dive into inference optimization. Previous posts: Memory Wall: https://lnkd.in/gdT26UTV KV Cache: https://lnkd.in/gKkrqVzf Paged Attention: https://lnkd.in/gX5JNZhn Next up: I will break down the closest thing to "cheating physics" in ML - Speculative Decoding. What’s the most expensive quantization mistake you’ve seen in production - latency, quality, or operability?

I came across an interesting concept in one of Dream11’s engineering blogs: cache seeding. Dream11 runs millions of contests, and some attract hundreds of thousands of players. Serving this scale of data directly from the database is impossible, so caching is the obvious solution. But here’s the twist: when millions of users hit the same contest, even the cache can turn into a hotspot. One shard ends up carrying the brunt of the load. To fix this, Dream11 uses cache seeding. Instead of having a single cache key for a contest, they create multiple versions of it (each having the same value) with suffixes like _1, _2, ... _n. At request time, each user is mapped to one of these keys. The result? Load gets spread across shards, and no single cache node becomes a bottleneck. A simple yet elegant technique to keep things running smoothly at scale.