Yashar Moradi

The Context Engineering Framework is quickly becoming one of the most important tools for anyone building reliable LLM systems. Getting the model to respond is the easy part. The real challenge is: → What should the model know right now? → Where should that info come from? → How should it be structured, stored, retrieved, or compressed? That’s exactly what this framework solves. 🧠 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 Context engineering = designing dynamic systems that deliver the right info, in the right structure, at the right time, so models can reason, retrieve, and respond effectively. This matters most in agents, copilots, retrieval-augmented pipelines, and anything with memory or tools. ⚙️ 𝗜𝗻𝘀𝗶𝗱𝗲 𝘁𝗵𝗲 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 Here’s the 3-layer system I use when designing end-to-end LLM workflows 👇 1️⃣ Context Retrieval & Generation → Prompt Engineering & Context Generation → External Knowledge Retrieval → Dynamic Context Assembly 2️⃣ Context Processing → Long Sequence Processing → Self-Refinement & Adaptation → Structured + Relational Information Integration 3️⃣ Context Management → Fundamental Constraints (tokens, latency, structure) → Memory Hierarchies & Storage Architectures → Context Compression & Trimming 🧱 All of this feeds into the Context Engine, which handles: → User Prompts → Retrieved Info → Available Tools → Long-Term Memory This is what gives your system continuity, task awareness, and reasoning depth across steps. ⚙️ Tools I would recommend: → LangGraph for orchestration + memory → Fireworks AI for fast, open-weight inference → LlamaIndex for modular retrieval → Redis & Vector DBs for scoped memory recall → Claude/Mistral for summarization and compression If your system is hallucinating, drifting, or missing the mark, it’s likely a context failure, not a prompt failure. 📌 Save this framework. 📩 Share it with your team before your next agent or RAG deployment. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for real-world GenAI system breakdowns, and subscribe to my Substack for deep dives and weekly insights: https://lnkd.in/dpBNr6Jg

321

59 Kommentare

𝟯 𝗺𝗶𝘀𝘁𝗮𝗸𝗲𝘀 𝘁𝗵𝗮𝘁 𝗺𝗼𝘀𝘁 𝗮𝗿𝗲 𝗱𝗼𝗶𝗻𝗴 𝗶𝗻 𝘁𝗵𝗲𝗶𝗿 𝗳𝗶𝗿𝘀𝘁 𝗟𝗟𝗠 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀 👇 I am no exception :) 𝟭. 𝗡𝗼𝘁 𝗸𝗻𝗼𝘄𝗶𝗻𝗴 𝗞𝗣𝗜𝘀 𝘁𝗼 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗲 𝗳𝗼𝗿 𝗯𝗲𝗳𝗼𝗿𝗲 𝘁𝗵𝗲 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻: especially if you are working on automation using LLMs. After a few months of development you are very likely to run into the scenario where your work is not moving the metrics fast enough. ❗️ Negative effect - lost trust from stakeholders. Likely to result in the project getting cut. ✅ Fix - align with stakeholders on the KPIs early. Dig deeper to understand what inputs will drive the required output. 𝟮. 𝗡𝗼𝘁 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗶𝗻𝗴 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗲𝗮𝗿𝗹𝘆 𝗶𝗻 𝘁𝗵𝗲 𝗽𝗿𝗼𝗷𝗲𝗰𝘁. ❗️ Negative effect - running blind, it will be impossible to know which part of the application is causing bad performance, what is slowing down the application, how much separate parts of the system are costing you. ✅ Fix - set up Observability tooling (LangSmith, LangFuse etc.) early, instrument your code. 𝟯. 𝗡𝗼𝘁 𝗵𝗮𝘃𝗶𝗻𝗴 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻/𝘁𝗲𝘀𝘁 𝗱𝗮𝘁𝗮𝘀𝗲𝘁𝘀 𝗿𝗲𝗮𝗱𝘆 𝘄𝗵𝗲𝗻 𝗶𝘁𝗲𝗿𝗮𝘁𝗶𝗻𝗴 𝗼𝗻 𝘁𝗵𝗲 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻: LLM based systems might become complex, changes in any part of the pipeline might derail the system and negatively impact cases that used to work. ❗️ Negative effect - Previously working case will inevitably stop working. You will spend days trying to figure out what caused this. ✅ Fix - construct your evaluation/test datasets early and include these in tests executed via your CI/CD pipelines. ℹ️ You will be able to reuse some of the data in these datasets in few shot examples so it’s not a waste of time! Are you guilty for not having the above? What other mistakes would you add? #LLM #MachineLearning #AI

81

13 Kommentare

📣 Reliable by Design: Fast, Fail-Safe AI Agents A few months ago, the team and I at AIMon asked ourselves a hard question: How can we make agentic systems highly reliable without compromising on latency? The reflection pattern is well understood, however, creating a good critique LLM is hard. We experimented with existing commercial and open-source LLMs, trying to push the limits of what they could do with some creative prompt engineering. But we consistently hit two major issues: * ⏳ Latency: Too slow for real-time agentic use cases. * 🗒️ Instruction Evaluation: Poor accuracy when evaluating dozens of instructions in a single prompt often resulting in subtle but critical failures. So we decided to build our own. We fine-tuned an open-source LLM using a custom data generation pipeline, optimized specifically for instruction following evaluation. We developed several post-generation optimization tricks that turned out to be key to achieving the best-in-class accuracy while keeping latency under 200ms. We are thrilled to share the result: 🧠 An optimized model that outperforms popular models like GPT-4o-mini at a fraction of the latency/cost. This model is available on AIMon via an API and will save you a ton of time and effort in building your own critique LLMs. Read about our metrics on our blog post (link in comments). A huge shoutout to our brilliant ML and engineering team - it’s been an incredible journey working on reliability in autonomous agents over the past year and a half 🚀

57

3 Kommentare

Code with LLM "SWE-Lancer: Can LLM earn 1M USD as a freelance developer?" is a new coding benchmark for LLMs. It uses real-world problems and e2e tests. The winning model is Claude, which managed to earn 400K USD. There was no real money. Another paper from OpenAI is Competitive Programming with LRM. The paper compares the results of o1, o1-ioi(a handcrafted model for competitive programming) and o3 in the International Olympiad in Informatics. The winning model is o3. On one side, it sounds like the model managed to beat humans in this exercise because o3 is in the top 99.8th percentile, but the real value of the paper is different. o1 is a model which was built to prove inference time scaling. o1-ioi is a fine-tuned version of o1, but o3 is the next generation of o1. o3 is an advanced general-purpose model which managed to beat domain-specific. So LLM wins in isolated, well-defined tasks but needs humans for complex and context-dependent ones. It means that we need to analyse use cases in the life of a software engineer and propose LLM-based solutions. Microsoft has released the paper Identifying Factors Contributing to Bad Days for Devs. The paper found factors which negatively impact dev productivity: tooling and infrastructure(flaky tests, slow builds), process inefficiencies(unclear ownership, changing priorities) and team dynamics. The one thing that improves dev performance is the state of flow. This is where LLMs help a lot. AI suggestions and chat help keep the flow state, answer questions, reduce repetitive tasks, increase progress and eliminate context switches. The primary stopper in adopting coding assistants is a lack of trust. Trust requires results (later about this) and visibility of how the result was achieved. The paper focuses on the last - describing how we can help LLMs establish trust in coding: generate tests, formal proofs, security guardrails and write code with the same intent as the rest of the project. One of the problems for LLM in generating working code from a description is ambiguity. The conducted experiment compared three prompts: full descriptions of a problem, summarized descriptions of a problem with hidden parts and interaction prompts. When we give the model a full description, we expect it to solve a problem for us. When we provide a summarized version, we hope the model will have problems generating code for us. In the last case, we give a model the ability to ask clarifying questions. Claude won this competition. Claude managed to solve more tasks, find ambiguity in a description and ask better questions to clarify the description. The point of the paper is that the model needs a way to ask clarifying questions. LLM solves more problems if they can interact with the user. The paper proposed including this in a prompt and be ready to clarify something for the model. It sounds like the next frontier after Large Language and Reasoning Models is Large Questioning Models.

50

2 Kommentare

🏇 𝗥𝗲𝗱𝘂𝗻𝗱𝗮𝗻𝗰𝘆 𝗜: 𝗟𝗼𝘀𝘁 𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 In German there is a phrase, "the left hand not knowing what the right hand does". Most software has deliberately been designed in a similar fashion. You are calling a function with a known effect but what happens inside is opaque to you. 😓 As a result, (temporary) data / information gathered on one side of the function call tree is not available on the other side. If your string class does not readily contain the length of the string, it's almost a certainty that different parts of the system will call strlen() or something similar over and over again. 💡 There are a few of things you can do to address that situation; all of them can fix "Big-O" algorithmic problems. From easy to hard: ✅ useful naming conventions Use getXYZ() only for O(1), quick access functions. If you need to traverse data, use xyz() for findXYZ() instead. The caller can then make sure to call the function e.g. once instead of N times. ✅ result caching, memoization If inputs/parameters for consecutive calls are related, e.g. in case of path traversal, you may store the latest input (path) and the respective output (graph location) inside your function. Next time the function gets called, check if you can simply continue the traversal from the prior end point. ✅ rich data structures The OOP version of memoization. Alongside the "core" object data, store any frequently requested derived data. E.g. strings should store their length information in some form. ✅ data flow object model, dynamic programming Eliminate the need for temporary data by representing the processing state inside the data /object model itself. Local processing functions are replaced by state transformations. Good examples for this are graph algorithms but you can extend the idea to larger systems. ❓ What are your strategies for dealing with redundant processing? Anything you would like to add? #FiveHorsemen #Redundancy #algorithms #performance

31

6 Kommentare

2nd order effects of GPT5 performance The results of GPT5 has been underwhelming to say the least. Here are the first order effects: 1. People will finalize realize that hardware scaling isn’t enough 2. Coding automation is close to be a solved problem. 3. Massive numbers of SLM is the path towards future progress . It will be messy and slow. There are several second order effects we will see: 1. Reasoning is not a solved problem. VCs should be on the hunt for the next OpenAI that is betting on a non-LLM architecture. My bet is still some variant energy based models that can perform neurosymbolic AI efficiently. 2. OSS will be going after the top of the stack - companies scaling fast due to their software stack (lovable / bolt / etc) will see their revenue erode with new entrants that can build new software for nearly free. We’ve seen OSS disrupt the bottom of the software stack, now it’s going after the top 3. Hardware is going to be the next big venture boom - this is happening for 3 reasons. 1) Software categories that has a network of effect has already transitioned to AI 2) OSS is going to kill all software revenues that does not have some sort of network effect 3) money is flowing to the hardware companies 4. There will be no “lift off”, meaning that humans will be needed to improve AI models over time. AI doomers and hypesters are equally wrong. 5. Policy makers will treat AI like any other technology before it. Namely, they will avoid regulation until they see negative effects on society. 6. Synthetic data is king - whoever can create the best data will win the AI race. Big tech will redirect their massive training clusters to synthetic data generation which will require a drastically different software and hardware stack than today (including quantum processors no doubt) There are massive 2nd order effect opportunities. It’s a great time to be in technology.

50

15 Kommentare