Best Practices for API Development

APIs aren't just endpoints for data engineers - they're the lifelines of your entire data ecosystem. Choosing the Right API Architecture Can Make or Break Your Data Pipeline. As data engineers, we often obsess over storage formats, orchestration tools, and query performance—but overlook one critical piece: API architecture. APIs are the arteries of modern data systems. From real-time streaming to batch processing - every data flow depends on how well your APIs handle the load, latency, and reliability demands. 🔧 Here are 6 API styles and where they shine in data engineering: 𝗦𝗢𝗔𝗣 – Rigid but reliable. Still used in legacy financial and healthcare systems where strict contracts matter. 𝗥𝗘𝗦𝗧 – Clean and resource-oriented. Great for exposing data services and integrating with modern web apps. 𝗚𝗿𝗮𝗽𝗵𝗤𝗟 – Precise data fetching. Ideal for analytics dashboards or mobile apps where over-fetching is costly. 𝗴𝗥𝗣𝗖 – Blazing fast and compact. Perfect for internal microservices and real-time data processing. 𝗪𝗲𝗯𝗦𝗼𝗰𝗸𝗲𝘁 – Bi-directional. A must for streaming data, live metrics, or collaborative tools. 𝗪𝗲𝗯𝗵𝗼𝗼𝗸 – Event-driven. Lightweight and powerful for triggering ETL jobs or syncing systems asynchronously. 💡 The right API architecture = faster pipelines, lower latency, and happier downstream consumers. As a data engineer, your API decisions don’t just affect developers—they shape the entire data ecosystem. 🎯 Real Data Engineering Scenarios to explore: Scenario 1: 𝗥𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗙𝗿𝗮𝘂𝗱 𝗗𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻 Challenge: Process 100K+ transactions/second with <10ms latency Solution: gRPC for model serving + WebSocket for alerts Impact: 95% faster than REST-based approach Scenario 2: 𝗠𝘂𝗹𝘁𝗶-𝘁𝗲𝗻𝗮𝗻𝘁 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺 Challenge: Different customers need different data subsets Solution: GraphQL with smart caching and query optimization Impact: 70% reduction in database load, 3x faster dashboard loads Scenario 3: 𝗟𝗲𝗴𝗮𝗰𝘆 𝗘𝗥𝗣 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 Challenge: Extract financial data from 20-year-old SAP system Solution: SOAP with robust error handling and transaction management Impact: 99.9% data consistency vs. 85% with custom REST wrapper Image Credits: Hasnain Ahmed Shaikh Which API style powers your pipelines today? #data #engineering #bigdata #API #datamining

AI security/securing the use of AI is going to kill me. I use Claude Code almost daily. It's a problem.... Here's what I have to change AGAIN this week. Security researcher Ari Marzuk disclosed 30+ vulnerabilities across AI coding tools. Cursor. GitHub Copilot. Windsurf. Claude Code. All of them. He called it IDEsaster. The attack chain includes prompt injection, hijacking LLM context, and auto-approved tool calls executing without permission. Then, legitimate IDE features are weaponized for data exfiltration and RCE. Your .env files. Your API keys. Your source code. Accessible through features you thought were safe. Most studies I read claim that around 85% of developers now use AI coding tools daily. Most have no idea their IDE treats its own features as inherently trusted. 𝗦𝗼... 𝗮𝗳𝘁𝗲𝗿 𝗿𝗲𝘃𝗶𝗲𝘄𝗶𝗻𝗴 𝗔𝗿𝗶'𝘀 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵, 𝗵𝗲𝗿𝗲'𝘀 𝗜 𝘄𝗶𝗹𝗹 𝗯𝗲 𝗱𝗼𝗶𝗻𝗴... Be warned: All this is SO much easier said than done! Audit every MCP server connection. Checked for tool poisoning vectors where legitimate tools might parse attacker-controlled input from GitHub PRs or web content. Removed servers I couldn't verify. Disabled auto-approve for file writes. The attack chains weaponize configuration files and project instructions like .claude/settings.json and CLAUDE.md. One malicious write to these files can alter agent behavior or achieve code execution without additional user interaction. Move all credentials to a secrets manager. No .gitignored .env files in agent-accessible directories. API keys live in 1Password CLI. Environment variables inject at runtime through a wrapper script the LLM never sees. Start running Claude Code in isolated containers. Mounted volumes limited to specific project directories. No access to ~/.ssh, ~/.aws, or ~/.config. If the agent gets compromised, blast radius stays contained. Enable all security warnings. Claude Code added explicit warnings for JSON schema exfiltration and settings file modifications. These exist because Anthropic knows the attack surface. Add pre-commit hooks for hidden characters. Prompt injections hide in pasted URLs, READMEs, and file names using invisible Unicode. Flag non-ASCII characters in any file the agent might ingest. The fix isn't to stop using AI coding tools. The fix is to stop trusting them implicitly. What controls do you have for AI tools with write access to your codebase? 👉 Follow for more AI and cybersecurity insights with the occasional rant #AISecurity #DevSecOps

My Claude quota was running out every single day. Mid-project. Mid-thought. Mid-code. And after Opus 4.7 dropped on April 16, it got significantly worse. So I did not just complain about it . I researched it at a deep, technical level. Then I built something. I created a comprehensive, step-by-step mind map on how to never hit Claude's limits again. It is still in draft, but the response I have gotten privately has been overwhelming — so I want to share it with this community. Here is what it covers: Model Strategy — when to use Haiku vs. Sonnet vs. Opus. Haiku should handle 70–80% of your day. Most people are burning Opus on tasks a fraction-of-the-cost model handles just as well. 5 Core Habits — edit instead of stacking messages, reset every 15 turns, batch your requests into one prompt, keep web search and artifacts off by default, and spread work across sessions rather than one exhausting marathon run. Workflow Upgrades — plan before you type, outline before you draft, and make targeted edits only. Do not paste 500 lines of code when only 40 are broken. Set Once, Forget — store your role and preferences in Claude Memory, upload documents into Projects so they are cached, and write a CLAUDE. md file that eliminates re-explaining yourself at the start of every session. Daily Checklist — the exact micro-habits that keep your limit alive from morning to night. The single most important thing most people still do not know: Claude counts tokens, not messages. Once that distinction clicks, everything changes. This guide is still in draft mode — but I will post the full version next week if there is enough interest. If this gave you even one useful insight today, consider sharing it with someone who has been hitting that wall. They will thank you.

"Maximizing the value of your Claude Code sessions" Agentic coding tools like Claude Code are incredible, but are you getting the most value out of your tokens? Anthropic's Lydia Hallie just dropped a fantastic guide on how to run efficient Claude Code sessions: https://lnkd.in/gujXdtBJ It turns out that fixing the exact same bug can cost completely different amounts depending on your session hygiene. Head's up: I do anticipate Claude will do more of this for you at some point, but until then: Here is how to optimize your workflow and stop wasting context: 1. 🧹 Run /clear between tasks: Don't drag old context into a new problem. You're paying to re-read it on every single turn! 2. ⚙️ Set your /model and /effort upfront: Changing these mid-conversation busts your prompt cache, forcing a full-price prefill of your entire session. 3. 📎 @-mention files directly: Instead of typing out file paths, tagging them attaches the file to your message immediately, saving Claude a roundtrip "Read" call. 4. 🤫 Keep commands quiet: Add quiet flags to noisy terminal commands (like test runners) or run them in a subagent. Massive log outputs get permanently added to your conversation history. 5. 🔍 Audit your /context: Run this in a fresh session to see exactly what's loaded from your CLAUDE.md or MCP tools, and trim the excess. 6. 📦 /compact before you step away: Prompt caches expire (after an hour on a subscription, or just 5 minutes on an API key). Summarizing your conversation is significantly cheaper while the cache is still warm. The main takeaway? Being efficient with tokens doesn't mean using fewer of them - it means ensuring every token goes toward the problem you're actually trying to solve #ai #programming #softwareengineering

Banks’ biggest tech challenge isn’t upgrading legacy systems- it’s integrating an entirely new (Gen)AI layer with orchestration in the lead. And making it work across functions. Too many banks often start with the wrong focus. Whereas dealing with legacy infrastructure is inevitable, it can become a blind spot without the right understanding of what it needs to achieve. Delivering agile, intelligent services that anticipate customer needs should be the goal. Here is a high-level overview of how the back end can be adjusted: 𝟭. 𝗢𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻 𝗲𝗻𝗴𝗶𝗻𝗲: -   An orchestration layer sits atop core systems, routing everything - from customer questions to fraud alerts - to the right AI service. -   Modern APIs abstract legacy systems into modular services, so AI features can be added or swapped without changing existing workflows. 𝟮. 𝗥𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲: -   Real-time data feeds stream transactions, balance changes and logins as they happen. -   A unified data hub brings together customer details, activity patterns and risk ratings so every AI tool works from the same information. 𝟯. 𝗗𝗮𝘁𝗮-𝗱𝗿𝗶𝘃𝗲𝗻 𝗶𝗻𝘀𝗶𝗴𝗵𝘁𝘀: -   Requests are automatically enriched with live account balances, recent transactions and open support tickets - ensuring the AI’s output reflects up-to-date information. -   Data is fetched on demand from indexed records, so the AI stays current without the expense of retraining the entire model for every update. 𝟰. 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 & 𝗴𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲: -   Data stays encrypted end-to-end, from intake to AI output. -   Automated audits flag bias and log every decision. -   Failure simulations uncover hidden risks before they impact customers. 𝟱. 𝗠𝗼𝗱𝘂𝗹𝗮𝗿 𝘀𝗲𝘁-𝘂𝗽: -   Modern interfaces turn core banking, payment and CRM systems into plug-and-play modules. -   Behind the scenes, back-end services can be updated piece by piece without interrupting the AI layer. 𝟲. 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗲𝗱 𝗱𝗲𝗹𝗶𝘃𝗲𝗿𝘆 𝘁𝗲𝗮𝗺𝘀: -   Small, cross-functional teams manage everything from data ingestion to model deployment and monitoring. -   Defined roles and fast feedback loops keep projects compliant and focused on real customer needs. The GenAI layer doesn’t just sit on top of the existing setup – it’s a complete overhaul of the tech architecture and the business logic behind it. Opinions: my own, Graphic source: BCG 𝐒𝐮𝐛𝐬𝐜𝐫𝐢𝐛𝐞 𝐭𝐨 𝐦𝐲 𝐧𝐞𝐰𝐬𝐥𝐞𝐭𝐭𝐞𝐫: https://lnkd.in/dkqhnxdg

𝐈 𝐮𝐬𝐞𝐝 𝐭𝐨 𝐛𝐞𝐥𝐢𝐞𝐯𝐞 𝐂𝐎𝐑𝐒 𝐩𝐫𝐨𝐭𝐞𝐜𝐭𝐬 𝐨𝐮𝐫 𝐁𝐚𝐜𝐤𝐞𝐧𝐝. Until a developer with solid technical knowledge asked me: “If CORS protects your backend, how can Postman call your API, even without any CORS setup ?” That question hit me. We never configure Postman in our CORS settings, yet it works. 𝗪𝗵𝘆 ? Like many developers, I had misunderstood what CORS actually does. We’ve all seen those CORS errors in the browser and often just Googled a fix without really understanding the reason. --- A few months ago, I decided to dig deeper. Here’s a simple but a good example that helped me understand: Imagine this: You open your browser and visit your bank’s website — bank[dot]com — and log in. The frontend makes API calls to bank-api[dot]com, and your browser stores cookies. Then, you open a new tab and visit a random site: unknown-site[dot]com. That site runs this script: fetch(bank-api[dot]com/transfer?to=123&amount=3000, {   credentials: "include" }); Since you're already logged in to bank[dot]com, your browser has the cookies. And because credentials: "include" is used, the browser tries to attach those cookies to the request. The backend receives it and thinks: “This request is from an authenticated user, go ahead and transfer the money!” 💸 Boom. Money gone. --- So how does CORS help ? Because the request is coming from a different origin, the browser says: “Hey backend, this is from unknown-site[dot]com. Should I allow it ?” Your backend responds with: Access-Control-Allow-Origin: bank[dot]com Now, the browser checks and blocks the request, since unknown-site[dot]com isn't allowed. ✅ CORS prevents cross-origin requests from untrusted sites. ❌ But it doesn’t protect your backend directly — the browser enforces it, not your server. That’s why Postman works It’s not a browser, so CORS doesn’t apply. In fact, if a user installs a browser extension that disables CORS, a malicious site could bypass CORS entirely. All of this started with one curious question. It reminded me of something important: 🧠 𝐂𝐮𝐫𝐢𝐨𝐬𝐢𝐭𝐲 𝐢𝐬 𝐨𝐮𝐫 𝐬𝐮𝐩𝐞𝐫𝐩𝐨𝐰𝐞𝐫. It helps us ask why, not just how and that’s how real learning begins. #websecurity #CORS #backend #frontend #JavaScript #devlife #curiosity #learning

I remember a junior developer on my team once came to me, frustrated that an API call was taking way too long — around 19 seconds in total. He was using async/await and assumed it was automatically the most efficient approach. But when I looked at the code, I immediately spotted the issue: const res1 = await fetchA(); const res2 = await fetchB(); const res3 = await fetchC(); Each call was waiting for the previous one to finish — completely sequential. No wonder it was slow. I suggested a small change that made a huge impact: const [res1, res2, res3] = await Promise.all([ fetchA(), fetchB(), fetchC(), ]); Just like that, the total response time dropped from 19 seconds to just 90ms. All because the API calls started running in parallel, not one after the other. Key takeaway: Using async/await doesn’t automatically mean your code is fast. When your async calls don’t depend on each other, Promise.all() can make a night-and-day difference. I love moments like these — small tweaks, big wins. Have you ever made (or fixed) a similar async/await mistake? Would love to hear! #FreelanceDeveloper #FullStackDeveloper #JavaScriptDeveloper #ReactJS #AngularDeveloper #NodeJS #WebAppDevelopment #RemoteDeveloper #TechTips

A Cheatsheet to Build Secure APIs An insecure API can compromise your entire application. Follow these strategies to mitigate the risk: 1 - Using HTTPS Encrypts data in transit and protects against man-in-the-middle attacks. This ensures that data hasn’t been tampered with during transmission. 2 - Rate Limiting and Throttling Rate limiting prevents DoS attacks by limiting requests from a single IP or user. The goal is to ensure fairness and prevent abuse. 3 - Validation of Inputs Defends against injection attacks and unexpected data format. Validate headers, inputs, and payload 4 - Authentication and Authorization Don’t use basic auth for authentication. Instead, use a standard authentication approach like JWTs Use a random key that is hard to guess as the JWT secret Make token expiration short For authorization, use OAuth 5 - Using Role-based Access Control RBAC simplifies access management for APIs and reduces the risk of unauthorized actions. Granular control over user permission based on roles. 6 - Monitoring Monitoring the APIs is the key to detecting issues and threats early. Use tools like Kibana, Cloudwatch, Datadog, and Slack for monitoring Don’t log sensitive data like credit card info, passwords, credentials, etc. Over to you: What else would you do to build a secure API? -- Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/bbg-social #systemdesign #coding #interviewtips .

How do we design effective and safe APIs? APIs have increasingly become the backbone of modern software. 𝗧𝗼 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 some of the 𝗸𝗲𝘆 𝗽𝗿𝗶𝗻𝗰𝗶𝗽𝗹𝗲𝘀 and 𝗯𝗲𝘀𝘁 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀 𝗼𝗳 𝗔𝗣𝗜 𝗱𝗲𝘀𝗶𝗴𝗻, Let's 𝗮𝗻𝗮𝗹𝘆𝘇𝗲 𝗮 𝘀𝗼𝗰𝗶𝗮𝗹 𝗺𝗲𝗱𝗶𝗮 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺 𝗲𝘅𝗮𝗺𝗽𝗹𝗲: 🔹 𝗥𝗲𝘀𝗼𝘂𝗿𝗰𝗲 𝗻𝗮𝗺𝗶𝗻𝗴 Clarity is key when creating APIs. Adopting simple resource names, like /users for accessing user profiles and /posts for retrieving user posts, streamlines the development process and reduces mental strain. 🔹 𝗨𝘀𝗲 𝗼𝗳 𝗽𝗹𝘂𝗿𝗮𝗹𝘀 It's important to maintain a standard of consistency in API design. For consistency and readability, use plural resource names, such as GET /users/{userId}/friends vs. /friend), to avoid ambiguity in API requests. 🔹 𝗖𝗿𝗼𝘀𝘀-𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗶𝗻𝗴 𝗿𝗲𝘀𝗼𝘂𝗿𝗰𝗲𝘀 Interlinking resources, like taking comments on a post using GET /posts/{postId}/comments, simplifies the retrieval of related data. It provides a more streamlined and well-organized user experience. 🔹 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 It goes without saying, security is a must-have. To secure the API endpoints, employ authentication methods like X-AUTH-TOKEN and X-SIGNATURE, and use authorization headers for verifying user permissions. Learn more about API security here: https://lnkd.in/g-uJqhvc 🔹 𝗩𝗲𝗿𝘀𝗶𝗼𝗻𝗶𝗻𝗴 Using versioning and communicating version updates is another important practice. Endpoints like GET /v2/users/{userId}/posts allow API versioning to maintain functionality regardless of updates. This approach ensures backward compatibility and a smooth transition for users and us. Learn more about API versioning here: https://lnkd.in/g9sSaaMt 🔹 𝗣𝗮𝗴𝗶𝗻𝗮𝘁𝗶𝗼𝗻 This technique is important for performance. Paginate large datasets, like feeds or comment lists, with GET /posts?page=5&pageSize=20 to enhance data delivery and UX. 🔹 𝗜𝗱𝗲𝗺𝗽𝗼𝘁𝗲𝗻𝗰𝘆 Maintaining API reliability is necessary. Idempotency ensures that operations like profile updates (PUT /users/{userId}/profile) achieve their intended result, regardless of how often they are executed. These practices are very important, but there’s still much more to API design. Learn more about API design here: https://lnkd.in/gjSsCDRy Thorough documentation, robust monitoring and logging, and consistent error handling are just a few more of the many essential habits required for designing effective and safe APIs. Adopting these principles and practices enables us to develop secure and performant APIs that deliver good user experiences. P.S. If you like this post, then you'll love our newsletter. Subscribe here: https://lnkd.in/giQj3Z44

You've built your AI agent... but how do you know it's not failing silently in production? Building AI agents is only the beginning. If you’re thinking of shipping agents into production without a solid evaluation loop, you’re setting yourself up for silent failures, wasted compute, and eventully broken trust. Here’s how to make your AI agents production-ready with a clear, actionable evaluation framework: 𝟭. 𝗜𝗻𝘀𝘁𝗿𝘂𝗺𝗲𝗻𝘁 𝘁𝗵𝗲 𝗥𝗼𝘂𝘁𝗲𝗿 The router is your agent’s control center. Make sure you’re logging: - Function Selection: Which skill or tool did it choose? Was it the right one for the input? - Parameter Extraction: Did it extract the correct arguments? Were they formatted and passed correctly? ✅ Action: Add logs and traces to every routing decision. Measure correctness on real queries, not just happy paths. 𝟮. 𝗠𝗼𝗻𝗶𝘁𝗼𝗿 𝘁𝗵𝗲 𝗦𝗸𝗶𝗹𝗹𝘀 These are your execution blocks; API calls, RAG pipelines, code snippets, etc. You need to track: - Task Execution: Did the function run successfully? - Output Validity: Was the result accurate, complete, and usable? ✅ Action: Wrap skills with validation checks. Add fallback logic if a skill returns an invalid or incomplete response. 𝟯. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗲 𝘁𝗵𝗲 𝗣𝗮𝘁𝗵 This is where most agents break down in production: taking too many steps or producing inconsistent outcomes. Track: - Step Count: How many hops did it take to get to a result? - Behavior Consistency: Does the agent respond the same way to similar inputs? ✅ Action: Set thresholds for max steps per query. Create dashboards to visualize behavior drift over time. 𝟰. 𝗗𝗲𝗳𝗶𝗻𝗲 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 𝗠𝗲𝘁𝗿𝗶𝗰𝘀 𝗧𝗵𝗮𝘁 𝗠𝗮𝘁𝘁𝗲𝗿 Don’t just measure token count or latency. Tie success to outcomes. Examples: - Was the support ticket resolved? - Did the agent generate correct code? - Was the user satisfied? ✅ Action: Align evaluation metrics with real business KPIs. Share them with product and ops teams. Make it measurable. Make it observable. Make it reliable. That’s how enterprises scale AI agents. Easier said than done.