Kubernetes Cluster Management

Here's a decision tree to help developers and admins who manage Microsoft 365, Azure and Entra tenants, pick the most secure authentication method for their apps. Managed Identities The best option for app secrets is to not have any secrets in the first place. This way, you're app credentials can never be leaked. Where possible, always choose to use managed identities in Azure and similar features in other platforms like AWS and GCP. Did you know that with Managed Identities in Azure, no one, including the Global Administrator, can access the underlying credential? With managed identities, identities are provided and deleted with Azure resources. Workload Identity Federation With workload identity federation, you can set up Microsoft Entra ID to trust 3rd-party identity provider tokens from services like AWS, GitHub, and Google Cloud. For example, using this approach, an app running on AWS with an identity in Amazon Cognito can present its Cognito token to Entra ID to access Entra-protected resources like Azure OpenAI or Microsoft Graph.  Certificates Not all services in Azure support Managed Identities yet, and there maybe instances where workload identity federation might not be an option. The next best method is to use certificates for app authentication. When using certificates, you are responsible for using short lived certificate expiry, protecting the private key from being leaked and rotating the cert when it expires (or if there is a compromise). You also need to worry about application outage while the cert is being rotated (unless your devs add support for multiple active keys). Client ID & Secret This is by far the most used app authentication method because of its simplicity. Unfortunately, this is also the reason credentials are leaked quite easily and unintentionally. Dangers include secrets checked into code, secrets left lying around on unencrypted text files, etc. Leaked credentials grant easy access to your data to threat actors. Where possible, help your devs and vendors move off client secrets to using stronger authentication like managed identities and certs. Learn more To learn more about avoiding common app-related security pitfalls, see this recent post by Brian Melton-Grace. https://lnkd.in/gNYz6zZx To learn about workload identity federation, see https://lnkd.in/gqRfC2bZ To learn about using Entra conditional access policies for workload identities, Workload Identity Protection, and setting tenant policies to block the use of secrets, see https://lnkd.in/gqvV5rGY If you found this helpful, please like and share this post with your network! Tx.

I think Red Hat’s launch of 𝗹𝗹𝗺-𝗱 could mark a turning point in 𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲 𝗔𝗜. While much of the recent focus has been on training LLMs, the real challenge is scaling inference, the process of delivering AI outputs quickly and reliably in production. This is where AI meets the real world, and it's where cost, latency, and complexity become serious barriers. 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗡𝗲𝘄 𝗙𝗿𝗼𝗻𝘁𝗶𝗲𝗿 Training models gets the headlines, but inference is where AI actually delivers value: through apps, tools, and automated workflows. According to Gartner, over 80% of AI hardware will be dedicated to inference by 2028. That’s because running these models in production is the real bottleneck. Centralized infrastructure can’t keep up. Latency gets worse. Costs rise. Enterprises need a better way. 𝗪𝗵𝗮𝘁 𝗹𝗹𝗺-𝗱 𝗦𝗼𝗹𝘃𝗲𝘀 Red Hat’s llm-d is an open source project for distributed inference. It brings together: 1. Kubernetes-native orchestration for easy deployment 2. vLLM, the top open source inference server 3. Smart memory management to reduce GPU load 4. Flexible support for all major accelerators (NVIDIA, AMD, Intel, TPUs) AI-aware request routing for lower latency All of this runs in a system that supports any model, on any cloud, using the tools enterprises already trust. 𝗢𝗽𝘁𝗶𝗼𝗻𝗮𝗹𝗶𝘁𝘆 𝗠𝗮𝘁𝘁𝗲𝗿𝘀 The AI space is moving fast. New models, chips, and serving strategies are emerging constantly. Locking into one vendor or architecture too early is risky. llm-d gives teams the flexibility to switch tools, test new tech, and scale efficiently without rearchitecting everything. 𝗢𝗽𝗲𝗻 𝗦𝗼𝘂𝗿𝗰𝗲 𝗮𝘁 𝘁𝗵𝗲 𝗖𝗼𝗿𝗲 What makes llm-d powerful isn’t just the tech, it’s the ecosystem. Forged in collaboration with founding contributors CoreWeave, Google Cloud, IBM Research and NVIDIA and joined by industry leaders AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI and university supporters at the University of California, Berkeley, and the University of Chicago, the project aims to make production generative AI as omnipresent as Linux. 𝗪𝗵𝘆 𝗜𝘁 𝗠𝗮𝘁𝘁𝗲𝗿𝘀 For enterprises investing in AI, llm-d is the missing link. It offers a path to scalable, cost-efficient, production-grade inference. It integrates with existing infrastructure. It keeps options open. And it’s backed by a strong, growing community. Training was step one. Inference is where it gets real. And llm-d is how companies can deliver AI at scale: fast, open, and ready for what’s next.

Microservice architecture has become a cornerstone of modern, cloud-native application development. Let's dive into the key components and considerations for implementing a robust microservice ecosystem: 1. Containerization:    - Essential for packaging and isolating services    - Docker dominates, but alternatives like Podman and LXC are gaining traction    2. Container Orchestration:    - Crucial for managing containerized services at scale    - Kubernetes leads the market, offering powerful features for scaling, self-healing, and rolling updates    - Alternatives include Docker Swarm, HashiCorp Nomad, and OpenShift 3. Service Communication:    - REST APIs remain popular, but gRPC is growing for high-performance, low-latency communication    - Message brokers like Kafka and RabbitMQ enable asynchronous communication and event-driven architectures 4. API Gateway:    - Acts as a single entry point for client requests    - Handles cross-cutting concerns like authentication, rate limiting, and request routing    - Popular options include Kong, Ambassador, and Netflix Zuul 5. Service Discovery and Registration:    - Critical for dynamic environments where service instances come and go    - Tools like Consul, Eureka, and etcd help services locate and communicate with each other 6. Databases:    - Polyglot persistence is common, using the right database for each service's needs    - SQL options: PostgreSQL, MySQL, Oracle    - NoSQL options: MongoDB, Cassandra, DynamoDB    7. Caching:    - Improves performance and reduces database load    - Distributed caches like Redis and Memcached are widely used 8. Security:    - Implement robust authentication and authorization (OAuth2, JWT)    - Use TLS for all service-to-service communication    - Consider service meshes like Istio or Linkerd for advanced security features 9. Monitoring and Observability:    - Critical for understanding system behavior and troubleshooting    - Use tools like Prometheus for metrics, ELK stack for logging, and Jaeger or Zipkin for distributed tracing    10. CI/CD:    - Automate builds, tests, and deployments for each service    - Tools like Jenkins, GitLab CI, and GitHub Actions enable rapid, reliable releases    - Implement blue-green or canary deployments for reduced risk 11. Infrastructure as Code:    - Use tools like Terraform or CloudFormation to define and version infrastructure    - Enables consistent, repeatable deployments across environments Challenges to Consider: - Increased operational complexity - Data consistency across services - Testing distributed systems - Monitoring and debugging across services - Managing multiple codebases and tech stacks Best Practices: - Design services around business capabilities - Embrace DevOps culture and practices - Implement robust logging and monitoring from the start - Use circuit breakers and bulkheads for fault tolerance - Automate everything possible in the deployment pipeline

If you don’t understand how traffic flows, you don’t really understand Kubernetes. In my years working across K8s clusters, one thing became very clear: 90% of the production issues.. live in the network path. Let’s simplify it. North–South traffic (External → Cluster) Client → Load Balancer → Ingress / Gateway API → Service (kube-proxy updates iptables) → Pod IP → CNI → Node Network This is where: • TLS termination happens • L7 routing decisions are made • Cloud load balancers get involved • Security teams start asking questions • eBPF (if using Cilium or advanced CNIs) If your app is “not reachable,” the problem is usually somewhere here. ⸻ East–West traffic (Pod ↔ Pod inside cluster) Pod → Service (ClusterIP) → [kube-proxy rules fire in kernel] → CNI → Target Pod This is where: • Service discovery (CoreDNS) kicks in • NetworkPolicies decide who can talk to whom • Service Mesh adds mTLS, retries, observability • Overlay networking (VXLAN / BGP) does its magic If your microservices randomly timeout, this path is where you look. ⸻ What most beginners miss: • Every Pod gets its own IP • Pods are routable (flat network model) • CNI handles IP allocation + container networking • kube-proxy programs iptables or IPVS rules • The node’s Linux networking stack is doing the heavy lifting Kubernetes networking is not “abstract magic.” It’s Linux networking + smart abstractions. Once you understand: ~ Service abstraction ~ kube-proxy behavior ~ CNI internals ~ Node routing tables You start thinking like a platform engineer ~ understanding how traffic actually moves inside your cluster, not just deploying applications on top of it. If you’re learning Kubernetes, don’t skip networking. It’s the difference between deploying YAML… and actually understanding production systems. What else would you add to this list?

I use this simple 3-step logs flow that helps me debug almost anything in Kubernetes under 30 minutes. 𝗦𝘁𝗲𝗽 1 → kubectl logs <pod> Ask: “Did the app fail inside the container?” If the pod is up, this is your first stop. Look for stack traces, startup errors, misconfigs. But if logs show nothing (or the pod never started), move on fast. 𝗦𝘁𝗲𝗽 2 → kubectl describe pod <pod> Ask: “Did Kubernetes kill the pod?” This one’s underrated. It shows you probe failures, CrashLoops, image pull issues, and mount errors. Basically, if K8s is mad at your pod, this will tell you why. 𝗦𝘁𝗲𝗽 3 → kubectl get events --sort-by=.metadata.creationTimestamp Ask: “What else is breaking in the cluster?” This is your timeline. It shows broader issues: node pressure, CNI problems, preemptions. If the problem isn’t in logs or describe, this one usually holds the clue. This is the exact flow we use inside incident war rooms. ➤ If the pod is running → check logs. ➤ If it’s crashing or pending → check describe. ➤ If you’re still lost → check events. Don’t waste 45 minutes staring at Grafana hoping something makes sense. Start with the logs. Ask better questions. Fix faster. I built a 1-page cheatsheet of this debugging flow. It’s part of our SRE onboarding at Infra360. Want it? Drop a “LOGS” in the comments and I’ll send it to you.

Using unverified container images, over-permissioning service accounts, postponing network policy implementation, skipping regular image scans and running everything on default namespaces…. What do all these have in common ? Bad cybersecurity practices! It’s best to always do this instead; 1. Only use verified images, and scan them for vulnerabilities before deploying them in a Kubernetes cluster. 2. Assign the least amount of privilege required. Use tools like Open Policy Agent (OPA) and Kubernetes' native RBAC policies to define and enforce strict access controls. Avoid using the cluster-admin role unless absolutely necessary. 3. Network Policies should be implemented from the start to limit which pods can communicate with one another. This can prevent unauthorized access and reduce the impact of a potential breach. 4. Automate regular image scanning using tools integrated into the CI/CD pipeline to ensure that images are always up-to-date and free of known vulnerabilities before being deployed. 5. Always organize workloads into namespaces based on their function, environment (e.g., dev, staging, production), or team ownership. This helps in managing resources, applying security policies, and isolating workloads effectively. PS: If necessary, you can ask me in the comment section specific questions on why these bad practices are a problem. #cybersecurity #informationsecurity #softwareengineering

In modern Kubernetes environments, RBAC alone isn’t enough. When you’re managing dozens or hundreds of clusters, you need Policy-as-Code, governance that’s automated, auditable, and scalable. With OPA Gatekeeper and Anthos Config Management (ACM), organizations can now: ✅ Define and enforce security & compliance policies as code. ✅ Sync configurations fleet-wide via GitOps. ✅ Block misconfigurations before they ever reach production. This shift moves us from manual guardrails to continuous, code-driven governance. https://lnkd.in/dsMkux_h RBAC grants access; Policy-as-Code defines responsibility. The result, secure, compliant, and self-healing GKE fleets. #PolicyAsCode #GKE #Kubernetes #Anthos #OPAGatekeeper #CloudGovernance #DevSecOps #GoogleCloud #AIGovernance

𝗧𝗟;𝗗𝗥: MCP has fixed the major authorization flaw I spoke about earlier! 🎉 The new specification addresses enterprise security concerns by implementing proper OAuth 2.x separation of concerns. 𝗘𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲𝘀 𝗰𝗮𝗻 𝗻𝗼𝘄 𝗰𝗼𝗻𝘀𝗶𝗱𝗲𝗿 𝗠𝗖𝗣 𝗱𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 𝘄𝗶𝘁𝗵 𝗰𝗼𝗻𝗳𝗶𝗱𝗲𝗻𝗰𝗲 as the authorization architecture now follows industry best practices. 𝗥𝗲𝗰𝗮𝗽: 𝗪𝗵𝗮𝘁 𝘄𝗮𝘀 𝘁𝗵𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺? In my earlier post (https://bit.ly/40fFoIW), I highlighted that MCP forced servers to function as both resource servers AND authorization servers—violating OAuth best practices and creating enterprise integration nightmares. 𝗪𝗵𝗮𝘁'𝘀 𝗯𝗲𝗲𝗻 𝗳𝗶𝘅𝗲𝗱? The latest specification (https://bit.ly/3G5cK6A) completely restructures authorization:  • MCP Servers no longer handle authentication or issue tokens  • Dedicated Authorization Server handles user auth and token issuance  • RFC9728 enables dynamic authorization server discovery 𝗞𝗲𝘆 𝗶𝗺𝗽𝗿𝗼𝘃𝗲𝗺𝗲𝗻𝘁𝘀:  • Proper separation of concerns  • Enterprise identity integration  • Dynamic client registration  • Resource binding with RFC 8707 𝗡𝗲𝘄 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲: Clean three-tier design:  • MCP Client (OAuth 2.x client)  • Authorization Server (auth/tokens)  • MCP Server (resource server) 𝗪𝗵𝗮𝘁 𝗰𝗮𝗻 𝘆𝗼𝘂 𝗱𝗼 𝗻𝗼𝘄?  • Start planning MCP pilots  • Review the updated specification  • Leverage existing identity infrastructure  • Continue with Amazon Web Services (AWS) solutions Kudos to the MCP community for responsive development and taking security feedback seriously! As always, consult with your security teams.

Agent architectures in the Real World ⬇️ 𝗪𝗵𝗮𝘁'𝘀 𝘁𝗵𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺? There is plenty of advice on how to build agent prototypes that > use third-party API, like OpenAI or Antrhopic. > encapsulate all the agent + tooling logic inside a single Python program > run locally with docker compose But the problem is, this design DOES NOT scale. Meaning, if you are a company trying to use this blueprint, it will either work > too slow > too expensive, or > BOTH So the question is, how can you design agentic systems that are cost efficient (both in time and money?). Here is a blueprint. 𝗦𝘆𝘀𝘁𝗲𝗺 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 📐 > 𝗔𝗴𝗲𝗻𝘁 𝗔𝗣𝗜 𝗴𝗮𝘁𝗲𝘄𝗮𝘆 Routes incoming requests to the appropriate agent using a lightweight LLM. > 𝗔𝗴𝗲𝗻𝘁 𝗹𝗼𝗴𝗶𝗰 𝘀𝗲𝗿𝘃𝗶𝗰𝗲𝘀 A task-specific agent gets the task, and uses a step-wise workflow (e.g. Langgraph, CrewAI, Langchain...) that invokes one or several LLMs, and a set of external tools. > 𝗧𝗼𝗼𝗹 𝘀𝗲𝗿𝘃𝗲𝗿𝘀 They act as an interface between your agents and the backend services these agents need to solve the task. Here you can use MCP clients and servers, and a library like FastMCP. > 𝗟𝗟𝗠 𝘀𝗲𝗿𝘃𝗲𝗿𝘀 They need to run on dedicated GPU nodes, using tools like vLLM or NVIDIA NIM. Every service is running as a containerised app in your Kubernetes cluster. BOOM! 𝗪𝗮𝗻𝗻𝗮 𝗹𝗲𝗮𝗿𝗻 𝗺𝗼𝗿𝗲 𝗿𝗲𝗮𝗹 𝘄𝗼𝗿𝗹𝗱 𝗟𝗟𝗠𝗢𝗽𝘀? In the next weeks Marius Rugan and I will dig deeper into LLMOps system architecture. In public. For FREE. Follow Pau Labarta Bajo so you don't miss what is coming next