helm charts for deploying models with llm-d

llm-d helm charts and deployment examples

Distributed KV cache scheduling & offloading libraries

A light weight vLLM simulator, for mocking out replicas.

Inference scheduler for llm-d

llm-d is a Kubernetes-native high-performance distributed LLM inference framework

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

NVIDIA Inference Xfer Library (NIXL)

A high-throughput and memory-efficient inference and serving engine for LLMs

Variant optimization autoscaler for distributed inference workloads

A K8s operator to reduce CO2 footprint of your clusters

Supercharge Your LLM with the Fastest KV Cache Layer

🎒 Token-Oriented Object Notation (TOON) – Compact, human-readable, schema-aware JSON for LLM prompts. Spec, benchmarks, TypeScript SDK.

Collection of Architectural Decision Records

Release tooling for OpenShift

community-operators metadata backing OpenShift OperatorHub

Composable building blocks to build Llama Apps

Install an OpenShift 4.x cluster

odh qe tier tests

🚀 Modern cross-platform HTTP load-testing tool written in Go

Controller for ModelMesh

AI-on-OpenShift website source code

TrustyAI Explainability Toolkit

A toolkit to run Ray applications on Kubernetes