Hardened LLM Stack with RAG & ML Training Pipeline

A production-ready Docker Compose deployment of a RAG (Retrieval-Augmented Generation) system with local or remote LLM inference, featuring comprehensive security hardening, TLS encryption, and flexible model configuration. The repository also includes a full ML training pipeline for generating synthetic domain-specific training data and fine-tuning embedding and reranking models.

---

Quick Start (5 minutes)

1. Prerequisites

• Docker & Docker Compose 3.9+
• NVIDIA GPU with CUDA support (for GPU acceleration)
• 16GB+ RAM, 100GB+ storage (32GB+ RAM recommended for model training)
• Port 3000 available (Web UI)

2. Setup

# Clone or download this deployment
cd /path/to/deployment

# Generate TLS certificates (required once)
./generate-certs.sh

# Copy environment template
cp example.env .env

# Edit .env with your settings
nano .env
# - Generate strong API keys: openssl rand -base64 32
# - Adjust GPU device IDs: nvidia-smi
# - Set model source (local or remote)

# Start services
docker compose up -d

# Wait for services to be healthy (~2-3 minutes)
docker compose ps

# Open Web UI
open http://localhost:3000

3. First Use

1. Open WebUI: http://localhost:3000
2. Configure LLM:

• Settings Models Add model
• For local: Select "model" (llama.cpp)
• For remote: Add API key and select provider model

3. Try RAG:

• Upload a document (if Docling enabled)
• Ask questions about it
• System will search, embed, and rerank documents

---

What's Included

Core Services

Service	Purpose	Status
Open WebUI	Web interface with RAG	Always running
Qdrant	Vector database for embeddings	Always running
vLLM Embedding	Embedding model — local file or HuggingFace Hub	Always running
vLLM Reranker	Reranking model — local file or HuggingFace Hub	Always running
SearXNG	Privacy-respecting metasearch	Always running
Redis	Cache for SearXNG	Always running
Llama.cpp	Local GGUF LLM inference (optional)	Profile: local-model
Docling	Document parsing & extraction (optional)	Profile: content-extraction
Unsloth	Jupyter + SSH environment for model fine-tuning (optional)	Profile: refine
SnapD Sandbox	Code execution environment for SnapD/Ubuntu Core (optional)	Profile: snapd_sandbox
STM32 Dev	Embedded development sandbox — STM32U5/N6, ThreadX, CMSIS (optional)	Profile: stm32_sandbox

Security Features

Non-root containers - All services run as dedicated users Dropped capabilities - Minimal required privileges only Read-only filesystems - Prevents malicious modifications TLS encryption - HTTPS between all critical services Private network - Only Web UI exposed to host Environment secrets - All keys in .env (not in code) Short JWT expiry - Tokens regenerated every 24 hours Resource limits - Memory/CPU caps per service Health checks - Automatic detection of failures Log rotation - Prevents disk space exhaustion

Note on vLLM: vLLM services (embedding & reranker) run as root due to NCCL library requirements documented in official vLLM documentation. This is mitigated by internal-only network isolation and capability dropping. See HARDENING.md for details.

---

Configuration Files

Essential Files

File	Purpose	Required
compose.yaml	Main service definitions	Yes
compose.swarm.yaml	Docker Swarm overlay (multi-node deployment)	Swarm only
compose.nvidia.yaml	NVIDIA GPU overlay	GPU only
compose.amd.yaml	AMD GPU overlay	GPU only
compose.cpu.yaml	CPU-only overlay	CPU only
compose.debug.yaml	Debug/development overrides	No
example.env	Configuration template	Yes (copy to .env)
generate-certs.sh	TLS certificate generation	Yes (run once)
setup.sh	Full first-run setup (certs + volume ownership)	Yes (run once)
swarm-setup.sh	Docker Swarm initialisation and secret creation	Swarm only

Documentation Files

File	Purpose	Read When
README.md	This file — overview & quick start	First time
MODEL_CONFIGURATION.md	Inference, embedding, and reranker model setup	Before first use
HARDENING.md	Detailed security measures	Planning production

Your Files

Directory	Purpose
./certs/	TLS certificates (created by generate-certs.sh)
./sandbox/	Code execution workspace
./searxng/	SearXNG configuration
.env	Your deployment configuration

---

Configuration Guide

1. Set LLM Inference Source

Option A: Local Inference (no API costs, requires GPU)

INFERENCE_MODEL_SOURCE=local
INFERENCE_PROFILE=local-model
LOCAL_MODEL_PATH=/home/username/models
LOCAL_MODEL_FILENAME=model.gguf

Start with: docker compose --profile local-model up -d

Option B: Remote Inference (uses external API, costs per token)

INFERENCE_MODEL_SOURCE=remote
ANTHROPIC_API_KEY=sk-ant-v0-...
INFERENCE_API_KEY=${ANTHROPIC_API_KEY}

Start with: docker compose up -d (llama-local not included)

2. Set Embedding & Reranker Model Source

Each vLLM service accepts either a local path or a HuggingFace Hub repo ID:

Option A: Local file

EMBEDDING_MODEL=/data
EMBEDDING_LOCAL_PATH=/home/username/Qwen3-Embedding-4B_4Bit
EMBEDDING_QUANTIZATION=bitsandbytes

RERANKING_MODEL=/data
RERANKING_LOCAL_PATH=/home/username/Qwen3-Reranker-0.6B
RERANKING_QUANTIZATION=bitsandbytes

Option B: HuggingFace Hub (public or private)

HF_TOKEN=hf_...        # required for private repos

EMBEDDING_MODEL=Rnfudge/snapd-embedder-v1
EMBEDDING_QUANTIZATION=bitsandbytes   # or awq/gptq, or comment out for full-precision

RERANKING_MODEL=Rnfudge/snapd-reranker-v1
RERANKING_QUANTIZATION=bitsandbytes

EMBEDDING_MODEL_NAME and RERANKING_MODEL_NAME are the stable aliases served to Open WebUI — change these only if you want a different name in the UI.

See MODEL_CONFIGURATION.md for detailed setup.

3. Generate Security Keys

# Generate strong random keys
for key in WEBUI QDRANT SEARXNG DOCLING LLAMA UNSLOTH; do
echo "${key}_KEY=$(openssl rand -base64 32)"
done

Copy outputs to .env file.

4. Configure GPU Assignment

# Check available GPUs
nvidia-smi

# In .env, assign devices
GPU_DEVICE_WEBUI=0
GPU_DEVICE_EMBEDDING=0
GPU_DEVICE_RERANKER=1
GPU_DEVICE_QDRANT=0
GPU_DEVICE_INFERENCE=0
GPU_DEVICE_DOCLING=1

5. Adjust Resource Limits

# For powerful workstation (16 cores, 64GB RAM)
WEBUI_MEMORY_LIMIT=16G
WEBUI_CPU_LIMIT=8.0
QDRANT_MEMORY_LIMIT=20G
INFERENCE_MEMORY_LIMIT=30G

# For laptop (4 cores, 16GB RAM)
WEBUI_MEMORY_LIMIT=4G
WEBUI_CPU_LIMIT=2.0
QDRANT_MEMORY_LIMIT=4G
INFERENCE_MEMORY_LIMIT=8G

---

Deployment

Standard Deployment (Recommended)

# Start all core services
docker compose up -d

# Verify health
docker compose ps

# Follow logs during startup
docker compose logs -f

With Optional Services

# Enable document processing
docker compose --profile content-extraction up -d

# Enable model fine-tuning
docker compose --profile refine up -d

# Enable code execution sandbox
docker compose --profile snapd_sandbox up -d

# Enable STM32 embedded development
docker compose --profile stm32_sandbox up -d

# Enable everything at once
docker compose \
--profile content-extraction \
--profile refine \
--profile snapd_sandbox \
--profile stm32_sandbox \
up -d

Verification

# Check all services are healthy
docker compose ps

# Test vector database
docker compose exec open-webui \
curl -kf https://qdrant:6333/health

# Test embeddings
docker compose exec open-webui \
curl -kf https://vllm-embedding:8000/health

# View logs
docker compose logs -f open-webui

Swarm Deployment (Multi-Node)

For deploying across multiple machines with different GPU/power profiles:

# 1. Initialise Docker Swarm on the manager node
docker swarm init

# 2. Run the setup script (interactive or non-interactive)
./swarm-setup.sh

# Non-interactive example (one GPU per role):
./swarm-setup.sh \
  --orchestrator manager-node \
  --gpu-inference gpu-node-1 \
  --gpu-retrieval gpu-node-2 \
  --sandbox cpu-node

# 3. Join worker nodes (run on each worker)
docker swarm join --token <token> <manager-ip>:2377

# 4. Deploy with Swarm overlay
docker compose -f compose.yaml -f compose.swarm.yaml --profile retrieval up -d

Node roles and service placement:

Role	Services	Typical Hardware
orchestrator	Open WebUI, SearXNG, Redis, Qdrant	CPU, minimal GPU
gpu-inference	llama-local, llama-orchestrator, Docling, Unsloth	NVIDIA/AMD GPU
gpu-retrieval	vLLM embedding, vLLM reranker	NVIDIA/AMD GPU
sandbox	SnapD sandbox, STM32 sandbox	CPU only

Key differences from single-host deployment:

• TLS certificates are distributed as Docker secrets (not bind-mounts) — swarm-setup.sh creates them from ./certs/
• Init containers (open-webui-init, qdrant-init) are disabled in Swarm; their logic runs as entrypoint wrappers inside the main container
• Network uses an encrypted overlay (llm-swarm-net) instead of a bridge network
• Model paths (EMBEDDING_LOCAL_PATH, LOCAL_MODEL_PATH, etc.) must exist on the node where the service is scheduled — placement constraints ensure this
• GPU overlays still apply: add -f compose.nvidia.yaml or -f compose.amd.yaml alongside the Swarm overlay

Single-node Swarm (all services on one machine):

docker node update --label-add llm_role=orchestrator \
  --label-add llm_role=gpu-inference \
  --label-add llm_role=gpu-retrieval \
  --label-add llm_role=sandbox \
  $(docker node ls -q)

docker compose -f compose.yaml -f compose.swarm.yaml --profile retrieval up -d

---

Usage

Web Interface

1. Open http://localhost:3000
2. Configure LLM model (Settings Models)
3. Upload documents (if extraction enabled)
4. Ask questions (system automatically searches/embeds/reranks)

Command Line

# View service status
docker compose ps

# Follow logs
docker compose logs -f service-name

# Execute in container
docker compose exec service-name bash

# Stop services
docker compose down

# Stop and remove volumes
docker compose down --volumes --remove-orphans

Testing Inter-Service Communication

# Test Qdrant connectivity with TLS
docker compose exec open-webui \
curl -k --cacert /etc/ssl/certs/local-ca.pem \
https://qdrant:6333/health

# Test embedding service
docker compose exec open-webui \
curl -kf https://vllm-embedding:8000/health

# Test SearXNG
docker compose exec open-webui \
curl -f http://searxng:8080/healthz

---

Security & Best Practices

Before Production

• Change all default secrets in .env
• Review GPU device assignments for your hardware
• Verify TLS certificates generated: ls -la ./certs/
• Test inter-service connectivity (see above)
• Set up log aggregation/monitoring
• Configure backups for volumes
• Test disaster recovery (restore from backup)
• Review HARDENING.md for compliance

Regular Maintenance

# Monthly: Update images
docker compose pull
docker compose up -d

# Quarterly: Rotate secrets
openssl rand -base64 32  # Generate new values
# Update .env and restart

# Quarterly: Renew certificates
./generate-certs.sh
docker compose restart qdrant vllm-embedding vllm-reranker

# As needed: Cleanup orphans
docker compose down --remove-orphans
docker volume prune -f
docker system prune -a

Monitoring

# Resource usage
docker stats

# Service logs
docker compose logs --follow service-name

# Volume usage
docker volume ls
du -sh docker-volumes-path

# Network connectivity
docker compose exec service-name curl other-service:port

---

Troubleshooting

Services Won't Start

# Check logs
docker compose logs -f

# Verify image pulls succeeded
docker images

# Check port conflicts
lsof -i :3000

# Validate configuration
docker compose config > /dev/null

TLS Certificate Errors

# Regenerate certificates
./generate-certs.sh

# Restart affected services
docker compose restart qdrant vllm-embedding vllm-reranker open-webui

# Verify certificate validity
openssl x509 -in ./certs/qdrant.pem -noout -text

Out of Memory

# Reduce context window (in .env)
RAG_CONTEXT_LENGTH=2048

# Lower GPU memory utilization
EMBEDDING_GPU_MEMORY_UTIL=0.7
RERANKING_GPU_MEMORY_UTIL=0.3

# Reduce service limits
QDRANT_MEMORY_LIMIT=8G

vLLM OOM during model weight loading: --gpu-memory-utilization controls KV-cache reservation after weights load — it does not reduce weight memory. If vLLM OOMs while constructing model layers (visible in the traceback as make_layers → create_weights), the model weights do not fit at full precision. Ensure EMBEDDING_QUANTIZATION=bitsandbytes (or awq/gptq) is set in .env and that the two --quantization lines in the vllm-embedding / vllm-reranker command: blocks in compose.yaml are uncommented. Those lines are commented out by default as a reminder that ROCm builds may require omitting them.

Model Not Loading

# For local llama.cpp model, verify file exists
ls -la ${LOCAL_MODEL_PATH}/${LOCAL_MODEL_FILENAME}
docker compose logs llama-local

# For vLLM embedding / reranker (local path)
ls -la ${EMBEDDING_LOCAL_PATH}
docker compose logs vllm-embedding
docker compose logs vllm-reranker

# For vLLM from HuggingFace Hub — check HF_TOKEN is set and valid
grep HF_TOKEN .env
docker compose exec vllm-embedding env | grep HUGGING_FACE_HUB_TOKEN

# Verify GPU is available
nvidia-smi

Remote API Key Invalid

# Test key manually
curl -H "x-api-key: $ANTHROPIC_API_KEY" \
https://api.anthropic.com/v1/models

# Verify key is in .env
grep ANTHROPIC_API_KEY .env

# Check key format (no quotes, spaces)

---

Architecture

Host Machine                          

Docker Network (172.28.0.0/16)             

Open WebUI (Port 3000)                 
- Web Interface                              
- RAG Pipeline                               
- TLS validation for backends                

Qdrant         vLLM     SearXNG     
(Vector DB)      Embedding (Search)    
TLS: 6333        TLS: 8000  HTTP       

vLLM Reranker                             
TLS: 8001                                

Optional Services (Profiles):                  
- Llama.cpp (local inference)                  
- Docling (document extraction)                
- Unsloth (model fine-tuning)                  
- Sandbox (code execution)                     
- STM32 Dev (embedded development)             

Legend:
TLS:  Encrypted (https://)
HTTP: Unencrypted (http://, internal only)

---

Performance Tips

For Speed

• Use Groq remote inference (ultra-fast)
• Disable reranking if not needed
• Reduce context window
• Use smaller embedding model

For Quality

• Use Anthropic Claude remote
• Enable reranking
• Increase context window
• Use larger embedding model

For Cost

• Use local model (Llama 3 8B)
• Use DeepSeek remote (cheapest)
• Batch queries
• Reuse embeddings

---

Common Tasks

Add a Document to RAG

1. Open Web UI Documents
2. Upload PDF/image/document
3. Wait for Docling to process
4. Ask questions about it

Fine-tune a Model (Unsloth)

# Enable Unsloth (Jupyter + SSH)
docker compose --profile refine up -d

# Access Jupyter at http://localhost:8000
# SSH access
ssh -p 2222 user@localhost

# The reinforcement/ directory is mounted at /workspace inside the container

Generate Training Data

The reinforcement/ notebooks implement a full synthetic dataset pipeline:

1. ingest.ipynb — Crawl documents into the Qdrant RAG knowledge base
2. generate-datasets.ipynb — Generate anchor_positive (embedding) and dpo (reranking) JSONL datasets using an LLM (Vertex AI / OpenAI-compatible)
3. ods-to-benchmark-jsonl.ipynb — Convert a hand-curated ODS spreadsheet into benchmark evaluation JSONL
4. embedding-reranking-model-training.ipynb — Fine-tune SentenceTransformer (embedding) or a causal reranker (DPO) using Unsloth; training data combines your domain JSONL, electroglyph/technical, nvidia/Retrieval-Synthetic-NVDocs-v1, and AllNLI (pair + triplet subsets via MultipleNegativesRankingLoss); saves merged safetensors + GGUF to local disk and pushes to HuggingFace Hub 4a. coder-reasoning-model-training.ipynb — Two-stage LoRA fine-tuning of Qwen3-series models for code generation with chain-of-thought reasoning via unsloth:
   • Stage 1 — Continued Pretraining: Domain-adapt on broad code/reasoning data (nvidia/OpenCodeReasoning-2, open-thoughts/OpenThoughts-114k) with anti-overfitting measures (LoRA dropout 0.1, lower LR 5e-5, weight decay 0.05, packing enabled); LoRA adapter is merged into base weights and the model is reloaded
   • Stage 2 — SFT: Curated domain-specific reasoning traces (local JSONL) with a fresh LoRA adapter (dropout 0, LR 2e-4, packing disabled); train/val splits with eval_strategy="steps" and load_best_model_at_end=True
   • SKIP_STAGE_1 flag to skip pretraining and go straight to SFT for faster iteration
   • Thinking token auto-detection: Compares apply_chat_template output with enable_thinking=True vs False to detect the model's thinking tokens — works across Qwen3, Qwen3.5, and any model with thinking-mode chat templates
   • IS_THINKING flag: True trains with reasoning traces wrapped in detected thinking tokens (reasoning variant); False trains answer-only (instruct variant); datasets marked requires_thinking=True are automatically excluded when IS_THINKING=False
   • MMLU/MMLU-Pro benchmark fallback: BENCH_FALLBACK="auto" selects MMLU-Pro for thinking models, MMLU for instruct models; accuracy tracking with per-category statistics; custom JSONL benchmarks also supported via BENCH_PATH
   • Validation monitoring: Both stages use eval_strategy="steps" with load_best_model_at_end=True; training analysis plots include train/val loss curves per stage plus pre/post accuracy comparison
   • Same export path as embedding notebook: merged 16-bit safetensors + GGUF via unsloth, HuggingFace Hub push, Google Drive transfer for Colab
5. hf-dataset-upload.ipynb — Package generated JSONL files into a HuggingFace dataset repo with four named configs (anchor_positive, dpo, benchmark) in Parquet format
6. visualize-model.ipynb — Inspect a trained model across five complementary views:
   • Captum attribution (Feature Ablation and Layer Integrated Gradients) — produces per-token heatmaps showing which input words most influenced a given completion
   • Attention pattern heatmap — shows which tokens each attention head focuses on across selected layers for a given prompt; configurable layer selection and head aggregation
   • Layer-wise representation drift — cosine similarity between adjacent layer outputs (last-token pooled) plotted as a line chart; pinpoints where the model does its most transformative work
   • Logit lens — projects each layer's residual stream through the unembedding matrix to reveal what token the model would predict if it stopped at that layer; colour-coded by entropy
   • UMAP 3D embedding space — projects all benchmark QC pairs (queries, ideal/questionable/incorrect candidates) plus configurable reference texts into an interactive 3D scatter plot; reveals how the model clusters semantically related concepts and how well it separates ideal from incorrect candidates

Prerequisites for the pipeline (set in .env):

HF_TOKEN=hf_...                       # HuggingFace token (write access)
VERTEX_PROJECT_ID=your-gcp-project    # Google Vertex AI project
VERTEX_LOCATION=us-central1

Graphical User Interface

user-interface/ contains a PySide6 desktop application that drives the full pipeline without touching the command line. Install and launch:

cd user-interface
pip install -r requirements.txt
python3 ui.py

The application has five tabs:

Tab	Purpose
Ingestion	Crawl GitHub repos or local paths into the RAG knowledge base and sync to Open WebUI
Generation	Generate synthetic training datasets from ingested knowledge; upload to HuggingFace Hub
Training	Fine-tune embedding or reranking models with configurable LoRA, quantization, and dataset paths
Visualization	Run Captum attribution analysis against a trained model and display the heatmap plots inline
Deployment	Start/stop the Docker Compose stack; shows live container status and health; auto-detects GPU architecture to select the correct compose overlay

GPU auto-detection (Deployment tab): On launch the app runs nvidia-smi to detect NVIDIA, checks /dev/kfd for AMD ROCm, and falls back to CPU. The detected architecture determines which compose overlay (compose.nvidia.yaml, compose.amd.yaml, or compose.cpu.yaml) is applied alongside the base compose.yaml.

Visualization tab: Runs visualize-model.ipynb via papermill, saves plots to output/viz/, and renders them directly in the UI. A Save images… button copies the outputs to a directory of your choice. The notebook can also be used standalone in Jupyter — set OUTPUT_DIR = None to display inline without saving.

The notebook exposes five analysis modes, each in its own cell with a config block at the top:

Mode	What it shows	Key config
Feature Ablation	Per-word attribution heatmap (perturbation-based)	EVAL_PROMPT, EVAL_TARGET
Layer Integrated Gradients	Per-token gradient attribution	N_STEPS, INTERNAL_BATCH_SIZE
Attention Heatmap	Token-to-token attention weights for selected layers	ATTN_LAYERS, ATTN_HEAD_AGGR
Layer-wise Drift	Cosine similarity between adjacent layer outputs	DRIFT_TEXTS
Logit Lens	Top predicted token at every layer, colour-coded by entropy	LOGIT_LENS_TOP_K
UMAP 3D	Embedding space scatter with QC pair roles and optional overlays	UMAP_TEXTS, OVERLAY_SOURCES

UMAP Embedding Space Comparison

The UMAP section of visualize-model.ipynb can compare embeddings produced by two different model checkpoints (e.g. a base model vs. its fine-tune) in the same 3D coordinate space. Because UMAP projections are not directly comparable across separate runs, one model serves as the reference frame — its embeddings are used to fit the UMAP projection — and the other model's embeddings are projected into that space using reducer.transform().

Outputs (written to OUTPUT_DIR when SAVE_EMBEDDINGS = True):

File	Purpose
embeddings.npz	Raw high-dimensional embeddings + role/text metadata; used to reload as an overlay in a future session
umap_reducer.pkl	Fitted UMAP reducer; preserved in case this session's coordinate space should be reused
embeddings.ply	3D point cloud with per-role colours for external viewers (MeshLab, CloudCompare, Open3D)
umap_3d.html	Interactive Plotly scatter (all pairs)
umap_3d.png	Static render

Two-pass comparison workflow:

1. Pass 1 — reference frame model (e.g. base model):

   MODEL_NAME       = "org/base-model"
   OUTPUT_DIR       = "./output/base-model"
   SAVE_EMBEDDINGS  = True
   OVERLAY_SOURCES  = []
   

   Run all cells. Produces embeddings.npz and umap_reducer.pkl in ./output/base-model/.

2. Restart kernel, load the second model (e.g. fine-tune):

   MODEL_NAME       = "org/finetuned-model"
   OUTPUT_DIR       = "./output/finetuned"
   SAVE_EMBEDDINGS  = True
   OVERLAY_SOURCES  = [
       {"path": "./output/base-model/embeddings.npz",
        "label": "Base model", "color": "#FF69B4"},
   ]
   

   Run all cells. The fine-tune's embeddings define the coordinate space; the base model's points are projected in as diamond-open markers with the specified colour.

The reference frame can be whichever model is more meaningful as an anchor — there is no requirement to run the base model first. Set OVERLAY_SOURCES = [] at any time to view a single model in isolation.

Debug a Service

# Enter container shell
docker compose exec service-name bash

# Check environment
env | sort

# Check volumes
mount | grep docker

# Test network
ping qdrant
curl https://qdrant:6333/health

Export Embeddings

# Snapshot Qdrant volume
docker compose exec qdrant tar czf - /qdrant/storage > qdrant-backup.tar.gz

# Restore later
docker compose exec qdrant tar xzf - < qdrant-backup.tar.gz

---

File Structure

.
├── compose.yaml                        # Main service definitions
├── compose.swarm.yaml                  # Docker Swarm overlay (multi-node deployment)
├── compose.amd.yaml                    # AMD GPU variant
├── compose.cpu.yaml                    # CPU-only variant
├── compose.debug.yaml                  # Debug/development overrides
├── example.env                         # Configuration template (copy to .env)
├── generate-certs.sh                   # TLS certificate generation
├── setup.sh                            # First-run setup (certs + volume ownership)
├── swarm-setup.sh                      # Docker Swarm init, labels, secrets, and network
├── README.md                           # This file
├── HARDENING.md                        # Security details
├── MODEL_CONFIGURATION.md              # Model setup guide
│
├── containers/                         # Custom Docker images
│   ├── Dockerfile.snapd                # SnapD/Ubuntu Core sandbox
│   ├── Dockerfile.stm32                # STM32U5/N6 development sandbox
│   ├── entrypoint/                      # Swarm entrypoint wrappers
│   │   ├── entrypoint-qdrant.sh         # Volume init + exec original Qdrant entrypoint
│   │   └── entrypoint-open-webui.sh     # Volume init + setpriv drop + exec Open WebUI
│   └── scripts/
│       ├── init-threadx.sh             # Azure RTOS ThreadX bootstrap
│       ├── init-u5-hal.sh              # STM32U5 HAL bootstrap
│       └── init-n6-hal.sh              # STM32N6 HAL bootstrap
│
├── user-interface/                     # PySide6 graphical management application
│   ├── ui.py                           # Main window: Ingestion, Generation, Training, Visualization, Deployment tabs
│   ├── orchestrator.py                 # QThread workers (notebook, compose, Docker SDK) and GPU detection
│   └── requirements.txt               # Python dependencies for the UI
│
├── reinforcement/                      # ML training pipeline
│   ├── ingest.ipynb                    # Ingest documents into Qdrant RAG knowledge base
│   ├── generate-datasets.ipynb         # Generate synthetic QA datasets via LLM
│   ├── ods-to-benchmark-jsonl.ipynb    # Convert ODS spreadsheet to benchmark JSONL
│   ├── embedding-reranking-model-training.ipynb  # Fine-tune embedding & reranking models (AllNLI pairs+triplets, MNRL)
│   ├── coder-reasoning-model-training.ipynb      # Fine-tune Qwen3-series coder with <think> reasoning traces
│   ├── hf-dataset-upload.ipynb         # Upload training datasets to HuggingFace Hub
│   ├── visualize-model.ipynb           # Model inspector: Feature Ablation, LIG, Attention Heatmap, Layer Drift, Logit Lens, UMAP 3D
│   ├── qc_tests.yaml                   # Quality-control test cases
│   ├── requirements.txt                # Python dependencies for training pipeline
│   ├── prompts/
│   │   ├── anchor_positive.md          # Prompt for anchor/positive pair generation
│   │   ├── dpo.md                      # Prompt for DPO preference pair generation
│   │   └── autobench.md                # Prompt for benchmark query generation
│   └── qa_generation/                  # Dataset generation library
│       ├── cli.py                      # CLI entry point
│       ├── config.py                   # Configuration dataclasses
│       ├── generator.py                # Core generation logic
│       ├── dataset.py                  # Dataset I/O utilities
│       ├── template.py                 # Prompt templating
│       ├── logger.py                   # Structured logging
│       └── providers/                  # LLM provider adapters
│           ├── base.py                 # Abstract provider interface
│           ├── factory.py              # Provider factory
│           ├── openai_compat.py        # OpenAI-compatible API adapter
│           ├── vertex.py               # Google Vertex AI adapter
│           └── rate_limiter.py         # Token/request rate limiting
│
├── certs/                              # TLS certificates (generated by generate-certs.sh)
│   ├── ca.pem
│   ├── qdrant.pem / qdrant-key.pem
│   ├── vllm-embedding.pem / vllm-embedding-key.pem
│   └── vllm-reranker.pem / vllm-reranker-key.pem
│
├── searxng/                            # SearXNG configuration
│   └── settings.yml
│
└── sandbox/                            # Persistent SnapD sandbox workspace

---

Support & Issues

Logs

docker compose logs -f service-name
docker compose logs --tail=100 service-name

Health Check

docker compose ps
docker compose exec service-name curl http://localhost:port/health

Resource Usage

docker stats
docker system df

---

References

• Docker Docs: https://docs.docker.com/
• NVIDIA Container Toolkit: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/
• Qdrant Documentation: https://qdrant.tech/documentation/
• Open WebUI: https://docs.openwebui.com/
• Llama.cpp: https://github.com/ggerganov/llama.cpp
• vLLM: https://docs.vllm.ai/
• Security Best Practices: See HARDENING.md

---

License

This deployment configuration is provided as-is. See LICENSE file for details.

---

Changelog

Version 4.0 (Current)

Graphical User Interface

• New PySide6 desktop application (user-interface/) covering the full pipeline in five tabs: Ingestion, Generation, Training, Visualization, Deployment
• Deployment tab auto-detects GPU architecture (NVIDIA / AMD ROCm / CPU) and applies the matching compose overlay; live container status table shows running/stopped/health state for every Docker container on the host
• Training tab exposes model type (embedding vs reranking), quantization, LoRA rank, dataset file pickers, and all training hyperparameters; driven via papermill against embedding-reranking-model-training.ipynb
• Visualization tab runs attribution analysis and renders the resulting heatmap PNGs directly in the UI with a "Save images…" export dialog

Attribution & Embedding Visualizer

• New reinforcement/visualize-model.ipynb — five analysis modes in one notebook:
  • Captum-based Feature Ablation and Layer Integrated Gradients attribution for any HuggingFace causal-LM checkpoint
  • Attention pattern heatmap: token-to-token attention weights for configurable layers and head aggregation (mean, max, or single head)
  • Layer-wise representation drift: cosine similarity between adjacent layer outputs plotted as a line chart; supports multiple texts on one chart for comparison
  • Logit lens: projects each layer's residual stream through the unembedding matrix; heatmap of top predicted tokens colour-coded by normalised entropy; targets Qwen/LLaMA/Mistral architecture with two clearly marked lines to adjust for other families
  • UMAP 3D interactive scatter showing benchmark QC pairs projected into the model's embedding space; configurable reference text groups colour-coded by semantic category
• Configurable via papermill parameters: MODEL_NAME, DEVICE_ID, EVAL_PROMPT, EVAL_TARGET, OUTPUT_DIR, N_STEPS, INTERNAL_BATCH_SIZE
• Memory optimizations for LIG: gradient checkpointing (use_reentrant=False), internal_batch_size=1, reduced default n_steps=20
• GPU selection via CUDA_VISIBLE_DEVICES set before torch import; supports multi-GPU hosts with non-uniform capability (e.g. mixed Ampere/non-Ampere cards)
• Fixed bugs in original visualizer: max_memory dict type, is_bf16_supported() call, bnb_4bit_use_double_quant parameter name, empty TextTemplateInput values, hardcoded skip token
• Embedding export and overlay comparison: SAVE_EMBEDDINGS=True writes embeddings.npz, umap_reducer.pkl, and embeddings.ply to OUTPUT_DIR; OVERLAY_SOURCES accepts any number of previously saved .npz files and projects them into the current session's UMAP coordinate space for side-by-side comparison of base vs. fine-tuned model embeddings

Bug fixes

• embedding-reranking-model-training.ipynb: corrected papermill parameter names (GRADIENT_ACCUMULATION, CTX_LENGTH, RANK) that did not match notebook variable names
• compose.yaml: documented that --quantization lines in vllm-embedding / vllm-reranker command blocks must be uncommented on CUDA hosts; EMBEDDING_QUANTIZATION=bitsandbytes in .env has no effect while those lines are commented out, causing OOM during weight loading on GPUs smaller than the model's full-precision footprint

Version 5.0 (Current)

Coder reasoning model training — two-stage pipeline

• Two-stage LoRA fine-tuning: Stage 1 (continued pretraining on nvidia/OpenCodeReasoning-2 and open-thoughts/OpenThoughts-114k with anti-overfitting measures) merges LoRA into base weights, then Stage 2 (SFT on domain-specific reasoning traces) applies a fresh LoRA adapter — industry-standard pretraining → SFT pipeline
• SKIP_STAGE_1 flag to skip continued pretraining and go straight to SFT
• Train/val splits with eval_strategy="steps" and load_best_model_at_end=True for both stages; training analysis plots include per-stage loss curves and pre/post accuracy comparison
• Thinking token auto-detection: compares apply_chat_template(..., enable_thinking=True) vs False output to detect the model's omos/amos tokens — works across Qwen3, Qwen3.5, and any model with thinking-mode chat templates
• IS_THINKING flag: True wraps reasoning in detected thinking tokens (reasoning variant); False trains answer-only (instruct variant); requires_thinking dataset flag automatically excludes reasoning-focused datasets when IS_THINKING=False
• MMLU/MMLU-Pro benchmark fallback: BENCH_FALLBACK="auto" selects MMLU-Pro for thinking models, MMLU for instruct models; per-category accuracy tracking via _extract_answer_letter()
• Formatters for HuggingFace datasets: format_open_code_reasoning() for nvidia/OpenCodeReasoning-2, format_open_thoughts() for open-thoughts/OpenThoughts-114k; existing load_reasoning_jsonl() for local JSONL

Docker Swarm deployment

• New compose.swarm.yaml overlay for multi-node deployment with encrypted overlay networking, Docker secrets for TLS certificate distribution, and deploy.placement.constraints for service placement by node role
• New swarm-setup.sh script: initialises Swarm, applies node labels (llm_role=orchestrator/gpu-inference/gpu-retrieval/sandbox), creates secrets and configs from ./certs/ and ./containers/entrypoint/, creates encrypted overlay network, prints join tokens and deploy commands
• Entrypoint wrappers (entrypoint-qdrant.sh, entrypoint-open-webui.sh) replace init sidecar containers; init containers are disabled (replicas: 0) in Swarm mode since depends_on is ignored
• Node role placement: orchestrator (Open WebUI, SearXNG, Redis, Qdrant), gpu-inference (llama-local, llama-orchestrator, Docling, Unsloth), gpu-retrieval (vLLM embedding, vLLM reranker), sandbox (SnapD, STM32)
• Rolling update and rollback policies on all services

Version 4.1

Embedding/Reranking training notebook

• Removed optional pretraining step — FastLanguageModel causal LM SFT is incompatible with Qwen3-Embedding-4B due to its YaRN RoPE configuration (max_position_embeddings = 40960) conflicting with unsloth's compiled Qwen3 kernel; the embedding fine-tuning path via FastSentenceTransformer is unaffected
• Added AllNLI dataset mix: sentence-transformers/all-nli pair subset (anchor/positive) concatenated with existing datasets; triplet subset (anchor/positive/negative) loaded separately; both passed to SentenceTransformerTrainer as a DatasetDict with per-key MultipleNegativesRankingLoss — triplets provide hard negatives on top of in-batch negatives, pairs use in-batch negatives only

New: Coder reasoning model training notebook

• New reinforcement/coder-reasoning-model-training.ipynb for supervised fine-tuning of Qwen3-series models on reasoning datasets
• IS_THINKING flag: True wraps the reasoning field of each assistant message in <think>...</think> (reasoning variant); False uses the content field only (non-reasoning variant)
• Dataset loader handles both array-per-line ([{...}, ...]) and object-per-line ({"messages": [...]}) JSONL formats with optional reasoning field on assistant turns
• Commented stubs for nvidia/OpenCodeReasoning-2 (Apache 2.0) and open-thoughts/OpenThoughts-114k (MIT) open reasoning datasets
• Pre/post training benchmark using code generation prompts; plots training loss, VRAM usage, and answer-length delta; saves per-prompt pre/post response comparison figures
• Same export path as embedding notebook: merged 16-bit safetensors + GGUF via unsloth, HuggingFace Hub push, Google Drive transfer for Colab

Version 3.0

• vLLM embedding and reranker support local files or HuggingFace Hub model sources via EMBEDDING_MODEL / RERANKING_MODEL — no separate containers needed
• HUGGING_FACE_HUB_TOKEN forwarded to vLLM containers for private repo access
• --task embed and --task score flags added to vLLM commands
• Fixed RERANKING_MODEL_API_URL pointing to wrong host (vllm-embedding → vllm-reranker) and wrong scheme (http → https)
• Full ML training pipeline in reinforcement/: document ingestion, synthetic dataset generation, embedding/reranking model fine-tuning, HuggingFace Hub dataset upload
• STM32 embedded development sandbox with ThreadX, STM32U5/N6 HAL support
• SnapD/Ubuntu Core code execution sandbox

Version 2.0

• TLS on embedding and reranker services
• Flexible LLM inference source (local llama.cpp or remote API)
• All configuration in .env
• GPU device assignment per service
• Comprehensive security hardening
• Qdrant volume ownership init container

Version 1.0

• Initial release with basic hardening
• OpenAI compatibility
• Basic TLS on vector database

---

Last Updated: 2026-06-07 Maintained By: Robert Fudge License: GPL-3.0-only