JL
Cambiar a español
Back to projects
Independent/portfolio experience

Local AI Inference Proxies

Modular, distributed local AI infrastructure deploying isolated services across dedicated machines, with dynamic proxy discovery and VRAM lifecycle management.

Evidence status: Approved written profilellama.cpp · Vulkan · OpenAI-compatible APIs
FIG. 04 · CLIENTS → ROUTE → ONE GPU ENGINE

LOCAL AI / INFERENCE PATCH-BAY

LOCAL AI / INFERENCE PATCH-BAYIllustrative local-inference patch-bay bench: clients, router, one Arc Pro B70 GPU engine, support modules, knowledge, VPN and observability. GPU ENGINE ROUTE HERMES OPENCODE WEBUI CLOUD KNOWLEDGE DOCUMENTS SCRAPPER VPN WATCHER QDRANT POSTGRES KNOWLEDGEOBSIDIAN FOLDER CLOUD PROVIDERS NEXTCLOUD TRAY SEARXNG + CACHECRAWL4AI → MDBROWSER · TBDSCRAPPER GLUETUN VPNINTERNET ARC PRO B7032 GB LLAMA-SERVER LLM / VLM ALA OCR XPU TTS · CPU STT EVAL 10 MIN IDLE LLAMA-PROXYBEARER SLOT RELAY ROUTELOCAL / CLOUD HERMES OPENCODE OPEN WEBUI WATCHER9106
100%
MODULES
0/11
VRAM
0/32 GB
ROUTE
LOCAL
GPU
OFF
IDLE
10 min

StatusChoose a service, then select a client

Illustrative cycle: choose a service, then activate Hermes, OpenCode or Open WebUI to see its complete route without moving the camera. Hermes and OpenCode use Route; Open WebUI reaches the proxy directly. Only ENGINE houses the 32 GB Arc Pro B70 GPU.

All values are illustrative demo values; they do not represent performance, latency, throughput or production data. The proxy discovers models dynamically and hides projector files. TTS currently runs on CPU only; STT and Browser remain under evaluation. Obsidian is attached as a folder, with no MCP yet.

01

A reliable, distributed local AI architecture running isolated services across dedicated hardware machines: a central proxy provides an OpenAI-compatible gateway, routes tasks to modular compute nodes, and reclaims GPU memory on idle.

02

Monolithic local AI setups often bundle inference, web scraping, document retrieval, and vector search onto a single host, leading to resource contention, unstable crash domains, and leaked GPU memory. A production-grade homelab setup requires distributing services across their own dedicated machines while exposing a single reliable, modular interface to client tools.

People evaluating self-hosted AI infrastructure who want to see how local inference, document processing and agent tooling can share one machine without cloud dependence.

03

Independent/portfolio work: designed and operated the proxies and the surrounding local services.

04

  1. Distribute specialized AI and data services across their own dedicated hardware machines (inference compute, scraping, vector store, and client hosts)
  2. Client applications (Hermes, OpenCode, Open WebUI) connect to a central OpenAI-compatible inference proxy
  3. Proxy verifies bearer tokens and dynamically discovers available models, filtering out internal projector files
  4. On request, the proxy signals the engine node, turning on inference and mounting model weights into GPU memory
  5. Route speech, OCR, web retrieval, and agent tasks to their dedicated modular services over isolated internal networks
  6. Stream responses back reliably with unified OpenAI-compatible formatting
  7. Automatically reclaim full VRAM and unload weights after 10 minutes of idle time to keep the GPU ready for other compute workloads

05

  • Distributed AI subsystems onto their own machines to isolate crash domains, prevent memory contention, and make individual services modular and independently upgradeable.
  • Standardized on an OpenAI-compatible proxy interface so any client tool connects interchangeably without custom wrappers.
  • Configured an automated 10-minute idle stop policy to fully reclaim VRAM for other system tasks.
  • Routed web scraping and extraction egress strictly through isolated VPN network boundaries.
  • Decoupled CPU-bound audio synthesis from GPU-intensive vision and language inference.