# Local AI Serving & Autonomous Agent Architecture Plan **Author:** Rudransh Shekhar (KaptaanLM Research Labs / mySphere) **Host Target:** `kaptaan45-Aspire-A515-52G` (Mobile VPS) **Date:** November 2026 **Document Status:** Approved Architecture / Execution Blueprint (Awaiting Order) **Workspace Directory:** [`/home/kaptaan45/ai-stack`](file:///home/kaptaan45/ai-stack) **Model Storage (1TB HDD Link):** [`/home/kaptaan45/ai-stack/models`](file:///home/kaptaan45/ai-stack/models) ➔ `/media/kaptaan45/Data/models` --- ## 1. Project Objective & Hardware Alignment The objective of this phase is to transform the Mobile VPS into an always-on **Local AI Inference Node** and a **24/7 Autonomous Personal Agent**, fully operational within the hardware constraints of the machine: * **Discrete GPU (dGPU):** NVIDIA GeForce MX130 (2048 MB VRAM, Maxwell GM108 architecture, Compute Capability 5.0 / `sm_50`). * **Host CPU:** Intel Core i5-8265U (4 Cores / 8 Threads, AVX2, FMA vector extensions). * **System RAM:** 12 GB DDR4 (~9.5 GB available in headless server mode). * **Storage Tiering:** Codebases, virtual environments, and compiler toolchains live on the fast **NVMe SSD** (`/home/kaptaan45/ai-stack`), while heavy GGUF model weights live on the **1 TB SATA HDD** (`/media/kaptaan45/Data/models`). --- ## 2. Core Architectural Decisions ### Pillar 1: The Serving Layer (Dual-Engine Benchmarking) We will deploy two complementary inference runtimes and benchmark them head-to-head: 1. **`llama.cpp` (`llama-server`)**: - Built natively with CUDA support targeting Maxwell architecture: `-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=50`. - Serves as the stable, low-overhead, OpenAI-compatible reference runtime. 2. **`InferenceOS` ([`rudy-07/InferenceOS`](https://github.com/rudy-07/InferenceOS))**: - Advanced adaptive inference runtime designed by Rudransh Shekhar. - Evaluates dynamic layer placement across CPU and dGPU, intelligent scheduling, and runtime memory budgeting. 3. **Dynamic Routing Strategy**: - Both engines will be profiled under tight VRAM constraints and expanded context windows. - If `InferenceOS` demonstrates superior scheduling or fewer memory overflow penalties on models that exceed the 2 GB VRAM ceiling, our serving gateway will dynamically route larger hybrid workloads to `InferenceOS`, while serving ultra-fast small models via the best-performing backend. --- ### Pillar 2: Model Selection & Quantization Strategy Models are categorized into two evaluation tiers: #### Phase 1: Benchmark Anchors (The First Two Models) To rigorously compare `llama.cpp` against `InferenceOS`, we start by deploying two models representing opposite memory paradigms: | Role | Model Target | Quantization | Est. Size | Expected Behavior on MX130 (2GB VRAM) | | :--- | :--- | :--- | :--- | :--- | | **Small Anchor (Fits in VRAM)** | **MiniCPM5-1B** | **`Q6_K`** | ~1.05 GB | **100% VRAM Ingestion**. High-precision weights; zero CPU layer penalty; tests maximum raw token throughput and low latency. | | **Big Anchor (VRAM Overflow)** | **K2-Horizon-3.7B** | **`Q4_K_M`** | ~2.3 GB | **Hybrid CPU/GPU Split**. Exceeds 2 GB VRAM; forces ~12–16 layers to dGPU with remainder in system RAM. Rigorously stress-tests layer placement and scheduler efficiency in `InferenceOS` vs `llama.cpp`. | #### Phase 2: Production & Experimentation Model Arsenal Once the serving engine baseline is established, the following models will be integrated based on specific operational needs: | Model Target | Quantization Format | Target Footprint | Primary VPS Use-Case | | :--- | :--- | :--- | :--- | | **MiniCPM5-2B** *(~2.6B actual)* | **`Q4_K_M`** | ~1.5 GB | Flagship compact reasoning & multi-turn dialog under 3B. | | **Qwen3.5-2B** | **`Q4_K_M`** | ~1.3 GB | High-accuracy structured data extraction and tool orchestration. | | **LFM2.5-2.6B** | **`Q3_K_M` / `IQ3_XS`** | ~1.1 – 1.3 GB | High-density parameter compression testing; perplexity vs speed analysis. | | **EXAONE-4.0-1.2B** | **`Q6_K`** | ~1.1 GB | High-fidelity dual-mode reasoning (thinking mode vs normal mode). | | **MiniCPM5-1B** | **`Q6_K`** | ~1.05 GB | Fast, general-purpose daily assistant model. | | **Granite-4.0-H-350M** | **`Q8_0` or FP16** | ~350 MB | Ultra-low latency, sub-second background classification, log parsing, and simple regex tasks. | --- ### Pillar 3: Agent Harness & Long-Term Memory Layer Once the serving layer and model routing are validated, we will deploy the agent infrastructure: ``` ┌─────────────────────────────────────────────────────────────────────────────────┐ │ MOBILE VPS │ │ │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ AGENT & MEMORY LAYER │ │ │ │ │ │ │ │ [ Hermes Agent ] (Nous Research) │ │ │ │ * Self-improving skill compilation (`~/.hermes/skills/`) │ │ │ │ * Autonomous bash execution, file editing, cron jobs │ │ │ │ ▲ │ │ │ │ │ (Context retrieval via MCP / REST) │ │ │ │ ▼ │ │ │ │ [ MemOS Daemon ] (MemTensor/MemOS) │ │ │ │ * Unified Memory OS (Episodic, Semantic, Skill memory) │ │ │ │ * Keeps context tight (<500 tokens) to prevent SLM attention drift │ │ │ └────────────────────────────────────┬────────────────────────────────────┘ │ │ │ OpenAI-compatible API (:8000) │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ │ SERVING & ROUTING LAYER │ │ │ │ │ │ │ │ [ llama-server ] (sm_50 cuBLAS) OR [ InferenceOS ] (Adaptive Split) │ │ │ │ │ │ │ │ * MiniCPM5-1B (Q6_K) -> 100% in MX130 VRAM (~1.05 GB) │ │ │ │ * K2-Horizon-3.7B (Q4) -> Dynamic Split (GPU VRAM + 12GB Host RAM) │ │ │ └─────────────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────────┘ ``` #### Evaluation Phases for Agent: 1. **Hermes Agent Standalone**: - Evaluate tool execution reliability, prompt size, and how well small models (1B–2.6B) follow system instructions without external memory layers. 2. **Hermes Agent + MemOS Integration**: - Connect MemOS as an MCP / vector memory service to Hermes. - Benchmark task recall, prompt token reduction, and context degradation over multi-session interactions. --- ## 3. Step-by-Step Execution Sequence (Roadmap) ``` [ PHASE A: Serving Runtimes & Anchors ] (Next Step upon Order) ├── 1. Build llama.cpp with CUDA 13.0 / sm_50 support in /home/kaptaan45/ai-stack ├── 2. Clone and configure InferenceOS in /home/kaptaan45/ai-stack ├── 3. Download MiniCPM5-1B (Q6_K) into /media/kaptaan45/Data/models └── 4. Download K2-Horizon-3.7B (Q4_K_M) into /media/kaptaan45/Data/models │ ▼ [ PHASE B: Benchmarking & Profiling ] ├── 1. Run MiniCPM5-1B on llama.cpp vs InferenceOS (tokens/sec, VRAM utilization) ├── 2. Run K2-Horizon-3.7B on llama.cpp vs InferenceOS (layer offloading, latency) ├── 3. Stress-test context expansion past 2048 tokens to monitor VRAM overflow └── 4. Establish dynamic routing mechanism based on benchmark findings │ ▼ [ PHASE C: Autonomous Agent & Memory Harness ] ├── 1. Install and test Hermes Agent standalone with local model endpoint ├── 2. Deploy MemOS daemon and configure MCP connection to Hermes Agent ├── 3. Compare Hermes standalone vs Hermes + MemOS on multi-turn autonomous tasks └── 4. Set up 24/7 background systemd service for persistent agent autonomy ``` --- ## 4. Current Environment State * **Workspace Prepared:** Directory [`/home/kaptaan45/ai-stack`](file:///home/kaptaan45/ai-stack) has been initialized on NVMe SSD. * **Storage Symlink Active:** [`/home/kaptaan45/ai-stack/models`](file:///home/kaptaan45/ai-stack/models) mapped to persistent 1 TB HDD (`/media/kaptaan45/Data/models`). * **Active Status:** Awaiting user order to begin Phase A.