vm-bench/AGENTS.md

AGENTS.md — VM Bench (VMware → Proxmox Image Converter)

Project Overview

VM Bench is a two-tier web application for converting VMware disk images (VMDK, VHD, etc.) into Proxmox-compatible QCOW2 disks and provisioning VMs with automatic OS and boot type detection.

• Repository: https://git.lohmar.co.uk/cclohmar/vm-bench.git
• Deployment root: /mnt/converter/ (bind-mounted between host and LXC)
• Frontend: FastAPI + Jinja2 web UI in an LXC container (vm-bench) on port 5000
• Backend: FastAPI REST API on the Proxmox host (srv2) on port 9000
• API spec: open-api.yaml (single source of truth — derive models from this)
• Python version: 3.13 (backend), 3.11 (frontend LXC)

Architecture

Browser            vm-bench LXC             Proxmox Host (srv2)
                   (this container)
──→ :5000 ──→  frontend/app.py  ──→  :9000  backend/app.py
                                       │
                                       ├─ qemu-img   (format, size)
                                       ├─ guestfish  (OS, EFI detection)
                                       ├─ 7z/unzip   (archive extraction)
                                       └─ qm create  (VM provisioning)

• Frontend (vm-bench LXC) — user-facing web UI, manages sessions, proxies API calls
• Backend (Proxmox host) — disk conversion, analysis, VM provisioning
• Shared storage — /mnt/converter/tmp/ (staging), /mnt/converter/logs/ (logs)

Directory Layout

/mnt/converter/
├── install.sh                    # Main deployment script (host-level)
├── clean.sh                      # Clean tmp dirs + restart backend
├── open-api.yaml                 # API specification (single source of truth)
├── README.md
├── AGENTS.md                     # This file
├── backend/                      # Backend API (runs on Proxmox host)
│   ├── app.py                    # FastAPI app, routes, logging
│   ├── models.py                 # Pydantic request/response models
│   ├── converter.py              # Archive extraction, disk probing, OS/EFI detection
│   ├── provisioner.py            # VM provisioning (qm create/importdisk/convert)
│   ├── requirements.txt          # fastapi, uvicorn, pydantic
│   └── vm-bench-backend.service  # Systemd unit
├── frontend/                     # Web UI (runs on vm-bench LXC)
│   ├── app.py                    # FastAPI app, routes, download/SCP handling
│   ├── api_client.py             # Typed REST client → backend
│   ├── setup.sh                  # Frontend installer (inside LXC)
│   ├── requirements.txt          # fastapi, uvicorn, jinja2, requests, python-multipart
│   ├── vm-bench.service          # Systemd unit
│   ├── templates/                # Jinja2 HTML templates
│   │   ├── base.html             # Base layout with session GUID + header/footer
│   │   ├── index.html            # New session form + download + progress
│   │   ├── _analysis.html        # Analysis result + confirm form (HTML fragment)
│   │   ├── polling.html          # Progress bar + job status + reuse section
│   │   └── scp.html              # SCP pull page
│   └── static/
│       └── proxmox.css           # Proxmox VE-inspired CSS theme
├── logs/                         # Shared log files (gitignored)
├── tmp/                          # Staging files (gitignored)
└── venv/                         # Backend Python venv (gitignored)

Key Files by Responsibility

Backend (backend/)

File	Purpose
backend/app.py:1	FastAPI app with routes: /api/v1/health, /analyze, /jobs, /jobs/{id}, /jobs/{id}/cleanup, /clone
backend/models.py:1	Pydantic models: JobSubmissionRequest, JobStatusResponse, AnalyzeRequest, AnalyzeResponse, DiskSpec, CleanupRequest, CloneRequest
backend/converter.py:1	Archive extraction (extract_if_needed), disk discovery (discover_disk), OS detection (detect_os), EFI detection (detect_efi)
backend/provisioner.py:1	Background job processor: submit_job, get_job_status, cleanup_staging, clone_vm, _process_job (qemu-img convert → virt-resize → qm create → qm importdisk)

Frontend (frontend/)

File	Purpose
frontend/app.py:1	FastAPI app with routes: /, /session/upload, /session/progress/{f}, /session/analyze, /session/confirm, /session/status/{id}, /session/cleanup/{id}, /session/clone, /scp, /scp/start, /scp/progress/{sid}/{f}
frontend/api_client.py:1	ApiClient class — typed HTTP client for backend API with ApiError exception

Deployment (install.sh)

File	Purpose
install.sh:1	Master deployment script (Proxmox host). Modes: --deploy, --update, --remove. Creates directory tree, clones repo, installs system packages (qemu-img, guestfish, archive tools), Python venv, backend systemd service, LXC container with bind mount.
frontend/setup.sh:1	Frontend installer (run inside LXC). Installs system packages (python3, aria2, openssh-client, sshpass), Python deps, systemd service.

Systemd Units

File	Port	Env Vars
backend/vm-bench-backend.service	9000	None
frontend/vm-bench.service	5000	BACKEND_URL=http://10.2.0.2:9000, TMPDIR=/mnt/converter/tmp

Usage Flow

1. User opens web UI → session form
2. Upload or download a source image → lands in /mnt/converter/tmp/{session_id}/in/
3. Backend analyses image → shows format, size, OS, EFI status
4. Configure VM settings → name, CPU, RAM, storage, boot type
5. Submit job → backend converts, shrinks (if needed), creates VM
6. Poll progress → real-time status bar updates every 2 seconds via AJAX
7. Reuse or cleanup → create another VM or delete staging files

Disk Sizing & Auto-Shrink Logic

• If target_disk_size_gb is omitted and the source disk is larger than its auto-shrink target (10% of virtual size, minimum 20 GB), the disk is shrunk using virt-resize --shrink --resize-force.
• If virt-resize fails, falls back to qemu-img resize --shrink.
• If target_disk_size_gb is larger than current, disk is expanded.

Session Management

• Frontend generates a UUID session ID stored in localStorage + cookie (vm_bench_sid)
• Files are staged at /mnt/converter/tmp/{session_id}/in/ and out/
• Cleanup can delete the entire session directory or preserve for reuse

Source Image Ingestion Methods

1. Download URL (index.html) — uses aria2c with 8 connections, kills download if ETA > 1 hour
2. SCP Pull (scp.html) — uses sshpass + scp directly from remote servers
3. File upload (legacy, referenced but not primary flow)

Backend Tools Required (on Proxmox host)

• qemu-img — disk format and size detection, format conversion
• guestfish / virt-inspector — OS type and EFI boot detection
• virt-resize — safe disk shrinking
• 7z, unzip, unrar, tar, gunzip, bunzip2, xz — archive extraction
• qm (Proxmox CLI) — VM creation, disk import, configuration

Frontend Tools Required (in LXC)

• python3, pip3, wget, curl, aria2c, openssh-client, sshpass

Configuration

Variable	Default	Description
BACKEND_URL	http://10.2.0.2:9000	Backend API base URL (set in systemd unit + env)
VM_ID_MIN	21000	Minimum allowed Proxmox VM ID
VM_ID_MAX	21100	Maximum allowed Proxmox VM ID
DOWNLOAD_TIMEOUT	14400 (4h)	Max download time (safety net)
SPEED_CHECK_AFTER	30 (seconds)	Wait before judging download speed
MAX_ETA_SECONDS	3600 (1h)	Kill download if ETA exceeds this

Job States

queued → processing_conversion → importing_storage → completed
                                                  → failed

Logging

Both services log to console (systemd journal) and rotating file handler:

• Frontend: /mnt/converter/logs/vm-bench.log
• Backend: /mnt/converter/logs/vm-bench-backend.log
• Rotation: 10 MB max, 5 backup files

Development Commands

Backend (on Proxmox host)

# Install/update
bash install.sh --deploy

# Restart
systemctl restart vm-bench-backend

# View logs
tail -f /mnt/converter/logs/vm-bench-backend.log
journalctl -u vm-bench-backend -f

# Health check
curl http://127.0.0.1:9000/api/v1/health

Frontend (inside LXC)

# Install/update
bash /mnt/converter/frontend/setup.sh

# Restart
systemctl restart vm-bench

# View logs
tail -f /mnt/converter/logs/vm-bench.log
journalctl -u vm-bench -f

Cleanup script

bash /mnt/converter/clean.sh  # Clears tmp dirs + restarts backend

Project Conventions

• No code comments unless strictly necessary (follow existing patterns)
• Models derived from open-api.yaml — update spec first, then code
• backend/provisioner.py uses in-memory job store (_jobs dict) with threading.Lock
• All paths use pathlib.Path
• Loggers use module-level naming: logging.getLogger("backend"), logging.getLogger("backend.converter"), etc.
• Frontend templates use pve- CSS prefix mimicking Proxmox VE design system
• HTML fragments (e.g. _analysis.html) are injected via innerHTML — no <script> tags
• Session-based isolation: each browser tab/session gets a UUID for file staging
• VM names are sanitized to DNS-safe characters: [^a-zA-Z0-9-] replaced with -
• VM ID range is validated client-side (21000–21100) and enforced in frontend routes