diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..e6acee8 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,214 @@ +# AGENTS.md — VM Bench (VMware → Proxmox Image Converter) + +## Project Overview + +VM Bench is a two-tier web application for converting VMware disk images (VMDK, VHD, +etc.) into Proxmox-compatible QCOW2 disks and provisioning VMs with automatic OS and +boot type detection. + +- **Repository**: `https://git.lohmar.co.uk/cclohmar/vm-bench.git` +- **Deployment root**: `/mnt/converter/` (bind-mounted between host and LXC) +- **Frontend**: FastAPI + Jinja2 web UI in an LXC container (`vm-bench`) on port 5000 +- **Backend**: FastAPI REST API on the Proxmox host (`srv2`) on port 9000 +- **API spec**: `open-api.yaml` (single source of truth — derive models from this) +- **Python version**: 3.13 (backend), 3.11 (frontend LXC) + +## Architecture + +``` +Browser vm-bench LXC Proxmox Host (srv2) + (this container) +──→ :5000 ──→ frontend/app.py ──→ :9000 backend/app.py + │ + ├─ qemu-img (format, size) + ├─ guestfish (OS, EFI detection) + ├─ 7z/unzip (archive extraction) + └─ qm create (VM provisioning) +``` + +- **Frontend** (`vm-bench` LXC) — user-facing web UI, manages sessions, proxies API calls +- **Backend** (Proxmox host) — disk conversion, analysis, VM provisioning +- **Shared storage** — `/mnt/converter/tmp/` (staging), `/mnt/converter/logs/` (logs) + +## Directory Layout + +``` +/mnt/converter/ +├── install.sh # Main deployment script (host-level) +├── clean.sh # Clean tmp dirs + restart backend +├── open-api.yaml # API specification (single source of truth) +├── README.md +├── AGENTS.md # This file +├── backend/ # Backend API (runs on Proxmox host) +│ ├── app.py # FastAPI app, routes, logging +│ ├── models.py # Pydantic request/response models +│ ├── converter.py # Archive extraction, disk probing, OS/EFI detection +│ ├── provisioner.py # VM provisioning (qm create/importdisk/convert) +│ ├── requirements.txt # fastapi, uvicorn, pydantic +│ └── vm-bench-backend.service # Systemd unit +├── frontend/ # Web UI (runs on vm-bench LXC) +│ ├── app.py # FastAPI app, routes, download/SCP handling +│ ├── api_client.py # Typed REST client → backend +│ ├── setup.sh # Frontend installer (inside LXC) +│ ├── requirements.txt # fastapi, uvicorn, jinja2, requests, python-multipart +│ ├── vm-bench.service # Systemd unit +│ ├── templates/ # Jinja2 HTML templates +│ │ ├── base.html # Base layout with session GUID + header/footer +│ │ ├── index.html # New session form + download + progress +│ │ ├── _analysis.html # Analysis result + confirm form (HTML fragment) +│ │ ├── polling.html # Progress bar + job status + reuse section +│ │ └── scp.html # SCP pull page +│ └── static/ +│ └── proxmox.css # Proxmox VE-inspired CSS theme +├── logs/ # Shared log files (gitignored) +├── tmp/ # Staging files (gitignored) +└── venv/ # Backend Python venv (gitignored) +``` + +## Key Files by Responsibility + +### Backend (`backend/`) + +| File | Purpose | +|------|---------| +| `backend/app.py:1` | FastAPI app with routes: `/api/v1/health`, `/analyze`, `/jobs`, `/jobs/{id}`, `/jobs/{id}/cleanup`, `/clone` | +| `backend/models.py:1` | Pydantic models: `JobSubmissionRequest`, `JobStatusResponse`, `AnalyzeRequest`, `AnalyzeResponse`, `DiskSpec`, `CleanupRequest`, `CloneRequest` | +| `backend/converter.py:1` | Archive extraction (`extract_if_needed`), disk discovery (`discover_disk`), OS detection (`detect_os`), EFI detection (`detect_efi`) | +| `backend/provisioner.py:1` | Background job processor: `submit_job`, `get_job_status`, `cleanup_staging`, `clone_vm`, `_process_job` (qemu-img convert → virt-resize → qm create → qm importdisk) | + +### Frontend (`frontend/`) + +| File | Purpose | +|------|---------| +| `frontend/app.py:1` | FastAPI app with routes: `/`, `/session/upload`, `/session/progress/{f}`, `/session/analyze`, `/session/confirm`, `/session/status/{id}`, `/session/cleanup/{id}`, `/session/clone`, `/scp`, `/scp/start`, `/scp/progress/{sid}/{f}` | +| `frontend/api_client.py:1` | `ApiClient` class — typed HTTP client for backend API with `ApiError` exception | + +### Deployment (`install.sh`) + +| File | Purpose | +|------|---------| +| `install.sh:1` | Master deployment script (Proxmox host). Modes: `--deploy`, `--update`, `--remove`. Creates directory tree, clones repo, installs system packages (qemu-img, guestfish, archive tools), Python venv, backend systemd service, LXC container with bind mount. | +| `frontend/setup.sh:1` | Frontend installer (run inside LXC). Installs system packages (python3, aria2, openssh-client, sshpass), Python deps, systemd service. | + +### Systemd Units + +| File | Port | Env Vars | +|------|------|----------| +| `backend/vm-bench-backend.service` | 9000 | None | +| `frontend/vm-bench.service` | 5000 | `BACKEND_URL=http://10.2.0.2:9000`, `TMPDIR=/mnt/converter/tmp` | + +## Usage Flow + +1. User opens web UI → session form +2. Upload or download a source image → lands in `/mnt/converter/tmp/{session_id}/in/` +3. Backend analyses image → shows format, size, OS, EFI status +4. Configure VM settings → name, CPU, RAM, storage, boot type +5. Submit job → backend converts, shrinks (if needed), creates VM +6. Poll progress → real-time status bar updates every 2 seconds via AJAX +7. Reuse or cleanup → create another VM or delete staging files + +## Disk Sizing & Auto-Shrink Logic + +- If `target_disk_size_gb` is omitted and the source disk is larger than its + auto-shrink target (10% of virtual size, minimum 20 GB), the disk is shrunk + using `virt-resize --shrink --resize-force`. +- If `virt-resize` fails, falls back to `qemu-img resize --shrink`. +- If `target_disk_size_gb` is larger than current, disk is expanded. + +## Session Management + +- Frontend generates a UUID session ID stored in `localStorage` + cookie (`vm_bench_sid`) +- Files are staged at `/mnt/converter/tmp/{session_id}/in/` and `out/` +- Cleanup can delete the entire session directory or preserve for reuse + +## Source Image Ingestion Methods + +1. **Download URL** (`index.html`) — uses `aria2c` with 8 connections, kills download if ETA > 1 hour +2. **SCP Pull** (`scp.html`) — uses `sshpass` + `scp` directly from remote servers +3. File upload (legacy, referenced but not primary flow) + +## Backend Tools Required (on Proxmox host) + +- `qemu-img` — disk format and size detection, format conversion +- `guestfish` / `virt-inspector` — OS type and EFI boot detection +- `virt-resize` — safe disk shrinking +- `7z`, `unzip`, `unrar`, `tar`, `gunzip`, `bunzip2`, `xz` — archive extraction +- `qm` (Proxmox CLI) — VM creation, disk import, configuration + +## Frontend Tools Required (in LXC) + +- `python3`, `pip3`, `wget`, `curl`, `aria2c`, `openssh-client`, `sshpass` + +## Configuration + +| Variable | Default | Description | +|----------|---------|-------------| +| `BACKEND_URL` | `http://10.2.0.2:9000` | Backend API base URL (set in systemd unit + env) | +| `VM_ID_MIN` | `21000` | Minimum allowed Proxmox VM ID | +| `VM_ID_MAX` | `21100` | Maximum allowed Proxmox VM ID | +| `DOWNLOAD_TIMEOUT` | `14400` (4h) | Max download time (safety net) | +| `SPEED_CHECK_AFTER` | `30` (seconds) | Wait before judging download speed | +| `MAX_ETA_SECONDS` | `3600` (1h) | Kill download if ETA exceeds this | + +## Job States + +``` +queued → processing_conversion → importing_storage → completed + → failed +``` + +## Logging + +Both services log to console (systemd journal) and rotating file handler: +- Frontend: `/mnt/converter/logs/vm-bench.log` +- Backend: `/mnt/converter/logs/vm-bench-backend.log` +- Rotation: 10 MB max, 5 backup files + +## Development Commands + +### Backend (on Proxmox host) +```bash +# Install/update +bash install.sh --deploy + +# Restart +systemctl restart vm-bench-backend + +# View logs +tail -f /mnt/converter/logs/vm-bench-backend.log +journalctl -u vm-bench-backend -f + +# Health check +curl http://127.0.0.1:9000/api/v1/health +``` + +### Frontend (inside LXC) +```bash +# Install/update +bash /mnt/converter/frontend/setup.sh + +# Restart +systemctl restart vm-bench + +# View logs +tail -f /mnt/converter/logs/vm-bench.log +journalctl -u vm-bench -f +``` + +### Cleanup script +```bash +bash /mnt/converter/clean.sh # Clears tmp dirs + restarts backend +``` + +## Project Conventions + +- **No code comments** unless strictly necessary (follow existing patterns) +- Models derived from `open-api.yaml` — update spec first, then code +- `backend/provisioner.py` uses in-memory job store (`_jobs` dict) with `threading.Lock` +- All paths use `pathlib.Path` +- Loggers use module-level naming: `logging.getLogger("backend")`, `logging.getLogger("backend.converter")`, etc. +- Frontend templates use `pve-` CSS prefix mimicking Proxmox VE design system +- HTML fragments (e.g. `_analysis.html`) are injected via `innerHTML` — no `