vm-bench/AGENTS.md

# AGENTS.md — VM Bench (VMware → Proxmox Image Converter)

## Project Overview

VM Bench is a two-tier web application for converting VMware disk images (VMDK, VHD,
etc.) into Proxmox-compatible QCOW2 disks and provisioning VMs with automatic OS and
boot type detection.

- **Repository**: `https://git.lohmar.co.uk/cclohmar/vm-bench.git`
- **Deployment root**: `/mnt/converter/` (bind-mounted between host and LXC)
- **Frontend**: FastAPI + Jinja2 web UI in an LXC container (`vm-bench`) on port 5000
- **Backend**: FastAPI REST API on the Proxmox host (`srv2`) on port 9000
- **API spec**: `open-api.yaml` (single source of truth — derive models from this)
- **Python version**: 3.13 (backend), 3.11 (frontend LXC)

## Architecture

```
Browser            vm-bench LXC             Proxmox Host (srv2)
                   (this container)
──→ :5000 ──→  frontend/app.py  ──→  :9000  backend/app.py
                                       │
                                       ├─ qemu-img   (format, size)
                                       ├─ guestfish  (OS, EFI detection)
                                       ├─ 7z/unzip   (archive extraction)
                                       └─ qm create  (VM provisioning)
```

- **Frontend** (`vm-bench` LXC) — user-facing web UI, manages sessions, proxies API calls
- **Backend** (Proxmox host) — disk conversion, analysis, VM provisioning
- **Shared storage** — `/mnt/converter/tmp/` (staging), `/mnt/converter/logs/` (logs)

## Directory Layout

```
/mnt/converter/
├── install.sh                    # Main deployment script (host-level)
├── clean.sh                      # Clean tmp dirs + restart backend
├── open-api.yaml                 # API specification (single source of truth)
├── README.md
├── AGENTS.md                     # This file
├── backend/                      # Backend API (runs on Proxmox host)
│   ├── app.py                    # FastAPI app, routes, logging
│   ├── models.py                 # Pydantic request/response models
│   ├── converter.py              # Archive extraction, disk probing, OS/EFI detection
│   ├── provisioner.py            # VM provisioning (qm create/importdisk/convert)
│   ├── requirements.txt          # fastapi, uvicorn, pydantic
│   └── vm-bench-backend.service  # Systemd unit
├── frontend/                     # Web UI (runs on vm-bench LXC)
│   ├── app.py                    # FastAPI app, routes, download/SCP handling
│   ├── api_client.py             # Typed REST client → backend
│   ├── setup.sh                  # Frontend installer (inside LXC)
│   ├── requirements.txt          # fastapi, uvicorn, jinja2, requests, python-multipart
│   ├── vm-bench.service          # Systemd unit
│   ├── templates/                # Jinja2 HTML templates
│   │   ├── base.html             # Base layout with session GUID + header/footer
│   │   ├── index.html            # New session form + download + progress
│   │   ├── _analysis.html        # Analysis result + confirm form (HTML fragment)
│   │   ├── polling.html          # Progress bar + job status + reuse section
│   │   └── scp.html              # SCP pull page
│   └── static/
│       └── proxmox.css           # Proxmox VE-inspired CSS theme
├── logs/                         # Shared log files (gitignored)
├── tmp/                          # Staging files (gitignored)
└── venv/                         # Backend Python venv (gitignored)
```

## Key Files by Responsibility

### Backend (`backend/`)

| File | Purpose |
|------|---------|
| `backend/app.py:1` | FastAPI app with routes: `/api/v1/health`, `/analyze`, `/jobs`, `/jobs/{id}`, `/jobs/{id}/cleanup`, `/clone` |
| `backend/models.py:1` | Pydantic models: `JobSubmissionRequest`, `JobStatusResponse`, `AnalyzeRequest`, `AnalyzeResponse`, `DiskSpec`, `CleanupRequest`, `CloneRequest` |
| `backend/converter.py:1` | Archive extraction (`extract_if_needed`), disk discovery (`discover_disk`), OS detection (`detect_os`), EFI detection (`detect_efi`) |
| `backend/provisioner.py:1` | Background job processor: `submit_job`, `get_job_status`, `cleanup_staging`, `clone_vm`, `_process_job` (qemu-img convert → virt-resize → qm create → qm importdisk) |

### Frontend (`frontend/`)

| File | Purpose |
|------|---------|
| `frontend/app.py:1` | FastAPI app with routes: `/`, `/session/upload`, `/session/progress/{f}`, `/session/analyze`, `/session/confirm`, `/session/status/{id}`, `/session/cleanup/{id}`, `/session/clone`, `/scp`, `/scp/start`, `/scp/progress/{sid}/{f}` |
| `frontend/api_client.py:1` | `ApiClient` class — typed HTTP client for backend API with `ApiError` exception |

### Deployment (`install.sh`)

| File | Purpose |
|------|---------|
| `install.sh:1` | Master deployment script (Proxmox host). Modes: `--deploy`, `--update`, `--remove`. Creates directory tree, clones repo, installs system packages (qemu-img, guestfish, archive tools), Python venv, backend systemd service, LXC container with bind mount. |
| `frontend/setup.sh:1` | Frontend installer (run inside LXC). Installs system packages (python3, aria2, openssh-client, sshpass), Python deps, systemd service. |

### Systemd Units

| File | Port | Env Vars |
|------|------|----------|
| `backend/vm-bench-backend.service` | 9000 | None |
| `frontend/vm-bench.service` | 5000 | `BACKEND_URL=http://10.2.0.2:9000`, `TMPDIR=/mnt/converter/tmp` |

## Usage Flow

1. User opens web UI → session form
2. Upload or download a source image → lands in `/mnt/converter/tmp/{session_id}/in/`
3. Backend analyses image → shows format, size, OS, EFI status
4. Configure VM settings → name, CPU, RAM, storage, boot type
5. Submit job → backend converts, shrinks (if needed), creates VM
6. Poll progress → real-time status bar updates every 2 seconds via AJAX
7. Reuse or cleanup → create another VM or delete staging files

## Disk Sizing & Auto-Shrink Logic

- If `target_disk_size_gb` is omitted and the source disk is larger than its
  auto-shrink target (10% of virtual size, minimum 20 GB), the disk is shrunk
  using `virt-resize --shrink --resize-force`.
- If `virt-resize` fails, falls back to `qemu-img resize --shrink`.
- If `target_disk_size_gb` is larger than current, disk is expanded.

## Session Management

- Frontend generates a UUID session ID stored in `localStorage` + cookie (`vm_bench_sid`)
- Files are staged at `/mnt/converter/tmp/{session_id}/in/` and `out/`
- Cleanup can delete the entire session directory or preserve for reuse

## Source Image Ingestion Methods

1. **Download URL** (`index.html`) — uses `aria2c` with 8 connections, kills download if ETA > 1 hour
2. **SCP Pull** (`scp.html`) — uses `sshpass` + `scp` directly from remote servers
3. File upload (legacy, referenced but not primary flow)

## Backend Tools Required (on Proxmox host)

- `qemu-img` — disk format and size detection, format conversion
- `guestfish` / `virt-inspector` — OS type and EFI boot detection
- `virt-resize` — safe disk shrinking
- `7z`, `unzip`, `unrar`, `tar`, `gunzip`, `bunzip2`, `xz` — archive extraction
- `qm` (Proxmox CLI) — VM creation, disk import, configuration

## Frontend Tools Required (in LXC)

- `python3`, `pip3`, `wget`, `curl`, `aria2c`, `openssh-client`, `sshpass`

## Configuration

| Variable | Default | Description |
|----------|---------|-------------|
| `BACKEND_URL` | `http://10.2.0.2:9000` | Backend API base URL (set in systemd unit + env) |
| `VM_ID_MIN` | `21000` | Minimum allowed Proxmox VM ID |
| `VM_ID_MAX` | `21100` | Maximum allowed Proxmox VM ID |
| `DOWNLOAD_TIMEOUT` | `14400` (4h) | Max download time (safety net) |
| `SPEED_CHECK_AFTER` | `30` (seconds) | Wait before judging download speed |
| `MAX_ETA_SECONDS` | `3600` (1h) | Kill download if ETA exceeds this |

## Job States

```
queued → processing_conversion → importing_storage → completed
                                                  → failed
```

## Logging

Both services log to console (systemd journal) and rotating file handler:
- Frontend: `/mnt/converter/logs/vm-bench.log`
- Backend: `/mnt/converter/logs/vm-bench-backend.log`
- Rotation: 10 MB max, 5 backup files

## Development Commands

### Backend (on Proxmox host)
```bash
# Install/update
bash install.sh --deploy

# Restart
systemctl restart vm-bench-backend

# View logs
tail -f /mnt/converter/logs/vm-bench-backend.log
journalctl -u vm-bench-backend -f

# Health check
curl http://127.0.0.1:9000/api/v1/health
```

### Frontend (inside LXC)
```bash
# Install/update
bash /mnt/converter/frontend/setup.sh

# Restart
systemctl restart vm-bench

# View logs
tail -f /mnt/converter/logs/vm-bench.log
journalctl -u vm-bench -f
```

### Cleanup script
```bash
bash /mnt/converter/clean.sh  # Clears tmp dirs + restarts backend
```

## Project Conventions

- **No code comments** unless strictly necessary (follow existing patterns)
- Models derived from `open-api.yaml` — update spec first, then code
- `backend/provisioner.py` uses in-memory job store (`_jobs` dict) with `threading.Lock`
- All paths use `pathlib.Path`
- Loggers use module-level naming: `logging.getLogger("backend")`, `logging.getLogger("backend.converter")`, etc.
- Frontend templates use `pve-` CSS prefix mimicking Proxmox VE design system
- HTML fragments (e.g. `_analysis.html`) are injected via `innerHTML` — no `<script>` tags
- Session-based isolation: each browser tab/session gets a UUID for file staging
- VM names are sanitized to DNS-safe characters: `[^a-zA-Z0-9-]` replaced with `-`
- VM ID range is validated client-side (21000–21100) and enforced in frontend routes