vm-bench/README.md

# VM Bench — Proxmox Image Conversion Frontend

Web GUI and REST API for converting VMware disk images (VMDK, VHD, etc.) into
Proxmox-compatible QCOW2 disks and provisioning VMs with automatic OS and boot
type detection.

## Architecture

```
Browser            vm-bench LXC             Proxmox Host (srv2)
                  (this container)
──→ :5000 ──→  frontend/app.py  ──→  :9000  backend/app.py
                                      │
                                      ├─ qemu-img   (format, size)
                                      ├─ guestfish  (OS, EFI detection)
                                      ├─ 7z/unzip   (archive extraction)
                                      └─ qm create  (VM provisioning)
```

- **Frontend** (`vm-bench` LXC) — FastAPI + Jinja2 web UI on port 5000
- **Backend** (`srv2` Proxmox host) — FastAPI REST API on port 9000
- **Shared storage** — `/mnt/converter/in` (staging) and `/mnt/converter/out` (output)

## Directory Layout

```
/mnt/converter/
├── frontend/                  # Web UI (runs on vm-bench LXC)
│   ├── app.py                 # FastAPI app, routes, session handling
│   ├── api_client.py          # Typed REST client → backend
│   ├── install.sh             # Frontend installer (system deps + service)
│   ├── vm-bench.service       # Systemd unit for the frontend
│   ├── requirements.txt
│   ├── templates/             # Jinja2 HTML templates
│   │   ├── base.html
│   │   ├── index.html         # New session form
│   │   ├── _analysis.html     # Analysis result + confirm form
│   │   └── polling.html       # Progress bar + job status
│   └── static/
│       └── proxmox.css
├── backend/                   # REST API (deploys to Proxmox host srv2)
│   ├── app.py                 # FastAPI app, routes
│   ├── models.py              # Pydantic request/response models
│   ├── converter.py           # Archive extraction, disk probing, OS/EFI detection
│   ├── provisioner.py         # VM provisioning (qm create/importdisk)
│   ├── install.sh             # Full deployment script (backend + LXC creation)
│   ├── vm-bench-backend.service
│   └── requirements.txt
├── open-api.yaml              # API spec (single source of truth)
├── in/                        # Shared staging directory (gitignored)
├── out/                       # Shared output directory (gitignored)
├── logs/                       # Shared log files (gitignored)
```

## Features

- **File upload or URL download** — ingest `.vmdk`, `.vhd`, `.7z`, `.zip`, `.tar.gz` archives
- **Automatic archive extraction** — 7z, unzip, tar, gunzip
- **Disk analysis** — detects format, virtual size, guest OS, EFI bootability
- **VM provisioning** — creates Proxmox VM with automatic disk import and boot config
- **Progress polling** — real-time job status with progress bar
- **Staging reuse** — keep source files to create multiple VMs from one image

## Prerequisites

### Proxmox host (`srv2`)

The host must be a Proxmox VE node with:
- `pct` — LXC container management (built-in)
- `git` — to clone the repository (install script handles this)
- Internet access to download container templates

### LXC container (`vm-bench`)

Created automatically by the install script. Requires:
- 3 CPU cores, 6 GB RAM, 24 GB disk, 2 GB swap
- Debian 12 template (downloaded by script)
- Network bridge `vmbr0`

## Deployment

All deployment is driven by a single script on the Proxmox host. It creates the
directory tree, clones the repo, installs the backend, and provisions the
`vm-bench` LXC container ready for the frontend.

### Step 1: Clone & deploy (on `srv2`)

```bash
# Create the shared directory and clone the repo
mkdir -p /mnt/converter
cd /mnt/converter
git clone https://git.lohmar.co.uk/cclohmar/vm-bench.git .

# Run the installer — handles everything interactively
bash backend/install.sh
```

The script will:
1. Create the `/mnt/converter/` directory tree
2. Clone or update the git repository
3. Install system packages (qemu-img, guestfish, archive tools, etc.)
4. Install Python dependencies and start the backend systemd service
5. **Prompt for container ID + network settings**
6. **Create and configure the `vm-bench` LXC container** with:
   - 3 cores, 6 GB RAM, 24 GB rootfs, 2 GB swap
   - Bind mount: `/mnt/converter` shared with host
   - Unprivileged mode off, nesting enabled
   - AppArmor unconfined, cgroup device access allowed
7. Optionally start the container

Example prompts:
```
Container ID (e.g. 20020): 20020
IP address/CIDR  [10.2.0.20/16]:
Gateway          [10.2.0.2]:
Bridge           [vmbr0]:
MAC address      [BC:24:11:80:5A:B1]:
```

Verify the backend is running:
```bash
curl http://127.0.0.1:9000/api/v1/health
# → {"status":"ok"}
```

### Step 2: Install frontend (inside the LXC)

After the container is created and started, enter it:

```bash
pct enter 20020   # use your container ID
```

Inside the container, run the frontend installer:

```bash
bash /mnt/converter/frontend/install.sh
```

This installs Python, pip, `wget`, `curl`, Python dependencies, and starts the
`vm-bench` systemd service on port 5000.

Verify:
```bash
curl http://127.0.0.1:5000
# → HTML response (the web UI)
```

Open `http://<container-ip>:5000` in a browser.

### Container LXC Config (created by install script)

| Setting | Value |
|---------|-------|
| Arch | `amd64` |
| Cores | `3` |
| Memory | `6144` MB |
| Swap | `2048` MB |
| Rootfs | `local-lvm:24G` |
| OS type | `debian` |
| Unprivileged | `0` (disabled) |
| Features | `nesting=1` |
| Network | `vmbr0`, firewall enabled, veth |
| Bind mount | `mp0: /mnt/converter,mp=/mnt/converter` |
| AppArmor | `unconfined` |
| Cgroup | `devices.allow: a`, `mount.auto: proc:rw sys:rw cgroup:rw` |

### Re-deploying / updating

The install scripts are idempotent — safe to re-run:

```bash
# On srv2: update backend
cd /mnt/converter && git pull
bash backend/install.sh    # skips LXC creation if container exists

# Inside LXC: update frontend
cd /mnt/converter && git pull
bash frontend/install.sh
```

## API Reference

All endpoints are under `/api/v1/`. Full specification in [`open-api.yaml`](open-api.yaml).

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/health` | Health check |
| `POST` | `/api/v1/analyze` | Probe source image (format, size, OS, EFI) |
| `POST` | `/api/v1/jobs` | Submit conversion + provisioning job |
| `GET` | `/api/v1/jobs/{id}` | Poll job status and progress |
| `POST` | `/api/v1/jobs/{id}/cleanup` | Delete or preserve staging files |

### Example: Analyze a source image

```bash
curl -X POST http://10.2.0.2:9000/api/v1/analyze \
  -H "Content-Type: application/json" \
  -d '{"vmid": 21050, "source_filename": "64bit/Debian 12.11.0 (64bit).vmdk"}'

# Response:
# {
#   "vmid": 21050,
#   "filename": "Debian 12.11.0 (64bit).vmdk",
#   "disk_format": "vmdk",
#   "disk_size_gb": 7.5,
#   "os_type": "Debian",
#   "efi_detectable": true
# }
```

### Example: Submit a conversion job

```bash
curl -X POST http://10.2.0.2:9000/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "vmid": 21050,
    "vm_name": "debian-test",
    "boot_disk": {
      "disk_type": "image_file",
      "source_filename": "64bit/Debian 12.11.0 (64bit).vmdk",
      "format": "vmdk"
    },
    "cpu_cores": 2,
    "ram_mb": 4096,
    "target_storage": "local-lvm"
  }'

# Response:
# { "job_id": "job_21050_1737480000", "vmid": 21050, "status": "queued", ... }
```

## Usage Flow

1. **Open the web UI** → new session form
2. **Upload or download a source image** → file lands in `/mnt/converter/in/`
3. **Backend analyses the image** → shows format, size, OS, EFI status
4. **Configure VM settings** → name, CPU, RAM, storage, boot type
5. **Submit job** → backend converts, shrinks (if needed), creates VM
6. **Poll progress** → real-time status bar updates every 2 seconds
7. **Reuse or cleanup** → create another VM from same source, or delete staging files

### Disk sizing and auto-shrink

Virtual disk images (VMDK, QCOW2) have two sizes:

| | Meaning | Example |
|---|---|---|
| **Virtual size** | Maximum the disk *could* grow to | 500 GB |
| **Actual size** | Real data in the image (sparse file) | ~7 GB |

During analysis, the UI shows the virtual size (e.g. *"500.0 GB detected. Omit to
auto-shrink to 30 GB"*). This is because the source image was created with a
large virtual disk, but only a fraction is used.

**Target Disk Size field — what happens:**

| You enter | Result |
|-----------|--------|
| **Blank** (default) | Auto-shrinks to 30 GB if the source is larger. Uses `virt-resize --shrink --resize-force` to safely reduce the disk container. |
| **A number** (e.g. `500`) | Uses that exact size. No shrinking. |
| **A number < virtual** (e.g. `40`) | Shrinks the disk to 40 GB. |
| **A number > virtual** (e.g. `600`) | Expands the disk. Safe operation. |

> **Tip:** For most Linux VMs (Debian, Ubuntu, etc.), 30 GB is plenty.
> You can always expand the disk later in Proxmox if you need more space.
> If you plan to store large datasets inside the VM, set a higher value.

## Configuration

| Variable | Default | Description |
|----------|---------|-------------|
| `BACKEND_URL` | `http://10.2.0.2:9000` | Backend API base URL (set in systemd unit) |
| `VM_ID_MIN` | `21000` | Minimum allowed Proxmox VM ID |
| `VM_ID_MAX` | `21100` | Maximum allowed Proxmox VM ID |

## Development

**Source of truth**: `open-api.yaml` — always derive models from this file.

**Backend tool requirements** (on srv2):
- `qemu-img` — disk format and size detection
- `guestfish` / `virt-inspector` — OS type and EFI boot detection
- `7z`, `unzip`, `tar`, `gunzip`, `bunzip2`, `xz` — archive extraction

**Provisioner note**: `backend/provisioner.py` is currently a stub with simulated
progress. Replace with real `qm create`, `qm importdisk`, and `qm set` commands
when deploying to the Proxmox host.

**Staging files**: `/mnt/converter/in/`, `/mnt/converter/out/`, and `/mnt/converter/logs/` are gitignored —
they contain runtime data, not code.

## Logging

Both services write to two locations simultaneously:

| Destination | How to access |
|-------------|---------------|
| **File** (shared) | `tail -f /mnt/converter/logs/vm-bench.log` (frontend) or `vm-bench-backend.log` (backend) |
| **Journal** (per-service) | `journalctl -u vm-bench -f` (frontend) or `journalctl -u vm-bench-backend -f` (backend) |

Log files rotate automatically at 10 MB, keeping 5 historical files
(e.g. `vm-bench.log`, `vm-bench.log.1`, `vm-bench.log.2`, …).

The file logs are the authoritative source — they're written to the shared
`/mnt/converter/logs/` directory, accessible from both the Proxmox host and
the LXC container.