cclohmar/vm-bench
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: update README with current directory layout, install script download link, SCP/clone features, and session isolation

Browse source
This commit is contained in:
Claus Lohmar 2026-07-25 18:17:09 +00:00
parent f1978008bd
commit 1e14d96052
1 changed files with 135 additions and 203 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

338
README.md
View file

@ -1,144 +1,118 @@
# VM Bench — Proxmox Image Conversion Frontend # VM Bench — Proxmox Image Conversion
Web GUI and REST API for converting VMware disk images (VMDK, VHD, etc.) into Web GUI and REST API for converting VMware disk images (VMDK, VHD, etc.) into
Proxmox-compatible QCOW2 disks and provisioning VMs with automatic OS and boot Proxmox-compatible QCOW2 disks and provisioning VMs with automatic OS and boot
type detection. type detection.
## Quick Start
Download and run the install script on your Proxmox host:
```bash
wget https://git.lohmar.co.uk/cclohmar/vm-bench/raw/branch/main/install.sh
bash install.sh --deploy
```
The script interactively:
1. Selects a storage pool
2. Installs the backend API (port 9000) on the host
3. Creates an LXC container with bind-mounted `/mnt/converter`
4. Inside the LXC, run `bash /mnt/converter/frontend/setup.sh` to start the web UI
## Architecture ## Architecture
``` ```
Browser vm-bench LXC Proxmox Host (srv2) Browser vm-bench LXC Proxmox Host
(this container) (this container)
──→ :5000 ──→ frontend/app.py ──→ :9000 backend/app.py ──→ :5000 ──→ frontend/app.py ──→ :9000 backend/app.py
│ │
├─ qemu-img (format, size) ├─ qemu-img (format, size)
├─ guestfish (OS, EFI detection) ├─ guestfish (OS, EFI detection)
├─ 7z/unzip (archive extraction) ├─ 7z/unzip (archive extraction)
└─ qm create (VM provisioning) └─ qm create (VM provisioning)
``` ```
- **Frontend** (`vm-bench` LXC) — FastAPI + Jinja2 web UI on port 5000 - **Frontend** (`vm-bench` LXC) — FastAPI + Jinja2 web UI on port 5000
- **Backend** (`srv2` Proxmox host) — FastAPI REST API on port 9000 - **Backend** (Proxmox host) — FastAPI REST API on port 9000
- **Shared storage** — `/mnt/converter/in` (staging) and `/mnt/converter/out` (output) - **Shared storage** — `/mnt/converter/tmp/` (staging, bind-mounted)
## Directory Layout ## Directory Layout
``` ```
/mnt/converter/ /mnt/converter/
├── frontend/ # Web UI (runs on vm-bench LXC) ├── install.sh # Master deployment script (run on Proxmox host)
│ ├── app.py # FastAPI app, routes, session handling ├── clean.sh # Clean tmp dirs + restart backend
│ ├── api_client.py # Typed REST client → backend ├── open-api.yaml # API spec (single source of truth)
│ ├── install.sh # Frontend installer (system deps + service) ├── AGENTS.md # Developer guide
│ ├── vm-bench.service # Systemd unit for the frontend ├── README.md # This file
│ ├── requirements.txt ├── backend/ # REST API (runs on Proxmox host)
│ ├── templates/ # Jinja2 HTML templates │ ├── app.py # FastAPI app, routes, logging
│ │ ├── base.html │ ├── models.py # Pydantic request/response models
│ │ ├── index.html # New session form │ ├── converter.py # Archive extraction, disk probing, OS/EFI detection
│ │ ├── _analysis.html # Analysis result + confirm form │ ├── provisioner.py # VM provisioning (qemu-img convert, qm create/importdisk)
│ │ └── polling.html # Progress bar + job status
│ └── static/
│ └── proxmox.css
├── backend/ # REST API (deploys to Proxmox host srv2)
│ ├── app.py # FastAPI app, routes
│ ├── models.py # Pydantic request/response models
│ ├── converter.py # Archive extraction, disk probing, OS/EFI detection
│ ├── provisioner.py # VM provisioning (qm create/importdisk)
│ ├── install.sh # Full deployment script (backend + LXC creation)
│ ├── vm-bench-backend.service │ ├── vm-bench-backend.service
│ └── requirements.txt │ └── requirements.txt
├── open-api.yaml # API spec (single source of truth) ├── frontend/ # Web UI (runs on vm-bench LXC)
├── in/ # Shared staging directory (gitignored) │ ├── app.py # FastAPI app, routes, download/SCP handling
├── out/ # Shared output directory (gitignored) │ ├── api_client.py # Typed REST client → backend
│ ├── setup.sh # Frontend installer (inside LXC)
│ ├── vm-bench.service # Systemd unit
│ ├── requirements.txt
│ ├── templates/ # Jinja2 HTML templates
│ │ ├── base.html # Base layout with session GUID
│ │ ├── index.html # New session form + download + progress
│ │ ├── _analysis.html # Analysis result + confirm form
│ │ ├── polling.html # Progress bar + job status + reuse section
│ │ └── scp.html # SCP pull page
│ └── static/
│ └── proxmox.css # Proxmox VE-inspired theme
├── logs/ # Shared log files (gitignored) ├── logs/ # Shared log files (gitignored)
└── tmp/ # Staging files per session (gitignored)
``` ```
## Features ## Features
- **File upload or URL download** — ingest `.vmdk`, `.vhd`, `.7z`, `.zip`, `.tar.gz` archives - **URL download** — ingest archives via `aria2c` with 8 connections
- **Automatic archive extraction** — 7z, unzip, tar, gunzip - **SCP pull** — transfer files directly from remote servers, no middle-hop
- **Archive extraction** — 7z, zip, rar, tar, gz, bz2, xz
- **Disk analysis** — detects format, virtual size, guest OS, EFI bootability - **Disk analysis** — detects format, virtual size, guest OS, EFI bootability
- **VM provisioning** — creates Proxmox VM with automatic disk import and boot config - **VM provisioning** — qemu-img convert + optional virt-resize shrink + qm create/importdisk
- **Progress polling** — real-time job status with progress bar - **VM cloning** — instant full clone of an existing VM
- **Progress polling** — real-time status bar updates every 2 seconds
- **Session isolation** — UUID per browser session for file staging
- **Staging reuse** — keep source files to create multiple VMs from one image - **Staging reuse** — keep source files to create multiple VMs from one image
## Prerequisites
### Proxmox host (`srv2`)
The host must be a Proxmox VE node with:
- `pct` — LXC container management (built-in)
- `git` — to clone the repository (install script handles this)
- Internet access to download container templates
### LXC container (`vm-bench`)
Created automatically by the install script. Requires:
- 3 CPU cores, 6 GB RAM, 24 GB disk, 2 GB swap
- Debian 12 template (downloaded by script)
- Network bridge `vmbr0`
## Deployment ## Deployment
All deployment is driven by a single script on the Proxmox host. It creates the ### Step 1: Backend (on Proxmox host)
directory tree, clones the repo, installs the backend, and provisions the
`vm-bench` LXC container ready for the frontend.
### Step 1: Clone & deploy (on `srv2`)
```bash ```bash
# Create the shared directory and clone the repo wget https://git.lohmar.co.uk/cclohmar/vm-bench/raw/branch/main/install.sh
mkdir -p /mnt/converter bash install.sh --deploy
cd /mnt/converter
git clone https://git.lohmar.co.uk/cclohmar/vm-bench.git .
# Run the installer — handles everything interactively
bash backend/install.sh
``` ```
The script will: The script interactively:
1. Create the `/mnt/converter/` directory tree 1. Scans available Proxmox storage pools and lets you pick one
2. Clone or update the git repository 2. Prompts for install path, container ID, IP, gateway, bridge, MAC
3. Install system packages (qemu-img, guestfish, archive tools, etc.) 3. Installs system packages (qemu-img, guestfish, 7z, unzip, etc.)
4. Install Python dependencies and start the backend systemd service 4. Creates Python venv, installs deps, generates and starts `vm-bench-backend` service
5. **Prompt for container ID + network settings** 5. Creates the `vm-bench` LXC container with bind mount
6. **Create and configure the `vm-bench` LXC container** with:
- 3 cores, 6 GB RAM, 24 GB rootfs, 2 GB swap
- Bind mount: `/mnt/converter` shared with host
- Unprivileged mode off, nesting enabled
- AppArmor unconfined, cgroup device access allowed
7. Optionally start the container
Example prompts: Verify:
```
Container ID (e.g. 20020): 20020
IP address/CIDR [10.2.0.20/16]:
Gateway [10.2.0.2]:
Bridge [vmbr0]:
MAC address [BC:24:11:80:5A:B1]:
```
Verify the backend is running:
```bash ```bash
curl http://127.0.0.1:9000/api/v1/health curl http://127.0.0.1:9000/api/v1/health
# → {"status":"ok"} # → {"status":"ok"}
``` ```
### Step 2: Install frontend (inside the LXC) ### Step 2: Frontend (inside the LXC)
After the container is created and started, enter it:
```bash ```bash
pct enter 20020 # use your container ID pct enter 20020 # use your container ID
bash /mnt/converter/frontend/setup.sh
``` ```
Inside the container, run the frontend installer: Installs system tools (python3, aria2, sshpass, etc.), Python deps, and starts
`vm-bench` on port 5000.
```bash
bash /mnt/converter/frontend/install.sh
```
This installs Python, pip, `wget`, `curl`, Python dependencies, and starts the
`vm-bench` systemd service on port 5000.
Verify: Verify:
```bash ```bash
@ -148,40 +122,44 @@ curl http://127.0.0.1:5000
Open `http://<container-ip>:5000` in a browser. Open `http://<container-ip>:5000` in a browser.
### Container LXC Config (created by install script)
| Setting | Value |
|---------|-------|
| Arch | `amd64` |
| Cores | `3` |
| Memory | `6144` MB |
| Swap | `2048` MB |
| Rootfs | `local-lvm:24G` |
| OS type | `debian` |
| Unprivileged | `0` (disabled) |
| Features | `nesting=1` |
| Network | `vmbr0`, firewall enabled, veth |
| Bind mount | `mp0: /mnt/converter,mp=/mnt/converter` |
| AppArmor | `unconfined` |
| Cgroup | `devices.allow: a`, `mount.auto: proc:rw sys:rw cgroup:rw` |
### Re-deploying / updating ### Re-deploying / updating
The install scripts are idempotent — safe to re-run: ```bash
# On Proxmox host
bash install.sh --update
# Inside LXC
bash /mnt/converter/frontend/setup.sh
```
### Uninstall
```bash ```bash
# On srv2: update backend bash install.sh --remove
cd /mnt/converter && git pull
bash backend/install.sh # skips LXC creation if container exists
# Inside LXC: update frontend
cd /mnt/converter && git pull
bash frontend/install.sh
``` ```
## Usage Flow
1. **Open web UI** → new session form (UUID auto-generated)
2. **Download or SCP pull** a source image → `/mnt/converter/tmp/{session_id}/in/`
3. **Analyse** → backend probes format, size, OS, EFI
4. **Configure VM** → name, CPU, RAM, storage, boot type, target disk size
5. **Submit job** → backend converts, shrinks (if needed), creates VM
6. **Poll progress** → real-time updates every 2 seconds
7. **Reuse or cleanup** → clone, re-convert from same source, or delete staging
### Disk sizing and auto-shrink
| You enter | Result |
|-----------|--------|
| **Blank** (default) | Auto-shrinks to 10% of virtual size (min 20 GB) via `virt-resize --shrink --resize-force` |
| **A number** (e.g. `500`) | Uses that exact size. No shrinking. |
| **A number < virtual** (e.g. `40`) | Shrinks the disk to 40 GB. |
| **A number > virtual** (e.g. `600`) | Expands the disk. |
## API Reference ## API Reference
All endpoints are under `/api/v1/`. Full specification in [`open-api.yaml`](open-api.yaml). All endpoints under `/api/v1/`. Full spec: [`open-api.yaml`](open-api.yaml).
| Method | Path | Description | | Method | Path | Description |
|--------|------|-------------| |--------|------|-------------|
@ -190,23 +168,17 @@ All endpoints are under `/api/v1/`. Full specification in [`open-api.yaml`](open
| `POST` | `/api/v1/jobs` | Submit conversion + provisioning job | | `POST` | `/api/v1/jobs` | Submit conversion + provisioning job |
| `GET` | `/api/v1/jobs/{id}` | Poll job status and progress | | `GET` | `/api/v1/jobs/{id}` | Poll job status and progress |
| `POST` | `/api/v1/jobs/{id}/cleanup` | Delete or preserve staging files | | `POST` | `/api/v1/jobs/{id}/cleanup` | Delete or preserve staging files |
| `POST` | `/api/v1/clone` | Clone an existing VM |
### Example: Analyze a source image ### Example: Analyze a source image
```bash ```bash
curl -X POST http://10.2.0.2:9000/api/v1/analyze \ curl -X POST http://10.2.0.2:9000/api/v1/analyze \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{"vmid": 21050, "source_filename": "64bit/Debian 12.11.0 (64bit).vmdk"}' -d '{"vmid": 21050, "source_filename": "session-id/in/image.vmdk"}'
# Response: # → {"vmid": 21050, "filename": "image.vmdk", "disk_format": "vmdk",
# { # "disk_size_gb": 7.5, "os_type": "Debian", "efi_detectable": true}
# "vmid": 21050,
# "filename": "Debian 12.11.0 (64bit).vmdk",
# "disk_format": "vmdk",
# "disk_size_gb": 7.5,
# "os_type": "Debian",
# "efi_detectable": true
# }
``` ```
### Example: Submit a conversion job ### Example: Submit a conversion job
@ -215,96 +187,56 @@ curl -X POST http://10.2.0.2:9000/api/v1/analyze \
curl -X POST http://10.2.0.2:9000/api/v1/jobs \ curl -X POST http://10.2.0.2:9000/api/v1/jobs \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
"vmid": 21050, "vmid": 21050, "vm_name": "debian-test",
"vm_name": "debian-test", "boot_disk": {"disk_type": "image_file", "source_filename": "session/in/image.vmdk", "format": "vmdk"},
"boot_disk": { "cpu_cores": 2, "ram_mb": 4096, "target_storage": "local-lvm"
"disk_type": "image_file",
"source_filename": "64bit/Debian 12.11.0 (64bit).vmdk",
"format": "vmdk"
},
"cpu_cores": 2,
"ram_mb": 4096,
"target_storage": "local-lvm"
}' }'
# Response: # → {"job_id": "job_21050_1737480000", "vmid": 21050, "status": "queued", ...}
# { "job_id": "job_21050_1737480000", "vmid": 21050, "status": "queued", ... }
``` ```
## Usage Flow
1. **Open the web UI** → new session form
2. **Upload or download a source image** → file lands in `/mnt/converter/in/`
3. **Backend analyses the image** → shows format, size, OS, EFI status
4. **Configure VM settings** → name, CPU, RAM, storage, boot type
5. **Submit job** → backend converts, shrinks (if needed), creates VM
6. **Poll progress** → real-time status bar updates every 2 seconds
7. **Reuse or cleanup** → create another VM from same source, or delete staging files
### Disk sizing and auto-shrink
Virtual disk images (VMDK, QCOW2) have two sizes:
| | Meaning | Example |
|---|---|---|
| **Virtual size** | Maximum the disk *could* grow to | 500 GB |
| **Actual size** | Real data in the image (sparse file) | ~7 GB |
During analysis, the UI shows the virtual size (e.g. *"500.0 GB detected. Omit to
auto-shrink"*). This is because the source image was created with a
large virtual disk, but only a fraction is used.
**Auto-shrink**: if you leave Target Disk Size blank, the disk shrinks to
**10% of its virtual size, minimum 20 GB**. A 500 GB disk becomes 50 GB,
an 80 GB disk becomes 20 GB. You can override this by entering a value.
| You enter | Result |
|-----------|--------|
| **Blank** (default) | Auto-shrinks to 30 GB if the source is larger. Uses `virt-resize --shrink --resize-force` to safely reduce the disk container. |
| **A number** (e.g. `500`) | Uses that exact size. No shrinking. |
| **A number < virtual** (e.g. `40`) | Shrinks the disk to 40 GB. |
| **A number > virtual** (e.g. `600`) | Expands the disk. Safe operation. |
> **Tip:** For most Linux VMs (Debian, Ubuntu, etc.), 30 GB is plenty.
> You can always expand the disk later in Proxmox if you need more space.
> If you plan to store large datasets inside the VM, set a higher value.
## Configuration ## Configuration
| Variable | Default | Description | | Variable | Default | Description |
|----------|---------|-------------| |----------|---------|-------------|
| `BACKEND_URL` | `http://10.2.0.2:9000` | Backend API base URL (set in systemd unit) | | `BACKEND_URL` | `http://10.2.0.2:9000` | Backend API base URL |
| `VM_ID_MIN` | `21000` | Minimum allowed Proxmox VM ID | | `VM_ID_MIN` | `21000` | Minimum allowed Proxmox VM ID |
| `VM_ID_MAX` | `21100` | Maximum allowed Proxmox VM ID | | `VM_ID_MAX` | `21100` | Maximum allowed Proxmox VM ID |
| `DOWNLOAD_TIMEOUT` | `14400` (4h) | Max download time |
| `MAX_ETA_SECONDS` | `3600` (1h) | Kill download if ETA exceeds this |
## Development ## Development
**Source of truth**: `open-api.yaml` — always derive models from this file. **Source of truth**: `open-api.yaml` — update spec first, then derive models.
**Backend tool requirements** (on srv2): ### Backend tools required (on Proxmox host)
- `qemu-img` — disk format and size detection - `qemu-img` — disk format, size detection, format conversion
- `guestfish` / `virt-inspector` — OS type and EFI boot detection - `guestfish` / `virt-inspector` — OS type and EFI boot detection
- `7z`, `unzip`, `tar`, `gunzip`, `bunzip2`, `xz` — archive extraction - `virt-resize` — safe disk shrinking
- `7z`, `unzip`, `unrar`, `tar`, `gunzip`, `bunzip2`, `xz` — archive extraction
- `qm` (Proxmox CLI) — VM creation, disk import, configuration
**Provisioner note**: `backend/provisioner.py` is currently a stub with simulated ### Frontend tools required (in LXC)
progress. Replace with real `qm create`, `qm importdisk`, and `qm set` commands - `python3`, `pip3`, `wget`, `curl`, `aria2c`, `openssh-client`, `sshpass`
when deploying to the Proxmox host.
**Staging files**: `/mnt/converter/in/`, `/mnt/converter/out/`, and `/mnt/converter/logs/` are gitignored — ### Job states
they contain runtime data, not code.
```
queued → processing_conversion → importing_storage → completed
→ failed
```
## Logging ## Logging
Both services write to two locations simultaneously: Both services log to systemd journal and rotating file:
| Destination | How to access | | Service | File Log | Journal |
|-------------|---------------| |---------|----------|---------|
| **File** (shared) | `tail -f /mnt/converter/logs/vm-bench.log` (frontend) or `vm-bench-backend.log` (backend) | | Backend | `/mnt/converter/logs/vm-bench-backend.log` | `journalctl -u vm-bench-backend -f` |
| **Journal** (per-service) | `journalctl -u vm-bench -f` (frontend) or `journalctl -u vm-bench-backend -f` (backend) | | Frontend | `/mnt/converter/logs/vm-bench.log` | `journalctl -u vm-bench -f` |
Log files rotate automatically at 10 MB, keeping 5 historical files Rotation: 10 MB max, 5 backup files.
(e.g. `vm-bench.log`, `vm-bench.log.1`, `vm-bench.log.2`, …).
The file logs are the authoritative source — they're written to the shared ## License
`/mnt/converter/logs/` directory, accessible from both the Proxmox host and
the LXC container. MIT