From 1e14d960528cbb8b0f426640b095e5413210ccd0 Mon Sep 17 00:00:00 2001 From: Claus Lohmar Date: Sat, 25 Jul 2026 18:17:09 +0000 Subject: [PATCH] docs: update README with current directory layout, install script download link, SCP/clone features, and session isolation --- README.md | 338 ++++++++++++++++++++++-------------------------------- 1 file changed, 135 insertions(+), 203 deletions(-) diff --git a/README.md b/README.md index 812d272..8710169 100644 --- a/README.md +++ b/README.md @@ -1,144 +1,118 @@ -# VM Bench — Proxmox Image Conversion Frontend +# VM Bench — Proxmox Image Conversion Web GUI and REST API for converting VMware disk images (VMDK, VHD, etc.) into Proxmox-compatible QCOW2 disks and provisioning VMs with automatic OS and boot type detection. +## Quick Start + +Download and run the install script on your Proxmox host: + +```bash +wget https://git.lohmar.co.uk/cclohmar/vm-bench/raw/branch/main/install.sh +bash install.sh --deploy +``` + +The script interactively: +1. Selects a storage pool +2. Installs the backend API (port 9000) on the host +3. Creates an LXC container with bind-mounted `/mnt/converter` +4. Inside the LXC, run `bash /mnt/converter/frontend/setup.sh` to start the web UI + ## Architecture ``` -Browser vm-bench LXC Proxmox Host (srv2) - (this container) +Browser vm-bench LXC Proxmox Host + (this container) ──→ :5000 ──→ frontend/app.py ──→ :9000 backend/app.py - │ - ├─ qemu-img (format, size) - ├─ guestfish (OS, EFI detection) - ├─ 7z/unzip (archive extraction) - └─ qm create (VM provisioning) + │ + ├─ qemu-img (format, size) + ├─ guestfish (OS, EFI detection) + ├─ 7z/unzip (archive extraction) + └─ qm create (VM provisioning) ``` - **Frontend** (`vm-bench` LXC) — FastAPI + Jinja2 web UI on port 5000 -- **Backend** (`srv2` Proxmox host) — FastAPI REST API on port 9000 -- **Shared storage** — `/mnt/converter/in` (staging) and `/mnt/converter/out` (output) +- **Backend** (Proxmox host) — FastAPI REST API on port 9000 +- **Shared storage** — `/mnt/converter/tmp/` (staging, bind-mounted) ## Directory Layout ``` /mnt/converter/ -├── frontend/ # Web UI (runs on vm-bench LXC) -│ ├── app.py # FastAPI app, routes, session handling -│ ├── api_client.py # Typed REST client → backend -│ ├── install.sh # Frontend installer (system deps + service) -│ ├── vm-bench.service # Systemd unit for the frontend -│ ├── requirements.txt -│ ├── templates/ # Jinja2 HTML templates -│ │ ├── base.html -│ │ ├── index.html # New session form -│ │ ├── _analysis.html # Analysis result + confirm form -│ │ └── polling.html # Progress bar + job status -│ └── static/ -│ └── proxmox.css -├── backend/ # REST API (deploys to Proxmox host srv2) -│ ├── app.py # FastAPI app, routes -│ ├── models.py # Pydantic request/response models -│ ├── converter.py # Archive extraction, disk probing, OS/EFI detection -│ ├── provisioner.py # VM provisioning (qm create/importdisk) -│ ├── install.sh # Full deployment script (backend + LXC creation) +├── install.sh # Master deployment script (run on Proxmox host) +├── clean.sh # Clean tmp dirs + restart backend +├── open-api.yaml # API spec (single source of truth) +├── AGENTS.md # Developer guide +├── README.md # This file +├── backend/ # REST API (runs on Proxmox host) +│ ├── app.py # FastAPI app, routes, logging +│ ├── models.py # Pydantic request/response models +│ ├── converter.py # Archive extraction, disk probing, OS/EFI detection +│ ├── provisioner.py # VM provisioning (qemu-img convert, qm create/importdisk) │ ├── vm-bench-backend.service │ └── requirements.txt -├── open-api.yaml # API spec (single source of truth) -├── in/ # Shared staging directory (gitignored) -├── out/ # Shared output directory (gitignored) +├── frontend/ # Web UI (runs on vm-bench LXC) +│ ├── app.py # FastAPI app, routes, download/SCP handling +│ ├── api_client.py # Typed REST client → backend +│ ├── setup.sh # Frontend installer (inside LXC) +│ ├── vm-bench.service # Systemd unit +│ ├── requirements.txt +│ ├── templates/ # Jinja2 HTML templates +│ │ ├── base.html # Base layout with session GUID +│ │ ├── index.html # New session form + download + progress +│ │ ├── _analysis.html # Analysis result + confirm form +│ │ ├── polling.html # Progress bar + job status + reuse section +│ │ └── scp.html # SCP pull page +│ └── static/ +│ └── proxmox.css # Proxmox VE-inspired theme ├── logs/ # Shared log files (gitignored) +└── tmp/ # Staging files per session (gitignored) ``` ## Features -- **File upload or URL download** — ingest `.vmdk`, `.vhd`, `.7z`, `.zip`, `.tar.gz` archives -- **Automatic archive extraction** — 7z, unzip, tar, gunzip +- **URL download** — ingest archives via `aria2c` with 8 connections +- **SCP pull** — transfer files directly from remote servers, no middle-hop +- **Archive extraction** — 7z, zip, rar, tar, gz, bz2, xz - **Disk analysis** — detects format, virtual size, guest OS, EFI bootability -- **VM provisioning** — creates Proxmox VM with automatic disk import and boot config -- **Progress polling** — real-time job status with progress bar +- **VM provisioning** — qemu-img convert + optional virt-resize shrink + qm create/importdisk +- **VM cloning** — instant full clone of an existing VM +- **Progress polling** — real-time status bar updates every 2 seconds +- **Session isolation** — UUID per browser session for file staging - **Staging reuse** — keep source files to create multiple VMs from one image -## Prerequisites - -### Proxmox host (`srv2`) - -The host must be a Proxmox VE node with: -- `pct` — LXC container management (built-in) -- `git` — to clone the repository (install script handles this) -- Internet access to download container templates - -### LXC container (`vm-bench`) - -Created automatically by the install script. Requires: -- 3 CPU cores, 6 GB RAM, 24 GB disk, 2 GB swap -- Debian 12 template (downloaded by script) -- Network bridge `vmbr0` - ## Deployment -All deployment is driven by a single script on the Proxmox host. It creates the -directory tree, clones the repo, installs the backend, and provisions the -`vm-bench` LXC container ready for the frontend. - -### Step 1: Clone & deploy (on `srv2`) +### Step 1: Backend (on Proxmox host) ```bash -# Create the shared directory and clone the repo -mkdir -p /mnt/converter -cd /mnt/converter -git clone https://git.lohmar.co.uk/cclohmar/vm-bench.git . - -# Run the installer — handles everything interactively -bash backend/install.sh +wget https://git.lohmar.co.uk/cclohmar/vm-bench/raw/branch/main/install.sh +bash install.sh --deploy ``` -The script will: -1. Create the `/mnt/converter/` directory tree -2. Clone or update the git repository -3. Install system packages (qemu-img, guestfish, archive tools, etc.) -4. Install Python dependencies and start the backend systemd service -5. **Prompt for container ID + network settings** -6. **Create and configure the `vm-bench` LXC container** with: - - 3 cores, 6 GB RAM, 24 GB rootfs, 2 GB swap - - Bind mount: `/mnt/converter` shared with host - - Unprivileged mode off, nesting enabled - - AppArmor unconfined, cgroup device access allowed -7. Optionally start the container +The script interactively: +1. Scans available Proxmox storage pools and lets you pick one +2. Prompts for install path, container ID, IP, gateway, bridge, MAC +3. Installs system packages (qemu-img, guestfish, 7z, unzip, etc.) +4. Creates Python venv, installs deps, generates and starts `vm-bench-backend` service +5. Creates the `vm-bench` LXC container with bind mount -Example prompts: -``` -Container ID (e.g. 20020): 20020 -IP address/CIDR [10.2.0.20/16]: -Gateway [10.2.0.2]: -Bridge [vmbr0]: -MAC address [BC:24:11:80:5A:B1]: -``` - -Verify the backend is running: +Verify: ```bash curl http://127.0.0.1:9000/api/v1/health # → {"status":"ok"} ``` -### Step 2: Install frontend (inside the LXC) - -After the container is created and started, enter it: +### Step 2: Frontend (inside the LXC) ```bash pct enter 20020 # use your container ID +bash /mnt/converter/frontend/setup.sh ``` -Inside the container, run the frontend installer: - -```bash -bash /mnt/converter/frontend/install.sh -``` - -This installs Python, pip, `wget`, `curl`, Python dependencies, and starts the -`vm-bench` systemd service on port 5000. +Installs system tools (python3, aria2, sshpass, etc.), Python deps, and starts +`vm-bench` on port 5000. Verify: ```bash @@ -148,40 +122,44 @@ curl http://127.0.0.1:5000 Open `http://:5000` in a browser. -### Container LXC Config (created by install script) - -| Setting | Value | -|---------|-------| -| Arch | `amd64` | -| Cores | `3` | -| Memory | `6144` MB | -| Swap | `2048` MB | -| Rootfs | `local-lvm:24G` | -| OS type | `debian` | -| Unprivileged | `0` (disabled) | -| Features | `nesting=1` | -| Network | `vmbr0`, firewall enabled, veth | -| Bind mount | `mp0: /mnt/converter,mp=/mnt/converter` | -| AppArmor | `unconfined` | -| Cgroup | `devices.allow: a`, `mount.auto: proc:rw sys:rw cgroup:rw` | - ### Re-deploying / updating -The install scripts are idempotent — safe to re-run: +```bash +# On Proxmox host +bash install.sh --update + +# Inside LXC +bash /mnt/converter/frontend/setup.sh +``` + +### Uninstall ```bash -# On srv2: update backend -cd /mnt/converter && git pull -bash backend/install.sh # skips LXC creation if container exists - -# Inside LXC: update frontend -cd /mnt/converter && git pull -bash frontend/install.sh +bash install.sh --remove ``` +## Usage Flow + +1. **Open web UI** → new session form (UUID auto-generated) +2. **Download or SCP pull** a source image → `/mnt/converter/tmp/{session_id}/in/` +3. **Analyse** → backend probes format, size, OS, EFI +4. **Configure VM** → name, CPU, RAM, storage, boot type, target disk size +5. **Submit job** → backend converts, shrinks (if needed), creates VM +6. **Poll progress** → real-time updates every 2 seconds +7. **Reuse or cleanup** → clone, re-convert from same source, or delete staging + +### Disk sizing and auto-shrink + +| You enter | Result | +|-----------|--------| +| **Blank** (default) | Auto-shrinks to 10% of virtual size (min 20 GB) via `virt-resize --shrink --resize-force` | +| **A number** (e.g. `500`) | Uses that exact size. No shrinking. | +| **A number < virtual** (e.g. `40`) | Shrinks the disk to 40 GB. | +| **A number > virtual** (e.g. `600`) | Expands the disk. | + ## API Reference -All endpoints are under `/api/v1/`. Full specification in [`open-api.yaml`](open-api.yaml). +All endpoints under `/api/v1/`. Full spec: [`open-api.yaml`](open-api.yaml). | Method | Path | Description | |--------|------|-------------| @@ -190,23 +168,17 @@ All endpoints are under `/api/v1/`. Full specification in [`open-api.yaml`](open | `POST` | `/api/v1/jobs` | Submit conversion + provisioning job | | `GET` | `/api/v1/jobs/{id}` | Poll job status and progress | | `POST` | `/api/v1/jobs/{id}/cleanup` | Delete or preserve staging files | +| `POST` | `/api/v1/clone` | Clone an existing VM | ### Example: Analyze a source image ```bash curl -X POST http://10.2.0.2:9000/api/v1/analyze \ -H "Content-Type: application/json" \ - -d '{"vmid": 21050, "source_filename": "64bit/Debian 12.11.0 (64bit).vmdk"}' + -d '{"vmid": 21050, "source_filename": "session-id/in/image.vmdk"}' -# Response: -# { -# "vmid": 21050, -# "filename": "Debian 12.11.0 (64bit).vmdk", -# "disk_format": "vmdk", -# "disk_size_gb": 7.5, -# "os_type": "Debian", -# "efi_detectable": true -# } +# → {"vmid": 21050, "filename": "image.vmdk", "disk_format": "vmdk", +# "disk_size_gb": 7.5, "os_type": "Debian", "efi_detectable": true} ``` ### Example: Submit a conversion job @@ -215,96 +187,56 @@ curl -X POST http://10.2.0.2:9000/api/v1/analyze \ curl -X POST http://10.2.0.2:9000/api/v1/jobs \ -H "Content-Type: application/json" \ -d '{ - "vmid": 21050, - "vm_name": "debian-test", - "boot_disk": { - "disk_type": "image_file", - "source_filename": "64bit/Debian 12.11.0 (64bit).vmdk", - "format": "vmdk" - }, - "cpu_cores": 2, - "ram_mb": 4096, - "target_storage": "local-lvm" + "vmid": 21050, "vm_name": "debian-test", + "boot_disk": {"disk_type": "image_file", "source_filename": "session/in/image.vmdk", "format": "vmdk"}, + "cpu_cores": 2, "ram_mb": 4096, "target_storage": "local-lvm" }' -# Response: -# { "job_id": "job_21050_1737480000", "vmid": 21050, "status": "queued", ... } +# → {"job_id": "job_21050_1737480000", "vmid": 21050, "status": "queued", ...} ``` -## Usage Flow - -1. **Open the web UI** → new session form -2. **Upload or download a source image** → file lands in `/mnt/converter/in/` -3. **Backend analyses the image** → shows format, size, OS, EFI status -4. **Configure VM settings** → name, CPU, RAM, storage, boot type -5. **Submit job** → backend converts, shrinks (if needed), creates VM -6. **Poll progress** → real-time status bar updates every 2 seconds -7. **Reuse or cleanup** → create another VM from same source, or delete staging files - -### Disk sizing and auto-shrink - -Virtual disk images (VMDK, QCOW2) have two sizes: - -| | Meaning | Example | -|---|---|---| -| **Virtual size** | Maximum the disk *could* grow to | 500 GB | -| **Actual size** | Real data in the image (sparse file) | ~7 GB | - -During analysis, the UI shows the virtual size (e.g. *"500.0 GB detected. Omit to -auto-shrink"*). This is because the source image was created with a -large virtual disk, but only a fraction is used. - -**Auto-shrink**: if you leave Target Disk Size blank, the disk shrinks to -**10% of its virtual size, minimum 20 GB**. A 500 GB disk becomes 50 GB, -an 80 GB disk becomes 20 GB. You can override this by entering a value. - -| You enter | Result | -|-----------|--------| -| **Blank** (default) | Auto-shrinks to 30 GB if the source is larger. Uses `virt-resize --shrink --resize-force` to safely reduce the disk container. | -| **A number** (e.g. `500`) | Uses that exact size. No shrinking. | -| **A number < virtual** (e.g. `40`) | Shrinks the disk to 40 GB. | -| **A number > virtual** (e.g. `600`) | Expands the disk. Safe operation. | - -> **Tip:** For most Linux VMs (Debian, Ubuntu, etc.), 30 GB is plenty. -> You can always expand the disk later in Proxmox if you need more space. -> If you plan to store large datasets inside the VM, set a higher value. - ## Configuration | Variable | Default | Description | |----------|---------|-------------| -| `BACKEND_URL` | `http://10.2.0.2:9000` | Backend API base URL (set in systemd unit) | +| `BACKEND_URL` | `http://10.2.0.2:9000` | Backend API base URL | | `VM_ID_MIN` | `21000` | Minimum allowed Proxmox VM ID | | `VM_ID_MAX` | `21100` | Maximum allowed Proxmox VM ID | +| `DOWNLOAD_TIMEOUT` | `14400` (4h) | Max download time | +| `MAX_ETA_SECONDS` | `3600` (1h) | Kill download if ETA exceeds this | ## Development -**Source of truth**: `open-api.yaml` — always derive models from this file. +**Source of truth**: `open-api.yaml` — update spec first, then derive models. -**Backend tool requirements** (on srv2): -- `qemu-img` — disk format and size detection +### Backend tools required (on Proxmox host) +- `qemu-img` — disk format, size detection, format conversion - `guestfish` / `virt-inspector` — OS type and EFI boot detection -- `7z`, `unzip`, `tar`, `gunzip`, `bunzip2`, `xz` — archive extraction +- `virt-resize` — safe disk shrinking +- `7z`, `unzip`, `unrar`, `tar`, `gunzip`, `bunzip2`, `xz` — archive extraction +- `qm` (Proxmox CLI) — VM creation, disk import, configuration -**Provisioner note**: `backend/provisioner.py` is currently a stub with simulated -progress. Replace with real `qm create`, `qm importdisk`, and `qm set` commands -when deploying to the Proxmox host. +### Frontend tools required (in LXC) +- `python3`, `pip3`, `wget`, `curl`, `aria2c`, `openssh-client`, `sshpass` -**Staging files**: `/mnt/converter/in/`, `/mnt/converter/out/`, and `/mnt/converter/logs/` are gitignored — -they contain runtime data, not code. +### Job states + +``` +queued → processing_conversion → importing_storage → completed + → failed +``` ## Logging -Both services write to two locations simultaneously: +Both services log to systemd journal and rotating file: -| Destination | How to access | -|-------------|---------------| -| **File** (shared) | `tail -f /mnt/converter/logs/vm-bench.log` (frontend) or `vm-bench-backend.log` (backend) | -| **Journal** (per-service) | `journalctl -u vm-bench -f` (frontend) or `journalctl -u vm-bench-backend -f` (backend) | +| Service | File Log | Journal | +|---------|----------|---------| +| Backend | `/mnt/converter/logs/vm-bench-backend.log` | `journalctl -u vm-bench-backend -f` | +| Frontend | `/mnt/converter/logs/vm-bench.log` | `journalctl -u vm-bench -f` | -Log files rotate automatically at 10 MB, keeping 5 historical files -(e.g. `vm-bench.log`, `vm-bench.log.1`, `vm-bench.log.2`, …). +Rotation: 10 MB max, 5 backup files. -The file logs are the authoritative source — they're written to the shared -`/mnt/converter/logs/` directory, accessible from both the Proxmox host and -the LXC container. +## License + +MIT