docs: update README with current directory layout, install script download link, SCP/clone features, and session isolation
This commit is contained in:
parent
f1978008bd
commit
1e14d96052
1 changed files with 135 additions and 203 deletions
338
README.md
338
README.md
|
|
@ -1,144 +1,118 @@
|
|||
# VM Bench — Proxmox Image Conversion Frontend
|
||||
# VM Bench — Proxmox Image Conversion
|
||||
|
||||
Web GUI and REST API for converting VMware disk images (VMDK, VHD, etc.) into
|
||||
Proxmox-compatible QCOW2 disks and provisioning VMs with automatic OS and boot
|
||||
type detection.
|
||||
|
||||
## Quick Start
|
||||
|
||||
Download and run the install script on your Proxmox host:
|
||||
|
||||
```bash
|
||||
wget https://git.lohmar.co.uk/cclohmar/vm-bench/raw/branch/main/install.sh
|
||||
bash install.sh --deploy
|
||||
```
|
||||
|
||||
The script interactively:
|
||||
1. Selects a storage pool
|
||||
2. Installs the backend API (port 9000) on the host
|
||||
3. Creates an LXC container with bind-mounted `/mnt/converter`
|
||||
4. Inside the LXC, run `bash /mnt/converter/frontend/setup.sh` to start the web UI
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Browser vm-bench LXC Proxmox Host (srv2)
|
||||
(this container)
|
||||
Browser vm-bench LXC Proxmox Host
|
||||
(this container)
|
||||
──→ :5000 ──→ frontend/app.py ──→ :9000 backend/app.py
|
||||
│
|
||||
├─ qemu-img (format, size)
|
||||
├─ guestfish (OS, EFI detection)
|
||||
├─ 7z/unzip (archive extraction)
|
||||
└─ qm create (VM provisioning)
|
||||
│
|
||||
├─ qemu-img (format, size)
|
||||
├─ guestfish (OS, EFI detection)
|
||||
├─ 7z/unzip (archive extraction)
|
||||
└─ qm create (VM provisioning)
|
||||
```
|
||||
|
||||
- **Frontend** (`vm-bench` LXC) — FastAPI + Jinja2 web UI on port 5000
|
||||
- **Backend** (`srv2` Proxmox host) — FastAPI REST API on port 9000
|
||||
- **Shared storage** — `/mnt/converter/in` (staging) and `/mnt/converter/out` (output)
|
||||
- **Backend** (Proxmox host) — FastAPI REST API on port 9000
|
||||
- **Shared storage** — `/mnt/converter/tmp/` (staging, bind-mounted)
|
||||
|
||||
## Directory Layout
|
||||
|
||||
```
|
||||
/mnt/converter/
|
||||
├── frontend/ # Web UI (runs on vm-bench LXC)
|
||||
│ ├── app.py # FastAPI app, routes, session handling
|
||||
│ ├── api_client.py # Typed REST client → backend
|
||||
│ ├── install.sh # Frontend installer (system deps + service)
|
||||
│ ├── vm-bench.service # Systemd unit for the frontend
|
||||
│ ├── requirements.txt
|
||||
│ ├── templates/ # Jinja2 HTML templates
|
||||
│ │ ├── base.html
|
||||
│ │ ├── index.html # New session form
|
||||
│ │ ├── _analysis.html # Analysis result + confirm form
|
||||
│ │ └── polling.html # Progress bar + job status
|
||||
│ └── static/
|
||||
│ └── proxmox.css
|
||||
├── backend/ # REST API (deploys to Proxmox host srv2)
|
||||
│ ├── app.py # FastAPI app, routes
|
||||
│ ├── models.py # Pydantic request/response models
|
||||
│ ├── converter.py # Archive extraction, disk probing, OS/EFI detection
|
||||
│ ├── provisioner.py # VM provisioning (qm create/importdisk)
|
||||
│ ├── install.sh # Full deployment script (backend + LXC creation)
|
||||
├── install.sh # Master deployment script (run on Proxmox host)
|
||||
├── clean.sh # Clean tmp dirs + restart backend
|
||||
├── open-api.yaml # API spec (single source of truth)
|
||||
├── AGENTS.md # Developer guide
|
||||
├── README.md # This file
|
||||
├── backend/ # REST API (runs on Proxmox host)
|
||||
│ ├── app.py # FastAPI app, routes, logging
|
||||
│ ├── models.py # Pydantic request/response models
|
||||
│ ├── converter.py # Archive extraction, disk probing, OS/EFI detection
|
||||
│ ├── provisioner.py # VM provisioning (qemu-img convert, qm create/importdisk)
|
||||
│ ├── vm-bench-backend.service
|
||||
│ └── requirements.txt
|
||||
├── open-api.yaml # API spec (single source of truth)
|
||||
├── in/ # Shared staging directory (gitignored)
|
||||
├── out/ # Shared output directory (gitignored)
|
||||
├── frontend/ # Web UI (runs on vm-bench LXC)
|
||||
│ ├── app.py # FastAPI app, routes, download/SCP handling
|
||||
│ ├── api_client.py # Typed REST client → backend
|
||||
│ ├── setup.sh # Frontend installer (inside LXC)
|
||||
│ ├── vm-bench.service # Systemd unit
|
||||
│ ├── requirements.txt
|
||||
│ ├── templates/ # Jinja2 HTML templates
|
||||
│ │ ├── base.html # Base layout with session GUID
|
||||
│ │ ├── index.html # New session form + download + progress
|
||||
│ │ ├── _analysis.html # Analysis result + confirm form
|
||||
│ │ ├── polling.html # Progress bar + job status + reuse section
|
||||
│ │ └── scp.html # SCP pull page
|
||||
│ └── static/
|
||||
│ └── proxmox.css # Proxmox VE-inspired theme
|
||||
├── logs/ # Shared log files (gitignored)
|
||||
└── tmp/ # Staging files per session (gitignored)
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **File upload or URL download** — ingest `.vmdk`, `.vhd`, `.7z`, `.zip`, `.tar.gz` archives
|
||||
- **Automatic archive extraction** — 7z, unzip, tar, gunzip
|
||||
- **URL download** — ingest archives via `aria2c` with 8 connections
|
||||
- **SCP pull** — transfer files directly from remote servers, no middle-hop
|
||||
- **Archive extraction** — 7z, zip, rar, tar, gz, bz2, xz
|
||||
- **Disk analysis** — detects format, virtual size, guest OS, EFI bootability
|
||||
- **VM provisioning** — creates Proxmox VM with automatic disk import and boot config
|
||||
- **Progress polling** — real-time job status with progress bar
|
||||
- **VM provisioning** — qemu-img convert + optional virt-resize shrink + qm create/importdisk
|
||||
- **VM cloning** — instant full clone of an existing VM
|
||||
- **Progress polling** — real-time status bar updates every 2 seconds
|
||||
- **Session isolation** — UUID per browser session for file staging
|
||||
- **Staging reuse** — keep source files to create multiple VMs from one image
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Proxmox host (`srv2`)
|
||||
|
||||
The host must be a Proxmox VE node with:
|
||||
- `pct` — LXC container management (built-in)
|
||||
- `git` — to clone the repository (install script handles this)
|
||||
- Internet access to download container templates
|
||||
|
||||
### LXC container (`vm-bench`)
|
||||
|
||||
Created automatically by the install script. Requires:
|
||||
- 3 CPU cores, 6 GB RAM, 24 GB disk, 2 GB swap
|
||||
- Debian 12 template (downloaded by script)
|
||||
- Network bridge `vmbr0`
|
||||
|
||||
## Deployment
|
||||
|
||||
All deployment is driven by a single script on the Proxmox host. It creates the
|
||||
directory tree, clones the repo, installs the backend, and provisions the
|
||||
`vm-bench` LXC container ready for the frontend.
|
||||
|
||||
### Step 1: Clone & deploy (on `srv2`)
|
||||
### Step 1: Backend (on Proxmox host)
|
||||
|
||||
```bash
|
||||
# Create the shared directory and clone the repo
|
||||
mkdir -p /mnt/converter
|
||||
cd /mnt/converter
|
||||
git clone https://git.lohmar.co.uk/cclohmar/vm-bench.git .
|
||||
|
||||
# Run the installer — handles everything interactively
|
||||
bash backend/install.sh
|
||||
wget https://git.lohmar.co.uk/cclohmar/vm-bench/raw/branch/main/install.sh
|
||||
bash install.sh --deploy
|
||||
```
|
||||
|
||||
The script will:
|
||||
1. Create the `/mnt/converter/` directory tree
|
||||
2. Clone or update the git repository
|
||||
3. Install system packages (qemu-img, guestfish, archive tools, etc.)
|
||||
4. Install Python dependencies and start the backend systemd service
|
||||
5. **Prompt for container ID + network settings**
|
||||
6. **Create and configure the `vm-bench` LXC container** with:
|
||||
- 3 cores, 6 GB RAM, 24 GB rootfs, 2 GB swap
|
||||
- Bind mount: `/mnt/converter` shared with host
|
||||
- Unprivileged mode off, nesting enabled
|
||||
- AppArmor unconfined, cgroup device access allowed
|
||||
7. Optionally start the container
|
||||
The script interactively:
|
||||
1. Scans available Proxmox storage pools and lets you pick one
|
||||
2. Prompts for install path, container ID, IP, gateway, bridge, MAC
|
||||
3. Installs system packages (qemu-img, guestfish, 7z, unzip, etc.)
|
||||
4. Creates Python venv, installs deps, generates and starts `vm-bench-backend` service
|
||||
5. Creates the `vm-bench` LXC container with bind mount
|
||||
|
||||
Example prompts:
|
||||
```
|
||||
Container ID (e.g. 20020): 20020
|
||||
IP address/CIDR [10.2.0.20/16]:
|
||||
Gateway [10.2.0.2]:
|
||||
Bridge [vmbr0]:
|
||||
MAC address [BC:24:11:80:5A:B1]:
|
||||
```
|
||||
|
||||
Verify the backend is running:
|
||||
Verify:
|
||||
```bash
|
||||
curl http://127.0.0.1:9000/api/v1/health
|
||||
# → {"status":"ok"}
|
||||
```
|
||||
|
||||
### Step 2: Install frontend (inside the LXC)
|
||||
|
||||
After the container is created and started, enter it:
|
||||
### Step 2: Frontend (inside the LXC)
|
||||
|
||||
```bash
|
||||
pct enter 20020 # use your container ID
|
||||
bash /mnt/converter/frontend/setup.sh
|
||||
```
|
||||
|
||||
Inside the container, run the frontend installer:
|
||||
|
||||
```bash
|
||||
bash /mnt/converter/frontend/install.sh
|
||||
```
|
||||
|
||||
This installs Python, pip, `wget`, `curl`, Python dependencies, and starts the
|
||||
`vm-bench` systemd service on port 5000.
|
||||
Installs system tools (python3, aria2, sshpass, etc.), Python deps, and starts
|
||||
`vm-bench` on port 5000.
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
|
|
@ -148,40 +122,44 @@ curl http://127.0.0.1:5000
|
|||
|
||||
Open `http://<container-ip>:5000` in a browser.
|
||||
|
||||
### Container LXC Config (created by install script)
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| Arch | `amd64` |
|
||||
| Cores | `3` |
|
||||
| Memory | `6144` MB |
|
||||
| Swap | `2048` MB |
|
||||
| Rootfs | `local-lvm:24G` |
|
||||
| OS type | `debian` |
|
||||
| Unprivileged | `0` (disabled) |
|
||||
| Features | `nesting=1` |
|
||||
| Network | `vmbr0`, firewall enabled, veth |
|
||||
| Bind mount | `mp0: /mnt/converter,mp=/mnt/converter` |
|
||||
| AppArmor | `unconfined` |
|
||||
| Cgroup | `devices.allow: a`, `mount.auto: proc:rw sys:rw cgroup:rw` |
|
||||
|
||||
### Re-deploying / updating
|
||||
|
||||
The install scripts are idempotent — safe to re-run:
|
||||
```bash
|
||||
# On Proxmox host
|
||||
bash install.sh --update
|
||||
|
||||
# Inside LXC
|
||||
bash /mnt/converter/frontend/setup.sh
|
||||
```
|
||||
|
||||
### Uninstall
|
||||
|
||||
```bash
|
||||
# On srv2: update backend
|
||||
cd /mnt/converter && git pull
|
||||
bash backend/install.sh # skips LXC creation if container exists
|
||||
|
||||
# Inside LXC: update frontend
|
||||
cd /mnt/converter && git pull
|
||||
bash frontend/install.sh
|
||||
bash install.sh --remove
|
||||
```
|
||||
|
||||
## Usage Flow
|
||||
|
||||
1. **Open web UI** → new session form (UUID auto-generated)
|
||||
2. **Download or SCP pull** a source image → `/mnt/converter/tmp/{session_id}/in/`
|
||||
3. **Analyse** → backend probes format, size, OS, EFI
|
||||
4. **Configure VM** → name, CPU, RAM, storage, boot type, target disk size
|
||||
5. **Submit job** → backend converts, shrinks (if needed), creates VM
|
||||
6. **Poll progress** → real-time updates every 2 seconds
|
||||
7. **Reuse or cleanup** → clone, re-convert from same source, or delete staging
|
||||
|
||||
### Disk sizing and auto-shrink
|
||||
|
||||
| You enter | Result |
|
||||
|-----------|--------|
|
||||
| **Blank** (default) | Auto-shrinks to 10% of virtual size (min 20 GB) via `virt-resize --shrink --resize-force` |
|
||||
| **A number** (e.g. `500`) | Uses that exact size. No shrinking. |
|
||||
| **A number < virtual** (e.g. `40`) | Shrinks the disk to 40 GB. |
|
||||
| **A number > virtual** (e.g. `600`) | Expands the disk. |
|
||||
|
||||
## API Reference
|
||||
|
||||
All endpoints are under `/api/v1/`. Full specification in [`open-api.yaml`](open-api.yaml).
|
||||
All endpoints under `/api/v1/`. Full spec: [`open-api.yaml`](open-api.yaml).
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
|
|
@ -190,23 +168,17 @@ All endpoints are under `/api/v1/`. Full specification in [`open-api.yaml`](open
|
|||
| `POST` | `/api/v1/jobs` | Submit conversion + provisioning job |
|
||||
| `GET` | `/api/v1/jobs/{id}` | Poll job status and progress |
|
||||
| `POST` | `/api/v1/jobs/{id}/cleanup` | Delete or preserve staging files |
|
||||
| `POST` | `/api/v1/clone` | Clone an existing VM |
|
||||
|
||||
### Example: Analyze a source image
|
||||
|
||||
```bash
|
||||
curl -X POST http://10.2.0.2:9000/api/v1/analyze \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"vmid": 21050, "source_filename": "64bit/Debian 12.11.0 (64bit).vmdk"}'
|
||||
-d '{"vmid": 21050, "source_filename": "session-id/in/image.vmdk"}'
|
||||
|
||||
# Response:
|
||||
# {
|
||||
# "vmid": 21050,
|
||||
# "filename": "Debian 12.11.0 (64bit).vmdk",
|
||||
# "disk_format": "vmdk",
|
||||
# "disk_size_gb": 7.5,
|
||||
# "os_type": "Debian",
|
||||
# "efi_detectable": true
|
||||
# }
|
||||
# → {"vmid": 21050, "filename": "image.vmdk", "disk_format": "vmdk",
|
||||
# "disk_size_gb": 7.5, "os_type": "Debian", "efi_detectable": true}
|
||||
```
|
||||
|
||||
### Example: Submit a conversion job
|
||||
|
|
@ -215,96 +187,56 @@ curl -X POST http://10.2.0.2:9000/api/v1/analyze \
|
|||
curl -X POST http://10.2.0.2:9000/api/v1/jobs \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"vmid": 21050,
|
||||
"vm_name": "debian-test",
|
||||
"boot_disk": {
|
||||
"disk_type": "image_file",
|
||||
"source_filename": "64bit/Debian 12.11.0 (64bit).vmdk",
|
||||
"format": "vmdk"
|
||||
},
|
||||
"cpu_cores": 2,
|
||||
"ram_mb": 4096,
|
||||
"target_storage": "local-lvm"
|
||||
"vmid": 21050, "vm_name": "debian-test",
|
||||
"boot_disk": {"disk_type": "image_file", "source_filename": "session/in/image.vmdk", "format": "vmdk"},
|
||||
"cpu_cores": 2, "ram_mb": 4096, "target_storage": "local-lvm"
|
||||
}'
|
||||
|
||||
# Response:
|
||||
# { "job_id": "job_21050_1737480000", "vmid": 21050, "status": "queued", ... }
|
||||
# → {"job_id": "job_21050_1737480000", "vmid": 21050, "status": "queued", ...}
|
||||
```
|
||||
|
||||
## Usage Flow
|
||||
|
||||
1. **Open the web UI** → new session form
|
||||
2. **Upload or download a source image** → file lands in `/mnt/converter/in/`
|
||||
3. **Backend analyses the image** → shows format, size, OS, EFI status
|
||||
4. **Configure VM settings** → name, CPU, RAM, storage, boot type
|
||||
5. **Submit job** → backend converts, shrinks (if needed), creates VM
|
||||
6. **Poll progress** → real-time status bar updates every 2 seconds
|
||||
7. **Reuse or cleanup** → create another VM from same source, or delete staging files
|
||||
|
||||
### Disk sizing and auto-shrink
|
||||
|
||||
Virtual disk images (VMDK, QCOW2) have two sizes:
|
||||
|
||||
| | Meaning | Example |
|
||||
|---|---|---|
|
||||
| **Virtual size** | Maximum the disk *could* grow to | 500 GB |
|
||||
| **Actual size** | Real data in the image (sparse file) | ~7 GB |
|
||||
|
||||
During analysis, the UI shows the virtual size (e.g. *"500.0 GB detected. Omit to
|
||||
auto-shrink"*). This is because the source image was created with a
|
||||
large virtual disk, but only a fraction is used.
|
||||
|
||||
**Auto-shrink**: if you leave Target Disk Size blank, the disk shrinks to
|
||||
**10% of its virtual size, minimum 20 GB**. A 500 GB disk becomes 50 GB,
|
||||
an 80 GB disk becomes 20 GB. You can override this by entering a value.
|
||||
|
||||
| You enter | Result |
|
||||
|-----------|--------|
|
||||
| **Blank** (default) | Auto-shrinks to 30 GB if the source is larger. Uses `virt-resize --shrink --resize-force` to safely reduce the disk container. |
|
||||
| **A number** (e.g. `500`) | Uses that exact size. No shrinking. |
|
||||
| **A number < virtual** (e.g. `40`) | Shrinks the disk to 40 GB. |
|
||||
| **A number > virtual** (e.g. `600`) | Expands the disk. Safe operation. |
|
||||
|
||||
> **Tip:** For most Linux VMs (Debian, Ubuntu, etc.), 30 GB is plenty.
|
||||
> You can always expand the disk later in Proxmox if you need more space.
|
||||
> If you plan to store large datasets inside the VM, set a higher value.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BACKEND_URL` | `http://10.2.0.2:9000` | Backend API base URL (set in systemd unit) |
|
||||
| `BACKEND_URL` | `http://10.2.0.2:9000` | Backend API base URL |
|
||||
| `VM_ID_MIN` | `21000` | Minimum allowed Proxmox VM ID |
|
||||
| `VM_ID_MAX` | `21100` | Maximum allowed Proxmox VM ID |
|
||||
| `DOWNLOAD_TIMEOUT` | `14400` (4h) | Max download time |
|
||||
| `MAX_ETA_SECONDS` | `3600` (1h) | Kill download if ETA exceeds this |
|
||||
|
||||
## Development
|
||||
|
||||
**Source of truth**: `open-api.yaml` — always derive models from this file.
|
||||
**Source of truth**: `open-api.yaml` — update spec first, then derive models.
|
||||
|
||||
**Backend tool requirements** (on srv2):
|
||||
- `qemu-img` — disk format and size detection
|
||||
### Backend tools required (on Proxmox host)
|
||||
- `qemu-img` — disk format, size detection, format conversion
|
||||
- `guestfish` / `virt-inspector` — OS type and EFI boot detection
|
||||
- `7z`, `unzip`, `tar`, `gunzip`, `bunzip2`, `xz` — archive extraction
|
||||
- `virt-resize` — safe disk shrinking
|
||||
- `7z`, `unzip`, `unrar`, `tar`, `gunzip`, `bunzip2`, `xz` — archive extraction
|
||||
- `qm` (Proxmox CLI) — VM creation, disk import, configuration
|
||||
|
||||
**Provisioner note**: `backend/provisioner.py` is currently a stub with simulated
|
||||
progress. Replace with real `qm create`, `qm importdisk`, and `qm set` commands
|
||||
when deploying to the Proxmox host.
|
||||
### Frontend tools required (in LXC)
|
||||
- `python3`, `pip3`, `wget`, `curl`, `aria2c`, `openssh-client`, `sshpass`
|
||||
|
||||
**Staging files**: `/mnt/converter/in/`, `/mnt/converter/out/`, and `/mnt/converter/logs/` are gitignored —
|
||||
they contain runtime data, not code.
|
||||
### Job states
|
||||
|
||||
```
|
||||
queued → processing_conversion → importing_storage → completed
|
||||
→ failed
|
||||
```
|
||||
|
||||
## Logging
|
||||
|
||||
Both services write to two locations simultaneously:
|
||||
Both services log to systemd journal and rotating file:
|
||||
|
||||
| Destination | How to access |
|
||||
|-------------|---------------|
|
||||
| **File** (shared) | `tail -f /mnt/converter/logs/vm-bench.log` (frontend) or `vm-bench-backend.log` (backend) |
|
||||
| **Journal** (per-service) | `journalctl -u vm-bench -f` (frontend) or `journalctl -u vm-bench-backend -f` (backend) |
|
||||
| Service | File Log | Journal |
|
||||
|---------|----------|---------|
|
||||
| Backend | `/mnt/converter/logs/vm-bench-backend.log` | `journalctl -u vm-bench-backend -f` |
|
||||
| Frontend | `/mnt/converter/logs/vm-bench.log` | `journalctl -u vm-bench -f` |
|
||||
|
||||
Log files rotate automatically at 10 MB, keeping 5 historical files
|
||||
(e.g. `vm-bench.log`, `vm-bench.log.1`, `vm-bench.log.2`, …).
|
||||
Rotation: 10 MB max, 5 backup files.
|
||||
|
||||
The file logs are the authoritative source — they're written to the shared
|
||||
`/mnt/converter/logs/` directory, accessible from both the Proxmox host and
|
||||
the LXC container.
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
|
|
|||
Loading…
Reference in a new issue