← All posts
Field reports

Field report vol. 1 (July 2026): 26 restores across 2 clouds, judged by an earlier standard

July 2026: one ComfyUI + SDXL environment packed, the machine destroyed, and the nest restored 26 times across 2 clouds, 7 datacenters and 6 GPU types, every file verified. The pass line used here has since been replaced by a stricter one; the current record is on the Proof page.

Renest field team July 14, 2026·4 min read
Pass this on XRedditLinkedIn

Update, September 2026. This report is kept as it was measured in July 2026, with one change of wording: we now call the packed environment a nest, not a bag. Since then: the pass/fail line of 0.98 SSIM used here has been replaced by a stricter standard (the restored machine must load the same extensions and produce output again), so these passes now count as recorded under an earlier standard, neither passes nor failures; the GPU generation a nest was packed on is now recorded in every nest, and the pre-check refuses a card the packed code can't run on; and we no longer quote a "typical" door-to-door time.

This first report answers the founding question: if you get a workflow running once on a rented GPU, nest the whole environment, and destroy the machine — can you actually bring it back later, anywhere, exactly as it was? How fast, and at what cost? We spent two days and about $1.21 in real GPU rentals (RunPod $1.12 + Vast $0.09) finding out.

The setup

We rented an RTX A4000 in RunPod's Iceland datacenter, installed ComfyUI + SDXL, fixed the random seed (seed=42), generated a baseline image, and packed the whole environment into object storage — 6.47 GiB, 5 files. Then we destroyed the machine and revived the nest on fresh machines across different GPUs, datacenters, times of day, and clouds, using nothing but the restore script. Four revives went a step further: boot ComfyUI, regenerate the image with the same seed and settings, and compare it to the baseline (SSIM, where 1.0 means pixel-identical).

Restore itself is very reliable: 26 for 26

26 revives, 26 byte-for-byte matches — across 2 clouds, 7 datacenters, 6 GPU types. Three attempts failed, and none failed in the restore step itself: two were our own experiment-design mistake (five back-to-back restores on one machine filled the disk; the preflight check refused to start, as it should), and one was an SSH connection dropping before the script even launched.

The revived environment is alive — and we can say exactly how alive

Revive conditionSSIM vs. baselineVerdict
Same GPU model (A4000 → A4000), across clouds (RunPod → Vast)1.000000 — pixel-identicalPass
One GPU generation apart (A4000 → RTX 4000 Ada)0.985489Pass (threshold 0.98)
Different model (A4000 → L4)0.983583Pass
Too-new GPU (RTX PRO 4500, Blackwell)Generation crashedFail

That last row is a real failure mode worth its own paragraph. On the Blackwell card, every file verified byte-for-byte — then image generation died with CUDA error: no kernel image is available. The PyTorch build in the nest (cu124) predates that chip. Identical files do not equal a working environment, which is exactly why the regenerate-an-image check can't be skipped. That $0.07 sample is also why the next format version gets a GPU-generation field.

Door to door: about 6–8 minutes — if you can get a machine

Across the 26 successful restores: transfer 69–216 seconds (median 145.5s, about 2.5 minutes), dependency install 28–65 seconds on first install, verification 18–33 seconds. Add 15–38 seconds of machine boot and roughly a minute to generate: 6–8 minutes door to door.

But: over two days, only 13 of 27 rental attempts got a machine immediately (48%). An accounting note we insist on: 2 of those 13 hits ran at $0.74/hr — one before our real-price check went live, one under a temporarily approved cap. Under the strict $0.5 price cap, the hit rate is 11/27 = 41%. Quote one number or the other; don't mix them.

  • When you do get a machine, it's fast and consistent: boot-to-usable p50 = 28s, p90 = 37s, worst 38s.
  • 11 of the 14 misses were "no stock anywhere" — and all 11 fell inside one 22-minute window on a European weekday afternoon. The other 3 had stock but were blocked by the price cap (the same $0.74/hr tier, repeatedly).
  • The tail isn't "waiting a long time" — it's "no machine exists." Which is the argument for a portable environment: go wherever there's stock.

Bucket location matters for upload, not download

Uploading from a European machine falls off steeply with distance: EU bucket 1069 Mbps, US-East 783, APAC 293, Oceania 239 — a 4.5× spread. Downloads show no correlation with distance: a Texas machine pulled 257–332 Mbps from buckets on five continents; a Georgia machine pulled 375–804 Mbps with the farthest bucket (Oceania) fastest. Neither pattern matches geography — downloads ride the storage provider's global backbone. Practical rule: put your bucket near where you pack; revive anywhere at the same speed.

Parallel downloads: 2.3–9.3× headroom

8 connections vs. 1 on the same 6.9GB model file: all 15 comparisons were significantly faster in parallel, peaking at 5062 Mbps (the whole file in 11 seconds) late at night in Europe. The planned multi-source downloader could take today's ~2-minute transfer to 15–25 seconds.

Prices are wild

The same A4000 traded between $0.076/hr (Vast, Japan) and $0.74/hr (RunPod, Europe at peak) — nearly 10×. At one moment, two datacenters differed 2.7× ($0.17 vs $0.46). A cloud's advertised "global lowest price" is not what you pay: a $0.34 listing opened at $0.74. Our test harness now checks the real price right after it rents a machine and cancels anything over budget — three interceptions, $0 lost. (That is our own testing setup; Renest itself never rents machines for you.) One measured cross-cloud revive cost $0.014 and took 7.3 minutes.

Caliber

Everything above used one minimal nest: 6.5GB, a single model, no custom nodes. Real workflows with custom nodes and LoRAs come next, and a 6–8 minute door-to-door time on a minimal nest says little about them. Networks inside China: completely untested. Most conditions ran only 1–3 times, so there is no reliable worst-case (p95/p99) number yet. Image timing also mixes two warmup effects we haven't separated. Samples are still small: treat these as reference points, not laws.

Pass this on XRedditLinkedIn

See a restore check itself.

Claim a starter nest — a run we packed and checked — and restore it on a machine you rent. Free, no card.