# If a restore stops.

> When renest restore stops, its last lines name the stage, a reason class such as [S0/DISK_INSUFFICIENT], and what to do. A refusal at the machine check means nothing has been downloaded yet, so rent a machine that fits and paste the same command. Machine faults are fixed by a replacement machine, not by downloading again. If you're stuck, run renest support --dir ./run and send us the last lines of the output at support@notify.renest.ai.

Canonical page: https://renest.ai/docs/troubleshooting

<h2 id="help">Stuck? Paste the last lines of the output to us</h2>

<p>Run this on the machine, before you stop or terminate it:</p>

<pre>$ renest support --dir ./run</pre>

<p>Read what it prints, then send it to
<a href="mailto:support@notify.renest.ai">support@notify.renest.ai</a>, or paste it into a
ticket from the <strong>Support</strong> page in the web console. If
<code>renest support</code> says there are no run records, send the last twenty or so lines
of the restore's own output instead.</p>

<h2 id="read">A stop names the stage, the reason, and usually the fix</h2>

<p>A restore runs its stages in order (S0 checking this machine, S1 downloading, S2 putting
files in place, S3 installing dependencies, S4 starting up, S5 test render). When one fails,
it prints a line like this and exits with a non-zero code:</p>

<pre>[restore] ✗ S0 checking this machine(…s)—— [S0/DISK_INSUFFICIENT] This machine did not pass the checks: …</pre>

<p>The part in brackets is the stage and the reason class. The text after it is written to
be acted on. If it names one thing to do, do that and run the same command again. A restore
carries on where it stopped, and files already on disk that match their checksums are kept.
The <a href="/docs/glossary#stages">glossary</a> lists what each stage does.</p>

<h2 id="disk">“Only … GiB free, and this nest needs … GiB”: rent a bigger disk</h2>

<p>The machine check compares the free space on the disk holding your <code>--dir</code>
with what the nest needs. It adds room for the Python environment to the nest's own size.
Nothing has been downloaded yet. The message goes on <em>Clear some space or rent a bigger
disk.</em> The exit code is <code>65</code>.</p>

<ul>
  <li>Check which disk <code>--dir</code> is on. <code>./run</code> is relative to the folder
  you pasted the command in. On RunPod, only paths under <code>/workspace</code> are on the
  volume disk.</li>
  <li>Rent a machine with a bigger disk, or restore into a folder on a disk with room, then
  paste the same command.</li>
  <li>Don't use <code>--force</code> here. The download would fill the disk partway.</li>
</ul>

<p>If a disk fills up during the restore anyway (another process writing, say), the class is
<code>S2/DISK_FULL</code> (exit code <code>25</code>). Free up space and run the same command
again.</p>

<h2 id="gpu">“This GPU (sm_…) is older / newer than … this nest's torch was built for”: rent a different card</h2>

<p>The nest's PyTorch build carries GPU code for a range of card generations. The check
compares this card's generation with that range before anything downloads. The class is
<code>S0/ARCH_UNSUPPORTED</code> (exit code <code>64</code>). The message names the range, for
example <em>Rent a card at sm_80 or newer</em>. Every file would still come back correctly,
and the app still could not run on this card.</p>

<p>Rent a card in the range the message names, and paste the same restore command. The
card the nest was packed on, shown as <em>Packed on</em> in the console, ran it. If the
mismatch only shows up once the app runs, the message is <em>Your nest rebuilt correctly, but this GPU
is not one the packed PyTorch build has kernels for</em>. The fix is the same: a different
card, not a second download.</p>

<h2 id="driver">“This nest needs CUDA …, and this machine's driver only goes up to CUDA …”: rent a newer driver</h2>

<p>The host's GPU driver is part of the machine, not of the nest, and the nest's dependency
lock needs a newer one. The class is <code>S0/CUDA_BLOCK</code> (exit code
<code>63</code>). A related message is <em>This machine's GPU driver … is older than the …
that … needs</em>. Rent a machine with a newer driver and paste the same command. Your card
is fine.</p>

<h2 id="node">“Full nvidia-smi failed here” or “No CUDA GPUs are available”: replace the machine</h2>

<p>Some rented GPU machines answer simple questions about their card, then fail when a
program actually uses it. The check asks the driver directly whether it can start CUDA, and stops (exit code <code>63</code>; <code>--force</code> goes ahead) if it answers with an error or finds no GPU. It looks for the wedged-node shape in two further ways and only warns, because the fault sometimes clears on its own:</p>

<ul>
  <li><em>Full nvidia-smi failed here (…), while the single-fact queries above still answer —
  that is the shape of a wedged GPU node.</em></li>
  <li><em>Asking this machine for … MiB of video memory failed: …</em></li>
</ul>

<p>If you see either one, or the app then fails at start-up with <code>No CUDA GPUs are
available</code> in its log, nothing is wrong with your nest. Stop or terminate that
machine, rent another, and paste the same restore command. The code is not tied to a
machine. Downloading again on the same machine won't help.</p>

<h2 id="libs">“This machine is missing a system library the app needs”: install it, then run again</h2>

<p>Some libraries belong to the operating system, so no nest carries them. The check looks
for the ones the working run loaded. Missing ones are reported before the download, and a library only named by the packages is never a stop on its own. If an install the restore attempted fails, it stops there at the first stage, and <code>--force</code> carries on anyway. If they are still missing
at the end, the closing lines say <em>Read this before you call it done</em>. If the app then
fails because of one, the class is <code>SYSLIB_MISSING</code> (exit code <code>46</code>
at start-up, <code>55</code> during the test render). Every message names the libraries and
the fix:</p>

<ul>
  <li><strong>Surest fix:</strong> when the nest records the image it was packed on, the
  message names it. Start a machine from that image.</li>
  <li>Or install the packages it names, for example
  <code>apt-get install -y libgl1</code>, then run the same restore command again. Your files
  are kept.</li>
  <li>When the restore runs as root, or with sudo that needs no password, it does this
  for you: before anything downloads it lists what is missing (libraries it knows the
  package for, and git, a C compiler or git-lfs when the nest needs them) and asks
  <code>Install them now? [Y/n]</code>. Add <code>--yes</code> to install without being asked. ffmpeg or ffprobe called by a custom node is installed the same way after the code is unpacked; if that fails the restore ends with "Restored, but not complete" instead of a stop. Without that access, or with nobody at the keyboard, it prints the command to run
  and carries on.</li>
  <li>A library that belongs to the GPU driver can't be installed as a package. The message
  says so. Use a machine whose driver is set up, or start the container with GPU
  access.</li>
</ul>

<p>The <a href="/docs/base-image">Renest base image</a> already includes the system
libraries that image and fine-tuning workflows commonly load at runtime.</p>

<h2 id="withheld">“… files this nest does not carry”: put them where it says, then run again</h2>

<p>A nest someone handed you leaves out files whose licence does not let the sender pass them
on. Before anything big downloads, the restore lists every one of them with its size, its
full checksum, the address the nest recorded and the reason. The ones with a public original
source are fetched for you and checked against the nest. If any cannot be had, the restore
stops with exit code <code>16</code> before the nest's own files download, and names all of
them at once, each with the exact path to put it at.</p>

<ul>
  <li>Download each named file yourself and put it at the path shown.</li>
  <li>Run the same restore command again. A file in place whose bytes match is kept, and the
  rest carries on.</li>
  <li>To see the list without moving any bytes, run it with <code>--check-only</code>.</li>
  <li>An address on a host Renest doesn't recognise is not contacted until you name the sender
  with <code>--trust-sender</code>.</li>
</ul>

<h2 id="different">“Restored. This machine differs a little from the one it was packed on”: this is a success</h2>

<p>This line (<em>Restored. This machine differs a little from the one it was packed on … every file checked; a render may differ slightly</em>) appears with exit code <code>0</code>. Every file is correct. The machine
differs from the one the nest was packed on, for example in a package version or a system
library, so a render here may differ slightly from the original. Nothing to fix unless it
also names a missing library (see above).</p>

<h2 id="untrusted">UNTRUSTED_SOURCE: the nest wants to install from a server nobody recognises</h2>

<p>The restore stopped before installing anything (<code>S3/UNTRUSTED_SOURCE</code>, exit
code <code>37</code>). Installing a dependency runs code the server sends. The message lists
the servers and, for a nest someone handed you, names the sender. Decide whether you trust
that source. If you do, run the same command again with the <code>--trust-host</code>
option the message spells out, plus <code>--trust-sender</code> for a nest someone handed
you.
<a href="/docs/dependency-sources">Unrecognised dependency sources</a> walks through the
decision. A nest someone handed you that wants to run setup commands stops the same way,
with <code>S2/UNTRUSTED_SETUP</code> (exit code <code>26</code>).</p>

<h2 id="code">“This restore code has expired or been revoked”: issue a new one</h2>

<p>The full message is <em>This restore code has expired or been revoked. Sign a new one from
your drive — your nest is still there.</em> The class is <code>S1/CREDENTIAL_EXPIRED</code>
(exit code <code>13</code>). Open the nest in your drive, choose <strong>Restore</strong>,
issue a new code and paste the new command with the same <code>--dir</code>. Everything
already downloaded is reused.</p>

<h2 id="network">A dropped connection: run the same command again</h2>

<p>Network trouble while downloading (<code>S1/NETWORK_INTERRUPTED</code>, exit code
<code>11</code>) or while installing dependencies (<code>S3/UPSTREAM_UNREACHABLE</code>, exit
code <code>38</code>) is temporary. Run the same command again once the network is back. It
carries on where it stopped. If installing dependencies is slow where you are,
<code>--package-source</code> points it at a closer package index. See
<a href="/docs/restore#options">Restore a nest</a>.</p>

<h2 id="data">“What it does not have is your data”: put your training data back</h2>

<p>For a fine-tuning nest, <code>S4/NEED_USER_DATA</code> (exit code <code>45</code>) means
the environment is back but the training data the config points at is not. Training data
never travels with a nest. Put it where the message says the config expects it, then run the
same command again.</p>

<h2 id="token">Exit code 3: no access key, or a refused one</h2>

<p>Exit code <code>3</code> is a configuration or credential problem. It comes from commands
that talk to your drive with an access key: <code>renest pack --dest hosted</code>,
<code>renest list</code> and <code>renest export</code>. The usual message is:</p>

<pre>✗ No access token. Generate one in the web console, then put it in the RENEST_TOKEN environment variable, or write it into [auth] token in &#126;/.config/renest/config.toml.</pre>

<p>Create a key in the web console under <strong>Settings → Access</strong> and set it
as the message says. If the message says the key was refused, it has probably been revoked.
Create a new one. Restoring with a restore code never needs an access key. See
<a href="/docs/cli#auth">Signing in to your drive</a>.</p>

<h2 id="support">renest support turns a failed run into text you can read and paste</h2>

<pre>$ renest support --dir ./run</pre>

<p>It reads the records the last restore in that folder left in
<code>.renest/evidence/</code>. It prints one block with the tool's version, facts about the
machine, the timings, the stage and reason it stopped, and the last lines of the app's or
training run's log. Before printing, it replaces things that look like credentials (keys,
tokens, signed-link signatures) with a visible marker, and replaces your home folder path
with <code>&#126;</code>. It only recognises secrets by their shape, so read the block before
you send it.</p>

<p>It works entirely on the machine. It doesn't go online, upload anything or write a file.
The block goes to standard output, so <code>renest support --dir ./run &gt;
support.txt</code> saves it. <code>--run</code> picks an earlier run by its folder name under
<code>.renest/evidence/</code>. By default it uses the newest one.</p>

<h2 id="codes">Exit codes: the tens digit is the stage</h2>

<p><code>0</code> is success. <code>2</code> is a usage problem and <code>3</code> a
configuration or credential problem; both happen before any stage starts. Every other code
has two digits. The tens digit is the stage (S1 to S5 give <code>1x</code> to
<code>5x</code>, and S0 gives <code>6x</code>), and the ones digit is the reason within it.
<code>x0</code> means a failure in that stage that has no more specific class. The ones on
this page:</p>

<ul>
  <li><code>11</code> network interrupted · <code>13</code> restore code expired or revoked ·
  <code>16</code> files that do not travel with the nest can't be fetched</li>
  <li><code>25</code> disk filled during the restore · <code>26</code> setup commands from a
  sender you haven't confirmed</li>
  <li><code>37</code> unrecognised install source · <code>38</code> package sources
  unreachable</li>
  <li><code>45</code> training data missing · <code>46</code> system library missing at
  start-up</li>
  <li><code>53</code> no GPU code for this card at run time · <code>55</code> system library
  missing during the test render</li>
  <li><code>63</code> driver too old, or the GPU can't start CUDA at all · <code>64</code> GPU generation not supported ·
  <code>65</code> not enough disk</li>
</ul>