Remote GPU training over SSH

Train on another GPU box over plain SSH — runs survive your laptop closing

Train on a remote GPU host over plain SSH. The run outlives your SSH session, so closing your laptop does not kill training.

Quick Start

You have a config, a dataset, and an SSH alias for a GPU box (gpubox in ~/.ssh/config).

1. Check the host is ready

   praisonai-train remote preflight gpubox

   Prints gpubox is ready to train. — or exits non-zero with the reason (no CUDA GPU, python3 not usable, host unreachable).

2. Start training

   praisonai-train remote start gpubox \
     -c config.yaml \
     -d data/train.jsonl

   Prints the run id and the stop command before streaming — so a Ctrl-C during the follow still leaves you with the handle:

   ✓ started run-1787... on gpubox
     tail:  praisonai-train remote tail gpubox run-1787...
     stop:  praisonai-train remote stop gpubox run-1787...
   step 1 | loss 2.14
   ...

3. Or start and get your shell back

   Use --no-follow if you want the shell back immediately.

   praisonai-train remote start gpubox -c config.yaml -d data/train.jsonl --no-follow

4. Check on it later

   Every command reattaches from just the host and run id, so a dropped laptop connection never kills training.

   praisonai train remote status gpu-box run-1787667005

5. Fetch the artefacts back

   Pull the adapter (or any file inside the run directory) home.

   praisonai train remote fetch gpu-box run-1787667005 lora_model --out ./out

Reattach, inspect, fetch, stop

The run is launched under setsid (or plain nohup on macOS, which has no setsid), so it outlives the SSH session that started it and its children — dataloader workers, torchrun ranks — sit in one process group stop can take down together. The run id names a directory on the far host; that id is enough to reattach, inspect, fetch, or stop the run later. The launch redirects its stdin from /dev/null, so remote start returns as soon as the trainer is running rather than holding the SSH connection open for the run’s duration.

Everything after start takes only host and run_id.

# Stream the log again (survives log rotation)
praisonai-train remote tail gpubox run-1787...

# Is it still going?
praisonai-train remote status gpubox run-1787...
# → running | completed | failed (exit N) | unknown

# Bring the adapter back
praisonai-train remote fetch gpubox run-1787... lora_model -o ./out

# Kill it
praisonai-train remote stop gpubox run-1787...

Deciding which command to run

How It Works

Runs launch under nohup in a detached session. The pid is written to <run_dir>/train.pid and the exit code to <run_dir>/status, so status, tail and stop work from any machine that can ssh to the host.

Commands

Every command takes the SSH host alias first. Runs are addressed by the run id start prints.

remote preflight

Check the host can train, before shipping anything to it.

praisonai train remote preflight gpu-box --gpus 1

remote start

Ship the job and start it. The run outlives this command.

praisonai train remote start gpu-box --config config.yaml --dataset data.jsonl

remote tail

Reattach to a running job’s log; prints the final status when it ends.

praisonai train remote tail gpu-box run-1787667005

remote status

Whether a run is still going, and how it ended if not — prints running, completed, failed (exit N), or unknown.

praisonai train remote status gpu-box run-1787667005

remote fetch

Bring an artefact back — the adapter, a checkpoint, the log.

praisonai train remote fetch gpu-box run-1787667005 lora_model --out ./out

remote stop

Stop a run. Signals the whole process group on the host — the wrapper, the trainer, and its dataloader / torchrun children — then re-probes to confirm the group is gone before reporting success. stopped means the run is actually off the GPU, not just that kill -TERM was issued.

praisonai train remote stop gpu-box run-1787667005

Three outcomes:

Run IDs

--run-id (when passed) must be a plain identifier: letters, digits, dot, dash and underscore only; starting with a letter or digit; up to 64 characters. Anything else is refused:

RemoteError: run id 'x;touch /tmp/pwned' is not usable: letters, digits, dot, dash and underscore only, starting with a letter or digit, up to 64 chars.

This validator was added in PR #4367: a shell metacharacter inside --run-id was interpolated into the remote path and executed by the far-side shell. Values like x;touch /tmp/pwned, $(id), ` id , a b, ../escape, /absolute, an empty string, and -leading-dash are all refused. run-1787667005 and my_run.2` are fine.

Workdir

--workdir (the remote directory the run lives in, default ~/.praisonai-train) is expanded by the remote shell, so it must survive being passed unquoted. The runner refuses anything outside a safe character set: letters, digits, ., _, -, /, and an optional leading ~. Anything else is rejected before any SSH is opened:

RemoteError: gpu-box: workdir '~/x; rm -rf /' may only contain letters, digits and ._-/~ -- it is expanded by the remote shell.

The ~ is expanded by the remote shell (not the local one), so ~/.praisonai-train lands in the remote user’s real home. The directory is created with mkdir -p -m 700, so on a shared GPU box the umask does not decide who can read the shipped config and dataset.

Fetch paths

fetch <run> <path> treats <path> as relative to the run directory on the remote host. These are refused up-front:

• absolute paths (/etc/passwd)
• any path containing .. (../../../../etc/passwd, a/../../b)
• empty or whitespace-only paths

RemoteError: '../../../../etc/passwd' points outside the run directory. Fetch paths are relative to it, e.g. 'lora_model' or 'train.log'.

The runner also shlex.quote()s the scp target, so a shell metacharacter that survives validation cannot be executed on the remote side — validation catches the typo, quoting catches everything else. lora_model and outputs/checkpoint-100 are accepted.

Command shape on the remote side

The remote invocation is:

python -m praisonai_train llm [DATASET] --config config.yaml

Not python -m praisonai_train llm config.yaml [DATASET] — that earlier shape produced two positionals, which Typer rejected with exit 2 before the GPU was touched. If a remote start log ends in Error: Got unexpected extra argument on an old version, upgrade to a release carrying PR #4367.

Why “not ready” is not a crash

Preflight failures and stopping something that isn’t running are answers, not errors — but they still exit non-zero (preflight) or print was not running (stop) so scripts can’t mistake them for success.

$ praisonai-train remote preflight gpubox
✗ gpubox cannot train
  needs 1 GPU(s), found 0 (no CUDA GPU visible to nvidia-smi)
$ echo $?
1

$ praisonai-train remote stop gpubox run-1787
run-1787 was not running on gpubox

$ praisonai-train remote start gpubox -c nope.yaml
Error: Invalid value for '--config' / '-c': Path 'nope.yaml' does not exist.

Prerequisites

ssh-copy-id gpubox
ssh gpubox echo ok   # must print `ok` without a prompt

Host gpubox
    HostName gpu-01.example.com
    User me
    IdentityFile ~/.ssh/id_ed25519

Python API

Script the same flow with RemoteRunner — each method reconnects, so a runner can be rebuilt from just host + run_id.

from praisonai_train.remote import RemoteError, RemoteRunner

runner = RemoteRunner(alias="gpubox")

# Is the box ready?
verdict = runner.preflight(expect_gpus=1)
if not verdict.ok:
    raise SystemExit(verdict.why_not())

# Start detached — returns immediately with a handle.
run = runner.start(config_path="config.yaml", dataset_path="data/train.jsonl")
print(run.run_id)

# Reattach, inspect, fetch, stop later.
runner.tail(run, on_line=print)
print(runner.status(run))              # running | completed | failed (exit N) | unknown
runner.fetch(run, remote_rel="lora_model", dest="./out")
runner.stop(run)                       # True only if something was actually killed

Best Practices

Related