> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Remote GPU training over SSH

> Train on another GPU box over plain SSH — runs survive your laptop closing

Train on a remote GPU host over plain SSH. The run outlives your SSH session, so closing your laptop does not kill training.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    subgraph "Remote SSH training"
        L[💻 Your laptop] -->|ssh/scp| H[🖥️ Remote GPU host]
        H --> R[🧠 Detached training run]
        R --> A[📦 Adapter]
        L -.->|reattach anytime| R
    end

    classDef local fill:#8B0000,stroke:#7C90A0,color:#fff
    classDef remote fill:#189AB4,stroke:#7C90A0,color:#fff
    classDef run fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef out fill:#10B981,stroke:#7C90A0,color:#fff

    class L local
    class H remote
    class R run
    class A out
```

<Note>
  Reachable through the integrated CLI (`praisonai train remote …`) from PraisonAI [PR #4367](https://github.com/MervinPraison/PraisonAI/pull/4367) onward — before that the whole group was unregistered and only worked from a standalone `praisonai-train` install. It speaks plain `ssh`/`scp` through `subprocess`, so there is **no third-party dependency**: any host in your `~/.ssh/config` works.
</Note>

## Quick Start

You have a config, a dataset, and an SSH alias for a GPU box (`gpubox` in `~/.ssh/config`).

<Steps>
  <Step title="Check the host is ready">
    ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    praisonai-train remote preflight gpubox
    ```

    Prints `gpubox is ready to train.` — or exits non-zero with the reason (no CUDA GPU, `python3` not usable, host unreachable).
  </Step>

  <Step title="Start training">
    ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    praisonai-train remote start gpubox \
      -c config.yaml \
      -d data/train.jsonl
    ```

    Prints the run id **and** the stop command **before** streaming — so a Ctrl-C during the follow still leaves you with the handle:

    ```
    ✓ started run-1787... on gpubox
      tail:  praisonai-train remote tail gpubox run-1787...
      stop:  praisonai-train remote stop gpubox run-1787...
    step 1 | loss 2.14
    ...
    ```
  </Step>

  <Step title="Or start and get your shell back">
    Use `--no-follow` if you want the shell back immediately.

    ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    praisonai-train remote start gpubox -c config.yaml -d data/train.jsonl --no-follow
    ```
  </Step>

  <Step title="Check on it later">
    Every command reattaches from just the host and run id, so a dropped laptop connection never kills training.

    ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    praisonai train remote status gpu-box run-1787667005
    ```
  </Step>

  <Step title="Fetch the artefacts back">
    Pull the adapter (or any file inside the run directory) home.

    ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    praisonai train remote fetch gpu-box run-1787667005 lora_model --out ./out
    ```
  </Step>
</Steps>

***

## Reattach, inspect, fetch, stop

The run is launched under `setsid` (or plain `nohup` on macOS, which has no `setsid`), so it **outlives the SSH session that started it** and its children — dataloader workers, torchrun ranks — sit in one process group `stop` can take down together. The run id names a directory on the far host; that id is enough to reattach, inspect, fetch, or stop the run later. The launch redirects its stdin from `/dev/null`, so `remote start` returns as soon as the trainer is running rather than holding the SSH connection open for the run's duration.

Everything after `start` takes only `host` and `run_id`.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Stream the log again (survives log rotation)
praisonai-train remote tail gpubox run-1787...

# Is it still going?
praisonai-train remote status gpubox run-1787...
# → running | completed | failed (exit N) | unknown

# Bring the adapter back
praisonai-train remote fetch gpubox run-1787... lora_model -o ./out

# Kill it
praisonai-train remote stop gpubox run-1787...
```

***

## Deciding which command to run

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph TB
    Start{What do you need?}
    Start -->|Is this box usable?| P[preflight]
    Start -->|Kick off training| S[start]
    Start -->|Watch a running job's log| T[tail]
    Start -->|Is it still running?| ST[status]
    Start -->|Get the adapter/log back| F[fetch]
    Start -->|Kill a run| K[stop]

    classDef q fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef c fill:#189AB4,stroke:#7C90A0,color:#fff
    class Start q
    class P,S,T,ST,F,K c
```

***

## How It Works

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
    participant User
    participant CLI as local CLI
    participant Host as remote GPU

    User->>CLI: remote start gpu-box -c config.yaml -d data.jsonl
    CLI->>Host: preflight (reachable? python? GPUs?)
    CLI->>Host: scp config.yaml + dataset
    CLI->>Host: nohup python -m praisonai_train llm data.jsonl --config config.yaml
    Host-->>CLI: run id
    CLI-->>User: tail log until status file appears
    User->>CLI: remote fetch gpu-box <run-id> lora_model
    CLI->>Host: scp run-dir/lora_model back
```

Runs launch under `nohup` in a detached session. The pid is written to `<run_dir>/train.pid` and the exit code to `<run_dir>/status`, so `status`, `tail` and `stop` work from any machine that can `ssh` to the host.

| Guarantee | How it's kept |
| - | - |
| Runs outlive your SSH session | `start` launches under `nohup` in its own session; a dropped connection can't kill training. |
| Reattach with just the run id | `tail`, `status`, `fetch`, `stop` reconnect from `host` + `run_id` alone. |
| No password prompt hangs | SSH runs in `BatchMode=yes` with a 10s connect timeout — a clean failure, never a stuck prompt. |
| No third-party dependency | Shells out to standard `ssh`/`scp` — no paramiko, no fabric. |

***

## Commands

Every command takes the SSH `host` alias first. Runs are addressed by the run id `start` prints.

### `remote preflight`

Check the host can train, before shipping anything to it.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai train remote preflight gpu-box --gpus 1
```

| Option | Type | Default | Description |
| - | - | - | - |
| `host` (arg) | str | **required** | SSH alias, as in `~/.ssh/config`. |
| `--gpus` | int | `1` | How many GPUs the host must have. |
| `--python` | str | `python3` | Interpreter to use on the host. |
| `--workdir` | str | `~/.praisonai-train` | Directory to work in on the host. See the [workdir rules](#workdir). |
| `--json` / `-j` | flag | `false` | Emit a machine-readable verdict. A not-ready host still exits `1`. |

### `remote start`

Ship the job and start it. The run outlives this command.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai train remote start gpu-box --config config.yaml --dataset data.jsonl
```

| Option | Type | Default | Description |
| - | - | - | - |
| `host` (arg) | str | **required** | SSH alias, as in `~/.ssh/config`. |
| `--config` / `-c` | path | **required** | Training config to ship (must exist locally). |
| `--dataset` / `-d` | path | — | Local dataset to ship alongside it (must exist). |
| `--gpus` | int | `1` | How many GPUs the run expects to find. |
| `--run-id` | str | auto (`run-<epoch>`) | Name the run directory. Must pass the [run-id rules](#run-ids). |
| `--python` | str | `python3` | Interpreter on the remote host. |
| `--workdir` | str | `~/.praisonai-train` | Directory to work in on the host. See the [workdir rules](#workdir). |
| `--follow` / `--no-follow` | flag | `--follow` | Stream the log until the run ends. |

### `remote tail`

Reattach to a running job's log; prints the final status when it ends.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai train remote tail gpu-box run-1787667005
```

| Option | Type | Default | Description |
| - | - | - | - |
| `host` (arg) | str | **required** | SSH alias. |
| `run_id` (arg) | str | **required** | The run id `start` printed. |
| `--python` | str | `python3` | Interpreter on the host. |
| `--workdir` | str | `~/.praisonai-train` | Directory to work in on the host. See the [workdir rules](#workdir). |

### `remote status`

Whether a run is still going, and how it ended if not — prints `running`, `completed`, `failed (exit N)`, or `unknown`.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai train remote status gpu-box run-1787667005
```

| Option | Type | Default | Description |
| - | - | - | - |
| `host` (arg) | str | **required** | SSH alias. |
| `run_id` (arg) | str | **required** | The run id. |
| `--python` | str | `python3` | Interpreter on the host. |
| `--workdir` | str | `~/.praisonai-train` | Directory to work in on the host. See the [workdir rules](#workdir). |

<Note>
  Fixed in PraisonAI [PR #4553](https://github.com/MervinPraison/PraisonAI/pull/4553) (closes [#4549](https://github.com/MervinPraison/PraisonAI/issues/4549)). Earlier releases recorded the pid of a transient wrapper subshell that could exit while training kept running, so `status` on a live run could report `unknown`. The pid file now names the wrapper that owns the trainer, so `status` probes a live process for the whole run.
</Note>

### `remote fetch`

Bring an artefact back — the adapter, a checkpoint, the log.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai train remote fetch gpu-box run-1787667005 lora_model --out ./out
```

| Option | Type | Default | Description |
| - | - | - | - |
| `host` (arg) | str | **required** | SSH alias. |
| `run_id` (arg) | str | **required** | The run id. |
| `remote_path` (arg) | str | **required** | Path inside the run directory. Must pass the [fetch-path rules](#fetch-paths). |
| `--out` / `-o` | path | `.` | Local directory to copy into. |
| `--python` | str | `python3` | Interpreter on the host. |
| `--workdir` | str | `~/.praisonai-train` | Directory to work in on the host. See the [workdir rules](#workdir). |

### `remote stop`

Stop a run. Signals the whole process group on the host — the wrapper, the trainer, and its dataloader / torchrun children — then re-probes to confirm the group is gone before reporting success. `stopped` means the run is actually off the GPU, not just that `kill -TERM` was issued.

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
praisonai train remote stop gpu-box run-1787667005
```

Three outcomes:

| Outcome | When | Output |
| - | - | - |
| Stopped | The group is gone after SIGTERM. | `stopped <run-id> on <host>` |
| Nothing to stop | No pid file, or the pid was already dead. | `<run-id> was not running on <host>` |
| Refused to lie | A process in the group survived SIGTERM, or the pid resolves to this SSH session's own group (a corrupt pid file). | `RemoteError: … did not stop …` or `RemoteError: refused to stop …` |

<Note>
  Fixed in PraisonAI [PR #4551](https://github.com/MervinPraison/PraisonAI/pull/4551) (closes [#4547](https://github.com/MervinPraison/PraisonAI/issues/4547)) — the remote counterpart of [#4491](https://github.com/MervinPraison/PraisonAI/pull/4491). Earlier releases signalled only the recorded pid (the wrapper shell), so real fine-tune children survived `stop`, reparented to init, and kept the GPU; and `stop` reported success the moment `kill -TERM` was issued, letting a UI label a live paid run "cancelled".
</Note>

<Note>
  The recording side of the same class of bug is fixed in PraisonAI [PR #4553](https://github.com/MervinPraison/PraisonAI/pull/4553) (closes [#4549](https://github.com/MervinPraison/PraisonAI/issues/4549)): the pid file now names the wrapper whose group contains the trainer, so the group `stop` resolves is the one the trainer actually runs in.
</Note>

| Option | Type | Default | Description |
| - | - | - | - |
| `host` (arg) | str | **required** | SSH alias. |
| `run_id` (arg) | str | **required** | The run id. |
| `--python` | str | `python3` | Interpreter on the host. |
| `--workdir` | str | `~/.praisonai-train` | Directory to work in on the host. See the [workdir rules](#workdir). |

***

## Run IDs

`--run-id` (when passed) must be a plain identifier: **letters, digits, dot, dash and underscore only; starting with a letter or digit; up to 64 characters**. Anything else is refused:

```text theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
RemoteError: run id 'x;touch /tmp/pwned' is not usable: letters, digits, dot, dash and underscore only, starting with a letter or digit, up to 64 chars.
```

This validator was added in [PR #4367](https://github.com/MervinPraison/PraisonAI/pull/4367): a shell metacharacter inside `--run-id` was interpolated into the remote path and executed by the far-side shell. Values like `x;touch /tmp/pwned`, `$(id)`, `` `id` ``, `a b`, `../escape`, `/absolute`, an empty string, and `-leading-dash` are all refused. `run-1787667005` and `my_run.2` are fine.

## Workdir

`--workdir` (the remote directory the run lives in, default `~/.praisonai-train`) is expanded by the remote shell, so it must survive being passed unquoted. The runner refuses anything outside a **safe character set: letters, digits, `.`, `_`, `-`, `/`, and an optional leading `~`.** Anything else is rejected before any SSH is opened:

```text theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
RemoteError: gpu-box: workdir '~/x; rm -rf /' may only contain letters, digits and ._-/~ -- it is expanded by the remote shell.
```

The `~` is expanded by the remote shell (not the local one), so `~/.praisonai-train` lands in the remote user's real home. The directory is created with `mkdir -p -m 700`, so on a shared GPU box the umask does not decide who can read the shipped config and dataset.

<Note>
  **Migration note (PraisonAI [PR #4550](https://github.com/MervinPraison/PraisonAI/pull/4550)).** Before this fix, the workdir was quoted before the remote shell saw it, so a directory **literally named `~`** was created in the login directory and every derived path (config, logs, checkpoints) landed there while `~/.praisonai-train` stayed empty. If you ran remote training on an older release, log in to the box and check:

  ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
  ssh gpu-box 'ls -ld ~ ~/\~ 2>/dev/null'
  ```

  A `~` directory (yes, one whose name is a single tilde character) is safe to remove after moving any artefacts you still want out of it:

  ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
  ssh gpu-box 'ls ~/\~'          # inspect
  ssh gpu-box 'rm -rf -- ~/\~'   # remove when you're done
  ```

  On the current release, `praisonai train remote` creates and uses the real `~/.praisonai-train` instead.
</Note>

## Fetch paths

`fetch <run> <path>` treats `<path>` as **relative to the run directory** on the remote host. These are refused up-front:

* absolute paths (`/etc/passwd`)
* any path containing `..` (`../../../../etc/passwd`, `a/../../b`)
* empty or whitespace-only paths

```text theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
RemoteError: '../../../../etc/passwd' points outside the run directory. Fetch paths are relative to it, e.g. 'lora_model' or 'train.log'.
```

The runner also `shlex.quote()`s the scp target, so a shell metacharacter that survives validation cannot be executed on the remote side — validation catches the typo, quoting catches everything else. `lora_model` and `outputs/checkpoint-100` are accepted.

## Command shape on the remote side

The remote invocation is:

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
python -m praisonai_train llm [DATASET] --config config.yaml
```

Not `python -m praisonai_train llm config.yaml [DATASET]` — that earlier shape produced two positionals, which Typer rejected with exit `2` before the GPU was touched. If a `remote start` log ends in `Error: Got unexpected extra argument` on an old version, upgrade to a release carrying [PR #4367](https://github.com/MervinPraison/PraisonAI/pull/4367).

***

## Why "not ready" is not a crash

Preflight failures and stopping something that isn't running are **answers**, not errors — but they still exit non-zero (preflight) or print `was not running` (stop) so scripts can't mistake them for success.

<AccordionGroup>
  <Accordion title="Preflight refuses an unready box">
    ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    $ praisonai-train remote preflight gpubox
    ✗ gpubox cannot train
      needs 1 GPU(s), found 0 (no CUDA GPU visible to nvidia-smi)
    $ echo $?
    1
    ```

    A script reading only the exit status will not ship a job to a box that can't run it.
  </Accordion>

  <Accordion title="Stop is honest about no-ops">
    ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    $ praisonai-train remote stop gpubox run-1787
    run-1787 was not running on gpubox
    ```

    Reporting success for a no-op would let a script believe it killed a run still burning GPU hours.
  </Accordion>

  <Accordion title="Nothing ships without a config">
    ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    $ praisonai-train remote start gpubox -c nope.yaml
    Error: Invalid value for '--config' / '-c': Path 'nope.yaml' does not exist.
    ```

    The missing config is refused before any SSH happens.
  </Accordion>
</AccordionGroup>

***

## Prerequisites

<AccordionGroup>
  <Accordion title="Key-based SSH (no password prompts)">
    The runner uses `ssh` in `BatchMode=yes` — password prompts are never accepted. Set up key auth:

    ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    ssh-copy-id gpubox
    ssh gpubox echo ok   # must print `ok` without a prompt
    ```

    Alias config lives in `~/.ssh/config`:

    ```
    Host gpubox
        HostName gpu-01.example.com
        User me
        IdentityFile ~/.ssh/id_ed25519
    ```
  </Accordion>

  <Accordion title="A usable Python and GPU on the host">
    The default `--python python3` must exist and be able to import `praisonai_train`. `nvidia-smi` must list at least `--gpus` GPUs.
  </Accordion>

  <Accordion title="Standard OpenSSH only">
    No paramiko, fabric, or SSH library required — the runner shells out to `ssh`/`scp`.
  </Accordion>
</AccordionGroup>

***

## Python API

Script the same flow with `RemoteRunner` — each method reconnects, so a runner can be rebuilt from just `host` + `run_id`.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonai_train.remote import RemoteError, RemoteRunner

runner = RemoteRunner(alias="gpubox")

# Is the box ready?
verdict = runner.preflight(expect_gpus=1)
if not verdict.ok:
    raise SystemExit(verdict.why_not())

# Start detached — returns immediately with a handle.
run = runner.start(config_path="config.yaml", dataset_path="data/train.jsonl")
print(run.run_id)

# Reattach, inspect, fetch, stop later.
runner.tail(run, on_line=print)
print(runner.status(run))              # running | completed | failed (exit N) | unknown
runner.fetch(run, remote_rel="lora_model", dest="./out")
runner.stop(run)                       # True only if something was actually killed
```

| Export | Purpose |
| - | - |
| `RemoteRunner` | Driver: `preflight` / `start` / `tail` / `status` / `fetch` / `stop`. |
| `RemoteRun` | Handle to one run (`host`, `run_id`, `remote_dir`, `log_path`). |
| `RemotePreflight` | Verdict of a readiness check (`ok`, `reasons`, `why_not()`). |
| `RemoteHost` | One SSH destination (`alias`, `python`, `workdir`). |
| `RemoteError` | Raised for actionable failures (unready host, refused start, missing artifact). |

***

## Best Practices

<AccordionGroup>
  <Accordion title="Preflight before you ship a big dataset">
    `praisonai-train remote preflight gpubox --gpus 2` catches a bad `~/.ssh/config`, a missing driver, or a mis-installed Python before you ship a multi-GB dataset.
  </Accordion>

  <Accordion title="Keep run ids to the safe alphabet">
    `--run-id my_run.2` is fine. Anything a shell would interpret is rejected up-front. Don't try to be clever — the safe alphabet is intentional.
  </Accordion>

  <Accordion title="Keep workdirs to the safe alphabet">
    `--workdir ~/.praisonai-train` (the default) is fine. Only letters, digits, `.`, `_`, `-`, `/`, and an optional leading `~` are allowed; `$(pwd)`, spaces, `;`, and quotes are refused before any SSH. See [Workdir](#workdir).
  </Accordion>

  <Accordion title="Ctrl-C the follow, not the run">
    `start` prints the id and stop hint before streaming. If you Ctrl-C the follow, the run keeps going — reattach with `remote tail <host> <run_id>`.
  </Accordion>

  <Accordion title="Script polling with status">
    `start --no-follow` returns immediately with the run id. Poll with `remote status` in a loop, then `fetch` when it reads `completed`.
  </Accordion>

  <Accordion title="Fetch only from the run directory">
    Fetch is scoped to the run's own artefacts. If you need something outside the run directory, log into the box directly with `ssh` — the CLI won't do it.
  </Accordion>

  <Accordion title="Fetch only what you need">
    `fetch` uses `scp -r` — pull `lora_model` for the adapter or `train.log` for the log. Avoid pulling the whole run directory.
  </Accordion>

  <Accordion title="Use a short ~/.ssh/config alias">
    Everything the CLI does is one hop, per-alias. A short entry with the host, user, and identity keeps the CLI arguments short.
  </Accordion>

  <Accordion title="Ship the config you want">
    `remote start` ships the config you pass and overwrites `config.yaml` on the box too. Don't rely on the box to already have a matching one.
  </Accordion>
</AccordionGroup>

***

## Related

<CardGroup cols={2}>
  <Card title="CLI: train" icon="terminal" href="/docs/cli/train">
    All `praisonai train` subcommands.
  </Card>

  <Card title="Train" icon="graduation-cap" href="/docs/train">
    `config.yaml` reference for the run that ships to the box.
  </Card>

  <Card title="Train Export" icon="upload" href="/docs/features/praisonai-train-export">
    Publish the adapter you fetched back — to Hugging Face, GGUF, or Ollama.
  </Card>

  <Card title="Multi-GPU Training" icon="microchip" href="/docs/features/praisonai-train-multigpu">
    Fine-tune across every GPU on the host with torchrun.
  </Card>

  <Card title="Checkpointing" icon="database" href="/docs/features/praisonai-train-checkpointing">
    Resume from checkpoints if a long run dies mid-way.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.