jobflow_remote.remote.host.separated module#

Host implementation that uses separate connections for commands and file transfers.

This is useful for HPC systems where login nodes have SFTP disabled but a dedicated data transfer node is available (e.g., LRC at LBNL).

class jobflow_remote.remote.host.separated.SeparatedTransferHost(command_host: RemoteHost, transfer_host: RemoteHost)[source]#

Bases: BaseHost

Host that delegates commands and file transfers to separate connections.

This enables HPC systems where:

Login node: SSH/scheduler commands work, but SFTP is disabled
Transfer node: SFTP works, but scheduler commands don’t work

Command execution (execute, shell) goes through the command_host. File operations (put, get, mkdir, etc.) go through the transfer_host.

Parameters:

command_host (RemoteHost) – Host for executing commands (e.g., login node with SLURM access).
transfer_host (RemoteHost) – Host for file transfers (e.g., data transfer node with SFTP).
sanitize – If True text a string will be prepended and appended to the output of the commands, to ease the parsing and avoid failures due to spurious text coming from the host shell.

close() → bool[source]#

Close connections to both hosts.

Returns:: True if both connections were closed successfully.
Return type:: bool

connect() → None[source]#: Open connections to both the command and transfer hosts.

copy(src, dst) → None[source]#

Copy a file on the command host.

Parameters:

src (str or Path) – Source path on remote host.
dst (str or Path) – Destination path on remote host.

execute(command: str | list[str], workdir: str | Path | None = None, timeout: int | None = None) → tuple[str, str, int][source]#

Execute the given command on the command host.

Parameters:

command (str or list of str) – Command to execute, as a str or list of str.
workdir (str or Path or None) – Path where the command will be executed.
timeout (int or None) – Timeout for the execution of the command.

Returns:

stdout (str) – Standard output of the command.
stderr (str) – Standard error of the command.
exit_code (int) – Exit code of the command.

exists(path: str | Path) → bool[source]#

Check if a path exists on the transfer host.

Parameters:: path (str or Path) – The path to check.
Returns:: True if the path exists.
Return type:: bool

get(src, dst) → None[source]#

Download a file from the transfer host.

Parameters:

src (str or Path) – Remote source path.
dst (str or Path or file-like) – Local destination path or file-like object.

property interactive_login: bool#

Check if either host requires interactive login.

Returns:: True if either host requires interactive login.
Return type:: bool

property is_connected: bool#

Check if both connections are open.

Returns:: True if both command and transfer hosts are connected.
Return type:: bool

listdir(path: str | Path) → list[str][source]#

List directory contents on the transfer host.

Parameters:: path (str or Path) – Path to the directory to list.
Returns:: List of filenames in the directory.
Return type:: list of str

mkdir(directory: str | Path, recursive: bool = True, exist_ok: bool = True) → bool[source]#

Create directory on the command host.

Parameters:

directory (str or Path) – Path of the directory to create.
recursive (bool) – If True, create parent directories as needed.
exist_ok (bool) – If True, do not raise an error if directory exists.

Returns:

True if the directory was created successfully.

Return type:

bool

move(src, dst) → None[source]#

Move a file on the command host.

Parameters:

src (str or Path) – Source path on remote host.
dst (str or Path) – Destination path on remote host.

put(src, dst) → None[source]#

Upload a file to the transfer host.

Parameters:

src (str or Path or file-like) – Local source path or file-like object.
dst (str or Path) – Remote destination path.

read_text_file(filepath: str | Path) → str[source]#

Read content from a file on the transfer host.

Parameters:: filepath (str or Path) – Path to the file to read.
Returns:: Content of the file.
Return type:: str

remove(path: str | Path) → None[source]#

Remove a file on the transfer host.

Parameters:: path (str or Path) – Path to the file to remove.

rmtree(path: str | Path, raise_on_error: bool = False) → bool[source]#

Recursively delete a directory tree on the command host.

Parameters:

path (str or Path) – Path to the directory tree to be removed.
raise_on_error (bool) – If False (default), errors will be ignored. Otherwise, errors will raise an exception.

Returns:

True if the directory tree was successfully removed.

Return type:

bool

shell(pre_cmd: str | None = None, shell: str = 'bash')[source]#

Open a shell on the command host.

Parameters:

pre_cmd (str or None) – Any command to be executed before starting the shell.
shell (str) – The name of the shell to start.

write_text_file(filepath: str | Path, content: str) → None[source]#

Write content to a file on the transfer host.

Parameters:

filepath (str or Path) – Path to the file to write.
content (str) – Content to write to the file.