jobflow_remote.remote.host package#

Submodules#

Module contents#

class jobflow_remote.remote.host.BaseHost(sanitize: bool = False)[source]#

Bases: MSONable

Base Host class.

Parameters:: sanitize – If True text a string will be prepended and appended to the output of the commands, to ease the parsing and avoid failures due to spurious text coming from the host shell.

abstractmethod close() → bool[source]#

abstractmethod connect()[source]#

abstractmethod copy(src, dst)[source]#

abstractmethod execute(command: str | list[str], workdir: str | Path | None = None, timeout: int | None = None) → tuple[str, str, int][source]#

Execute the given command on the host.

Parameters:

command (str or list of str) – Command to execute, as a str or list of str
workdir (str or None) – path where the command will be executed.
timeout – Timeout for the execution of the commands.

abstractmethod exists(path: str | Path) → bool[source]#

Test whether a path exists.

Parameters:: path – The path to check whether it exists.
Returns:: True if the path exists
Return type:: bool

abstractmethod get(src, dst)[source]#

property interactive_login: bool#: True if the host requires interactive actions upon login. False by default. Subclasses should override the method to customize the value.

abstract property is_connected: bool#

abstractmethod listdir(path: str | Path) → list[str][source]#

abstractmethod mkdir(directory: str | Path, recursive: bool = True, exist_ok: bool = True) → bool[source]#: Create directory on the host.

abstractmethod move(src, dst)[source]#

abstractmethod put(src, dst)[source]#

abstractmethod read_text_file(filepath) → str[source]#: Read content from a file on the host.

abstractmethod remove(path: str | Path)[source]#

abstractmethod rmtree(path: str | Path, raise_on_error: bool = False) → bool[source]#

Recursively delete a directory tree on a host.

This method must be implemented by subclasses of BaseHost. It is intended to remove an entire directory tree, including all files and subdirectories, on the host represented by the subclass.

Parameters:

path (str or Path) – The path to the directory tree to be removed.
raise_on_error (bool) – If set to False (default), errors will be ignored, and the method will attempt to continue removing remaining files and directories. Otherwise, any errors encountered during the removal process will raise an exception.

Returns:

True if the directory tree was successfully removed, False otherwise.

Return type:

bool

sanitize_command(cmd: str) → str[source]#

Sanitizes a command by adding a prefix and suffix to the command string if sanitization is enabled. The prefix and suffix are the same and are used to mark the parts of the output that should be sanitized. The prefix and suffix are defined by SANITIZE_KEY.

Parameters:: cmd – The command string to be sanitized
Returns:: The sanitized command string
Return type:: str

sanitize_output(output: str) → str[source]#

Sanitizes the output of a command by selecting the section between the SANITIZE_KEY strings. If the second instance of the key is not found, the part of the output after the key is returned. If the key is not present, the entire output is returned.

Parameters:: output – The output of the command to be sanitized
Returns:: The sanitized output
Return type:: str

property sanitize_regex: Pattern#: Regular expression to sanitize sensitive info in command outputs.

abstractmethod shell(pre_cmd: str | None = None, shell: str = 'bash')[source]#

Open a connection to the host and starts the selected shell

Parameters:

pre_cmd – Any command to be executed before starting the shell
shell – The name of the shell to start

test() → str | None[source]#

abstractmethod write_text_file(filepath, content)[source]#: Write content to a file on the host.

class jobflow_remote.remote.host.LocalHost(timeout_execute: int = None, sanitize: bool = False)[source]#

Bases: BaseHost

Parameters:: sanitize – If True text a string will be prepended and appended to the output of the commands, to ease the parsing and avoid failures due to spurious text coming from the host shell.

close() → bool[source]#

connect() → None[source]#

copy(src, dst) → None[source]#

execute(command: str | list[str], workdir: str | Path | None = None, timeout: int | None = None)[source]#

Execute the given command on the host.

Note that the command is executed with shell=True, so commands can be exposed to command injection. Consider whether to escape part of the input if it comes from external users.

Parameters:

command (str or list of str) – Command to execute, as a str or list of str
workdir (str or None) – path where the command will be executed.
timeout – Timeout for the execution of the commands.

Returns:

stdout (str) – Standard output of the command
stderr (str) – Standard error of the command
exit_code (int) – Exit code of the command.

exists(path: str | Path) → bool[source]#

Test whether a path exists.

Parameters:: path – The path to check whether it exists.
Returns:: True if the path exists
Return type:: bool

get(src, dst) → None[source]#

property is_connected: bool#

listdir(path: str | Path) → list[str][source]#

mkdir(directory: str | Path, recursive: bool = True, exist_ok: bool = True) → bool[source]#: Create directory on the host.

move(src, dst) → None[source]#

put(src, dst) → None[source]#

read_text_file(filepath) → str[source]#: Read content from a file on the host.

remove(path: str | Path) → None[source]#

rmtree(path: str | Path, raise_on_error: bool = False) → bool[source]#

Recursively delete a directory tree on a local host.

It is intended to remove an entire directory tree, including all files and subdirectories, on this local host.

Parameters:

path (str or Path) – The path to the directory tree to be removed.
raise_on_error (bool) – If set to False (default), errors will be ignored, and the method will attempt to continue removing remaining files and directories. Otherwise, any errors encountered during the removal process will raise an exception.

Returns:

True if the directory tree was successfully removed, False otherwise.

Return type:

bool

shell(pre_cmd: str | None = None, shell: str = 'bash')[source]#

Open a connection to the host and starts the selected shell

Parameters:

pre_cmd – Any command to be executed before starting the shell
shell – The name of the shell to start

write_text_file(filepath, content) → None[source]#: Write content to a file on the host.

class jobflow_remote.remote.host.RemoteHost(host, user=None, port=None, config=None, gateway=None, forward_agent=None, connect_timeout=None, connect_kwargs=None, inline_ssh_env=None, timeout_execute=None, keepalive=60, shell_cmd='bash', login_shell=True, retry_on_closed_connection=True, interactive_login=False, sanitize: bool = False)[source]#

Bases: BaseHost

Execute commands on a remote host. For some commands assumes the remote can run unix.

Parameters:: sanitize – If True text a string will be prepended and appended to the output of the commands, to ease the parsing and avoid failures due to spurious text coming from the host shell.

close() → bool[source]#

connect() → None[source]#

property connection#

copy(src, dst) → None[source]#

execute(command: str | list[str], workdir: str | Path | None = None, timeout: int | None = None)[source]#

Execute the given command on the host.

Parameters:

command (str or list of str) – Command to execute, as a str or list of str.
workdir (str or None) – path where the command will be executed.

Returns:

stdout (str) – Standard output of the command
stderr (str) – Standard error of the command
exit_code (int) – Exit code of the command.

exists(path: str | Path) → bool[source]#

Test whether a path exists.

Parameters:: path – The path to check whether it exists.
Returns:: True if the path exists
Return type:: bool

get(src, dst) → None[source]#

property interactive_login: bool#: True if the host requires interactive actions upon login. False by default. Subclasses should override the method to customize the value.

property is_connected: bool#

listdir(path: str | Path)[source]#

mkdir(directory: str | Path, recursive: bool = True, exist_ok: bool = True) → bool[source]#: Create directory on the host.

move(src, dst) → None[source]#

put(src, dst) → None[source]#

read_text_file(filepath: str | Path) → str[source]#: Read content from a file on the host.

remove(path: str | Path) → None[source]#

rmtree(path: str | Path, raise_on_error: bool = False) → bool[source]#

Recursively delete a directory tree on a remote host.

It is intended to remove an entire directory tree, including all files and subdirectories, on this remote host.

Parameters:

path (str or Path) – The path to the directory tree to be removed.
raise_on_error (bool) – If set to False (default), errors will be ignored, and the method will attempt to continue removing remaining files and directories. Otherwise, any errors encountered during the removal process will raise an exception.

Returns:

True if the directory tree was successfully removed, False otherwise.

Return type:

bool

shell(pre_cmd: str | None = None, shell: str = 'bash')[source]#

Open a connection to the host and starts the selected shell

Parameters:

pre_cmd – Any command to be executed before starting the shell
shell – The name of the shell to start

write_text_file(filepath: str | Path, content: str) → None[source]#: Write content to a file on the host.

class jobflow_remote.remote.host.SeparatedTransferHost(command_host: RemoteHost, transfer_host: RemoteHost)[source]#

Bases: BaseHost

Host that delegates commands and file transfers to separate connections.

This enables HPC systems where:

Login node: SSH/scheduler commands work, but SFTP is disabled
Transfer node: SFTP works, but scheduler commands don’t work

Command execution (execute, shell) goes through the command_host. File operations (put, get, mkdir, etc.) go through the transfer_host.

Parameters:

command_host (RemoteHost) – Host for executing commands (e.g., login node with SLURM access).
transfer_host (RemoteHost) – Host for file transfers (e.g., data transfer node with SFTP).
sanitize – If True text a string will be prepended and appended to the output of the commands, to ease the parsing and avoid failures due to spurious text coming from the host shell.

close() → bool[source]#

Close connections to both hosts.

Returns:: True if both connections were closed successfully.
Return type:: bool

connect() → None[source]#: Open connections to both the command and transfer hosts.

copy(src, dst) → None[source]#

Copy a file on the command host.

Parameters:

src (str or Path) – Source path on remote host.
dst (str or Path) – Destination path on remote host.

execute(command: str | list[str], workdir: str | Path | None = None, timeout: int | None = None) → tuple[str, str, int][source]#

Execute the given command on the command host.

Parameters:

command (str or list of str) – Command to execute, as a str or list of str.
workdir (str or Path or None) – Path where the command will be executed.
timeout (int or None) – Timeout for the execution of the command.

Returns:

stdout (str) – Standard output of the command.
stderr (str) – Standard error of the command.
exit_code (int) – Exit code of the command.

exists(path: str | Path) → bool[source]#

Check if a path exists on the transfer host.

Parameters:: path (str or Path) – The path to check.
Returns:: True if the path exists.
Return type:: bool

get(src, dst) → None[source]#

Download a file from the transfer host.

Parameters:

src (str or Path) – Remote source path.
dst (str or Path or file-like) – Local destination path or file-like object.

property interactive_login: bool#

Check if either host requires interactive login.

Returns:: True if either host requires interactive login.
Return type:: bool

property is_connected: bool#

Check if both connections are open.

Returns:: True if both command and transfer hosts are connected.
Return type:: bool

listdir(path: str | Path) → list[str][source]#

List directory contents on the transfer host.

Parameters:: path (str or Path) – Path to the directory to list.
Returns:: List of filenames in the directory.
Return type:: list of str

mkdir(directory: str | Path, recursive: bool = True, exist_ok: bool = True) → bool[source]#

Create directory on the command host.

Parameters:

directory (str or Path) – Path of the directory to create.
recursive (bool) – If True, create parent directories as needed.
exist_ok (bool) – If True, do not raise an error if directory exists.

Returns:

True if the directory was created successfully.

Return type:

bool

move(src, dst) → None[source]#

Move a file on the command host.

Parameters:

src (str or Path) – Source path on remote host.
dst (str or Path) – Destination path on remote host.

put(src, dst) → None[source]#

Upload a file to the transfer host.

Parameters:

src (str or Path or file-like) – Local source path or file-like object.
dst (str or Path) – Remote destination path.

read_text_file(filepath: str | Path) → str[source]#

Read content from a file on the transfer host.

Parameters:: filepath (str or Path) – Path to the file to read.
Returns:: Content of the file.
Return type:: str

remove(path: str | Path) → None[source]#

Remove a file on the transfer host.

Parameters:: path (str or Path) – Path to the file to remove.

rmtree(path: str | Path, raise_on_error: bool = False) → bool[source]#

Recursively delete a directory tree on the command host.

Parameters:

path (str or Path) – Path to the directory tree to be removed.
raise_on_error (bool) – If False (default), errors will be ignored. Otherwise, errors will raise an exception.

Returns:

True if the directory tree was successfully removed.

Return type:

bool

shell(pre_cmd: str | None = None, shell: str = 'bash')[source]#

Open a shell on the command host.

Parameters:

pre_cmd (str or None) – Any command to be executed before starting the shell.
shell (str) – The name of the shell to start.

write_text_file(filepath: str | Path, content: str) → None[source]#

Write content to a file on the transfer host.

Parameters:

filepath (str or Path) – Path to the file to write.
content (str) – Content to write to the file.