jobflow_remote.jobs.batch module

class jobflow_remote.jobs.batch.LocalBatchManager(files_dir: str | Path, process_id: str, multiprocess_lock=None)

Bases: object

Manager of local files containing information about Jobs to be handled by a batch worker.

Used in the worker to executes the batch Jobs.

Parameters:

• files_dir – The full path to directory where the files to handle the jobs to be executed in batch processes are stored.

• process_id – The uuid associated to the batch process.

• multiprocess_lock – A lock from the multiprocessing module to be used when executing jobs in parallel with other processes of the same worker.

get_job() → str | None

Select randomly a job from the submitted directory to be executed. Move the file to the running directory.

Locks will prevent the same job from being executed from other processes. If no job can be executed, None is returned.

Returns:

The name of the job that was selected, or None if no job can be executed.

Return type:

str | None

terminate_job(job_id: str, index: int) → None

Terminate a job by removing the corresponding file from the running directory and adding a new file in the terminated directory.

Parameters:

• job_id – The uuid of the job to terminate.

• index – The index of the job to terminate.

class jobflow_remote.jobs.batch.RemoteBatchManager(host: BaseHost, files_dir: str | Path)

Bases: object

Manager of remote files containing information about Jobs to be handled by a batch worker.

Used by the Runner.

Parameters:

• host – The host where the files are.

• files_dir – The full path to directory where the files are stored.

delete_terminated(ids: list[tuple[str, int, str]]) → None

get_running() → list[tuple[str, int, str]]

Get job ids and process ids of the running jobs from the corresponding directory on the host.

Returns:

The list of job ids, job indexes and batch process uuids in the host running directory.

Return type:

list

get_submitted() → list[str]

Get a list of files present in the submitted directory.

Return type:

The list of file names in the submitted directory.

get_terminated() → list[tuple[str, int, str]]

Get job ids and process ids of the terminated jobs from the corresponding directory on the host.

Returns:

The list of job ids, job indexes and batch process uuids in the host terminated directory.

Return type:

list

submit_job(job_id: str, index: int) → None

Submit a Job by uploading the corresponding file.

Parameters:

• job_id – Uuid of the Job.

• index – Index of the Job.