jobflow_remote.jobs.batch module#

class jobflow_remote.jobs.batch.LocalBatchManager(files_dir: str | Path, process_id: str, multiprocess_lock=None)[source]#

Manager of local files containing information about Jobs to be handled by a batch worker.

Used in the worker to executes the batch Jobs.

Parameters:

files_dir – The full path to directory where the files to handle the jobs to be executed in batch processes are stored.
process_id – The uuid associated to the batch process.
multiprocess_lock – A lock from the multiprocessing module to be used when executing jobs in parallel with other processes of the same worker.

get_job() → str | None[source]#

Select randomly a job from the submitted directory to be executed. Move the file to the running directory.

Locks will prevent the same job from being executed from other processes. If no job can be executed, None is returned.

Returns:: The name of the job that was selected, or None if no job can be executed.
Return type:: str | None

terminate_job(job_id: str, index: int) → None[source]#

Terminate a job by removing the corresponding file from the running directory and adding a new file in the terminated directory.

Parameters:

class jobflow_remote.jobs.batch.RemoteBatchManager(host: BaseHost, files_dir: str | Path)[source]#

Manager of remote files containing information about Jobs to be handled by a batch worker.

Used by the Runner.

Parameters:

cleanup() → bool[source]#

Remove the files directory on the host.

Returns:: True if the directory was successfully deleted or was not existing.
Return type:: bool

delete_running(process_id: str) → None[source]#

Remove job files from the running folder for a specific process uuid.

Should be used only for jobs that failed and left dangling running files.

Parameters:: process_id – The uuid of the process for the running files to be removed.

Get job ids and process ids of the running jobs from the corresponding directory on the host.

Returns:: The list of job ids, job indexes and batch process uuids in the host running directory.
Return type:: list

get_submitted() → list[str][source]#

Get a list of files present in the submitted directory.

get_terminated() → list[tuple[str, int, str]][source]#

Get job ids and process ids of the terminated jobs from the corresponding directory on the host.

Returns:: The list of job ids, job indexes and batch process uuids in the host terminated directory.
Return type:: list

submit_job(job_id: str, index: int) → None[source]#

Submit a Job by uploading the corresponding file.

Parameters: