Core Concepts#

Understanding the core components of qtoolkit is essential for effectively managing computational jobs across different environments.

The Package Architecture#

qtoolkit is organized into three main layers:

  1. Core Data Objects: Standardized representations of jobs, resources, and states.

  2. IO Layer (SchedulerIO): Generators and parsers for scheduler-specific commands.

  3. Manager Layer (QueueManager): The high-level API that orchestrates command execution and data parsing.

Key Components#

SchedulerIO#

Each supported queue manager has a corresponding SchedulerIO implementation (e.g., SlurmIO, PBSIO, SGEIO). These classes are responsible for:

  • Creating the shell script for job submission.

  • Generating the CLI commands.

  • Converting CLI output into SubmissionResult, CancelResult, or QJob objects.

Significantly, SchedulerIO classes do not execute any commands themselves. They only handle the conversion between Python objects and shell-consumable strings.

QJob and QState#

A QJob object represents a job’s status and metadata as retrieved from the scheduler. It includes:

  • job_id: The unique identifier assigned by the scheduler.

  • state: A standardized QState enum (e.g., QUEUED, RUNNING, DONE, FAILED).

  • sub_state: The raw, scheduler-specific state defined as a sub-class of QSubState.

  • info: A QJobInfo object with resource utilization and limits.

By using the standardized QState, users can write scheduler-independent logic for job monitoring.

QResources#

The QResources class can be used to specify the computational requirements for a job. It includes attributes such as:

  • job_name: Identifying the job.

  • nodes, processes, processes_per_node: Defining the parallelism.

  • time_limit: Maximum execution time.

  • memory_per_thread: Memory requirements.

  • queue_name: The target partition or queue.

QResources also supports different process placement strategies (e.g., scattered, same_node, evenly_distributed) through factory methods.

Note that while QResources provides a standardized interface, most methods in qtoolkit also accept a simple dictionary of scheduler-specific keywords as an alternative.

Interface#

qtoolkit also provides a high-level interface for submitting and managing jobs through the QueueManager class. It combines a SchedulerIO instance with a Host instance to provide a complete workflow for job management.

QueueManager#

The QueueManager is the primary interface for users that do not want to deal with the low-level details of job submission and management. It combines a SchedulerIO instance with a Host instance to provide a complete workflow for job management.

Methods like submit, cancel, and get_job handle the process of generating commands, executing them on the specified host, and parsing the results.

Hosts#

The Host abstraction represents where the commands generated by the SchedulerIO are executed.

  • LocalHost: Executes commands on the machine where the Python script is running.

  • Other host implementations (like SSH hosts) can be used to execute commands on remote clusters.

JSON Serialization#

If needed, all the main objects representing the data can be serialized to JSON using the methods exposed by the MSONable interface from monty. The MSONable interface is optional and available only if the monty package is installed. See Installation for details.