stages – Build System Task Abstractions

stages is part of a component in a complete a complete system tool that provides abstractions for groups

Tasks (i.e. functions) are the smallest unit of work, and are typically Python functions, although a task may be shell commands in some cases. stages allows you to specify two basic relationships between tasks:

Sequence

A series of tasks which must be executed in a specific order. Provided by BuildSequence().

A group of tasks that can be executed in any order, and potentially concurrently. Provided by BuildStage().

Stage

A series of tasks that may execute in parallel.

BuildSequence() and BuildStage() share the same base class (BuildSteps()) and have a common interface despite different behaviors.

class stages.BuildSequence(initial_stage=None)

A subclass of :class:~stages.BuildSteps` that executes jobs in the order they were added to the :class:~stages.BuildSteps` object.

Returns:True upon completion.

run(*args, **kwargs)

Runs all jobs in :class:~stages.BuildSteps` in the order they were added to the object. Ignores all arguments.

class stages.BuildStage(initial_stage=None)

A subclass of :class:~stages.BuildSteps` that executes jobs using a multiprocessing worker pool.

run(workers=None)

Parameters:workers (int) – Overrides the :meth:~stages.BuildSteps.workers` value, which is typically the number of CPU cores your system has.

Runs all jobs in :attr:~stages.BuildSteps.stages` using a worker pool.

Returns:True upon completion.

class stages.BuildSteps(initial_stage=None)

Parameters:initial_stage (BuildStage) – Optional. You may optionally pass a BuildSequence() or BuildStage() object to BuildSteps(). Otherwise, the new object will initialize an empty task list.

BuildSteps() is an object that stores a representation of a group of tasks. In practice, BuildSteps() is the base class for specific implementations for groups of build tasks. In practice sub-classes simply implement the run() method which raises a NotImplementedError.

add(func, args, strict=True)

Parameters:

• func (callable) – A callable object (e.g. function or method) to run as a job.
• args (tuple) – A tuple or dict of arguments to pass to the callable.
• strict (bool) – Defaults to True. When True, raises exceptions when attempting to perform inadmissible actions. If strict is False, will continue to operate silently.

Raises :

err.StageClosed in strict mode, if attempting to add to an already closed stage.

Adds a job to a :class:~stages.BuildSteps` object. If strict is True, :meth:~stages.BuildSteps.add()` raises exceptions if you attempt to add a task to a non-open :class:~stages.BuildSteps` object or add a malformed job to the :class:~stages.BuildSteps` object.

close()

If BuildSteps() is mutable, finalizes the object to prevent further editing. Returns False to reflect the current mutability state of the object.

closed

Is True if the BuildSteps() is mutable, and False otherwise.

count()

Returns:The number of items in the :class:~stages.BuildSteps` object.

extend(jobs, strict=True)

Parameters:

• jobs (list) – A list of job specifications, or tuples where the first item is a callable and the second item is a tuple or dict of arguments.
• strict (bool) – Toggles strict mode for :meth:~stages.BuildSteps.add()`.

A wrapper around :meth:~stages.BuildSteps.add()` for adding multiple jobs to a :class:~stages.BuildSteps` object. To add jobs from one :class:~stages.BuildSteps` object to another, use the following operation:

seq = BuildStage()
st = BuildStage()
seq.add(func, args)
st.extend(seq.stage, strict=False)

grow(func, arg_list, strict=True)

Parameters:

• func (callable) – A callable oject (e.g. function or method) to run as a job.
• arg_list (list) – A list of tuples, which are arguments to func for a sequence of jobs.
• strict (bool) – Toggles strict mode for :meth:~stages.BuildSteps.add()`.

Use to add a sequence of jobs to a :class:~stages.BuildSteps` object that use the same func but different arguments.

run()

Raises :NotImplementedError()

stage = None

A list of task objects. See add() for information on the form of a task.

static validate(stage, strict=False)

Parameters:

• stage (iterable) – A stage object to validate.
• strict (bool) – Defaults to False. When True, validate() raises exceptions when it detects an invalid object. Otherwise, validate() returns False for invalid stage objects and True for valid stage objects

Raises :

InvalidStage in strict-mode.

Used internally to ensure that stage objects are well formed before inserting them into the build :class:~stages.BuildSteps` object.

workers

The size of the worker pool that will process the tasks in this group or stage of tasks. Set workers to the number of jobs you want to execute in parallel. By default this is the number of cores/threads available on your system.

The minimum value for workers() is 2, and cannot be set to a value lower than 2.