system – Build System Generation

system is an interface for specifying and executing a build process with concurrent semantics. It provides abstractions for running build processes that are BuildSequence() and BuildStage() objects.

BuildSystemGenerator() uses the tools in stages to produce a build system and provide the foundation of the command-line build system tool buildc – Command Line Build System Interface.

class system.BuildSystem(initial_system=None)

A representation of a multi-stage build system that combines a number build stages to be constructed in any order, and executed sequentially.

_error_or_return(msg, strict=None, exception=None)

Parameters:

• msg (str) – A description of the exception.
• strict (bool) – Optional. Defaults to strict
• exc (exception) – Optional. The exception to raise. Defaults to InvalidSystem

Raises :

InvalidStage if strict is True, otherwise, returns False.

_validate_stage(name)

Parameters:name (string) – The name of a stage. Returns:False when name already exists in the build system, and True otherwise.

add_stage(name, stage=None, stage_type='stage', strict=None)

Parameters:

• name (string) – The name of a stage.
• stage (BuildStage) – A :class:~stages.BuildSteps` object. Default’s to None, which has the same effect as :meth:~system.BuildSystem.new_stage()`.
• stage_type (string) – Defaults to stage. Either sequence (or seq) or stage to determine what kind of :class`~stages.BuildSteps()` object to instantiate if stage is None
• strict (bool) – Overrides the default strict attribute. When True, prevents callers from adding duplicate stages to a :class:~system.BuildSystem` object.

Raises :

InvalidStage() in strict mode if name or stage is malformed.

Creates a new stage and optionally populates the stage with tasks. Note the following behaviors:

• If name exists in the current :class:~system.BuildSystem`, strict is False, and stage is None; then :meth:~system.BuildSystem.add_stage()` adds the stage with name to the current :class:~system.BuildSystem` object. This allows you to perform the same stage multiple times in one build process.

• If `name does not exist in the current :class:~system.BuildSystem`, and stage is None, :meth:~system.BuildSystem.add_stage()` adds the stage and instantiates a new :class`~stages.BuildSteps()` (either :class`~stages.BuildSequence()` or :class`~stages.BuildStage()` depending on the value of stage_type) object with name to the current :class:~system.BuildSystem` object.

  :meth:~system.BuildSystem.add_stage()` must specify a stage_type if stage is None. When strict is True not doing so raises an exception.

• Adds the stage with the specified :class`~stages.BuildSteps()` object to the :class:~system.BuildSystem` object. Raises an exception if stage is not a :class`~stages.BuildSteps()` object.

close()

Sets the open value to False.

closed

Returns:True when after finalizing a :class:~system.BuildSystem` object.

count()

Returns:The number of stages in the :class:~system.BuildSystem` object.

extend(system)

Parameters:system (BuildSystem) – A BuildSystem() object.

Takes system and grows the current BuildSystem() object to have the objects stages. If there are existing stages and job definitions that collide with stages and jobs in system, the data in system wins.

get_order()

Returns:List of names of build stages in the order they’d run.

get_stage_index(stage)

Parameters:stage (string) – The name of a stage in the BuildSystem. Returns:The position of the stage in the BuildSystem, or None if it doesn’t exist in the BuildSystem object.

new_stage(name)

Parameters:name (string) – The name of a stage.

Special case and wrapper of :meth:~system.BuildSystem.add_stage()` that adds a new stage without adding any jobs to that stage.

open = None

Boolean. If True, (and strict is True,)then it’s safe to add additional stages and execute the build

run(strict=None)

Parameters:strict (bool) – Defaults to strict. Raises :StageRunError if strict is True and open is True.

Calls the run() method of each stage object.

run_part(stop=0, start=0, run_all=False, strict=None)

Parameters:

• stop (int) – The last job in the zero-indexed list of the tasks in the stages. Defaults to zero, which
• start (int) – The first job in the zero-indexed list of the tasks in the stages. Defaults to 0, or the first step.
• run_all (bool) – Defaults to Run
• strict (bool) – Defaults to strict.

Raises :

StageRunError if strict is True and open is True.

Returns:

The return value of the last run() method called.

Calls the run() method of each stage object until the idx^th item of the _stages array.

run_stage(name, strict=None)

Parameters:

• name (string) – The potential name of a stage in the :class:~system.BuildSystem` object.
• strict (bool) – Defaults to strict.

Returns:

False if name is not in the :class:~system.BuildSystem` object, and strict is False. In all other cases run_stages() returns True after the stage runs.

Raises :

StageRunError if strict is True and name does not exist.

Runs a single stage from the :class:~system.BuildSystem` object.

stage_exists(name)

Parameters:name (string) – The name of a possible stage in the :class:~system.BuildSystem` object. Returns:True if the stage exists in :class:~system.BuildSystem` and False otherwise.

stages = None

A mapping of the names of stages to their stage objects.

strict

A Getter and setter for the :attr:~system.BuildSystem._strict` setting.

class system.BuildSystemGenerator(funcs=None)

BuildSystemGenerator objects provide a unifying interface for generating and running BuildSystem objects from easy-to-edit sources. BuildSystemGenerator is the fundamental basis of the buildc – Command Line Build System Interface tool.

Parameters:funcs (dict) – Optional. A dictionary that maps callable objects (values), to identifiers used by build system definitions. If you do not specify.

BuildSystemGenerator is designed to parse streams of YAML documents that resemble the following:

job: <func>
args: <<arg>,<list>>
stage: <str>
---
job: <func>
args: <<arg>,<list>>
target: <str>
dependency: <<str>,<list>>
---
dir: <<path>,<list>>
cmd: <str>
args: <<arg>,<list>>
stage: <str>
---
dir: <<path>,<list>>
cmd: <str>
args: <<arg>,<list>>
dependency: <<str>,<list>>
---
tasks:
  - job: <func>
    args: <<arg>,<list>>
  - dir: <<path>,<list>>
    cmd: <str>
    args <<args>,<list>>
stage: <str>
...

See Define and Run Builds With buildc for documentation of the input data format and its use.

_add_tasks_to_stage(rebuilds_needed, idx, total, task, stack)

Parameters:

• rebuild_needed (bool) – True, when the dependencies require a rebuild of this target.
• idx (int) – The index of the current task in the build dependency chain.
• total (int) – The total number of tasks in the dependency chain.
• task (string) – The name of a task in the build process.
• stack (list) – The list of non-dependent tasks that must be rebuilt, not including the current task.

Raises :

InvalidSystem if its impossible to add a task in the BuildSystemGenerator object to the finalized build system object.

Called by _finalize_process_tree() to add and process tasks.

_finalize_process_tree()

Loops over the _process list tree created in the main finalize() method, and adds tasks to system object, which is itself a BuildSystem object.

While constructing the system object, _finalize_process_tree() combines adjacent tasks that do not depend upon each other to increase the potential for parallel execution.

_finalize_process(), calls _add_tasks_to_stage() which may raise InvalidSystem in the case of malformed tasks.

_process_dependency(spec)

Parameters:spec (dict) – The task specification imported from user input.

Wraps _process_stage() and modifies the internal strucutres associated with the dependency tree.

_process_job(spec, strings=None)

Parameters:spec (dict) – A dictionary of strings that describe a build job.

Processes the spec and attempts to create and add the resulting task to the build system.

_process_stage(spec, spec_keys=None, strings=None)

Parameters:

• spec (dict) – The task specification imported from user input.
• spec_keys (set) – A set containing the top level keys in the set.
• strings (dict) – Optional. A dictionary of strings mapping to strings to use as replacement keys for spec content.

Returns:

A two item tuple, containing a function and a list of arguments.

Raises :

InvalidJob if the schema of spec is not recognized.

Based on the schema of the spec, creates tasks and sequences to add to a BuildSystem using, as appropriate one of the following methods:

• generate_job(),
• generate_shell_job(), or
• generate_sequence().

add_task(name, func)

Parameters:

• name (string) – The identifier of a callable.
• func (callable) – A callable object.

Raises :

InvaidJob if func is not callable.

Adds a callable object to the funcs attribute with the identifier name.

finalize()

Raises :InvalidSystem if you call finalize() more than once on a single BuildSystemGenerator() object.

You must call finalize() before running the build system.

finalize() compiles the build system. If there are no tasks with dependencies, _stages becomes the system object. Otherwise, finalize() orders the tasks with dependencies, inserts them into a BuildSequence object before inserting the _stages tasks.

static generate_job(spec, funcs)

Parameters:

• spec (dict) – A job specification.
• funcs (dict) – A dictionary mapping names (i.e. spec.job to functions.)

Returns:

A tuple where the first item is a callable and the second item is either a dict or a tuple of arguments, depending on the type of the spec.args value.

Raises :

InvalidStage if spec.args is neither a dict, list, or tuple.

static generate_sequence(spec, funcs)

Parameters:

• spec (dict) – A job specification.
• funcs (dict) – A dictionary mapping names to callable objects.

Returns:

A BuildSequence() object.

Raises :

An exception when spec does not have a task element or when task.spec is not a list.

Process a job spec that describes a sequence of tasks and returns a BuildSequence() object for inclusion in a BuildSystem() object.

static generate_shell_job(spec)

Parameters:spec (dict) – A job specification. Returns:A tuple where the first element is subprocess call(), and the second item is a dict args, a dir, and cmd keys.

Takes a spec dict and returns a tuple to define a task.

static get_dependency_list(spec)

Parameters:spec (dict) – A specification of a build task with a target, to process. Returns:A list that specifies all specified dependencies of that task.

ingest(jobs, strings=None)

Parameters:

• jobs (iterable) – An interable object that contains build job specifications.
• strings (dict) – Optional. A dictionary of strings mapping to strings to use as replacement keys for spec content.

Returns:

The number of jobs added to the build system.

For every document

ingest_json(filename, strings=None)

Wraps ingest().

Parameters:

• filename (string) – The fully qualified path name of a JSON file that contains a build system specification.
• strings (dict) – Optional. A dictionary of strings mapping to strings to use as replacement keys for spec content.

For every document in the JSON file specified by filename, calls _process_job() to add that job to the system object.

ingest_yaml(filename, strings=None)

Wraps ingest().

Parameters:

• filename (string) – The fully qualified path name of a YAML file that contains a build system specification.
• strings (dict) – Optional. A dictionary of strings mapping to strings to use as replacement keys for spec content.

For every document in the YAML file specified by filename, calls _process_job() to add that job to the system object.

static process_strings(spec, strings)

Parameters:

• spec (string) – A string, possibly with format() style tokens in curly braces.
• strings (dict) – A mapping of strings to replacement values.

Raises :

TypeError if strings is not a dict and there may be a replacement string.

specs = None

Mapping of job specs targets to specs.