Plugins

This page documents the plugins that ship with the PyBuilder distribution.

QA plugins

Running Python Unittests

PyBuilder ships with a plugin to execute unittests written using Python’s unittest module during the build. Use the python.unittest plugin to enable unittest support.

The plugin executes all test cases found in modules ending with _tests.py in the directory src/unittest/python.

Python unittest properties

Name Type Default Value Description
teamcity_output bool False Output teamcity service messages with test names and errors
dir_source_unittest_python string src/unittest/python Directory where unittest modules are located.
unittest_module_glob string *_tests Pattern used to filter Python modules that should be imported by the unittest runner.
unittest_test_method_prefix string None This allows you to define how the module unittest identifies tests. The plugin sets defaultTestLoader.testMethodPrefix to the given value. When set to None the plugin will not set the value at all.
unittest_file_suffix string None Suffix used to filter files that should be executed as tests.
Deprecated: This exists for backwards compatibility with existing build.py files. If it is set to a non-None value, it will override unittest_module_glob with an appropriate glob pattern by prefixing the value with an asterisk (*) and removing the ".py" extension.

Measuring unittest coverage

PyBuilder adds support to measure the coverage of your unittest using the coverage module. Use the python.coverage module to activate coverage.

Coverage properties

Name Type Default Value Description
coverage_threshold_warn integer 70 Warn if the overall coverage drops below this threshold.
coverage_break_build bool True Break the build (i.e. fail it) if the coverage is below the given threshold.
coverage_reload_modules bool True Reload modules during coverage to also count definition lines.
coverage_fork bool False Run coverage in a different process.
coverage_exceptions list of strings empty List of package names to exclude from coverage analyzation.

SonarQube integration

Using the plugin python.sonarqube will add the task run_sonar_analysis to your project. This task has a run-time dependency to the sonar-runner program and will submit your project to sonarqube for analysis.

Sonarqube configuration

Name Type Default Value Description
sonarqube_project_key string `project.name` The SonarQube project key to use for analysis.
sonarqube_project_name string `project.name` The SonarQube project name to use for analysis.

Linting python sources

Frosted plugin

Frosted is a fork of pyflakes (originally created by Phil Frost) that aims at more open contribution from the outside public, a smaller more maintainable code base, and a better Python checker for all.

Frosted configuration
Name Type Default Value Description
frosted_break_build boolean False Break the build if warnings are found
frosted_include_test_sources boolean False Also run frosted on integrationtest and unittest sources
frosted_include_scripts boolean False Also run the frosted linter on all files in $dir_source_main_scripts
frosted_ignore List of strings [ ] List of warnings to exclude
Example: ["E205", "W101"].
See here, section Frosted error codes, for reference (or use frosted --verbose to check).

Flake8 plugin

PyBuilder can lint your files and verify (or enforce) compliance with PEP8 and PyFlakes through the flake8 module. Use the python.flake8 module to activate linting.

Flake8 configuration
Name Type Default Value Description
flake8_break_build boolean False Break the build if warnings are found
flake8_ignore string None Comma separated list of error messages to ignore
Example: "F403,W404,W801"
flake8_include_scripts boolean False Also run the flake8 linter on all files in $dir_source_main_scripts
flake8_include_test_sources boolean False Also run flake8 on integrationtest and unittest sources
flake8_max_line_length integer 120 Maximum allowed line length
flake8_exclude_patterns string None, but flake8 comes with the default:
.svn,CVS,.bzr,.hg,.git,__pycache__
Comma separated list of file patterns to exclude
Example: ".git,__pycache__"
flake8_verbose_output boolean False Display flake8 warnings and errors in command line output.

Pychecker plugin

Using the plugin python.pychecker will let pychecker run on python source modules.

Pychecker configuration
Name Type Default Value Description
pychecker_break_build boolean True Break the build if warnings are found.
pychecker_break_build_threshold integer 0 Threshold to break the build. A threshold of `n` means: "break the build if more than `x` warnings are found".

Pylint plugin

Using the plugin python.pylint will let pylint run on python source modules.

Pylint configuration
Name Type Default Value Description
pylint_options list of strings `["--max-line-length=100", "--no-docstring-rgx=.*"]` Options to be passed to pylint

Pep8 plugin

Using the plugin python.pep8 will let pep8 run on python sources. There is no configuration possible currently, consider using the flake8 plugin instead.

Pymetrics plugin

Using the plugin python.pymetrics will let pymetrics run on all python source files. There is no configuration possible currently, consider using the flake8 plugin instead.

Running Python Integration Tests

PyBuilder ships with a plugin to run integration tests written in Python. The plugin is named python.integrationtest. It executes all python modules named *_tests.py in the src/integrationtest/python directory.

However, the default run_integration_tests job is just a stub per default , which won’t run your integration tests. In order to use integration tests, you will have to enable the module by specifying use_plugin('python.integrationtest'). If you don’t specify this, your integration tests will not be run.

Every module is executed as a Python module. The Python path contains the integration test directory as well as the production source directory.

Integration test properties

Name Type Default Value Description
teamcity_output bool False Output teamcity service messages with test names and errors
dir_source_integrationtest_python string src/integrationtest/python Directory where integration test modules are located
integrationtest_file_glob string *_tests.py Pattern used to filter files that should be executed as tests.
integrationtest_additional_environment map empty Map containing additional environment variables used when executing the integration tests.
integrationtest_additional_commandline string Not set Additional command line arguments/commands to pass to the integration test call
integrationtest_inherit_environment boolean False Inherit the current environment to integration tests.
integrationtest_parallel boolean False Run integration tests in parallel.
integrationtest_cpu_scaling_factor integer 4 The amount of workers to use per cpu for integration test parallelization
integrationtest_file_suffix string None Suffix used to filter files that should be executed as tests.
Deprecated: This exists for backwards compatibility with existing build.py files. If it is set to a non-None value, it will override integrationtest_file_glob with an appropriate glob pattern by prefixing the value with an asterisk (*).

Travis CI warning

If you use PyBuilder with Travis CI and run integration tests in parallel, you will need a workaround due to a travis issue with POSIX semaphores. Please refer to the related travis issue.

Python 3.2 warning

We have experienced strange problems (EOFErrors) on python 3.2. This seems to be due to a bug with pickling randomly raising EOFError. We recomment updating to python 3.3 or 3.4, where it works fine.

Running Cram tests

Cram is a functional testing framework for command line applications based on Mercurial’s unified test format.

The plugin python.cram can be used to run tests written for cram. Cram tests run during run_integration_tests but there is also a separate task available (run_cram_tests).

Python 3.2.3 warning

Cram crashes on python 3.2.3 : see the relevant bitbucket issue. An easy workaround is to not use the plugin on python 3.2 :

if not sys.version_info[0:2] == (3,2):
    use_plugin("python.cram")

Cram properties

Name Type Default Value Description
dir_source_cmdlinetest string src/cmdlinetest Directory where cram test files are located.
cram_test_file_glob string *.t Glob matching the cram test files
cram_fail_if_no_tests boolean True Fail if there are no cram tests
cram_run_test_from_target boolean True Run tests from the target or src directory.

Python deployment

Python core

The python core plugin deals with sources and distributions. It copies the sources into the distribution directory.

Python core properties

Name Type Default Value Description
dir_source_main_python string src/main/python Directory where python modules are located.
dir_source_main_scripts string src/main/scripts Directory where runnable scripts are located.
dir_dist string $dir_target/dist/$name-$version Directory where distributions are built.
dir_dist_scripts string scripts (starting with 0.10.40, before that scripts are simply copied to the dir_dist by default)</em> Directory where scripts are copied to (relative to distribution directory).

Installing dependencies

PyBuilder manages build and runtime dependencies for you. Use the python.install_dependencies module to activate dependency management. This will make the following tasks available:

Task Effect
install_build_dependencies Installs build dependencies only
install_runtime_dependencies Installs runtime dependencies only
install_dependencies Installs all dependencies (build and runtime)

Declaring dependencies

The project object exposes the following methods for declaring dependencies

Requirements file :
project.depends_on_requirements("requirements.txt")
project.build_depends_on_requirements("requirements-dev.txt")
Runtime dependencies :
project.depends_on("awesome-library")
Build dependencies :
project.build_depends_on("awesome-library")

Note that the *_depends_on methods accept the following arguments :

Argument Effect
name Required positional argument. It's the name of the dependency on PyPI.
version Optional keyword argument (None default). Set it to a version string (example "==1.5.4" or ">=0.10" if necessary.
url Optional keyword argument (None default). Set it to an editable URL where the dependency should be downloaded from.

The logic of version goes as follows:

  • If you don’t specify any version, you don’t care about the version and any version will do, so PyB will get whatever version if one doesn’t exist and then don’t bother ever updating.
  • If you specify an exact version once it is obtained there is no point in updating it.
  • If you specify a fuzzy version, you want the latest and greatest that satisfies the expression specified. This aids rapid incompatibility detection, as PyBuilder tracks the latest version that always satisfies the version specifier.

Install dependencies configuration

Name Type Default Value Description
dir_install_logs string $dir_logs/install_dependencies Where installation logs should be saved.
install_dependencies_index_url string None Index URL to use with pip (None means use the pip default).
install_dependencies_extra_index_url string None Extra index url to use with pip.
install_dependencies_use_mirrors boolean True Enables the use of PyPI mirros for pip operations.
install_dependencies_upgrade boolean False If dependencies are already available, try to upgrade them instead.
install_dependencies_insecure_installation List of strings [ ] Tell newer versions of pip that it's OK to install those dependencies insecurely (externally hosted, potentially unverified)

Creating a source distribution

PyBuilder can build a source distribution of your code with the plugin source_distribution. Activating this plugin will expose a task, build_source_distribution.

Source distribution configuration

Name Type Default Value Description
dir_source_dist string $dir_target/dist/$(project.name)-$(project.version)-src Where the source distribution should be created.
source_dist_ignore_patterns list of strings ["*.pyc", ".hg*", ".svn", ".CVS"] Patterns to ignore when copying sources for the source distribution.

Building a Python package

PyBuilder ships a plugin that generates and executes setup.py files using with distutils or setuptools (distribute will be supported in future releases). In order to generate a setup.py file use the python.distutils plugin.

The distutils plugin will consider many of the project’s attributes, i.e. * name * version * summary * description * author * license * url

Moving scripts to a nested directory

By default, executable scripts are located at the egg’s root. The setup.py contains the bare script names. Sometimes though you may want to move the scripts inside a directory to avoid naming collisions (i.e. with packages).

The python.core and python.distutils plugins support the property dir_dist_scripts that can name a directory to contain scripts inside the egg. This property is empty by default. Set it to the local name of the directory you want the scripts to be moved to:

@init
def initialize (project):
    project.set_property("dir_dist_scripts", 'scripts')

...

Distutils properties

Name Type Default Value Description
distutils_commands list of strings sdist, bdist_dump Commands to execute using the generated setup.py script during publish
distutils_classifiers list of strings Development Status :: 3 - Alpha
Programming Language :: Python
Classifiers describing the project; see http://pypi.python.org/pypi?%3Aaction=list_classifiers
distutils_upload_repository string None The PyPI repository to upload the package to
distutils_upload_sign boolean False Sign each uploaded file using GPG (GNU Privacy Guard)
distutils_upload_sign_identity string None The identity to sign each file with, when distutils_upload_sign is True
distutils_use_setuptools boolean True Use setuptools instead of distutils
distutils_issue8876_workaround_enabled boolean False Deny distutils the use of hardlinks for creating binary distributions

VirtualBox pitfall with binary dists

If you build your project on a shared filesystem (e.G. with vagrant/Virtualbox virtualization) then distutils is unable to build a binary distribution due to a python stdlib bug. Setting the property distutils_issue8876_workaround_enabled to True (off by default) will work around this for you.

Copying resources into a distribution

In some cases you want to include non-source files in a distribution. An example is adding a setup.cfg to your project. It is not a source file, but it needs to be in the distribution to work (beneath the setup.py file). The copy_resources plugin will do this for you at build-time.

Copy resources configuration

Name Type Default Value Description
copy_resources_target string $dir_target Where resources should be copied to.
copy_resources_glob list of strings [] A list of globs matching files that should be copied to $copy_resources_target.

Filtering files

With the filter_resources plugin, it is possible to replace placeholder values of type ${version} with actual values at build-time. The actual values are attributes of the project object, so ${version} will be replaced with the value of project.version. In order to make resource filtering explicit, all files that go through filtering need to be specified with a glob.

Filter resources configuration

Name Type Default Value Description
filter_resources_target string $dir_target Toplevel directory containing the resources to be filtered.
filter_resources_glob list of strings [] A list of globs matching files that should be filtered.

Filter resources example

A common use case is keeping a __version__ attribute in your main module. But you don’t want to specify the version both in build.py and in __init__.py because build.py should be the single point of truth.

So just use __version__ = '${version}' in __init__.py.

We can tell the plugin to filter this file by adding ['**/YOUR-PACKAGE-NAME/__init__.py'] to the filter_resources_glob. The correct version will be patched in at run-time since the version is defined in project.version. For filtering other attributes like ${myattribute}, just add a project.myattribute = "foobar" in build.py’s initializer.

Creating documentation with sphinx

The plugin python.sphinx will expose the task sphinx_generate_documentation which will generate the project documentation.

Sphinx plugin configuration

Name Type Default Value Description
sphinx_builder string html The sphinx builder to use
sphinx_config_path string "" The path to the sphinx configuration
sphinx_source_dir string "" The path to the sphinx sources
sphinx_output_dir string "_build/" The path to the sphinx output directory where the documentation will be generated

Generic build plugins

Executing shell commands

PyBuilder ships with a plugin that allows you to incorporate arbitrary shell commands in the build process.

This plugin can be activated using use_plugin("exec") and is configured through the project properties.

Adding a shell command to the build process

Adding commands is done by setting properties and is discrete towards the project lifecycles, thus you may have only one shell command for the analyze lifecycle for example.

Name Type Description
run_unit_tests_command string Run a shell command during run_unit_tests
run_integration_tests_command string Run a shell command during run_integration_tests
analyze_command string Run a shell command during analyze
package_command string Run a shell command during package
publish_command string Run a shell command during publish

The output of the plugin can also be customized by using the properties $PHASE_propagate_stdout and $PHASE_propagate_stderr.

See the properties for the phase run_unit_tests for example:

Name Type Default Description
run_unit_tests_propagate_stdout string False Propagate the stdout of the command to the PyBuilder output
run_unit_tests_propagate_stderr string False Propagate the stdout of the command to the PyBuilder output

IDE integration

Project files for Eclipse PyDev

If the plugin python.pydev is used, PyBuilder provides the task pydev_generate. This task can be used to generate PyDev project files in the project root directory. The project can then be imported easily.

Project files for Jetbrains PyCharm

If the plugin python.pycharm is used, PyBuilder provides the task pycharm_generate. This task can be used to generate a PyCharm project files in the project root directory. The project can then be imported easily through the Open... command.

Developer tools

Visual feedback for tests

The python.pytddmon plugin provides the task pytddmon which will start a graphical application that gives real-time feedback about passing or failing unit tests. Please see the pytddmon page for more information about pytddmon.

Additional project structure

Splitting up your build.py with bldsup

Maybe you want to have project specific plugins and support code while keeping your build.py file small, while avoiding to clutter your project’s root directory. This can be done by activating bldsup support :

from pybuilder.core import use_bldsup
use_bldsup(build_support_dir="bldsup")

If it’s not called in a project’s build.py, then no “bldsup” directory is searched. If it is called, then it confirms that the build_support_dir directory exists and that includes a __init__.py file so imports will work. It just needs to be called before you try to import/use_plugin() any modules from the build_support_dir.