Integration with TeamCity

The main design goal of Universum is to integrate with various CI systems. E.g. continuous integration/automation, version control, static analysis, testing, etc.

One of popular continuous integration systems is TeamCity, and this particular tutorial will explain how to integrate it with Universum.

The proposed scenario includes the following steps:

0. Install and configure both TeamCity and Universum systems, prepare TeamCity build servers for project building/running/testing/etc.

1. Create a project on TeamCity. Configure common parameters for the project:

   1. project parameters, if any

   2. parameters for Universum

   TeamCity configurations automatically inherit all project settings, so configuring them once on project-level allows to avoid multiple reconfiguring of same parameters in each configuration, and all changes applied to project settings are automatically applied to all inherited settings

2. Create a common TeamCity meta-runner. TeamCity offers two ways of using meta-runners:

   1. creating a new configuration out of existing meta-runner

   2. using an existing meta-runner as a build step

   Both of this scenarios have their own benefits: when creating a configuration, all parameters are inherited too; but when using a build step, any change in meta-runner automatically affects all configurations using it

3. Create all needed configurations, such as:

   • precommit check

   • postcommit check

   • testing

   • etc.

This scenario pays a lot of attention to reusing settings instead of just copying them. It is most important in cases when some of these settings have to be changed: if generalized on project level, there’s only one place to fix. And if all similar settings are duplicated, tracking changes becomes more difficult the more times they are copied.

Install Universum on build agents

1. Install and configure TeamCity server

2. Install build agents on build server(s), add them to the project pool

3. Install Universum on corresponding build server(s) by running sudo pip3.8 install universum

Please refer to TeamCity official manuals for details.

Create a top-level project

0. Create new project for all Universum configurations or add a sub-project to an existing one

1. Go to Parameters section in Project Settings and add the following parameters:

   env.BUILD_ID:

   %teamcity.build.id%

   env.CONFIGURATION_ID:

   %system.teamcity.buildType.id%

   env.TEAMCITY_SERVER:

   server URL to refer to in reports

   env.CONFIGURATION_PARAMETERS:

   will be used in meta-runner; leave empty so far

Making all projects using Universum sub-projects to this one will automatically add all these parameters to their settings.

Create a common meta-runner

0. Create an .xml file with the following content:

❯     Click here to show/hide

<?xml version="1.0" encoding="UTF-8"?>
<meta-runner name="Run build using CI system">
  <description>Basic project configuration</description>
  <settings>
    <build-runners>
      <runner name="Download and run" type="simpleRunner">
        <parameters>
          <param name="script.content"><![CDATA[
#!/bin/bash

EXITCODE=0

HOST=`hostame | sed -e "s/_/-/"`
USER=`whoami | sed -e "s/_/-/"`
P4CLIENT="Disposable_workspace_"$HOST"-"$USER

cmd="python3.8 -u -m universum --p4-client ${P4CLIENT} --p4-force-clean %env.CONFIGURATION_PARAMETERS%"
echo "==> Run: ${cmd}"
${cmd} || EXITCODE=1

echo "##teamcity[setParameter name='STOPPED_BY_USER' value='false']"

exit $EXITCODE]]></param>
          <param name="teamcity.step.mode" value="default" />
          <param name="use.custom.script" value="true" />
        </parameters>
      </runner>
      <runner name="Clean" type="simpleRunner">
        <parameters>
          <param name="script.content"><![CDATA[
#!/bin/bash

if [ %STOPPED_BY_USER% == true ]
then
echo "==> User interrupted, force cleaning"

EXITCODE=0

HOST=`hostame | sed -e "s/_/-/"`
USER=`whoami | sed -e "s/_/-/"`
P4CLIENT="Disposable_workspace_"$HOST"-"$USER

cmd="python3.8 -u -m universum --p4-client ${P4CLIENT} --p4-force-clean %env.CONFIGURATION_PARAMETERS% --finalize-only --artifact-dir finalization_artifacts"
echo "==> Run: ${cmd}"
${cmd}

else
echo "==> Additional cleaning not needed, skipping"
fi
          ]]></param>
          <param name="teamcity.step.mode" value="execute_always" />
          <param name="use.custom.script" value="true" />
        </parameters>
      </runner>
    </build-runners>
  </settings>
</meta-runner>

Note

Universum default VCS type is Perforce, so this meta-runner is oriented to be used with P4. But the same meta-runner can be used for configurations using any other VCS type. Unused P4 parameters will be just ignored.

1. In Project Settings find Meta-Runners page and press Upload Meta-Runner

2. Select your newly created .xml file as a Meta-Runner file

Configure project using Perforce

0. Create a sub-project to a created earlier top-level project

1. Go to Parameters in Project Settings

2. If not using the default (.universum.py), add env.CONFIG_PATH: a relative path to project configuration file, starting from project root

3. Also add all required project-wide Perforce parameters:

   env.P4USER:

   Perforce user ID

   env.P4PASSWD:

   user <env.P4USER> password

   env.P4PORT:

   Perforce server URL (including port if needed)

   env.P4_MAPPINGS:

   Perforce mappings in special format. Also can be replaced with legacy env.P4_PATH (but not both at a time)

Note

If using Perforce, install P4 CLI on build agent(s), and use pip3.8 install -U universum[p4] to make sure all VCS-dependant modules are installed.

Create basic postcommit configuration

0. After creating new build configuration, go to Build Configuration Settings

1. To get artifacts from default artifact directory, go to General Settings, find Artifact paths field and add artifacts/* line there

2. To trigger builds via TeamCity but download via Universum, go to Version Control Settings, attach required VCS Root and set VCS checkout mode to Do not checkout files automatically

3. Go to Triggers and add VCS Trigger with required settings

4. Go to Build steps, press Add build step, in Runner type scroll down to your project runners and select a meta-runner created earlier

After setting up all the environment variables right, you must get the fully working configuration.

Create configuration for custom builds

0. As in postcommit, specify artifacts/* in Artifact paths and add your meta-runner as a Build step

1. Attaching VCS root is not necessary because custom build configurations usually do not use VCS Trigger; instead of this, add the following parameters to configuration:

   env.SYNC_CHANGELIST:

   can be a CL number or a list of sync CLs for several different P4_MAPPINGS, see ‘–p4-sync’ option description

   env.SHELVE_CHANGELIST:

   one or several coma-separated CLs to unshelve for the build

Integrate with Swarm

0. Go to Build Configuration Settings (or to Project Settings, if you plan on having more than one Swarm-related configuration)

1. Create env.REVIEW, env.PASS and env.FAIL parameters and leave them empty

2. In Build Configuration Settings –> Parameters and add --report-to-review option in env.CONFIGURATION_PARAMETERS

3. If needed, add other Swarm options, such as --report-build-start and --report-build-success

4. Go to Swarm project settings, check in Automated tests check-box and follow this instruction

The resulting URL you should insert in text field. The URL should look like:

http://<user>:<password>@<TeamCity URL>/httpAuth/action.html?add2Queue=<configuration> &name=env.SWARM_CHANGELIST&value={change}&name=env.PASS&value={pass}&name=env.FAIL&value={fail} &name=env.REVIEW&value={review}

where

user:

is a name of a TeamCity user triggering Swarm builds (preferably some bot)

password:

is that user’s password

TeamCity URL:

is actual server URL, including port if needed

configuration:

is an ID of your Swarm configuration (see Build configuration ID in settings)

or, if your TeamCity supports anonymous build triggering, user & password can be omitted along with httpAuth/ parameter.

1. Probably, in the POST Body field you should additionally insert below line:

   name=STATUS&value={status}

or, any other parameter. This is a workaround for TeamCity requirement for using POST method to trigger builds.

Configure project and configurations using Git

0. Create a sub-project to a top-level project for Universum configurations

1. In Parameters set env.CONFIG_PATH relative to project root

2. Add oject-wide Git parameters:

   env.GIT_REPO:

   a parameter to pass to git clone, e.g. ssh://user@server/project-name/

   env.GIT_REFSPEC:

   if some non-default git refspec is needed for project, specify it here

3. Create post-commit configurations as described above

4. When creating custom build configurations, use the following parameters instead of P4-specific:

   env.GIT_CHECKOUT_ID:

   parameter to be passed to git checkout; can be commit hash, branch name, tag, etc. (see official manual for details)

   env.GIT_CHERRYPICK_ID:

   one or several coma-separated commit IDs to cherry-pick (see official manual for details)

Note

If using Git, install Git on build agent(s), and use pip3.8 install -U universum[git] to make sure all VCS-dependant modules are installed.