Jetbrains Teamcity and HP ALM Integration Guide¶
Why you need Bumblebee
In many organizations, TeamCity is used to build software, run unit tests, and run various kinds of testing frameworks. Example: Selenium, JUnit, Pytest, TestNG, Visual Studio Test, etc. These test metrics are very important and should be reflected in HP ALM to show the true health of any project. Bumblebee plugin for TeamCity allows users to integrate any testing framework with HP ALM without making any code changes or writing custom tools. In addition, the plugin allows you to run HP ALM tests directly from TeamCity. Many organization use Bumblebee and TeamCity to achieve CICD.
Integrating TeamCity with HP ALM is super simple by using Bumblebee's plugin for TeamCity. Bumblebee's TeamCity plugin allows you to do the following:
- Export test results to HP ALM and automatically create TestPlan, TestLabs, TestsSets, and TestRuns. This is extremely useful for running tests directly in TeamCity and automatically these test results in HP ALM.
- Schedule Tests in HP ALM from TeamCity and show their status in TeamCity and HP ALM
- Run local UFT tests and report execution results in both TeamCity and HP ALM
- Run HP Performance Center tests from TeamCity
Install Bumblebee Server before configuring the TeamCity plugin
Bumblebee TeamCity plugin communicates with HP ALM via the Bumblebee server. You must install Bumblebee server before configuring the TeamCity plugin. Bumblebee server setup instructions
- Install plugin¶
- Download bumblebee-teamcity-plugin.zip file using the following link: Download
- Open TeamCity web GUI and navigate to Administration -> Plugins List and click on "Upload plugin zip" link
- Select bumblebee-teamcity-plugin.zip and click on "Save" button
- Restart TeamCity server
- Configure plugin¶
Configure the TeamCity plugin settings. Administration → Server Administration → Bumblebee System
|BumbleBee URL||URL for bumblebee server. Must end with /bumblebee. Example: http://some-server:8888/bumblebee|
|HP ALM URL||URL for your HP ALM instance. Must end with /qcbin. Example: http://your-alm-server:8080/qcbin|
|HP PC URL||URL of HP Performance Center. Leave it empty if you will not be using plugin for running tests in HP Performance Center.|
|HP ALM User||Name of the HP ALM user to connect to ALM instance.|
|HP ALM User Password||Password for HP ALM user to login into ALM|
|UFT Batch Runner||The default path to the UFT Batch Runner (UFTBatchRunnerCMD.exe) on all agents. It can be overridden by agent properties. Leave it empty if you will not be using plugin for running local UFT tests.|
|Upload timeout||The number of minutes to wait for the Bumblebee server to process the request. 0 means wait indefinitely. Uploading of test results into HP ALM might take a few minutes depending upon the number of tests that must be processed.|
|PC Timeout||The number of minutes to wait for the Performance Center test to finish. 0 means wait indefinitely.|
Save button to save your changes. Saving may take a few seconds because the Bumblebee plugin will validate the configuration before saving.
- Export Tests Results¶
Bumblebee plugins allows automatic export of TeamCity build's test results to HP ALM. Bumblebee supports a variety of test results formats like JUnit, NUnit, TestNG, Fitnesse, Cucumber, Serenity and JBehave. To enable this feature, configure the TeamCity job and add and configure Bumblebee HP ALM Uploader build step.
|Task Description||Yes||Standard description for Atlassian Bamboo® task|
|Domain in HP ALM||Yes||The name of HP ALM domain where to export test results|
|Project in HP ALM||Yes||The name of HP ALM project|
|TestPlan in HP ALM||Yes||TestPlan path in HP ALM where tests should be saved. Must start with "Subject\"|
|TestLab in HP ALM||Yes||TestLab path in HP ALM where test results should be saved. Must start with "Root\"|
|Test Set||Yes||The name of test set in HP ALM|
|Format||Yes||Format of test results generated by previous tasks in the same job. Available values: junit, nunit, testng, cucumber, serenity, jbehave|
|Results File Pattern||Yes||Provide path to the XML file(s) generated during the build.This is a comma separated list of test result directories.You can also use Ant style patterns such as **/surefire-reports/*.xml|
|HP ALM Mappings||No||If ALM Mappings are configured on the Bumblebee server, their values can be specified in this field.The format is [alm field label 1]=[value 1], [alm field label 2]=[value 2]. You can also use Atlassian Bamboo® environment variables.You can also directly specify the default value on the Bumblebee server.Please refer to Bumblebee server documentation for details on configuring HP ALM mappings.|
|Fail build if upload unsuccessful||N/A||If checked, the build will be marked as failed if for any reason the add-on was not able to upload the results.This could be due to Bumblebee server issues, HP ALM server issues, network issues, etc.|
|Process offline||N/A||If checked, Bumblebee will send test reports to the special asynchronous,processing queue on Bumblebee server and process it afterwards. Use,this when upload of test results takes much time and you don't want the,build to wait until it finishes|
configure ALM mappings (optional)
If your ALM project has custom user fields and you have already configured ALM Mappings, you can easily specify them in the TeamCity build step as name-value pairs. If you don't specify them in the TeamCity step, Bumblebee will use the custom field values from the server (if they exist).
Once the job is configured, simply build the job. The build console will show the Bumblebee TeamCity plugin to Bumblebee REST API activity. This output is quite useful for troubleshooting.
All the results will be processed by Bumblebee server and corresponding TesPlan, TestLab, Testset, and TestRun will be created automatically.
- Offline processing¶
Processing of test results can add a few minutes to your build time. If this is not desirable, Bumblebee provides an option for processing tests results in a background thread on the Bumblebee server. Simply check the Process offline checkbox in the job configuration to enable this feature.
When Process offline is enabled, the console output will be
The console output will contain a link to the processing queue item.
To view all the offline tasks being processing, simply navigate to Offline Processing Queue from the Bumblebee Server main page
- Running ALM Tests from TeamCity¶
Bumblebee TeamCity plugin task allows you to run tests stored in HP ALM (UFT, LeanFT, QTP, etc) directly from TeamCity and view the results in both TeamCity and HP ALM.
- Job must run on TeamCity agent running Windows OS
- TeamCity slave must have launch method: Launch slave agents via Java Web Start
- TeamCity slave must NOT run as windows service
- Appropriate version of HP ALM Connectivity Tool must be installed on TeamCity agent. Tool is available at http://your_alm_server_and_port/qcbin/PlugIns/TDConnectivity/TDConnect.exe
- Appropriate version of HP ALM Client must be installed on TeamCity agent. Available at http://your_alm_server_and_port/qcbin/start_a.jsp?common=true
To run HP Testsets from TeamCity, add Bumblebee HP ALM Test Set Runner build step to TeamCity job configuration
|Domain||The name of HP ALM domain|
|Project||The name of HP ALM project|
|Test Sets||A list of test sets to execute. Each test set path must start with new,line. Patch must start with Root and contains full path to the test set,in HP ALM TestLab. E.g. Root\folder1\testset1, where Root\folder1 is,TestLab folder and folder1 is the name of the test set to execute|
|Run Mode||How to run test sets. Possible values: LOCAL - run all tests on agent's,machine, SCHEDULED - run tests on planned host, REMOTE - run on remote,host|
|Run Host||The name of host on which tests shall be run. May be blank if Run Mode is LOCAL or SCHEDULED|
|JUnit Results Directory||Directory where JUnit-like execution reports will be placed. If it does not exist, Bumblebee will create it|
|Timeout||The number of minutes to wait for test sets execution. 0 means wait indefinitely.|
When TeamCity build is executed, the Bumblebee HP ALM Test Set Runner step connects to HP ALM server and runs the specified ALM Testsets. The build waits till the test runs is completed in HP ALM.
Example: TeamCity build log
Once the build is complete, Bumblebee HP ALM Test Set Runner transforms the HP ALM test results into a JUnit report compliant schema so that it can be published within TeamCity, using the standard JUnit Parser step.
- Adding tests from HP ALM TestPlan to TestSet¶
If a single test or all tests in a TestPlan folder need to be added to HP ALM TestSet, then Bumblebee: Add test to set build step can be used:
The step has the following configuration fields:
|Domain||The name of HP ALM domain|
|Project||The name of HP ALM project|
|Test Plan Path||Path to a test or test folder in HP ALM TestPlan, e.g. Subject\testfolder\test1|
|Test Set Path||Path to a HP ALM TestSet into which test(s) shall be added. If TestSet does not exist, Bumblebee will try to create it on the fly. Example: Root\testfolder\testSet|
Results in HP ALM:
- Running ALM Tests inside TeamCity¶
Bumblebee also allows you to run HP Unified Functional Testing tests directly from TeamCity and reports resutls back to TeamCity.
- TeamCity Agent runs on Windows machine and have appropriate HP Unified Functional Testing version installed. Please see UFT requirements for a particular version of OS and other software.
- TeamCity agent must run as a console application (not as a windows service)
- Set UFT Batch Runner property of Global Configuration or com.agiletestware.bumblebee.uft_runner_path agent property
To override path to UFT Batch Runner, defined in the Global Configuration, you need to set a com.agiletestware.bumblebee.uft_runner_path agent property for a remote agent (please see TeamCity documentation).
To set a property to an agent:
- Shutdown the agent
- Add the following string to the end of file:
example: com.agiletestware.bumblebee.uft_runner_path=c:\Program Files (x86)\HP\Unified Functional Testing\bin\UFTBatchRunnerCMD.exe
- Start agent
To run UFT tests locally inside of TeamCity, add the Bumblebee: Run Local UFT Tests step in the TeamCity job configuration.
Bumblebee: Run local UFT tests build step has the following configuration parameters:
- Test Path: The path to a test folder or test batch file (.mtb) which shall be executed
If you use .mtb file from GIT repository, you need to make sure paths to tests are correct and point to tests in current build directory. You can use "File content replacer" Build Feature of TeamCity to replace paths in your .mtb file (please see example below)
When TeamCity runs Bumblebee UFT step, it will trigger local HP UFT Batch runner and record its output:
Bumblebee UFT task captures results of test execution and produces a simple JUnit report which are then attached to the build report and can be seen on Tests tab:
For failed tests, report contains an error message reported by UFT:
UFT also produces detailed reports with description of all steps, screenshots, etc. You can configure TeamCity jobs to capture these as build artifacts and attach them to a build results.
- Open General Settings of job configuration
- Specify appropriate value for Artifact paths field
- Save job
After build has finished, artifacts can be found in the Artifacts tab
To view the UFT HTML Report, open run_results.html artifact
- Running HP Performance Center Tests from TeamCity¶
HP Performance Center Tests + TeamCity
HP Performance Center is a powerful set of tools for composing and running performance tests and used by many companies. Bumblebee TeamCity plugin allows an easy way to schedule HP Performance Center tests and report results back to TeamCity.
Before creating a Bamboo job for running HP Performance Center tests, validate the Bumblebee Global Settings
- Configure Bumblebee HP PC Test Runner step¶
To add a new Bumblebee: Run HP PC tests build step, add a new build step in TeamCity build configuration and add Bumblebee: Run HP PC tests Runner type.
Bumblebee: Run HP PC tests build step has the following configuration parameters:
|Domain||Domain name in HP ALM|
|Project||Project name in HP ALM|
|Results Directory||Directory to which test result files will be saved|
|Path To Test||Path to a test in HP ALM TestPlan, e.g. "Subject\folder1\test", where "Subject\folder" is a path to a test folder and "test" is the name of a test to run|
|Test Set||Path to a test set in HP ALM TestLab, containing correspondent test instance, e.g. "Root\folder1\testSet", where "Root\folder1" is a path to a test lab folder and "testSet" is the name of a test set. If test set does not exist or test is not assigned to it, Bumblebee task will try to create a new test set and assign a test to it.|
|Post Run Action||Defines what PC should do after a test run. Available options: Collate And Analyze, Collate Results and Do Not Collate.|
|Time Slot Duration||Time to allot for the test (PC parameter). It cannot be less than 30 minutes (limitation by PC).|
|Use VUD Licenses||If true, the test consumes Virtual User Day (VUD) licenses.|
|Timeout||overrides a global PC timeout value and represents the number of minutes to wait for the Performance Center test to finish. 0 means wait indefinitely.|
|Retry Attempts||Number of retry attempts, before task completely fails.|
|Retry Interval||Number of seconds to wait between retry attempts.|
|Interval Increase Factor||Increase factor for retry interval. E.g. if it is set to 2, then each subsequent wait interval between attempts will be twice bigger than the previous one.|
|Collate/Analyze retry attempts||Number of retry attempts for Collate/Analyze phases, before task completely fails|
|Collate/Analyze retry interval||Number of seconds to wait between retry attempts|
|Polling Interval||The number of minutes between two test state requests.|
|Fail Build If Task Fails||If true and task has failed (or timeout has reached), then the whole build will be failed. If false, then build will not be failed even if task has failed.|
- HP PC Results Configuration¶
HP Performance Center produces test reports that can be published inside TeamCity builds. Bumblebee HP PC Test Runner task automatically downloads these reports from the HP Performance Center server and copies them to the specified Results Directory.
To publish these reports inside TeamCity, configure the Artifacts Paths in the job configuration and define appropriate values.
- Running HP PC TeamCity Job¶
When the TeamCity job is triggered it starts a new run in HP Performance Center for the test specified by Path To Test and Test Set properties of the task. Once the HP PC test has started, Bumblebee HP PC Task waits for it to finish and checks the run's possible Run States from time to time. If the test reaches one of the following states, Bumblebee assumes that test has passed:
- Before Collating Results (if Post Run Action = Do Not Collate)
- Before Creating Analysis Data (if Post Run Action = Collate Results)
If test reaches one of the following states or timeout has occurred, Bumblebee treats test as failed:
- Run Failure
- Failed Collating Results
- Failed Creating Analysis Data
If a test has failed, Bumblebee marks the build as passed or failed based on the Fail Build If Task Fails job configuration property. If true, the build is marked as FAILED and build is aborted. If false, Bumblebee simply proceeds with the next test.
If an error occurs during fetching runs status from PC, Bumblebee will retry failed action according to the retry settings defined for a task.
Sample execution log:
- Pulling test results from HP ALM to TeamCity¶
If you want to pull test results from HP ALM and display them as JUnit report of your build, you can use "Bumblebee: Import HP ALM Test Results" step.
- Bumblebee version 4.1.5 and higher
- Configure Import HP ALM Test Results step¶
|Domain||Domain name in HP ALM|
|Project||Project name in HP ALM|
|Login||User name in HP ALM. If it is set, it will override global settings|
|Password||Password in HP ALM. If it is set, it will override global settings|
|Test Set Path||Path to a TestSet in HP ALM TestLab to pull results from it|
During the execution of "Bumblebee: Import HP ALM Test Results" test step, Bumblebee searches for a Test Set by path given in "Test Set Path" parameter, creates JUnit XML report file and publishes it to TeamCity: