Jenkins and HP ALM Integration Guide

Why you need Bumblebee

In many organizations, Jenkins is used to build software, run unit tests, and run various kinds of testing frameworks. Example: Selenium, JUnit, Pytest, TestNG, Visual Studio Test, etc. These test metrics are very important and should be reflected in HP ALM to show the true health of any project. Bumblebee plugin for Jenkins allows users to integrate any testing framework with HP ALM without making any code changes or writing custom tools. In addition, the plugin allows you to run HP ALM tests directly from Jenkins. Many organization use Bumblebee and Jenkins to achieve CICD.

Integrating Jenkins with HP ALM is super simple by using Bumblebee's plugin for Jenkins. Bumblebee's Jenkins plugin allows you to do the following:

1. Export test results to HP ALM and automatically create TestPlan, TestLabs, TestsSets, and TestRuns. This is extremely useful for running tests directly in Jenkins and automatically these test results in HP ALM.
2. Schedule Tests in HP ALM from Jenkins and show their status in Jenkins and HP ALM
3. Run local UFT tests and report execution results in both Jenkins and HP ALM
4. Run HP Performance Center tests from Jenkins

- Install plugin

Bumblebee Jenkins plugin can be installed via the Jenkins update center

Install Bumblebee Server before configuring the Jenkins plugin

Bumblebee Jenkins plugin communicates with HP ALM via the Bumblebee server. You must install Bumblebee server before configuring the Jenkins plugin. Bumblebee server setup instructions

- Configure plugin

Configure the Jenkins plugin global settings. manage Jenkins → Manage Jenkins → Configure System

Install Bumblebee Server before configuring the Jenkins plugin

Bumblebee Jenkins plugin communicates with HP ALM via the Bumblebee server. You must install Bumblebee server before configuring the Jenkins plugin. Bumblebee server setup instructions

Field Name	Description
BumbleBee URL	URL for bumblebee server. Must end with /bumblebee. Example: http://some-server:8888/bumblebee
HP ALM URL	URL for your HP ALM instance. Must end with /qcbin. Example: http://your-alm-server:8080/qcbin
HP PC URL	URL of HP Performance Center. Leave it empty if you will not be using plugin for running tests in HP Performance Center
HP ALM User	Name of the HP ALM user to connect to ALM instance.
HP ALM User Password	Password for HP ALM user to login into ALM
UFT Batch Runner	The default path to the UFT Batch Runner (UFTBatchRunnerCMD.exe) on all slaves. It can be overridden by setting UFT_RUNNER environment variable on an agent. Leave it empty if you will not be using plugin for running local UFT tests.
Upload timeout	The number of minutes to wait for the Bumblebee server to process the request. 0 means wait indefinitely. Uploading of test results into HP ALM might take a few minutes depending upon the number of tests that must be processed.
Skipping connectivity diagnostic	If it's set to true, bumblebee will not check if Bumblebee and ALM servers are available and will not encrypt HP ALM User Password, so please encrypt password manually (just navigate to http://bumblebee_server/bumblebee/#!/admin/encrypt)
Trust self-signed certificates	Indicates if Bumblebee Plugin trusts to self-signed SSL\TLS certificates installed on Bumblebee Server. If it is set to true, Bumblebee Plugin will accept any self-signed certificate from Bumblebee Server. This option should not be used in production environments as trusting to self-signed certificates brings security risks

Save Configuration to save your changes. Saving may take a few seconds because the Bumblebee plugin will validate the configuration before saving.

---

- Export Tests Results

- Test step configuration

Bumblebee's Jenkins plugin allows automatic export of Jenkins build's test results to HP ALM. Bumblebee supports a variety of test results formats like JUnit, NUnit, TestNG, Cucumber, JBehave and Serenity. To enable this feature, configure the Jenkins job and add Bumblebee HP ALM Uploader post-build step. In this step, configure a few fields and specify the results that need to be published.

Bumblebee HP ALM Uploader configuration fields details

Field Name (pipeline property)	Required	Description
Domain (domain)	Yes	The name of HP ALM domain where to export test results
Project (projectName)	Yes	The name of HP ALM project
Test Plan (testPlan)	Yes	TestPlan path in HP ALM where tests should be saved. Must start with Subject\
Test Lab (testLab)	Yes	TestLab path in HP ALM where test results should be saved. Must start with Root\
Test Set (testSet)	Yes	The name of test set in HP ALM
Format (format)	Yes	Format of test results generated by previous tasks in the same job. Available values: junit, nunit, testng, cucumber, serenity, jbehave
Results File Pattern (resultPattern)	Yes	Provide path to the XML file(s) generated during the build.This is a comma separated list of test result directories.You can also use Ant style patterns such as **/surefire-reports/*.xml
HP ALM Mappings (customProperties)	No	If ALM Mappings are configured on the Bumblebee server, their values can be specified in this field.The format is [alm field label 1]=[value 1], [alm field label 2]=[value 2].. The label values can be Jenkins environment variables. You can also directly specify the default value on the Bumblebee server. Please refer to Bumblebee server documentation for details on configuring HP ALM mappings.
Fail build if upload unsuccessful (failIfUploadFailed)	N/A	If checked, the build will be marked as failed if for any reason the plugin was not able to upload the results. This could be due to Bumblebee server issues, HP ALM server issues, network issues, etc.
Process offline (offline)	N/A	If checked, Bumblebee will send test reports to be processed in a background thread on the Bumblebee server. This option is useful if you notice a large increase in build time due to Bumblebee's real-time test results processing.

configure ALM mappings (optional)

If your ALM project has custom user fields and you have already configured ALM Mappings, you can easily specify them in the Jenkins post-build step as name-value pairs. If you don't specify them in the Jenkins step, Bumblebee will use the custom field values from the server (if they exist).

- Pipeline configuration

Here is an example of pipeline configuration for exporting test results to HP ALM:

  step([$class: 'BumblebeePublisher',
     configs:[[
         domain: 'DEFAULT',
         projectName: 'demo',
         testPlan: 'Subject\\testPlan',
         testLab: 'Root\\testLab',
         testSet : 'testSet',
         format: 'JUNIT',
         resultPattern:'**/surefire-reports/*.xml',
         customProperties:'key=value',
         failIfUploadFailed: false,
         offline: false]],
     ])

- Execution

Once the job is configured, simply build the job. The build console will show the Bumblebee Jenkins plugin to Bumblebee REST API activity. This output is quite useful for troubleshooting.

All the results will be processed by Bumblebee server and corresponding TestPlan, TestLab, Testset, and TestRun will be created automatically.

---

- Offline processing

Processing of test results can add a few minutes to your build time. If this is not desirable, Bumblebee provides an option for processing tests results in a background thread on the Bumblebee server. Simply check the Process offline checkbox in the job configuration to enable this feature.

When Process offline is enabled, the console output will be

The console output will contain a link to the processing queue item.

To view all the offline tasks being processing, simply navigate to Offline Processing Queue from the Bumblebee Server main page

---

- Automatic Defect Management (since 5.6)

Bumblebee plugin allows users to create and resolve defects in HP ALM automatically and provides a special "HP ALM Auto Defect Management" configuration section in Bumblebee HP ALM Uploader post-build action for that:

It contains the following parameters:

Field Name (pipeline property)	Description
Defect Create Policy (defectCreatePolicy)	Defines if defects should be created or reopened. Available values: Create - creates a new defect for failing test or updates the existing one which status is New, Open or Reopen Reopen - creates a new defect for failing test or reopens the existing one
Create Status (defectCreateStatus)	Status in ALM which will be assigned to a newly created defect
Create Defect Mappings (defectCreateProperties)	Set of properties which will be set to a newly created defect. The format is as follows: [alm field label]=[value]. Each name/value pair must start with the new line. ALM field labels must be configured on Bumblebee Server. See setting up ALM Mappings for the details.
Reopen Status (defectReopenStatus)	Status in ALM which will be assigned to a reopened defect. Effective only if create policy is 'Reopen'
Reopen Defect Mappings (defectReopenProperties)	Set of properties which will be set to a reopened defect. The format is as follows: [alm field label]=[value]. Each name/value pair must start with the new line. ALM field labels must be configured on Bumblebee Server. See setting up ALM Mappings for the details.
Severity (defectSeverity)	Value for 'Severity' property of defect
Defect Resolve Policy (defectResolvePolicy)	If set to 'Close', Bumblebee will search for an existing defects with "Bumblebee" prefix for passed tests and resolve them
Resolve Status (defectResolveStatus)	Status in ALM which will be assigned to a resolved defect
Resolve Defect Mappings (defectResolveProperties)	Set of properties which will be set to a resolved defect. The format is as follows: [alm field label]=[value]. Each name/value pair must start with the new line. ALM field labels must be configured on Bumblebee Server. See setting up ALM Mappings for the details.

Pipeline configuration:

step([$class: 'BumblebeePublisher',
     configs:[[
         domain: 'DEFAULT',
         projectName: 'demo',
         testPlan: 'Subject\\defcts',
         testLab: 'Root\\defects',
         testSet : 'defects',
         format: 'cucumber',
         resultPattern:'target/*.json',
         defectCreatePolicy: 'REOPEN',
         defectCreateStatus: 'Open',
         defectSeverity: "1-Low",
         defectReopenStatus: "Open",
         defectResolvePolicy: "CLOSE",
         defectResolveStatus: "Closed",
         defectCreateProperties: """defect parameter=Created from Jenkins\ndefect project=Test\ndefect priority=2-Medium""",
         defectReopenProperties: "defect parameter=Reopened from Jenkins",
         defectResolveProperties: "defect parameter=Closed from Jenkins",
         failIfUploadFailed: false,
         offline: false]],
     ])

Creating defects for failed tests

To create defects for failing test, Defect Create Policy property must be set to either Create or Reopen, Create Status must contain a valid value for defect's Status field.

It is also possible to provide values for some system or custom fields (e.g. if they are mandatory) using Create Defect Mappings configuration field. To be able to do that, those fields should be first configured on ALM Mappings page of Bumblebee Server, e.g.:

After, fields have been configured on Bumblebee Server, values from their 'label' fields can be used in mappings:

If test report contains any failed tests, then Bumblebee will create a new defect for each of them and assign field values according to the configuration above:

If a particular test instance in ALM already has linked defect with "Bumblebee" prefix and status is not "Closed", "Fixed" or "Rejected", Bumblebee will add comments into it for each failed run.

Resolving defects for passed tests

To resolve defects for passed tests, the value of Defect Resolve Policy must be set to Close. In such a case, Bumblebee will search for linked defects with "Bumblebee" prefix in "Summary" and which status is not "Closed", "Fixed" or "Rejected" and set the status, configured with Resolve Status property.
It is also possible to provide values to system and user fields using Resolve Defect Mappings property. It works in the same way as Create Defect Mappings one.

Results in ALM:

Reopening defects

To reopen existing defects which have been closed already, the value of Defect Create Policy property must be set to "Reopen" and Reopen Status must contain a valid value for defect's Status field.

It is also possible to provide values to system and user fields using Reopen Defect Mappings property. It works in the same way as Create Defect Mappings one.

For failed test, Bumblebee searches for an existing defect with "Bumblebee" prefix in "Summary" and either reopens it (if it has been already closed) or updates. If defect is being reopened, then Bumblebee sets new status defined in "Reopen Status" property and appies values from "Reopen Defect Mappings":

---

- Running ALM Tests from Jenkins

Bumblebee's Jenkins build step allows you to run tests stored in HP ALM (UFT, LeanFT, QTP, etc) directly from Jenkins and view the results in both Jenkins and HP ALM.

- Prerequisites

• Job must run on the Jenkins slave installed on Windows machine
• Jenkins slave must have launch method: Launch slave agents via Java Web Start
• Jenkins slave must NOT run as windows service
• Appropriate version of HP ALM Connectivity Tool must be installed on Jenkins slave machine. Tool is available at http://your_alm_server_and_port/qcbin/PlugIns/TDConnectivity/TDConnect.exe
• Appropriate version of HP ALM Client must be installed on Jenkins slave machine. Available at http://your_alm_server_and_port/qcbin/start_a.jsp?common=true

- Step configuration

To run ALM Testset from Jenkins, add Bumblebee HP ALM Test Set Runner build step to the Jenkins job configuratoin

Field Name (pipeline property)	Description
Domain (domain)	The name of HP ALM domain
Project (project)	The name of HP ALM project
Test Sets (testSets)	A list of test sets to execute. Each test set path must start on a new line. Path must start with Root and specify the full path of HP ALM Testlab's TestSet. E.g. Root\folder1\testset1. In this example Root\folder1 is the TestLab folder and folder1 is the name of the TestSet to execute
Run Mode (runMode)	How to run test sets. Possible values: LOCAL - run all tests on agent's,machine, SCHEDULED - run tests on planned host, REMOTE - run on remote,host
Run Host (host)	The name of host on which tests shall be run. May be blank if Run Mode is LOCAL or SCHEDULED
JUnit Results Directory (outputDirPath)	Directory where JUnit-like execution reports will be placed. If it does not exist, Bumblebee will create it
Timeout (timeOut)	The number of minutes to wait for test sets execution. 0 means wait indefinitely.

- Pipeline configuration

step([$class: 'RunTestSetBuildStep',
         domain:'DEFAULT',
         project:'demo',
         runMode:'LOCAL',
         host:'',
         testSets:'Root\\folder1\\test',
         outputDirPath:'alm-results',
         timeOut: 0
     ])

- Execution

When Jenkins build is executed, the Bumblebee HP ALM Test Set Runner step connects to HP ALM server and runs the specified ALM Testsets. The build waits till the test runs is completed in HP ALM.

Example: Jenkins build console

Once the build is complete, Bumblebee HP ALM Test Set Runner transforms the HP ALM test results in a JUnit report compliant schema so that it can be published within Jenkins, using the standard Publish JUnit test result report post-build step.

- Adding tests from HP ALM TestPlan to TestSet

If a single test or all tests in a TestPlan folder need to be added to HP ALM TestSet, then Bumblebee: Add Test to Test Set build step can be used:

The step has the following configuration fields:

Field (pipeline property)	Description
Domain (domain)	The name of HP ALM domain
Project (project)	The name of HP ALM project
Test Plan Path (testPlanPath)	Path to a test or test folder in HP ALM TestPlan, e.g. Subject\testfolder\test1
Test Set Path (testSetPath)	Path to a HP ALM TestSet into which test(s) shall be added. If TestSet does not exist, Bumblebee will try to create it on the fly. Example: Root\testfolder\testSet

Or it can be also configured to run as a part of pipeline:

 step([$class: 'AddTestToSetStep',
      domain: 'DEFAULT',
     project: 'demo',     
     testPlanPath: 'Subject\\testPlan\\testFail1',
     testSetPath: 'Root\\testLab\\testSet'
     ])

Console output:

Results in HP ALM:

---

- Running ALM Tests inside Jenkins

Bumblebee allows you to run HP Unified Functional Testing tests directly from Jenkins and reports results back to Jenkins.

- Prerequisites

• Jenkins slave runs on Windows machine and have appropriate HP Unified Functional Testing version installed. Please see UFT requirements for a particular version of OS and other software.
• Jenkins slave must run as a console application (not as a windows service)
• UFT Batch Runner property of Global Configuration or UFT_RUNNER environment variable on slave must be set

To override path to UFT Batch Runner, defined in the Global Configuration, you need to set a UFT_RUNNER environment variable on a Jenkins slave.

To set a value to UFT_RUNNER environment variable of Jenkins slave:

• Open Jenkins slave configuration page
• Check "Environment variables" checkbox
• Add a new variable and type "UFT_RUNNER" as "Name" and path to UFT Batch Runner on that slave as a "Value"
• Click on "Save" button

To run UFT tests locally inside of Jenkins, add the Bumblebee Local UFT Test Runner build step in your jenkins job configuration.

Bumblebee Local UFT Test Runner build step has the following configuration parameters:

• Test Path: The path to a test folder or test batch file (.mtb) which needs to be executed
• Results Directory: Directory inside your project where Bumblebee can add JUnit-like execution reports. If it does not exist, Bumblebee will create it automatically.

Pipeline:

step([$class: 'RunUftTestBuildStep',
      testPath:'test.mtb',
      outputDirPath:'uft-results'
     ])

mtb file paths

If your .mtb files are located in some GIT repository, make sure paths to tests are correct and point to tests in build directory. You can use windows batch script for this.

Bumblebee Local UFT Test Runner step outputs the results of test execution as a standard JUnit style report in the JUnit Results Directory.

These reports can then be published using Publish JUnit test result report Post-build step.

When Jenkins runs Bumblebee Local UFT Test Runner step, it will trigger local HP UFT Batch runner and record its output:

Bumblebee Local UFT Test Runner step captures test results from the Jenkins build and produces a standard JUnit style report. If the job is configured to publish reports, you can view the tests results in Jenkins like any other JUnit style reports by going to the Test Results section:

Bumblebee captures detailed errors as reported by UFT and dumps them in the reports.

UFT also produces detailed reports with description of all steps, screenshots, etc. The Jenkins job can be configured to capture these as build artifacts. To enable this, add the Archive the artifacts Post-build step to the job configuration and define appropriate value for Files to archive

After the build has finished, artifacts are displayed in the Artifacts build tab:

---

- Running HP Performance Center Tests from Jenkins

HP Performance Center Tests + Jenkins

HP Performance Center is a powerful set of tools for composing and running performance tests and used by many companies. Bumblebee Jenkins plugin allows an easy way to schedule HP Performance Center tests and report results back to Jenkins.

- Prerequisites

Before creating a Jenkins job for running HP Performance Center tests, validate the Bumblebee Global Settings

- Configure Bumblebee HP PC Test Runner step

To configure a Jenkins job to run HP Performance Center tests, add the Bumblebee HP PC Test Runner build step.

Bumblebee HP PC Test Runner build step has the following configuration parameters:

Parameter name (pipeline property)	Description
Domain (domain)	Domain name in HP ALM
Project (project)	Project name in HP ALM
Results Directory (outputDir)	Directory to which test result files will be saved
Path To Test (testPlanPath)	Path to a test in HP ALM TestPlan. Example: Subject\folder1\foo, where Subject\folder is the path to the test folder and foo is the name of a test to run
Test Set (testLabPath)	Path to the TestSet in HP ALM TestLab, containing corresponding test instances. Example: Root\folder1\testSet, where Root\folder1 is the path to the testlab folder and testSet is the name of the testset. If testset does not exist or test is not assigned to it, Bumblebee task will automatically create a new testset and assign a test to it.
Post Run Action (postRunActionString)	Defines what PC should do after a test run. Available options: Collate And Analyze, Collate Results and Do Not Collate
Time Slot Duration (timeslotDuration)	Time to allot for the test (PC parameter). It cannot be less than 30 minutes (limitation by PC).
Use VUD Licenses (vudsMode)	If true, the test consumes Virtual User Day (VUD) licenses.
Timeout (timeout)	Overrides a global PC timeout value and represents the number of minutes to wait for the Performance Center test to finish. 0 means wait indefinitely.
Retry Attempts (retryCount)	Number of retry attempts, before task completely fails.
Retry Interval (retryInterval)	Number of seconds to wait between retry attempts.
Interval Increase Factor (retryIntervalMultiplier)	Increase factor for retry interval. E.g. if it is set to 2, then each subsequent wait interval between attempts will be 2x the previous one.
Collate/Analyze retry attempts (retryCollateAndAnalysisAttempts)	Number of retry attempts for Collate/Analyze phases, before task completely fails
Collate/Analyze retry interval (retryCollateAndAnalysisInterval)	Number of seconds to wait between retry attempts
Polling Interval (pollingInterval)	The number of minutes between two test state requests.
Fail Build If Task Fails (failIfTaskFails)	If true and task has failed (or timeout has reached), then the whole build will be failed. If false, then build will not be failed even if task has failed.

- Pipeline configuration

 step([$class: 'RunPcTestBuildStep',
      domain: 'default',
     project: 'demo',     
     outputDir: 'pc-results',
     testPlanPath: 'Subject\\pc\\test',
     testLabPath : 'Root\\pc\\test',
     postRunActionString: 'Collate Results',
     timeslotDuration:30,
     vudsMode:false,
     timeout: 0,
     retryCount: 0,
     retryInterval:0,
     retryIntervalMultiplier:0,
     retryCollateAndAnalysisFlag: false,
     retryCollateAndAnalysisAttempts:0,
     retryCollateAndAnalysisInterval:0,
     pollingInterval: 1,
     failIfTaskFails:true,
     ])

- HP PC Results Configuration

HP Performance Center produces test reports that can be published inside Jenkins builds. Bumblebee HP PC Test Runner task automatically downloads these reports from the HP Performance Center server and copies them to the specified Results Directory.

To publish these reports inside Jenkins, add the Archive Artifacts post-build step to job configuration and define appropriate values for Files to Archive

- Running HP PC Jenkins Job

When the Jenkins job is triggered it starts a new run in HP Performance Center for the test specified by Path To Test and Test Set properties of the task. Once the HP PC test has started, Bumblebee HP PC Task waits for it to finish and checks the run's possible Run States from time to time. If the test reaches one of the following states, Bumblebee assumes that test has passed:

• Finished
• Before Collating Results (if Post Run Action = Do Not Collate)
• Before Creating Analysis Data (if Post Run Action = Collate Results)

If test reaches one of the following states or timeout has occurred, Bumblebee treats test as failed:

• Canceled
• Run Failure
• Aborted
• Failed Collating Results
• Failed Creating Analysis Data

If a test has failed, Bumblebee marks the build as passed or failed based on the Fail Build If Task Fails job configuration property. If true, the build is marked as FAILED and build is aborted. If false, Bumblebee simply proceeds with the next test.

If an error occurs during fetching runs status from PC, Bumblebee will retry failed action according to the retry settings defined for a task.

Sample execution log:

- Pulling test results from HP ALM to Jenkins

If you want to pull test results from Jenkins and display them as JUnit report of your build, you can use "Bumblebee: Import HP ALM Test Results" step.

- Prerequisites

• Bumblebee version 4.1.5 and higher

- Configure Import HP ALM Test Results step

Parameter name (pipeline property)	Description
Domain (domain)	Domain name in HP ALM
Project (project)	Project name in HP ALM
Login	User name in HP ALM. If it is set, it will override global settings
Password	Password in HP ALM. If it is set, it will override global settings
Results Directory (resultsDir)	Path to the directory where to put JUnit-like reports containing results of tests in HP ALM
Test Set Path (configurations:[[testSetPath]]	Path to a TestSet in HP ALM TestLab to pull results from it

- Pipeline configuration

step([$class: 'GetTestResults',
     domain:'DEFAULT',
     project:'demo',
     resultsDir:'alm-results',
     configurations:[[
            testSetPath: 'Root\\testLab\\testSet'
     ]]])

- Execution

During the execution of "Bumblebee: Import HP ALM Test Results" test step, Bumblebee searches for a Test Set by path given in "Test Set Path" parameter, creates JUnit XML report file and puts it into "Results Directory" folder. This folder can be used by JUnit publisher to build test trends.