Skip to content

TestNg and HP ALM Integration Guide

Description

Bumblebee provides a set of Java annotations which allows you to send your TestNG test XML reports right after test execution in your favorite IDE.

Technically it consists of the following parts:

  • @Bumblebee annotation
  • TestNG test run reporter: com.agiletestware.bumblebee.annotations.testng.BumblebeeTestNGReporter
  • TestNG test listener: com.agiletestware.bumblebee.annotations.testng.BumblebeeTestNGListener
  • XML configuration file which contains ALM connection and other details needed for functioning of bumblebee-annotations

Prerequisites

  • Bumblebee Server must be installed in your network

Installation and configuration

There are two different approaches on how to use Bumblebee annotation package with TestNG tests:

  • Adding Bumblebee-related reporter and listener to a specific TestNG test class (or to base class) or into TestNG suite XML file.
  • Configuring your Maven surefire plugin to use Bumblebee annotations only for tests running within Maven build.

Adding reporter and listener to TestNG test

This feature allows to run tests and export their results into ALM right from their IDE. To start using it, follow the steps below:

1) Modify your pom.xml file and add Agiletestware repository and bumblebee-annotations dependency:

<dependencies>
    <dependency>
        <groupId>com.agiletestware</groupId>
        <artifactId>bumblebee-annotations</artifactId>
        <version>0.1.6</version>
        <scope>test</scope>
    </dependency>
</dependencies>
<repositories>
    <repository>
        <id>nexus.agiletestware.com</id>
        <url>https://nexus.agiletestware.com/repository/maven-public</url>
    </repository>
</repositories>

2) Create bumblebee_config.xml file and put into your project root folder. Here is a description of configuration file:

<?xml version="1.0"?>
<bumblebee>
  <!-- URL of the Bumblebee Server -->
  <bumblebee_url>http://server_name:port/bumblebee</bumblebee_url>
  <!-- URL of HP ALM Server -->
  <alm_url>http://server_name:port/qcbin</alm_url>
  <!-- Name of HP ALM User -->
  <alm_user>qcuser</alm_user>
  <!-- Encrypted password: please use http://server_name:port/bumblebee/password/encrypt to encrypt your plain text password  -->
  <alm_encrypted_pass>fd4OMOXLJjkMR6e64RJh3Q==</alm_encrypted_pass>
  <!-- HP ALM Domain -->
  <alm_domain>DEFAULT</alm_domain>
  <!-- HP ALM Project -->
  <alm_project>annotations_demo</alm_project>
  <!-- Asynchronous (offline) update -->
  <async_update>false</async_update>
</bumblebee>

3) Add Bumblebee com.agiletestware.bumblebee.annotations.testng.BumblebeeTestNGReporter and com.agiletestware.bumblebee.annotations.testng.BumblebeeTestNGListener listeners to a test class:

import org.testng.annotations.Listeners;

import com.agiletestware.bumblebee.annotations.testng.BumblebeeTestNGListener;
import com.agiletestware.bumblebee.annotations.testng.BumblebeeTestNGReporter;

@Listeners(value = { BumblebeeTestNGReporter.class, BumblebeeTestNGListener.class })
public class BaseClass {

}

Now, each time a test class is run its results are uploaded to HP ALM

Please note that in such a case listeners are added globally and apply to the whole TestNG suite

It is also possible to add listeners to your TestNG test suite XML file, e.g.:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Bumblebee test suite" parallel="false">
    <listeners>
        <listener class-name="com.agiletestware.bumblebee.annotations.testng.BumblebeeTestNGListener" />
        <listener class-name="com.agiletestware.bumblebee.annotations.testng.BumblebeeTestNGReporter" />
    </listeners>
    <test name="Test">
        <classes>
            <class
                name="com.agiletestware.bumblebee.dummytest.MethodsAsStepsTest" />
        </classes>
    </test>
</suite>

Configuring Maven surefire plugin

In case of Maven project, Bumblebee might be configured on surefire plugin level. In such a case test results will be exported to ALM only when they are run as a pasrt of Maven build.
This might be useful if sending results to ALM should happen only under certain conditions (e.g. from CI system only) and controlled by Maven profiles.

To setup Maven surefire plugin please follow the steps below:

1) Modify your pom.xml file and add agiletestware repository and bumblebee-annotations dependency:

<dependencies>
    <dependency>
        <groupId>com.agiletestware</groupId>
        <artifactId>bumblebee-annotations</artifactId>
        <version>0.1.6</version>
        <scope>test</scope>
    </dependency>
</dependencies>
<repositories>
    <repository>
        <id>nexus.agiletestware.com</id>
        <url>https://nexus.agiletestware.com/repository/maven-public</url>
    </repository>
</repositories>

2) Configure maven surefire plugin:

<build>
  <plugins>   
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.19</version>
        <configuration>
          <properties>
            <property>
              <name>listener</name>
              <value>com.agiletestware.bumblebee.annotations.testng.BumblebeeTestNGReporter</value>
            </property>
          </properties>
        </configuration>
      </plugin>
  </plugins>
</build>

3) Create bumblebee_config.xml file and put into your project root folder. Here is a description of configuration file:

<?xml version="1.0"?>
<bumblebee>
  <!-- URL of the Bumblebee Server -->
  <bumblebee_url>http://server_name:port/bumblebee</bumblebee_url>
  <!-- URL of HP ALM Server -->
  <alm_url>http://server_name:port/qcbin</alm_url>
  <!-- Name of HP ALM User -->
  <alm_user>qcuser</alm_user>
  <!-- Encrypted password: please use http://server_name:port/bumblebee/password/encrypt to encrypt your plain text password  -->
  <alm_encrypted_pass>fd4OMOXLJjkMR6e64RJh3Q==</alm_encrypted_pass>
  <!-- HP ALM Domain -->
  <alm_domain>DEFAULT</alm_domain>
  <!-- HP ALM Project -->
  <alm_project>annotations_demo</alm_project>
  <!-- Asynchronous (offline) update -->
  <async_update>false</async_update>
</bumblebee>

Overriding configuration options with Java Properties

Due to security or other reasons, you might want to override or replace XML configuration with Java Properties.

The following Java configuration properties are available:

Name Description
bumblebee.config Path to a configuration file. If not present, then ${project.basedir}/bumblebee_config.xml is used
bumblebee.config.bumblebee_url URL of the Bumblebee Server
bumblebee.config.alm_url URL of ALM Server
bumblebee.config.alm_user Name of ALM User
bumblebee.config.alm_password Encrypted user password
bumblebee.config.alm_domain Name of ALM Domain
bumblebee.config.alm_project Name of ALM Project
bumblebee.config.async_update Asynchronous (offline) update
bumblebee.config.take_screenshot Boolean flag indicating whether a screenshot of a failure should be automatically done (for WebDriver tests only)
bumblebee.config.enabled Boolean flag indicating whether Bumblebee functionality is enabled or not
bumblebee.config.update_testplan Boolean flag indicating whether Bumblebee should update existing tests in ALM TestPlan module

To provide values to Maven via command line, please use -D, e.g.

mvn clean test -Dbumblebee.config.bumblebee_url=http://localhost:8080/bumblebee

Note

Values provided with Java Properties override the ones provided in XML configuration.

Add Bumblebee annotations to your TestNG test classes/methods

@Bumblebee Java annotation can be added on class or method level When class or method is marked with Bumblebee annotation, its result will be exported to HP ALM.

Bumblebee annotation parameters

Name Description Applicable Required
testplan Path to test in a Test Plan in HP ALM. e.g. Subject\Test1 Class, method Yes
testlab Path to test set in a test lab in HP ALM. e.g. Root\Test1 Class, method Yes
testset Name of test set in HP ALM Class, method Yes
testname If specified, then the value of testname parameter will be set as the,name of test in HP ALM. If not set, test name will be set to fully,qualified method name, e.g. com.annotations.Demo.method1 Method No
almid Defines id of a test in a HP ALM test plan which needs to be updated. If specified then testplan and testname are ignored Method No
description Description for test in HP ALM Method No
overwriteAlmSteps If set to true, all existing test steps will be deleted Class, method No
parameters An array of custom parameters which are passed to the bumblebee server and then mapped to HP ALM custom fields. Please refer to documentation documentation on how to setup mappings on Bumblebee server Class, method No

Bumblebee annotation on method level overrides values set by Bumblebee annotation on class level.

Defining dynamic values for annotation parameters (since 0.1.0)

The following parameters can contain dynamic values:

  • testplan
  • testlab
  • testset
  • testname
  • description

To define a dynamic value, just add a placeholder in format: ${property_name}, where property_name is the name of property containing value for the parameter.
Bumblebee resolves dynamic values in run time using Java System Properties or XML configuration file.

Providing values with bumblebee_config.xml file

To set a value for dynamic parameter in bumblebee_config.xml file, just add new property element. E.g. to provide a value for ${"example"} placeholer

@Bumblebee(testset="Test set: ${example}")

<property name="example">some value</property>

Providing values with Java System Properties

To provide a value with Java System Property, just pass then into your Maven build, e.g.

mvn clean test -Dexample=Something

If both XML configuration and Java System Property are provided, value in Java System Property will be used.

Run Maven build

To run TestNG tests and send results to HP ALM, just run Maven test phase on your project: e.g.: mvn test

Examples

Bumblebee annotation on class level:

@Bumblebee(testlab = "Root\\testng_simple", testset = "class_annotations", testplan = "Subject\\testng_simple")
public class SomePassSomeFailTest {

  @Bumblebee(testname = "Test with changed name")
  @Test
  public void test1() {
    Assert.fail("test1 failed");
  }

  @Test
  public void test2() {
    System.out.println("test2 passed");
  }

  @Test
  public void test3() {
    System.out.println("test3 passed");
  }

  @Test
  public void test4() {
    Assert.fail("test4 failed");
  }

  @Test
  public void test5() {
    System.out.println("test5 passed");
  }
}

After export to HP ALM:

Result in HP ALM

Bumblebee annotation on method level:

public class SomePassSomeFailTest {

  @Test
  public void test1() {
    Assert.fail("test1 failed");
  }

  @Bumblebee(testname = "TestNG test", testlab = "Root\\testng_simple", testset = "method_annotation", testplan = "Subject\\testng_simple")
  @Test
  public void test2() {
    System.out.println("test2 passed");
  }
}

Results in HP ALM:

Result in HP ALM

Overriding annotation values on methods:

public class AllPassedTest {

  @Bumblebee(testlab = "Root\\testng_simple", testset = "method_annotation", testplan = "Subject\\testng_simple")
  @Test
  public void test1() {
    System.out.println("test1");
  }

  @Bumblebee(testlab = "Root\\testng_simple", testset = "method_annotation", testplan = "Subject\\testng_simple")
  @Test
  public void test2() {
    System.out.println("test1");
  }

  @Bumblebee(testlab = "Root\\testng_simple", testset = "method_annotation", testplan = "Subject\\testng_simple")
  @Test
  public void test3() {
    System.out.println("test1");
  }

  @Test
  public void test4() {
    System.out.println("test1");
  }
}

Results:

Result in HP ALM

Map TestNG test method to existing test in HP ALM:

Here is the existing Manual test in HP ALM Test Plan with id = 141:

Map to existing test

It contains two design steps:

Map to existing test

To map test method in java class to the test in HP ALM, almid property can be used. Also it is necessary to define testlab and testset properties :

public class UpdateExistingTest {
  @Bumblebee(almid = 141, description = "Updated description", testlab = "Root\\testNG existing test", testset = "map existing test")
  @Test
  public void test1() {
    Assert.fail("test1 failed");
  }
}

Results in HP ALM: Test description in Test Plan is updated with the value of Bumblebee description field:

Results in HP ALM

And results are exported to the HP ALM Test Lab. In such a case, all steps get the same status. Note that the new step was added:

Results in HP ALM

Mapping single test to multiple tests in ALM

Bumblebee almid field can accept an array of test IDs, e.g.: almid={42,24}. In such a case test result will be mapped into two tests in ALM.

Setting custom fields in HP ALM from TestNG class

Bumblebee annotation allows users to pass custom parameters to Bumblebee Server which then are mapped to custom user fields in HP ALM. Please refer to documentation on how to setup mappings on Bumblebee server. Parameter name shall match to the "label" attribute of mapping in mapping XML on the server:

<mapping alm_field="TS_USER_01" label="user field"/>

To map some value to TS_USER_01 field in HP ALM, define a parameter with name="user field":

public class UpdateExistingTest {
  @Bumblebee(testplan = "Subject\\custom fields", description = "Test description",
      testlab = "Root\\custom fields", testset = "map custom fields",
      parameters = {@Parameter(name = "user field", value = "value from TestNG") })
  @Test
  public void test1() {
    Assert.fail("test1 failed");
  }
}

After export TS_USER_01 field of the test in HP ALM test plan gets the value from Java class:

Results in HP ALM

Currently Bumblebee supports setting of custom fields for HP ALM Test, Test Set, Run, Design Step, Run Step, Test Instance

Overriding Bumblebee custom fields configuration (since 0.1.3)

It is also possible to override server side configuration with the bumblebee XML configuration. To do that, just add a new set of mapping elements into configuration:

<bumblebee>
    <bumblebee_url>http://bbe:8888/bumblebee</bumblebee_url>
    <alm_url>http://alm:8080/qcbin</alm_url>
    <alm_user>user</alm_user>
    <alm_encrypted_pass>fd4OMOXLJjkMR6e64RJh3Q==</alm_encrypted_pass>
    <alm_domain>DEFAULT</alm_domain>
    <alm_project>demo</alm_project>

    <mapping field_type="test" field_label="user field" bumblebee_label="test user field" default_value="something"/>   
</bumblebee>

mapping element has the following attributes:

  • field_type - a type of entity in ALM for which mapping is provided. The following are supported:
    • test - test in TestPlan
    • test-set - test set in TestLab
    • test-instance - test instance in TestLab
    • design-step - step of a test in TestPlan
    • run-step - step of a test in a run in TestLab
    • run - run in TestLab
  • field_label - label of a field in ALM (not a system name as in mapping configuration on server), e.g. 'Status' or 'User field', etc...
  • bumblebee_label - this is a label which should be used as @Parameter.name to pass a different value
  • default_value - default value

E.g. if you wish to provide value for test's "User field", the following element should be used:

<mapping field_type="test" field_label="user field" bumblebee_label="test user field" default_value="something"/>

When running Maven build, the following notification is displayed in the console output:

Bumblebee: Passing mappings configuration to the server: [Mapping [fieldLabel=user field, fieldType=TEST, defaultValue=something, bumblebeeLabel=test user field]]
Bumblebee: Mapping configuration on Bumblebee Server will be ignored

And results in ALM:

Result in HP ALM

Exporting all test methods into one HP ALM test

Sometimes test classes are designed in the way that each of them represents a single test case and test methods are just a test steps.
While some can say that it is not a best test design practice, Bumblebee supports such a use case.
To export all methods of a single test class to HP ALM, this class shall be marked with @Bumblebee annotation with hasSteps attribute set to BooleanValue.TRUE.
e.g.:

import org.testng.annotations.Test;
import com.agiletestware.bumblebee.annotations.BooleanValue;
import com.agiletestware.bumblebee.annotations.Bumblebee;

@Bumblebee(testname = "SingleTest", testplan = "Subject\\testng_simple", testlab = "Root\\testng_simple", testset = "method_annotations", hasSteps = BooleanValue.TRUE)
public class SingleTest {

    @Test(priority = 1)
    public void methodOne() {
        System.out.println("Passed");
    }

    @Test(priority = 2)
    public void methodTwo() {
        throw new RuntimeException("ooops");
    }

    @Test(priority = 3)
    public void methodThree() {
        System.out.println("Passed");
    }
}

In such a case, the new test with three steps will be created in HP ALM Test Plan:

Result in HP ALM

and results will also appear in Test Lab:

Result in HP ALM

Note: Please note that Bumblebee relies on test method execution order provided by TestNG, so you have to provide a fixed test methods execution order somehow (e.g. via priorities).
Otherwise your test steps will have incorrect execution order and consequently wrong order in HP ALM.

By default, test step names are the same as names of corresponding test methods in your Java code.
To change test step names and some other export settings, a @BumblebeeTestStep annotation was introduced.
This annotation can be applied to test methods only and has the following properties:

Name Description Required
name Name of a test step in HP ALM Yes
description Description of a test step in HP ALM No
expected Value for Expected field of a test step in HP ALM No

e.g.:

import org.testng.annotations.Test;
import com.agiletestware.bumblebee.annotations.BooleanValue;
import com.agiletestware.bumblebee.annotations.Bumblebee;
import com.agiletestware.bumblebee.annotations.BumblebeeTestStep;

@Bumblebee(testname = "SingleTest", testplan = "Subject\\testng_simple", testlab = "Root\\testng_simple", testset = "method_annotations", hasSteps = BooleanValue.TRUE)
public class SingleTest {

    @Test(priority = 1)
    public void methodOne() {
        System.out.println("Passed");
    }

    @Test(priority = 2)
    public void methodTwo() {
        throw new RuntimeException("ooops");
    }

    @BumblebeeTestStep(name = "method three", description = "method three description", expected = "expected")
    @Test(priority = 3)
    public void methodThree() {
        System.out.println("Passed");
    }
}

And it produces the following test steps in HP ALM Test Plan:

Result in HP ALM

Adding arbitrary attachments

To add an arbitrary attachment to a run in ALM, Bumblebee provides a static utility class com.agiletestware.bumblebee.annotations.util.CurrentTest with a set of methods:

  • addAttachment(File) - add a single file
  • addAttachment(String, InputStream) - add a single attachment from a stream
  • addAttachments(List) - add list of files

Here is an example of adding attachments:

@Bumblebee(testlab = "Root\\testng", testset = "class_annotations", testplan = "Subject\\testng")
public class SomePassSomeFailTest {

    @Test
    public void test1() {
        CurrentTest.addAttachment(new File("test.log"));
    }
}

Results in ALM:

Result in HP ALM

Adding arbitrary test steps (since 0.1.1)

To add additional test steps to test in ALM, Bumblebee provides com.agiletestware.bumblebee.annotations.util.CurrentTest.addStep(String) method which returns a builder object which provides convenient way to set attributes of step.

@Bumblebee(testlab = "Root\\testng", testset = "class_annotations", testplan = "Subject\\testng")
public class SomePassSomeFailTest {

    @Test
    public void test_additional_step() {
        final Step step = CurrentTest.addStep("Additional step")
                .description("description")
                .expected("expected")
                .actual("actual")
                .status(Status.FAILED)
                .build();
        step.addAttachment(new File("test.log"));
    }
}

Results in ALM:

Result in HP ALM

Adding log output into test step's actual (since 0.1.3)

To add some log messages into test step's actual field, use com.agiletestware.bumblebee.annotations.util.CurrentTest.log(String) method:

import org.testng.annotations.Test;

import com.agiletestware.bumblebee.annotations.Bumblebee;
import com.agiletestware.bumblebee.annotations.util.CurrentTest;

@Bumblebee(testlab = "Root\\testng_simple", testset = "class_annotations", testplan = "Subject\\testng_simple")
public class SomePassSomeFailTest {

    @Test
    public void test1() {
        CurrentTest.log("Log line one");
        CurrentTest.log("Log line two");
    }
}

Results in ALM:

Result in HP ALM

Disabling update of existing tests in TestPlan (since 0.1.3)

Sometimes it is not desired to update your existing test in TestPlan, but just export results into TestLab. For this purpose, Bumblebee offers an update_testplan configuration switch. By default, it is set to true, so Bumblebee updates test in TestPlan for each run. To disable update, just add true element into bumblebee XML configuration:

<bumblebee>
    <bumblebee_url>http://bbe:8888/bumblebee</bumblebee_url>
    <alm_url>http://alm:8080/qcbin</alm_url>
    <alm_user>user</alm_user>
    <alm_encrypted_pass>fd4OMOXLJjkMR6e64RJh3Q==</alm_encrypted_pass>
    <alm_domain>DEFAULT</alm_domain>
    <alm_project>demo</alm_project>
    <update_testplan>false</update_testplan>
</bumblebee>

In such a case, Bumblebee will not update existing tests in TestPlan and add a new Comments sections into a run in TestLab:

Result in HP ALM

while existing test steps will get No Run status:

Result in HP ALM

Disabling Bumblebee functionality

There are two ways to disable Bumblebee functionality:

Disable exporting of all test results into HP ALM

To disable exporting all test results into HP ALM, just add <enabled>false</enabled> into bumblebee_config.xml file:

<?xml version="1.0"?>
    <bumblebee>
      <!-- URL of the Bumblebee Server -->
      <bumblebee_url>http://server_name:port/bumblebee</bumblebee_url>
      <!-- URL of HP ALM Server -->
      <alm_url>http://server_name:port/qcbin</alm_url>
      <!-- Name of HP ALM User -->
      <alm_user>qcuser</alm_user>
      <!-- Encrypted password: please use http://server_name:port/bumblebee/password/encrypt to encrypt your plain text password  -->
      <alm_encrypted_pass>fd4OMOXLJjkMR6e64RJh3Q==</alm_encrypted_pass>
      <!-- HP ALM Domain -->
      <alm_domain>DEFAULT</alm_domain>
      <!-- HP ALM Project -->
      <alm_project>annotations_demo</alm_project>
      <enabled>false</enabled>
    </bumblebee>

During the execution the following message will be shown in console output: Bumblebee: Bumblebee has been disabled in configuration → Results will not be exported to HP ALM

Disable exporting results for a particular test class or method

To disable exporting results for a particular test the one just need to add enabled = BooleanValue.FALSE attribute to Bumblebee annotation:

@Bumblebee(testlab = "Root\\testng_simple", testset = "class_annotations", testplan = "Subject\\testng_simple", enabled = BooleanValue.FALSE)
    public class SomePassSomeFailTest {

      @Bumblebee(testname = "Test with changed name")
      @Test
      public void test1() {
        Assert.fail("test1 failed");
      }
    }