Fail Maven Build if JUnit Coverage Falls Below Certain Threshold

1. Overview

In this tutorial, we’ll see how we can cause a Maven build to fail when the JaCoCo code coverage falls below certain thresholds. We’ll start by looking at the vanilla JaCoCo plugin with no thresholds. Then we’ll add a new execution to the existing JaCoCo plugin that focuses on checking the coverage.

Here we’ll touch upon some of the salient elements of this new execution. We’ll then extend a simple example of a ProductService to see the effect of adding rules for BRANCH and INSTRUCTION coverage. We’ll see the build failing with the specific rules in place. Finally, we’ll conclude with the potential benefits of enforcing rules with JaCoCo for quality control.

2. JaCoCo Maven Plugin

Let’s first use the JaCoCo plugin in its simple form. It means that we’ll use it to calculate the code coverage and generate a report when we do a mvn clean install:

<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>${jacoco.version}</version>
    <configuration>
        <excludes>
            <exclude>com/baeldung/**/ExcludedPOJO.class</exclude>
            <exclude>com/baeldung/**/*DTO.*</exclude>
            <exclude>**/config/*</exclude>
        </excludes>
    </configuration>
    <executions>
        <execution>
            <id>jacoco-initialize</id>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
        </execution>
        <execution>
            <id>jacoco-site</id>
            <phase>package</phase>
            <goals>
                <goal>report</goal>
            </goals>
        </execution>
    </executions>
</plugin>

3. Setting up the Example

Next,  let’s take an example of two simple services, namely CustomerService and ProductService. Let’s add a getSalePrice() method in our ProductService which has two branches, one for when the flag is set to true and the other one with the flag set to false:

public double getSalePrice(double originalPrice, boolean flag) {
    double discount;
    if (flag) {
        discount = originalPrice - originalPrice * DISCOUNT;
    } else {
        discount = originalPrice;
    }
    return discount;
}

Let’s write two separate tests that cover both the conditions with boolean true and false:

@Test
public void givenOriginalPrice_whenGetSalePriceWithFlagTrue_thenReturnsDiscountedPrice() {
    ProductService productService = new ProductService();
    double salePrice = productService.getSalePrice(100, true);
    assertEquals(salePrice, 75);
}

@Test
public void givenOriginalPrice_whenGetSalePriceWithFlagFalse_thenReturnsDiscountedPrice() {
    ProductService productService = new ProductService();
    double salePrice = productService.getSalePrice(100, false);
    assertEquals(salePrice, 100);
}

Similarly, CustomerService comprises a simple method getCustomerName():

public String getCustomerName() {
    return "some name";
}

Next, here’s the corresponding unit test to show the coverage for this method in the report:

@Test
public void givenCustomer_whenGetCustomer_thenReturnNewCustomer() {
    CustomerService customerService = new CustomerService();
    assertNotNull(customerService.getCustomerName());
}

4. Setting the Test Coverage Baseline With JaCoCo

With a basic set of classes and tests setup, let’s do a mvn clean install first and observe the results of the JaCoCo report. This helps us set the baseline for the current module test coverage:

Here the module builds successfully. Next, let’s look at the JaCoCo report to see the current coverage:

As we can see from the report, the current coverage is 72% and we have a successful Maven build.

5. Adding Rules to JaCoCo Plugin

Now let’s apply the rules to the plugin so that the overall coverage based on the number of Instructions doesn’t go below 70% and the Branch coverage shouldn’t be below 68%.

Let’s define a new execution in the plugin for this purpose:

<execution>
    <id>check-coverage</id>
    <phase>verify</phase>
    <goals>
        <goal>check</goal>
    </goals>
    <configuration>
        <rules>
            <rule>
                <element>BUNDLE</element>
                <limits>
                    <limit>
                        <counter>INSTRUCTION</counter>
                        <value>COVEREDRATIO</value>
                        <minimum>0.70</minimum>
                    </limit>
                    <limit>
                        <counter>BRANCH</counter>
                        <value>COVEREDRATIO</value>
                        <minimum>0.68</minimum>
                    </limit>
                </limits>
            </rule>
        </rules>
    </configuration>
</execution>

Here, we have defined a new execution within the JaCoCo plugin with the id check-coverage. We have associated it with verify phase of the Maven build.  The goal associated with this execution is set to check, which means that jacoco:check shall be executed at this point. Next, we’ve defined the rules that need to be checked during jacoco:check.

Next, let’s look closely into the components defined within a <rule> tag:

5.1. <element> in JaCoCo

The first important part of the rule is the <element> tag which defines the coverage element the rule is applied on. Coverage elements represent different levels of granularity within our code, such as classes, packages, methods, or lines. We can set specific coverage criteria for that particular level by specifying the appropriate coverage element.

Here we have set the coverage threshold at the BUNDLE level which represents the overall coverage of the entire codebase being analyzed. It combines the coverage information from different units, such as classes, packages, or modules, into a single aggregated result. It’s typically used to define coverage rules or thresholds at the highest level, giving an overview of the code coverage for the entire project

Additionally, if we want to set coverage thresholds at the class level, we would use <element> CLASS </element>.

Similarly, if we want to set thresholds at the line level, we can use <element> LINE </element>.

5.2. <limits> in JaCoCo

Once we’ve defined the granularity (BUNDLE) here, we define one or more limits. Essentially, limit encapsulates the information about how we want to enforce the code coverage using one or more counters.

Each <limit> encompasses the corresponding  <counter>, < value>, and <minimum> or <maximum> tags. The possible options for the <value> tag are COVEREDRATIO, COVEREDCOUNT, MISSEDCOUNT, and TOTALCOUNT. In our example, we’re using COVEREDRATIO which represents the coverage ratio.

We’ll discuss the counters in a bit more detail in the next section.

5.3. <counter> in JaCoCo

Essentially, here we want to enforce thresholds on BUNDLE based on two counters; the BRANCH  counter and the INSTRUCTION counter. For both these counters we’re using the COVEREDRATIO as the measure of the threshold. For this example, we want the INSTRUCTION coverage ratio to be a minimum of 72% and the BRANCH coverage ratio to be a minimum of 68%.

The INSTRUCTION counter is valuable for understanding the level of code execution and identifying areas that may have been missed by the tests. The BRANCH counter enables the identification of decision point coverage, ensuring that it exercises both true and false branches.

The LINE counter provides insight into the granularity of code coverage, allowing us to determine which individual lines of code are covered.

Other possible options for counters are LINE, COMPLEXITY, METHOD, and CLASS. These counters provide different perspectives on code coverage and can be used to analyze different aspects of the codebase.

The choice of which counters are more useful depends on the specific goals and requirements of the project. However, the INSTRUCTION, BRANCH, and LINE  counters are typically the most commonly used and provide a good overall understanding of code coverage.

6. Failing the Build With Decreased Coverage

To see our build fail with the newly enforced rules, let’s  disable the following test:

@Test
@Disabled
public void givenOriginalPrice_whenGetSalePriceWithFlagFalse_thenReturnsDiscountedPrice() {//...}

Now, let’s run the command mvn clean install again:

We find that the build fails at this point, referring to the failed coverage checks:

[ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.6:check (check-coverage) on project testing-libraries-2: Coverage checks have not been met.

Importantly, we observe that the jacoco: check goal executes. It uses Jacoco.exec as a source for the analysis. Next, it performs the analysis on the BUNDLE level as we specified in the element tag of the rule and finds that the Maven build fails both for the INSTRUCTION and BRANCH counter.

As shown in the results, the INSTRUCTION  counter value of COVEREDRATIO is down to 0.68 or 68% whereas the minimum was 0.70 or 70%. Additionally, the BRANCH counter value is 0.50 or 50% whereas the expected minimum is 68%.

To fix, the build, we need to bring back the overall coverage as dictated by the BUNDLE element to have INSTRUCTION covered to be >= 70% and BRANCH covered to be >=68%.

7. Conclusion

In this article, we saw how we can make the Maven build fail if our code coverage falls below certain thresholds. We looked at various options provided by the JaCoCo Maven plugin to enforce the thresholds at different granularity levels. The example focused on BRANCH and INSTRUCTION counters with the value being COVEREDRATIO.

Enforcing coverage rules ensures that our codebase meets a minimum level of test coverage all the time. All in all, this contributes towards improving the overall quality of the software by identifying areas that lack sufficient testing. By failing the build when coverage rules aren’t met, we prevent the progression of code with inadequate coverage further in the development lifecycle, thereby reducing the likelihood of releasing untested or low-quality code

As always, the source code is available over on GitHub.