Difference between revisions of "Sonar + maven configuration + Jenkins"

Line 47: Line 47:
  
 
By default, the Failsafe Plugin will automatically include all test classes with the following wildcard patterns:
 
By default, the Failsafe Plugin will automatically include all test classes with the following wildcard patterns:
* "<code>**/IT*.java</code>" - includes all of its subdirectories and all Java filenames that start with "IT".
+
* <code>**/IT*.java</code> - includes all of its subdirectories and all Java filenames that start with "IT".
* "<code>**/*IT.java</code>" - includes all of its subdirectories and all Java filenames that end with "IT".
+
* <code>**/*IT.java</code> - includes all of its subdirectories and all Java filenames that end with "IT".
* "<code>**/*ITCase.java</code>" - includes all of its subdirectories and all Java filenames that end with "ITCase".
+
* <code>**/*ITCase.java</code> - includes all of its subdirectories and all Java filenames that end with "ITCase".
 
If the test classes do not follow any of these naming conventions, then configure Failsafe Plugin and specify the tests you want to include.
 
If the test classes do not follow any of these naming conventions, then configure Failsafe Plugin and specify the tests you want to include.
  
  
 
Results are saved '''in each Maven module''':  
 
Results are saved '''in each Maven module''':  
* Results are saved in <code>XML</code> and <code>TXT</code> formats in <code>$module/target/failsafe-reports/*</code>
+
* Results are saved in XML and TXT formats in <code>$module/target/failsafe-reports/*</code>
* Results are saved in <code>Binary (jacoco format)</code> in <code>$module/target/jacoco-it.exec</code>
+
* Results are saved in BINARY (jacoco format) in <code>$module/target/jacoco-it.exec</code>
  
  
Line 64: Line 64:
  
  
<syntaxhighlight lang="xml">
+
<syntaxhighlight lang="XML">
 
     <build>
 
     <build>
 
         <plugins>
 
         <plugins>

Revision as of 14:54, 9 April 2019


This article explains how to configure your maven projects to produce reports + how to configure Jenkins with SonarQube, bringing both Unit and Integration tests coverage.


Requirements

  • Maven must run on Java 8 or +
  • You must use Sonarqube 7.x or +


Maven parent POM

You just have to define the configuration once, in the parent POM.


Maven plugins

Surefire UNIT Tests

Surefire is for UNIT tests = tests that should only use: plain java / Mockito / PowerMock / etc.


By default, the Surefire Plugin will automatically include all test classes with the following wildcard patterns:

  • "**/Test*.java" - includes all of its subdirectories and all Java filenames that start with "Test".
  • "**/*Test.java" - includes all of its subdirectories and all Java filenames that end with "Test".
  • "**/*Tests.java" - includes all of its subdirectories and all Java filenames that end with "Tests".
  • "**/*TestCase.java" - includes all of its subdirectories and all Java filenames that end with "TestCase".

If the test classes do not follow any of these naming conventions, then configure Surefire Plugin and specify the tests you want to include.

See Maven Surefire plugin documentation


Results are saved in each Maven module:

  • Results are saved in XML and TXT formats in $module/target/surefire-reports/*
  • Results are saved in Binary (jacoco format) in $module/target/jacoco.exec


Failsafe INTEGRATION tests

Failsafe will process the Integration Tests = tests that should use Spring test / black box tests / etc.


By default, the Failsafe Plugin will automatically include all test classes with the following wildcard patterns:

  • **/IT*.java - includes all of its subdirectories and all Java filenames that start with "IT".
  • **/*IT.java - includes all of its subdirectories and all Java filenames that end with "IT".
  • **/*ITCase.java - includes all of its subdirectories and all Java filenames that end with "ITCase".

If the test classes do not follow any of these naming conventions, then configure Failsafe Plugin and specify the tests you want to include.


Results are saved in each Maven module:

  • Results are saved in XML and TXT formats in $module/target/failsafe-reports/*
  • Results are saved in BINARY (jacoco format) in $module/target/jacoco-it.exec


Notes:

  • No tests will be run if mvn clean install -DskipIntegrationTests
  • The exclusion | inclusion list is something that depends on your own situation


    <build>
        <plugins>

            <!-- To run INTEGRATION tests and generate execution reports. These reports are required for SonarQube -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>2.22.1</version>
                <configuration>
                    <includes>
                        <include>**/*TestAPI</include>
                        <include>**/*IntegrationTest</include>
                    </includes>
                </configuration>
                <executions>
                    <execution>
                        <id>default-integration-test</id>
                        <goals>
                            <goal>integration-test</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            ...
        </plugins>
    </build>




SONAR maven plugin

SonarQube released a maven plugin to work with SONAR. This plugin already includes most of the configuration and default behavior is correct.

  • Sonar will read Jacoco "exec" binaries files
  • Sonar will process the Surefire & Failsafe reports



Source code POM example

    <properties>
        <!-- Project configuration -->
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>

        <!-- Sonar -->
        <!-- Tell sonar where to look for the binaries coverage files. Property inherited by submodules -->
        <sonar.jacoco.reportPaths>${project.basedir}/../target/jacoco.exec</sonar.jacoco.reportPaths>
    </properties>


    <build>
        <plugins>
            <!-- Build settings -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.3</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>

            <!-- Always generate the source, it is better for debugging -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-source-plugin</artifactId>
                <version>3.0.1</version>
                <executions>
                    <execution>
                        <id>attach-sources</id>
                        <goals>
                            <goal>jar-no-fork</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>attach-test-sources</id>
                        <goals>
                            <goal>test-jar-no-fork</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <!-- To run UNIT tests and generate execution reports. These reports are required for SonarQube -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
                <configuration>
                    <excludes>
                        <exclude>**/*TestAPI</exclude>
                        <exclude>**/*IntegrationTest</exclude>
                    </excludes>
                </configuration>
            </plugin>

            <!-- To run INTEGRATION tests and generate execution reports. These reports are required for SonarQube -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>2.22.1</version>
                <configuration>
                    <includes>
                        <include>**/*TestAPI</include>
                        <include>**/*IntegrationTest</include>
                    </includes>
                </configuration>
                <executions>
                    <execution>
                        <id>default-integration-test</id>
                        <goals>
                            <goal>integration-test</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <!-- JACOCO test coverage plugin.
                 Use it to compile SUREFIRE (unit tests) and FAILSAFE (integration tests) reports for SonarQube
                 (i) attach that plugin to Maven TEST phase
                 Reports are generated in "${project.build.directory}/site/jacoco/*" by default
                 Good documentations:
                    https://wiki.onap.org/display/DW/Implementing+Code+Coverage
                    https://www.eclemma.org/jacoco/trunk/doc/maven.html
                 -->
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>0.8.3</version>
                <configuration>
                    <append>true</append>
                </configuration>
                <executions>
                    <!-- ## UNIT TESTS ## -->
                    <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven SUREFIRE plugin is executed. -->
                    <execution>
                        <id>pre-unit-tests</id>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                    </execution>
                    <!-- Create reports -->
                    <execution>
                        <id>report-unit-tests</id>
                        <goals>
                            <goal>report</goal>
                        </goals>
                    </execution>
                    <!-- ## INTEGRATION TESTS ## -->
                    <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven FAILSAFE plugin is executed. -->
                    <execution>
                        <id>pre-integration-tests</id>
                        <goals>
                            <goal>prepare-agent-integration</goal>
                        </goals>
                        <configuration>
                            <destFile>${project.basedir}/target/jacoco-it.exec</destFile>
                            <append>true</append>
                        </configuration>
                    </execution>
                    <!-- Create reports -->
                    <execution>
                        <id>report-integration-tests</id>
                        <goals>
                            <goal>report-integration</goal>
                        </goals>
                    </execution>
                    <!-- ## MERGE UNIT TESTS reports ## -->
                    <!-- This will merge all unit tests reports into 1 so it can be considered as another "integration" tests report -->
                    <execution>
                        <id>merge</id>
                        <goals>
                            <goal>merge</goal>
                        </goals>
                        <configuration>
                            <fileSets>
                                <fileSet implementation="org.apache.maven.shared.model.fileset.FileSet">
                                    <directory>${project.basedir}</directory>
                                    <includes>
                                        <include>**/*.exec</include>
                                    </includes>
                                </fileSet>
                            </fileSets>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <!-- SonarQube engine -->
            <plugin>
                <groupId>org.sonarsource.scanner.maven</groupId>
                <artifactId>sonar-maven-plugin</artifactId>
                <version>3.6.0.1398</version>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        ...
  
        <!-- jUnit -->
        <dependency>
             <groupId>junit</groupId>
             <artifactId>junit</artifactId>
             <version>${junit.version}</version>
             <scope>test</scope>
        </dependency>      
    </dependencies>



Maven / Sonar key points

Maven properties

POM.xml extract:

    <properties>
                <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>    
                <junit.version>4.12</junit.version>

                <!-- ================================================================= -->
                <!-- ==== FAILSAFE (IT tests) + SUREFIRE (Unit tests) + SONARQUBE ==== -->
                <!-- ================================================================= -->
                <maven-surefire-plugin.version>2.18.1</maven-surefire-plugin.version>
                <maven-failsafe-plugin.version>2.18.1</maven-failsafe-plugin.version>
                <maven.jacoco.version>0.7.7.201606060606</maven.jacoco.version>        
                
                <!-- ========================================== -->
                <!-- Global Sonar settings. Do not change them! -->
                <!-- ========================================== -->
                <sonar.language>java</sonar.language>
                <sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
                <sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
                <jacoco.lib.path>
                    ${settings.localRepository}/org/jacoco/org.jacoco.agent/${maven.jacoco.version}/org.jacoco.agent-${maven.jacoco.version}-runtime.jar
                </jacoco.lib.path>
                <javaagent>${jacoco.lib.path}</javaagent>
                <!-- Don't let Sonar execute tests! Maven will run them and produce some reports for Sonar -->
                <sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
                
                <!-- =============== -->
                <!-- Quality reports -->
                <!-- =============== -->
                <!-- XML reports (go to $/target/site/*-reports) -->
                <sonar.surefire.reportsPath>${project.reporting.outputDirectory}/surefire-reports</sonar.surefire.reportsPath>
                <sonar.failsafe.reportsPath>${project.reporting.outputDirectory}/failsafe-reports</sonar.failsafe.reportsPath>
                <!-- Web reports (= go to $/target/site/jacoco-*/index.html to browse results) -->
                <jacoco.ut.summary>${project.reporting.outputDirectory}/jacoco-ut</jacoco.ut.summary>
                <jacoco.it.summary>${project.reporting.outputDirectory}/jacoco-it</jacoco.it.summary>
                <!-- Binary format [= Sonar format] -->
                <!--    1) Unit tests reports must be in each module $/target/ project -->
                <jacoco.ut.execution.data.file>${project.build.directory}/jacoco.exec</jacoco.ut.execution.data.file>
                <!--    2) Integration tests reports must be consolidated together. If not then you will probably lost multi-project coverage -->
                <jacoco.it.execution.data.file>${session.executionRootDirectory}/target/jacoco-it.exec</jacoco.it.execution.data.file>
            </properties>


Jacoco plugin (coverage agent)

This is how you should configure this plugin:

    <build>
        <plugins>
                    <!-- #################### -->
                    <!-- SONAR # CODE METRICS -->
                    <!-- #################### -->
                    <!-- This will auto-generate the right jacoco command for Unit | Integration tests -->
                    <plugin>
                        <groupId>org.jacoco</groupId>
                        <artifactId>jacoco-maven-plugin</artifactId>
                        <version>${maven.jacoco.version}</version>
                        <executions>
                        
                            <!-- #- - - - - - - - - - - - # -->
                            <!-- Unit tests configuration -->
                            <!-- #- - - - - - - - - - - - # -->           
                            <!-- JaCoCo runtime agent which is passed as VM argument when Maven Surefire plugin is executed. -->
                            <execution>
                                <id>pre-unit-test</id>                  
                                <goals>
                                    <goal>prepare-agent</goal>
                                </goals>
                                <configuration>
                                    <!-- Surefire (Unit tests) reports will be saved as XML in... -->
                                    <destFile>${jacoco.ut.execution.data.file}</destFile>
                                    <propertyName>surefireArgLine</propertyName>
                                    <append>true</append>
                                </configuration>
                            </execution>
                             <!-- Ensures code coverage report for unit tests is created after unit tests have been run. -->
                            <execution>
                                <id>post-unit-test</id>
                                <phase>test</phase>
                                <goals>
                                    <goal>report</goal>
                                </goals>
                                <configuration>
                                    <!-- Surefire (Unit tests) reports will be saved as Binary in... -->
                                    <dataFile>${jacoco.ut.execution.data.file}</dataFile>
                                    <outputDirectory>${jacoco.ut.summary}</outputDirectory>
                                    <append>true</append>
                                </configuration>
                            </execution>
                            
                            <!-- #- - - - - - - - - - - - # -->
                            <!-- Integration tests configuration -->
                            <!-- #- - - - - - - - - - - - # -->
                            <!-- JaCoCo runtime agent which is passed as VM argument when Maven Failsafe plugin is executed. -->                   
                            <execution>
                                <id>pre-integration-test</id>
                                <phase>pre-integration-test</phase>
                                <goals>
                                    <goal>prepare-agent</goal>
                                </goals>
                                <configuration>
                                    <!-- Failsafe (Integration tests) reports will be saved as XML in... -->
                                    <destFile>${jacoco.it.execution.data.file}</destFile>
                                    <propertyName>failsafeArgLine</propertyName>                            
                                    <append>true</append>                 
                                </configuration>
                            </execution>
                             <!-- Ensures code coverage report for integration tests is created after integration tests have been run. -->
                            <execution>
                                <id>post-integration-test</id>
                                <phase>post-integration-test</phase>
                                <goals>
                                    <goal>report</goal>
                                </goals>
                                <configuration>
                                    <!-- Failsafe (Integration tests) reports will be saved as Binary in... -->
                                    <dataFile>${jacoco.it.execution.data.file}</dataFile>
                                    <outputDirectory>${jacoco.it.summary}</outputDirectory>                         
                                    <append>true</append>                 
                                </configuration>
                            </execution>
                        </executions>
                    </plugin>

            <plugin> Surefire (UT) </plugin>
            <plugin> Failsafe (IT) </plugin>

        </plugins>
    </build>

This configuration force the generation of both XML and Binary reports.



Failsafe (Integration tests)

    <build>
        <plugins>

            <plugin> Maven-jacoco </plugin>
            <plugin> Surefire (UT) </plugin>

                    <!-- == process INTEGRATION tests == -->
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-failsafe-plugin</artifactId>
                        <version>${maven-failsafe-plugin.version}</version>              
                        <configuration>                        
                            <!-- Sets the VM argument line used when integration tests are run. -->
                            <argLine>-Xms256m -Xmx1G -XX:PermSize=256m -XX:MaxPermSize=512m ${failsafeArgLine}</argLine>
                            <!-- Skip integration tests if asked -->
                            <skipTests>${skipIntegrationTests}</skipTests>
                            <includes>
                                <include>**/*IntegrationTest.java</include>
                                <include>**/*IT.java</include>                         
                            </includes>      
                            <!-- Excludes GWT crap -->
                            <excludes>
                                <exclude>**/*GWT*.java</exclude>
                            </excludes>                   
                            <!-- Save reports in particular directory -->
                            <reportsDirectory>${sonar.failsafe.reportsPath}</reportsDirectory>
                        </configuration>
                        <!-- Only run failsafe when required -->
                        <executions>
                            <execution>
                                <id>integration-tests</id>
                                <goals>
                                    <goal>integration-test</goal>
                                    <goal>verify</goal>
                                </goals>
                            </execution>
                        </executions>
                    </plugin> 

        </plugins>
    </build>



Maven dependencies

At last, you need to add the jacoco dependency:

(i) 2016-07-19 # this step is NOT required anymore on latest Jenkins & Sonarqube


Maven profile

Put all the following elements inside a Maven profile:

        <profile>
            <id>sonar</id>
            <activation>
                    <activeByDefault>false</activeByDefault>
            </activation>

            <properties> ... </properties>
            <build> <plugins> ... </plugins> </build>
            <dependencies> ... </dependencies>
        </profile>
    </profiles>

To use it:

  • Standard use = mvn clean install -P sonar
  • Skip Unit tests = mvn clean install -DskipTests -P sonar
  • Skip Integration tests = mvn clean install -DskipIntegrationTests -P sonar


Jenkins configuration

Now that you can generate these reports, especially the integration results, you need to configure Jenkins to use them!

Jenkins SonarQube configuration


To set up the Sonar Jenkins plugin to process already generated JaCoCo data files, the following properties must be specified in the "Additional properties" text box, under the Sonar installation being configured:

  • -Dsonar.dynamicAnalysis=reuseReports ==> Tells SonarQube to reuse existing reports for unit tests execution and coverage reports
  • -Dsonar.jacoco.itReportPath=${ABSOLUTE_PATH_TO_JENKINS}/workspace/${JENKINS_JOB}/target/reports/jacoco-it.exec => integration tests report to process


Real life example

You can download a multi-project POM example: http://www.daxiongmao.eu/wiki_upload_files/code/sonar/pom.xml


You can check out these excellent tutorials:

Appendices

Run tests in specific order

If for some reasons you need to run your tests in a particular order you can say do so in Maven and/or Java

Reminder : if you have to run your tests in a particular order then you should refactor your tests! These tricks should not let you escape from that!


Maven

The runOrder is a nice trick.

    <build>
        <plugins>
            <plugin>
                SUREFIRE or FAILSAFE

                <configuration>
                    <!-- JVM settings -->
                    <argLine>-Xmx1024m -XX:maxPermSize:256m</argLine>
                    <forkCount>1</forkCount>
                    <reuseForks>true</reuseForks>
                    <runOrder>alphabetical</runOrder>
                    <!-- Jacoco execution -->                 
                    <argLine>${jacoco.agent.it.arg}</argLine>
                    ...
                </configuration>
            </plugin>

        </plugins>
    </build>


Java

Since jUnit 4.11 you can set the order of your tests.


References

This article is based on my daily work in VEHCO + European Parliament.

To update to SonarQube I used the following articles and code examples:

Jenkins configuration:

StackOverflow