Difference between revisions of "Sonar + maven configuration + Jenkins"

 
(35 intermediate revisions by the same user not shown)
Line 8: Line 8:
 
=Requirements=
 
=Requirements=
  
 +
* Testing libraries: '''jUnit 4.12 or +''' / optional, for static class mocking: '''Powermock 1.7.x or +'''
 
* Maven must run on '''Java 8 or +'''  
 
* Maven must run on '''Java 8 or +'''  
 
* You must use '''Sonarqube 7.x or +'''
 
* You must use '''Sonarqube 7.x or +'''
 +
* Jenkins must be '''v2.164.1 or +''' (LTS version)
  
  
=Maven parent POM=
+
=Principle=
 +
 
 +
TODO
  
 
You just have to define the configuration once, in the parent POM.
 
You just have to define the configuration once, in the parent POM.
 +
 +
* Jenkins
 +
** Requirements: plugins / setup
 +
** Job configuration
 +
* Maven plugins list
 +
** Reports generation : unit / integration tests
 +
** Jacoco binaries : many unit tests "jacoco.exec" / 1 single aggregation for "jacoco-it.exec"
 +
** SonarQube plugin
 +
* Jenkins usage
 +
 +
  
  
==Concepts==
+
='''Maven plugins'''=
  
  
==Source code==
+
==Surefire UNIT Tests==
 +
 
 +
Surefire is for '''UNIT tests''' = tests that should only use: plain java / ''Mockito'' / ''PowerMock'' / etc.
 +
 
 +
 
 +
By default, the Surefire Plugin will automatically '''include''' all test classes with the following wildcard '''patterns''':
 +
* <code>**/Test*.java</code> - includes all of its subdirectories and all Java filenames that start with "Test".
 +
* <code>**/*Test.java</code> - includes all of its subdirectories and all Java filenames that end with "Test".
 +
* <code>**/*Tests.java</code> - includes all of its subdirectories and all Java filenames that end with "Tests".
 +
* <code>**/*TestCase.java</code> - includes all of its subdirectories and all Java filenames that end with "TestCase".
 +
If the test classes do not follow any of these naming conventions, then configure Surefire Plugin and specify the tests you want to include.
 +
 
 +
See [https://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html Maven Surefire plugin documentation]
 +
 
 +
 
 +
(i) It is safer to '''exclude''' the integration tests '''patterns''' as well, depending on your own development setup:
 +
* <code>**/*TestAPI.java</code>
 +
* <code>**/*IntegrationTest.java</code>
 +
 
 +
 
 +
Results are saved '''in each Maven module''', in XML and TXT formats: <code>$module/target/surefire-reports/*</code>
 +
 
  
 
<syntaxhighlight lang="xml">
 
<syntaxhighlight lang="xml">
  
 
     <properties>
 
     <properties>
         <!-- Project configuration -->
+
         <!-- Set the reporting settings -->
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
 
 
         <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
 
         <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
 
 
        <!-- Sonar -->
 
        <!-- Tell sonar where to look for the binaries coverage files. Property inherited by submodules -->
 
        <sonar.jacoco.reportPaths>${project.basedir}/../target/jacoco.exec</sonar.jacoco.reportPaths>
 
 
     </properties>
 
     </properties>
  
Line 38: Line 68:
 
     <build>
 
     <build>
 
         <plugins>
 
         <plugins>
             <!-- Build settings -->
+
             <!-- To run UNIT tests and generate execution reports. These reports are required for SonarQube -->
 
             <plugin>
 
             <plugin>
 
                 <groupId>org.apache.maven.plugins</groupId>
 
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-compiler-plugin</artifactId>
+
                 <artifactId>maven-surefire-plugin</artifactId>
                 <version>3.3</version>
+
                 <version>2.22.1</version>
 
                 <configuration>
 
                 <configuration>
                     <source>${java.version}</source>
+
                     <excludes>
                    <target>${java.version}</target>
+
                        <exclude>**/*TestAPI</exclude>
 +
                        <exclude>**/*IntegrationTest</exclude>
 +
                    </excludes>
 
                 </configuration>
 
                 </configuration>
 
             </plugin>
 
             </plugin>
  
             <!-- Always generate the source, it is better for debugging -->
+
             ...
            <plugin>
+
        </plugins>
                <groupId>org.apache.maven.plugins</groupId>
+
    </build>
                <artifactId>maven-source-plugin</artifactId>
+
</syntaxhighlight>
                <version>3.0.1</version>
+
 
                <executions>
+
 
                    <execution>
+
 
                        <id>attach-sources</id>
+
 
                        <goals>
+
==Failsafe INTEGRATION tests==
                            <goal>jar-no-fork</goal>
+
 
                        </goals>
+
Failsafe will process the '''Integration Tests''' = tests that should use Spring test / black box tests / etc.
                    </execution>
+
 
                    <execution>
+
 
                        <id>attach-test-sources</id>
+
By default, the Failsafe Plugin will automatically include all test classes with the following wildcard patterns:
                        <goals>
+
* <code>**/IT*.java</code> - includes all of its subdirectories and all Java filenames that start with "IT".
                            <goal>test-jar-no-fork</goal>
+
* <code>**/*IT.java</code> - includes all of its subdirectories and all Java filenames that end with "IT".
                        </goals>
+
* <code>**/*ITCase.java</code> - includes all of its subdirectories and all Java filenames that end with "ITCase".
                    </execution>
+
If the test classes do not follow any of these naming conventions, then configure Failsafe Plugin and specify the tests you want to include.
                </executions>
+
 
            </plugin>
+
 
 +
Results are saved '''in each Maven module''':
 +
* Results are saved in XML and TXT formats in <code>$module/target/failsafe-reports/*</code>
 +
* Results are saved in BINARY (jacoco format) in <code>$module/target/jacoco-it.exec</code>
 +
 
 +
 
 +
Notes:
 +
* No tests will be run if <code>mvn clean install -DskipIntegrationTests</code>
 +
* The exclusion | inclusion list is something that depends on your own situation
 +
 
 +
 
 +
<syntaxhighlight lang="xml">
 +
 
 +
    <properties>
 +
        <!-- Set the reporting settings -->
 +
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
 +
    </properties>
 +
 
  
            <!-- To run UNIT tests and generate execution reports. These reports are required for SonarQube -->
+
    <build>
            <plugin>
+
        <plugins>
                <groupId>org.apache.maven.plugins</groupId>
 
                <artifactId>maven-surefire-plugin</artifactId>
 
                <version>2.22.1</version>
 
            </plugin>
 
  
 
             <!-- To run INTEGRATION tests and generate execution reports. These reports are required for SonarQube -->
 
             <!-- To run INTEGRATION tests and generate execution reports. These reports are required for SonarQube -->
Line 85: Line 130:
 
                     <includes>
 
                     <includes>
 
                         <include>**/*TestAPI</include>
 
                         <include>**/*TestAPI</include>
 +
                        <include>**/*IntegrationTest</include>
 
                     </includes>
 
                     </includes>
 
                 </configuration>
 
                 </configuration>
Line 96: Line 142:
 
                 </executions>
 
                 </executions>
 
             </plugin>
 
             </plugin>
 +
 +
            ...
 +
        </plugins>
 +
    </build>
 +
</syntaxhighlight>
 +
 +
 +
 +
 +
==Jacoco code coverage==
 +
 +
Jacoco is a plugin that will:
 +
# Parse SUREFIRE and FAILSAFE results
 +
# Extract and generate code coverage reports (XML format)
 +
# Convert the XML into binaries (*.exec) for SonarQube
 +
 +
 +
Jacoco will generate the following binaries reports:
 +
* '''1 unit test report per project''', for every module: <code>$parent/$module/target/jacoco.exec</code>
 +
* '''1 centralized integration test report''': <code>$parent/target/jacoco-it.exec</code>
 +
 +
 +
 +
<syntaxhighlight lang="xml">
 +
 +
    <properties>
 +
        <!-- Project configuration -->
 +
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
 +
        <!-- Sonar -->
 +
        <!-- Tell sonar where to look for the binaries coverage files. Property inherited by submodules -->
 +
        <sonar.junit.reportPaths>${project.build.directory}/surefire-reports</sonar.junit.reportPaths>
 +
        <sonar.jacoco.reportPath>${project.build.directory}/jacoco.exec</sonar.jacoco.reportPath>
 +
        <sonar.coverage.jacoco.xmlReportPaths>${project.build.directory}/site/jacoco/jacoco.xml</sonar.coverage.jacoco.xmlReportPaths>
 +
        <!-- Integration tests -->
 +
        <sonar.jacoco.itReportPath>${project.build.directory}/jacoco-it.exec</sonar.jacoco.itReportPath>
 +
    </properties>
 +
 +
 +
    <build>
 +
        <plugins>
 +
            ...
  
 
             <!-- JACOCO test coverage plugin.
 
             <!-- JACOCO test coverage plugin.
Line 103: Line 190:
 
                 Good documentations:
 
                 Good documentations:
 
                     https://wiki.onap.org/display/DW/Implementing+Code+Coverage
 
                     https://wiki.onap.org/display/DW/Implementing+Code+Coverage
 +
                    https://www.devcon5.ch/en/blog/2015/05/29/multi-module-integration-test-coverage-sonar-jacoco/
 
                     https://www.eclemma.org/jacoco/trunk/doc/maven.html
 
                     https://www.eclemma.org/jacoco/trunk/doc/maven.html
 
                 -->
 
                 -->
Line 111: Line 199:
 
                 <configuration>
 
                 <configuration>
 
                     <append>true</append>
 
                     <append>true</append>
 +
                    <!-- Use offline bytecode (with powermock changes) -->
 +
                    <excludes>
 +
                        <exclude>*</exclude>
 +
                    </excludes>
 
                 </configuration>
 
                 </configuration>
 
                 <executions>
 
                 <executions>
 +
                    <!-- Support for PowerMock tests -->
 +
                    <!-- See https://www.igorkromin.net/index.php/2018/03/06/quick-look-at-jacoco-vs-cobertura-performance-and-coverage-results/ -->
 +
                    <execution>
 +
                        <id>jacoco-instrument</id>
 +
                        <goals>
 +
                            <goal>instrument</goal>
 +
                        </goals>
 +
                    </execution>
 +
                    <execution>
 +
                        <id>jacoco-restore-instrumented-classes</id>
 +
                        <goals>
 +
                            <goal>restore-instrumented-classes</goal>
 +
                        </goals>
 +
                    </execution>
 +
 
                     <!-- ## UNIT TESTS ## -->
 
                     <!-- ## UNIT TESTS ## -->
 
                     <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven SUREFIRE plugin is executed. -->
 
                     <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven SUREFIRE plugin is executed. -->
Line 136: Line 243:
 
                         </goals>
 
                         </goals>
 
                         <configuration>
 
                         <configuration>
                             <destFile>${project.basedir}/target/jacoco-it.exec</destFile>
+
                            <!-- Only 1 destination file to aggregate ALL integration tests reports -->
 +
                            <!-- the "session.executionRootDirectory" = parent folder that is being build by Jenkins -->
 +
                             <destFile>${session.executionRootDirectory}/target/jacoco-it.exec</destFile>
 
                             <append>true</append>
 
                             <append>true</append>
 
                         </configuration>
 
                         </configuration>
Line 146: Line 255:
 
                             <goal>report-integration</goal>
 
                             <goal>report-integration</goal>
 
                         </goals>
 
                         </goals>
                    </execution>
 
                    <!-- ## MERGE UNIT TESTS reports ## -->
 
                    <!-- This will merge all unit tests reports into 1 so it can be considered as another "integration" tests report -->
 
                    <execution>
 
                        <id>merge</id>
 
                        <goals>
 
                            <goal>merge</goal>
 
                        </goals>
 
                        <configuration>
 
                            <fileSets>
 
                                <fileSet implementation="org.apache.maven.shared.model.fileset.FileSet">
 
                                    <directory>${project.basedir}</directory>
 
                                    <includes>
 
                                        <include>**/*.exec</include>
 
                                    </includes>
 
                                </fileSet>
 
                            </fileSets>
 
                        </configuration>
 
 
                     </execution>
 
                     </execution>
 
                 </executions>
 
                 </executions>
 
             </plugin>
 
             </plugin>
  
            <!-- SonarQube engine -->
 
            <plugin>
 
                <groupId>org.sonarsource.scanner.maven</groupId>
 
                <artifactId>sonar-maven-plugin</artifactId>
 
                <version>3.6.0.1398</version>
 
            </plugin>
 
 
         </plugins>
 
         </plugins>
 
     </build>
 
     </build>
 
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 183: Line 267:
  
  
=Maven / Sonar key points=
+
==SonarQube plugin==
  
 +
SonarQube released a maven plugin to work with SONAR. This plugin already includes most of the configuration and default behavior is correct.
 +
* Sonar will read Jacoco "exec" binaries files
 +
* Sonar will process the ''Surefire'' & ''Failsafe'' reports
  
==Maven key properties==
 
  
Here are some properties you must know and use to configure Sonar:
+
<syntaxhighlight lang="xml">
* <code>${project.reporting.outputDirectory}</code> == module/target/site/
 
* <code>${project.build.directory}</code> == module/target/
 
* <code>${session.executionRootDirectory}</code> == root folder where <code>mvn clean install</code> is launched
 
* <code><project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding></code>
 
  
 +
    <properties>
 +
        <!-- Sonar -->
 +
        <sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
 +
        <sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
 +
        <sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
 +
    </properties>
  
==Sonar properties==
 
  
First of all you need to configure SONAR by setting up some properties and paths, see ''Surefire'' and ''Failsafe'' below.
+
    <build>
 +
        <plugins>
 +
            ..
  
To optimize SONAR execution we will set the following behaviour:
+
            <!-- SonarQube engine -->
* Language == Java
+
            <plugin>
* Coverage tool == JaCoCo
+
                <groupId>org.sonarsource.scanner.maven</groupId>
* Behaviour == re-use Surefire and Failsafe reports. Do NOT re-compile for Sonar.
+
                <artifactId>sonar-maven-plugin</artifactId>
 +
                <!-- Do not forget to change version in JenkinsFile as well -->
 +
                <version>3.6.0.1398</version>
 +
            </plugin>
 +
        </plugins>
 +
    </build>
 +
</syntaxhighlight>
  
  
==Surefire Unit Tests==
 
  
Results are saved '''in each Maven module''':
 
* Results are saved in <code>XML</code> format in <code>$module/target/site/surefire-reports</code>
 
* Results are saved in <code>HTML</code> format in <code>$module/target/site/jacoco-ut</code>. You can open <code>index.html</code> to browse the results.
 
* Results are saved in <code>Binary (jacoco format)</code> in <code>$module/target/jacoco.exec</code>
 
  
  
==Failsafe Integration tests==
+
='''Maven complete POM''' example=
  
Only XML and HTML results are saved in each Maven module. '''Consolidate results are saved in the execution path ./target'''
+
This is how the '''parent POM''' should look like to use SonarQube with correct coverage in dedicated PROFILE.  
* Results are saved in <code>XML</code> format in <code>$module/target/site/failsafe-reports</code>
 
* Results are saved in <code>HTML</code> format in <code>$module/target/site/jacoco-it</code>. You can open <code>index.html</code> to browse the results.
 
* Results are saved in <code>Binary (jacoco format)</code> in <code>$maven_execution_place/target/jacoco-it.exec</code>
 
  
 +
To execute: <code>mvn clean install -Pquality_control</code>
  
  
=Maven '''properties'''=
+
<syntaxhighlight lang="xhtml">
  
POM.xml extract:
+
    <properties>
 +
        <!-- Project configuration -->
 +
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
 +
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
 +
        <java.version>1.8</java.version>
 +
    </properties>
  
<syntaxhighlight lang="xml">
+
    <dependencies>
    <properties>
+
        ... 
                <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>  
+
        <!-- jUnit -->
                <junit.version>4.12</junit.version>
+
        <dependency>
 +
            <groupId>junit</groupId>
 +
            <artifactId>junit</artifactId>
 +
            <version>${junit.version}</version>
 +
            <scope>test</scope>
 +
        </dependency>     
 +
    </dependencies>
  
                 <!-- ================================================================= -->
+
    <profiles>
                <!-- ==== FAILSAFE (IT tests) + SUREFIRE (Unit tests) + SONARQUBE ==== -->
+
        <profile>
                 <!-- ================================================================= -->
+
            <id>quality_control</id>
                 <maven-surefire-plugin.version>2.18.1</maven-surefire-plugin.version>
+
            <activation>
                <maven-failsafe-plugin.version>2.18.1</maven-failsafe-plugin.version>
+
                 <activeByDefault>true</activeByDefault>
                 <maven.jacoco.version>0.7.7.201606060606</maven.jacoco.version>       
+
            </activation>
               
+
            <properties>
                <!-- ========================================== -->
+
                 <!-- SonarQube and Jacoco tests reports -->
                 <!-- Global Sonar settings. Do not change them! -->
+
                 <surefire.version>2.22.2</surefire.version>
                <!-- ========================================== -->
+
                 <jacoco.version>0.8.3</jacoco.version>
 +
                 <!-- Sonar global configuration -->
 
                 <sonar.language>java</sonar.language>
 
                 <sonar.language>java</sonar.language>
 +
                <sonar.sourceEncoding>${project.reporting.outputEncoding}</sonar.sourceEncoding>
 
                 <sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
 
                 <sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
 
                 <sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
 
                 <sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
                <jacoco.lib.path>
 
                    ${settings.localRepository}/org/jacoco/org.jacoco.agent/${maven.jacoco.version}/org.jacoco.agent-${maven.jacoco.version}-runtime.jar
 
                </jacoco.lib.path>
 
                <javaagent>${jacoco.lib.path}</javaagent>
 
                <!-- Don't let Sonar execute tests! Maven will run them and produce some reports for Sonar -->
 
 
                 <sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
 
                 <sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
               
+
                 <!-- Dependencies checks (OWASP) reports -->
                 <!-- =============== -->
+
                 <!-- sonar.dependencyCheck.reportPath>${project.build.directory}/dependency-check-report.xml</sonar.dependencyCheck.reportPath -->
                <!-- Quality reports -->
+
                 <!-- Tell sonar where to look for the UNIT coverage files. Property inherited by submodules -->
                 <!-- =============== -->
+
                 <sonar.junit.reportPaths>${project.build.directory}/surefire-reports</sonar.junit.reportPaths>
                <!-- XML reports (go to $/target/site/*-reports) -->
+
                 <sonar.jacoco.reportPath>${project.build.directory}/jacoco.exec</sonar.jacoco.reportPath>
                <sonar.surefire.reportsPath>${project.reporting.outputDirectory}/surefire-reports</sonar.surefire.reportsPath>
+
                 <sonar.coverage.jacoco.xmlReportPaths>${project.build.directory}/site/jacoco/jacoco.xml</sonar.coverage.jacoco.xmlReportPaths>
                <sonar.failsafe.reportsPath>${project.reporting.outputDirectory}/failsafe-reports</sonar.failsafe.reportsPath>
+
                 <!-- Integration tests -->
                 <!-- Web reports (= go to $/target/site/jacoco-*/index.html to browse results) -->
+
                 <sonar.jacoco.itReportPath>${project.build.directory}/jacoco-it.exec</sonar.jacoco.itReportPath>
                 <jacoco.ut.summary>${project.reporting.outputDirectory}/jacoco-ut</jacoco.ut.summary>
 
                 <jacoco.it.summary>${project.reporting.outputDirectory}/jacoco-it</jacoco.it.summary>
 
                 <!-- Binary format [= Sonar format] -->
 
                <!--    1) Unit tests reports must be in each module $/target/ project -->
 
                <jacoco.ut.execution.data.file>${project.build.directory}/jacoco.exec</jacoco.ut.execution.data.file>
 
                 <!--   2) Integration tests reports must be consolidated together. If not then you will probably lost multi-project coverage -->
 
                 <jacoco.it.execution.data.file>${session.executionRootDirectory}/target/jacoco-it.exec</jacoco.it.execution.data.file>
 
 
             </properties>
 
             </properties>
</syntaxhighlight>
 
  
 +
            <!-- To generate units and integrations test reports -->
 +
            <build>
 +
                <plugins>
 +
                    <!-- To run UNIT tests and generate execution reports. These reports are required for SonarQube -->
 +
                    <plugin>
 +
                        <groupId>org.apache.maven.plugins</groupId>
 +
                        <artifactId>maven-surefire-plugin</artifactId>
 +
                        <version>${surefire.version}</version>
 +
                        <configuration>
 +
                            <excludes>
 +
                                <exclude>**/*TestAPI</exclude>
 +
                                <exclude>**/*IntegrationTest</exclude>
 +
                            </excludes>
 +
                        </configuration>
 +
                    </plugin>
  
 +
                    <!-- To run INTEGRATION tests and generate execution reports. These reports are required for SonarQube -->
 +
                    <plugin>
 +
                        <groupId>org.apache.maven.plugins</groupId>
 +
                        <artifactId>maven-failsafe-plugin</artifactId>
 +
                        <version>${surefire.version}</version>
 +
                        <configuration>
 +
                            <includes>
 +
                                <include>**/*TestAPI</include>
 +
                                <include>**/*IntegrationTest</include>
 +
                            </includes>
 +
                        </configuration>
 +
                        <executions>
 +
                            <execution>
 +
                                <id>default-integration-test</id>
 +
                                <goals>
 +
                                    <goal>integration-test</goal>
 +
                                </goals>
 +
                            </execution>
 +
                        </executions>
 +
                    </plugin>
  
='''Jacoco plugin''' (coverage agent)=
+
                    <!-- JACOCO test coverage plugin.
 
+
                        Use it to compile SUREFIRE (unit tests) and FAILSAFE (integration tests) reports for SonarQube
This is how you should configure this plugin:
+
                        (i) attach that plugin to Maven TEST phase
 
+
                        Reports are generated in "${project.build.directory}/site/jacoco/*" by default
<syntaxhighlight lang="xml">
+
                        Good documentations:
    <build>
+
                            https://wiki.onap.org/display/DW/Implementing+Code+Coverage
        <plugins>
+
                            https://www.devcon5.ch/en/blog/2015/05/29/multi-module-integration-test-coverage-sonar-jacoco/
                    <!-- #################### -->
+
                            https://www.eclemma.org/jacoco/trunk/doc/maven.html
                    <!-- SONAR # CODE METRICS -->
+
                        -->
                    <!-- #################### -->
 
                    <!-- This will auto-generate the right jacoco command for Unit | Integration tests -->
 
 
                     <plugin>
 
                     <plugin>
 
                         <groupId>org.jacoco</groupId>
 
                         <groupId>org.jacoco</groupId>
 
                         <artifactId>jacoco-maven-plugin</artifactId>
 
                         <artifactId>jacoco-maven-plugin</artifactId>
                         <version>${maven.jacoco.version}</version>
+
                         <version>${jacoco.version}</version>
 +
                        <configuration>
 +
                            <append>true</append>
 +
                            <!-- Use offline bytecode (with powermock changes) -->
 +
                            <excludes>
 +
                                <exclude>*</exclude>
 +
                            </excludes>
 +
                        </configuration>
 
                         <executions>
 
                         <executions>
                       
+
                             <!-- Support for PowerMock tests -->
                             <!-- #- - - - - - - - - - - - # -->
+
                            <!-- See https://www.igorkromin.net/index.php/2018/03/06/quick-look-at-jacoco-vs-cobertura-performance-and-coverage-results/ -->
                             <!-- Unit tests configuration -->
+
                             <execution>
                            <!-- #- - - - - - - - - - - - # -->          
+
                                <id>jacoco-instrument</id>
                             <!-- JaCoCo runtime agent which is passed as VM argument when Maven Surefire plugin is executed. -->
+
                                <goals>
 +
                                    <goal>instrument</goal>
 +
                                </goals>
 +
                             </execution>
 
                             <execution>
 
                             <execution>
                                 <id>pre-unit-test</id>                
+
                                <id>jacoco-restore-instrumented-classes</id>
 +
                                <goals>
 +
                                    <goal>restore-instrumented-classes</goal>
 +
                                </goals>
 +
                            </execution>
 +
                            <!-- ## UNIT TESTS ## -->
 +
                            <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven SUREFIRE plugin is executed. -->
 +
                            <execution>
 +
                                 <id>pre-unit-tests</id>
 
                                 <goals>
 
                                 <goals>
 
                                     <goal>prepare-agent</goal>
 
                                     <goal>prepare-agent</goal>
 
                                 </goals>
 
                                 </goals>
                                <configuration>
 
                                    <!-- Surefire (Unit tests) reports will be saved as XML in... -->
 
                                    <destFile>${jacoco.ut.execution.data.file}</destFile>
 
                                    <propertyName>surefireArgLine</propertyName>
 
                                    <append>true</append>
 
                                </configuration>
 
 
                             </execution>
 
                             </execution>
                            <!-- Ensures code coverage report for unit tests is created after unit tests have been run. -->
+
                            <!-- Create reports -->
 
                             <execution>
 
                             <execution>
                                 <id>post-unit-test</id>
+
                                 <id>report-unit-tests</id>
                                <phase>test</phase>
 
 
                                 <goals>
 
                                 <goals>
 
                                     <goal>report</goal>
 
                                     <goal>report</goal>
 
                                 </goals>
 
                                 </goals>
                                <configuration>
 
                                    <!-- Surefire (Unit tests) reports will be saved as Binary in... -->
 
                                    <dataFile>${jacoco.ut.execution.data.file}</dataFile>
 
                                    <outputDirectory>${jacoco.ut.summary}</outputDirectory>
 
                                    <append>true</append>
 
                                </configuration>
 
 
                             </execution>
 
                             </execution>
                           
+
                             <!-- ## INTEGRATION TESTS ## -->
                             <!-- #- - - - - - - - - - - - # -->
+
                             <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven FAILSAFE plugin is executed. -->
                             <!-- Integration tests configuration -->
+
                             <execution>
                             <!-- #- - - - - - - - - - - - # -->
+
                                <id>pre-integration-tests</id>
                             <!-- JaCoCo runtime agent which is passed as VM argument when Maven Failsafe plugin is executed. -->                  
+
                                <goals>
 +
                                    <goal>prepare-agent-integration</goal>
 +
                                </goals>
 +
                            </execution>
 +
                             <!-- Create reports -->
 
                             <execution>
 
                             <execution>
                                 <id>pre-integration-test</id>
+
                                 <id>report-integration-tests</id>
                                <phase>pre-integration-test</phase>
 
 
                                 <goals>
 
                                 <goals>
                                     <goal>prepare-agent</goal>
+
                                     <goal>report-integration</goal>
 
                                 </goals>
 
                                 </goals>
                                <configuration>
 
                                    <!-- Failsafe (Integration tests) reports will be saved as XML in... -->
 
                                    <destFile>${jacoco.it.execution.data.file}</destFile>
 
                                    <propertyName>failsafeArgLine</propertyName>                           
 
                                    <append>true</append>               
 
                                </configuration>
 
 
                             </execution>
 
                             </execution>
                            <!-- Ensures code coverage report for integration tests is created after integration tests have been run. -->
+
                            <!-- ## MERGE ALL TESTS reports ## -->
 
                             <execution>
 
                             <execution>
                                 <id>post-integration-test</id>
+
                                 <id>merge</id>
                                <phase>post-integration-test</phase>
 
 
                                 <goals>
 
                                 <goals>
                                     <goal>report</goal>
+
                                     <goal>merge</goal>
 
                                 </goals>
 
                                 </goals>
 
                                 <configuration>
 
                                 <configuration>
                                     <!-- Failsafe (Integration tests) reports will be saved as Binary in... -->
+
                                     <!-- Only 1 destination file to aggregate ALL integration tests reports -->
                                     <dataFile>${jacoco.it.execution.data.file}</dataFile>
+
                                    <!-- the "session.executionRootDirectory" = parent folder that is being build by Jenkins -->
                                     <outputDirectory>${jacoco.it.summary}</outputDirectory>                        
+
                                     <destFile>${session.executionRootDirectory}/target/jacoco-it.exec</destFile>
                                    <append>true</append>                
+
                                     <fileSets>
 +
                                        <fileSet implementation="org.apache.maven.shared.model.fileset.FileSet">
 +
                                            <directory>${project.build.directory}</directory>
 +
                                            <includes>
 +
                                                <include>**/*.exec</include>
 +
                                            </includes>
 +
                                        </fileSet>
 +
                                    </fileSets>
 
                                 </configuration>
 
                                 </configuration>
 
                             </execution>
 
                             </execution>
Line 352: Line 481:
 
                     </plugin>
 
                     </plugin>
  
            <plugin> Surefire (UT) </plugin>
+
                    <!-- SonarQube engine -->
             <plugin> Failsafe (IT) </plugin>
+
                    <plugin>
 +
                        <groupId>org.sonarsource.scanner.maven</groupId>
 +
                        <artifactId>sonar-maven-plugin</artifactId>
 +
                        <!-- Do not forget to change version in JenkinsFile as well -->
 +
                        <version>3.6.0.1398</version>
 +
                    </plugin>
 +
                </plugins>
 +
             </build>
 +
        </profile>
 +
 
 +
 
  
        </plugins>
 
    </build>
 
 
</syntaxhighlight>
 
</syntaxhighlight>
  
This configuration force the generation of both XML and Binary reports.
+
='''Jenkins'''=
  
  
 +
==Jenkins plugins==
  
='''Surefire (Unit tests)'''=
 
  
Surefire will process the Unit Tests.  
+
To work with SonarQube, Jenkins requires the following plugins to be installed and enabled:
 +
* Maven Integration plugin
 +
* SonarQube scanner for Jenkins. See [https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner+for+Jenkins SonarQube documentation]
  
!! Do not use Surefire for your integration tests because Surefire produce different outputs per maven module !!
 
  
 +
Actions:
 +
* Log into Jenkins as an administrator
 +
* Go to '''Manage Jenkin''' > '''Manage plugins'''
  
<syntaxhighlight lang="xml">
 
    <build>
 
        <plugins>
 
  
            <plugin> Maven-jacoco </plugin>
 
  
                    <!-- == process UNIT tests == -->
+
[[File:Jenkins settings plugins 2.PNG|1024px|Jenkins settings # Maven Integration plugin]]
                    <plugin>
+
 
                        <groupId>org.apache.maven.plugins</groupId>
+
[[File:Jenkins settings plugins 1.PNG|1024px|Jenkins settings # SonarQube scanner for Jenkins]]
                        <artifactId>maven-surefire-plugin</artifactId>
+
 
                        <version>${maven-surefire-plugin.version}</version>
+
 
                        <configuration>              
+
 
                            <!-- Sets the VM argument line used when unit tests are run. -->
+
 
                            <argLine>-Xms256m -Xmx1G -XX:PermSize=256m -XX:MaxPermSize=512m ${surefireArgLine}</argLine>
+
==Global configuration==
                            <!-- Skip unit tests if asked -->
+
 
                            <skipTests>${skipTests}</skipTests>
+
* Log into Jenkins as an administrator
                            <!-- Excludes integration tests when unit tests are run. -->
+
* Go to '''Manage Jenkins''' > '''Configure System'''
                            <excludes>
+
* Scroll to the '''SonarQube servers section'''
                                <exclude>**/*IT.java</exclude>
+
* Check ''Enable injection of SonarQube server configuration as build environment variables''
                                <exclude>**/*IntegrationTest.java</exclude>
+
 
                                <exclude>**/*GWT*.java</exclude>
+
 
                             </excludes>
+
[[File:Jenkins settings global.PNG|1024px|Jenkins Global configuration # SonarQube]]
                             <!-- Save reports in particular directory -->
+
 
                             <reportsDirectory>${sonar.surefire.reportsPath}</reportsDirectory>
+
 
                         </configuration>
+
 
                     </plugin>
+
 
 +
==Job configuration==
 +
 
 +
Now that you can generate these reports, especially the integration tests results, you need to configure Jenkins to use them!
 +
 
 +
 
 +
===Maven build (legacy)===
 +
 
 +
This explain how to configure a '''Maven build''' (legacy) with SonarQube Runner.
 +
 
 +
 
 +
In the '''Build Environment''' section > check ''Prepare SonarQube Scanner environment''
 +
 
 +
[[File:Jenkins_build_maven_1.PNG|770px|Jenkins Build configuration #1]]
 +
 
 +
 
 +
 
 +
Then, set the '''Build''' command line:
 +
 
 +
[[File:Jenkins_build_maven_2.PNG|1024px|Jenkins Build configuration #2]]
 +
 
 +
 
 +
Adjust the SonarQube server URL + path to integration tests reports!
 +
 
 +
* Maven build: <code>clean install</code> or <code>clean deploy</code>
 +
* Use SonarQube
 +
** Enable plugin: <code>$SONAR_MAVEN_GOAL</code>  
 +
** Set SonarQube server URL: <code>-Dsonar.host.url=http://localhost:9000/sonarqube</code>
 +
** Give path of Integration tests: <code>-Dsonar.jacoco.itReportPath=/var/lib/jenkins/workspace/Flow.e.r.s/target/jacoco-it.exec</code>
 +
 
 +
 
 +
 
 +
 
 +
=JenkinsFile (Pipelines)=
 +
 
 +
'''Requirements''': have a local SonarQube instance configured in Jenkins, called "''sonarqube''"
 +
 
 +
<syntaxhighlight lang="groovy">
 +
pipeline {
 +
 
 +
    parameters {
 +
        booleanParam(defaultValue: false, description: 'Enable quality control', name: 'qualityControlEnabled')
 +
        booleanParam(defaultValue: false, description: 'Enable Maven dependencies checks (missing declaration / unused libraries / etc.)', name: 'mvnDependenciesChecksEnabled')
 +
        booleanParam(defaultValue: false, description: 'Enable OWASP checks (search for known vulnerabilities in libraries)', name: 'owaspChecksEnabled')
 +
    }
 +
 
 +
 
 +
    stages {
 +
        stage('Checkout') {
 +
            steps {
 +
                checkout scm
 +
            }
 +
        }
 +
 
 +
        stage('MVN dependencies checks') {
 +
            when {
 +
                expression { params.mvnDependenciesChecksEnabled }
 +
            }
 +
            steps {
 +
                withMaven(
 +
                        maven: 'Maven 3.5.2',
 +
                        mavenLocalRepo: '.repository'
 +
                ) {
 +
                    echo "MVN dependencies checks [unused / missing / bad scope / etc.]"
 +
                    sh "mvn org.apache.maven.plugins:maven-dependency-plugin:analyze-report"
 +
                }
 +
            }
 +
        }
 +
 
 +
        stage('OWASP vulnerabilities checks') {
 +
            when {
 +
                expression { params.owaspChecksEnabled }
 +
            }
 +
            steps {
 +
                withMaven(
 +
                        maven: 'Maven 3.5.2',
 +
                        mavenLocalRepo: '.repository'
 +
                ) {
 +
                    echo "OWASP vulnerabilities checks [lib. with security issue(s)]"
 +
                    sh "mvn org.owasp:dependency-check-maven:check"
 +
                    // sh "mvn org.owasp:dependency-check-maven:aggregate"
 +
 
 +
                    // Publish report in Jenkins
 +
                    dependencyCheckPublisher failedTotalHigh: '0', unstableTotalHigh: '1', failedTotalNormal: '2', unstableTotalNormal: '5'
 +
                }
 +
            }
 +
        }
 +
 
 +
        stage('Build') {
 +
            /*
 +
            when {
 +
                // optional: my condition(s)
 +
            }
 +
            */
 +
            steps {
 +
                withMaven(
 +
                        maven: 'Maven 3.5.2',
 +
                        mavenLocalRepo: '.repository'
 +
                ){
 +
                    script {
 +
                        if (params.qualityControlEnabled) {
 +
                            // On human request only
 +
                             echo "Building with QUALITY controls from ${BRANCH_NAME} ..."
 +
                             sh "mvn clean install -Pquality_control"
 +
                        } else {
 +
                            // Default case - for automatic builds
 +
                            echo "Standard build"
 +
                             sh "mvn clean install"
 +
                        }
 +
                    }
 +
                }
 +
            }
 +
        }
 +
 
 +
        stage('SonarQube') {
 +
            when {
 +
                expression { params.qualityControlEnabled }
 +
            }
 +
            steps {
 +
                withMaven(
 +
                        maven: 'Maven 3.5.2',
 +
                         mavenLocalRepo: '.repository'
 +
                ){
 +
                     withSonarQubeEnv('sonarqube') {
 +
                        echo "Static code analysis with SonarQube runner"
 +
                        // TODO ensure the version of the plugin matches the one in the PARENT pom
 +
                        sh "mvn org.sonarsource.scanner.maven:sonar-maven-plugin:3.6.0.1398:sonar -Dsonar.projectName=\"my_project ($BRANCH_NAME)\" -Dsonar.projectKey=my_project:$BRANCH_NAME -Pquality_control"
 +
                    }
 +
                }
 +
            }
 +
        }
 +
    }
  
             <plugin> Failsafe (IT) </plugin>
+
    post {
 +
        always {
 +
             echo 'Cleaning workspace'
 +
            deleteDir()
 +
        }
 +
    }
  
        </plugins>
 
    </build>
 
 
</syntaxhighlight>
 
</syntaxhighlight>
  
  
Notes:
 
* No tests will be run if <code>mvn clean install -DskipTests</code>
 
* The exclusion list is something that depends on your own situation
 
  
 +
See [https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner+for+Jenkins#AnalyzingwithSonarQubeScannerforJenkins-AnalyzinginaJenkinspipeline SonarQube documentation]
 +
 +
='''SonarQube'''=
 +
 +
 +
By default there will be '''1 sonarqube''' entry for every Jenkins '''artifactId'' that is build (parent name).
 +
 +
 +
==How to see the code coverage by module?==
 +
 +
To have a quick glance of the project's modules:
 +
 +
* SonarQube > project > '''Code'''
 +
 +
[[File:Sonarqube results 2.PNG|1024px|SonarQube view results #1]]
  
  
='''Failsafe (Integration tests)'''=
+
* SonarQube > project > '''Measures''' > ''Expand'' '''coverage'' > '''Conditions coverage'''
  
Failsafe will process the Integration Tests.  
+
[[File:Sonarqube results 1.PNG|1024px|SonarQube view results #2]]
  
  
<syntaxhighlight lang="xml">
 
    <build>
 
        <plugins>
 
  
            <plugin> Maven-jacoco </plugin>
+
==How to see measures by module?==
            <plugin> Surefire (UT) </plugin>
 
  
                    <!-- == process INTEGRATION tests == -->
+
By default SonarQube display measures for all the project's modules. This is sometimes not convenient... The old "code" dashboard has been removed, but you can still see the measure by sub-module with a little trick.
                    <plugin>
 
                        <groupId>org.apache.maven.plugins</groupId>
 
                        <artifactId>maven-failsafe-plugin</artifactId>
 
                        <version>${maven-failsafe-plugin.version}</version>             
 
                        <configuration>                       
 
                            <!-- Sets the VM argument line used when integration tests are run. -->
 
                            <argLine>-Xms256m -Xmx1G -XX:PermSize=256m -XX:MaxPermSize=512m ${failsafeArgLine}</argLine>
 
                            <!-- Skip integration tests if asked -->
 
                            <skipTests>${skipIntegrationTests}</skipTests>
 
                            <includes>
 
                                <include>**/*IntegrationTest.java</include>
 
                                <include>**/*IT.java</include>                       
 
                            </includes>     
 
                            <!-- Excludes GWT crap -->
 
                            <excludes>
 
                                <exclude>**/*GWT*.java</exclude>
 
                            </excludes>                 
 
                            <!-- Save reports in particular directory -->
 
                            <reportsDirectory>${sonar.failsafe.reportsPath}</reportsDirectory>
 
                        </configuration>
 
                        <!-- Only run failsafe when required -->
 
                        <executions>
 
                            <execution>
 
                                <id>integration-tests</id>
 
                                <goals>
 
                                    <goal>integration-test</goal>
 
                                    <goal>verify</goal>
 
                                </goals>
 
                            </execution>
 
                        </executions>
 
                    </plugin>
 
  
        </plugins>
 
    </build>
 
</syntaxhighlight>
 
  
 +
1. Select a module to analyze from the "code" menu
  
Notes:
+
[[File:Sonarqube module selection 1.PNG|1024px|SonarQube select module, step 1]]
* No tests will be run if <code>mvn clean install -DskipIntegrationTests</code>
 
* The exclusion | inclusion list is something that depends on your own situation
 
  
  
=Maven '''dependencies'''=
+
2. Notice the SonarQube URL and the selection parameter
  
At last, you need to add the jacoco dependency:
+
[[File:Sonarqube module selection 2.PNG|800px|SonarQube select module, step 2]]
  
(i) 2016-07-19 # this step is NOT required anymore on latest Jenkins & Sonarqube
 
  
<syntaxhighlight lang="xml">
+
3. Copy that <code>&selected=..</code> parameter
    <dependencies>
 
        <!-- jUnit -->
 
        <dependency>
 
            <groupId>junit</groupId>
 
            <artifactId>junit</artifactId>
 
            <version>${junit.version}</version>
 
            <scope>test</scope>
 
        </dependency>    
 
  
        <!-- SonarQube dependencies -->
 
        <dependency>
 
            <groupId>org.jacoco</groupId>
 
            <artifactId>org.jacoco.agent</artifactId>
 
            <version>${maven.jacoco.version}</version>
 
            <classifier>runtime</classifier>
 
            <scope>test</scope>
 
        </dependency>
 
    </dependencies>
 
</syntaxhighlight>
 
  
 +
4. Parse the value on the measures page
  
=Maven '''profile'''=
+
[[File:Sonarqube module selection 3.PNG|925px|SonarQube select module, step 3]]
  
Put all the following elements inside a Maven profile:
 
  
<syntaxhighlight lang="xml">
+
5. All good! You can browse the measures for that particular module.
        <profile>
 
            <id>sonar</id>
 
            <activation>
 
                    <activeByDefault>false</activeByDefault>
 
            </activation>
 
  
            <properties> ... </properties>
 
            <build> <plugins> ... </plugins> </build>
 
            <dependencies> ... </dependencies>
 
        </profile>
 
    </profiles>
 
</syntaxhighlight>
 
  
To use it:
 
* Standard use = <code>mvn clean install -P sonar</code>
 
* Skip Unit tests = <code>mvn clean install -DskipTests -P sonar</code>
 
* Skip Integration tests = <code>mvn clean install -DskipIntegrationTests -P sonar</code>
 
  
 +
==SonarQube plugins==
  
='''Jenkins''' configuration=
+
Go to SonarQube > Administration > Marketplace > "All"
  
Now that you can generate these reports, especially the integration results, you need to configure Jenkins to use them!
 
  
[[File:SonarQube jenkins configuration.png|900px|Jenkins SonarQube configuration]]
+
I advised you to install:  
 +
* Code smells
 +
* FindBugs
 +
* Git
 +
* JaCoCo (mandatory)
 +
* PMD
  
  
To set up the Sonar Jenkins plugin to process already generated JaCoCo data files, the following properties must be specified in the "Additional properties" text box, under the Sonar installation being configured:
 
  
* <code>-Dsonar.dynamicAnalysis=reuseReports</code> ==> Tells SonarQube to reuse existing reports for unit tests execution and coverage reports
+
=How to disable rule(s) in SonarQube=
* <code>-Dsonar.jacoco.itReportPath=${ABSOLUTE_PATH_TO_JENKINS}/workspace/${JENKINS_JOB}/target/reports/jacoco-it.exec</code> => integration tests report to process
 
  
 +
(i) This operation is only for logged user with administrators rights.
  
  
=Real life example=
+
==Quality profile==
  
You can download a multi-project POM example: http://www.daxiongmao.eu/wiki_upload_files/code/sonar/pom.xml
+
First of all you have to create a custom quality profile.  
 +
* Go to '''Quality profiles'''
 +
* Click on the '''wheel''' of the quality profile you'd like to adjust > '''copy'''
 +
* Name the new profile > Click on the '''wheel''' top right > '''set as default'''
  
  
You can check out these excellent tutorials:  
+
[[File:Create quality profile.png|750px|SonarQube # create quality profile]]
* https://github.com/dgageot/coverage
 
* https://github.com/pkainulainen/maven-examples/tree/master/code-coverage-jacoco
 
  
=Appendices=
 
  
  
==Run tests in specific order==
+
==Disable / Enable rule for quality profile==
  
If for some reasons you need to run your tests in a particular order you can say do so in Maven and/or Java
+
* Go to '''Rules'''
 +
* Search for the rule you want to modify
 +
* Click on the rule name
  
'''Reminder''' : if you have to run your tests in a particular order then you should refactor your tests! These tricks should not let you escape from that!
+
[[File:Disable rule 1.PNG|1024px|SonarQube # disable rule screen 1]]
  
 +
* Scroll down to the quality profile view
 +
* Disable rule for particular quality profile(s)
  
===Maven===
+
[[File:Disable rule 2.PNG|1024px|SonarQube # disable rule screen 2]]
  
The <code>runOrder</code> is a nice trick.
 
  
<syntaxhighlight lang="xml">
 
    <build>
 
        <plugins>
 
            <plugin>
 
                SUREFIRE or FAILSAFE
 
  
                <configuration>
 
                    <!-- JVM settings -->
 
                    <argLine>-Xmx1024m -XX:maxPermSize:256m</argLine>
 
                    <forkCount>1</forkCount>
 
                    <reuseForks>true</reuseForks>
 
                    <runOrder>alphabetical</runOrder>
 
                    <!-- Jacoco execution -->               
 
                    <argLine>${jacoco.agent.it.arg}</argLine>
 
                    ...
 
                </configuration>
 
            </plugin>
 
  
        </plugins>
+
=Other=
    </build>
 
</syntaxhighlight>
 
  
 +
==jUnit tests ordering==
  
===Java===
+
Since jUnit 4.11 you can set the order of your tests. '''If you have to use that it means something is probably wrong with your tests design'''!
  
Since jUnit 4.11 you can set the order of your tests.
 
  
 
* Set order to the Class level: https://github.com/junit-team/junit/wiki/Test-execution-order
 
* Set order to the Class level: https://github.com/junit-team/junit/wiki/Test-execution-order
* Set custom order: http://memorynotfound.com/run-junit-tests-method-order/  
+
* Set custom order: http://memorynotfound.com/run-junit-tests-method-order/
 +
 
  
  
Line 588: Line 788:
 
=References=
 
=References=
  
This article is based on my daily work in VEHCO + European Parliament.
+
This article is based on my daily work in VEHCO + European Parliament + LuxTrust.
 +
 
 +
This article is based on a lot of research and code browsing. Following articles are very good:
 +
* [https://wiki.onap.org/display/DW/Implementing+Code+Coverage ONAP wiki: how to setup code coverage for a lot of languages: including java, python & javascript]
 +
* [https://www.devcon5.ch/en/blog/2015/05/29/multi-module-integration-test-coverage-sonar-jacoco/ Multi module integration test coverage with Sonar and Jacoco]
 +
* [https://www.eclemma.org/jacoco/trunk/doc/maven.html Jacoco official documentation]
  
To update to SonarQube I used the following articles and code examples:
+
PowerMock support
* http://www.aheritier.net/maven-failsafe-sonar-and-jacoco-are-in-a-boat/
+
* [https://www.igorkromin.net/index.php/2018/02/20/jacoco-reports-missing-code-coverage-for-tests-using-powermock/ How to include PowerMock tests coverage into JaCoCo]
* VERY good GitHub example, provided by the same author: https://github.com/dgageot/coverage
+
* [https://snmaddula.github.io/tuning-jacoco-for-powermock/ PowerMock with Jacoco]
 +
 
 +
 
 +
Other article that are relevant for older versions of Sonar (versions < 6.x with Java 7)
 
* http://stackoverflow.com/questions/26253277/jacoco-agent-for-integration-tests-on-remote-machine
 
* http://stackoverflow.com/questions/26253277/jacoco-agent-for-integration-tests-on-remote-machine
 
* Nice tutorial: https://www.petrikainulainen.net/programming/maven/creating-code-coverage-reports-for-unit-and-integration-tests-with-the-jacoco-maven-plugin/
 
* Nice tutorial: https://www.petrikainulainen.net/programming/maven/creating-code-coverage-reports-for-unit-and-integration-tests-with-the-jacoco-maven-plugin/
 
Jenkins configuration:
 
* http://blog.ccbill.com/2014/07/code-coverage-with-surefire-and-jacoco.html
 
 
StackOverflow
 
* http://stackoverflow.com/questions/31388856/jacoco-reports-path-not-found-by-sonar-runner
 

Latest revision as of 08:18, 22 May 2019


This article explains how to configure your maven projects to produce reports + how to configure Jenkins with SonarQube, bringing both Unit and Integration tests coverage.


Requirements

  • Testing libraries: jUnit 4.12 or + / optional, for static class mocking: Powermock 1.7.x or +
  • Maven must run on Java 8 or +
  • You must use Sonarqube 7.x or +
  • Jenkins must be v2.164.1 or + (LTS version)


Principle

TODO

You just have to define the configuration once, in the parent POM.

  • Jenkins
    • Requirements: plugins / setup
    • Job configuration
  • Maven plugins list
    • Reports generation : unit / integration tests
    • Jacoco binaries : many unit tests "jacoco.exec" / 1 single aggregation for "jacoco-it.exec"
    • SonarQube plugin
  • Jenkins usage



Maven plugins

Surefire UNIT Tests

Surefire is for UNIT tests = tests that should only use: plain java / Mockito / PowerMock / etc.


By default, the Surefire Plugin will automatically include all test classes with the following wildcard patterns:

  • **/Test*.java - includes all of its subdirectories and all Java filenames that start with "Test".
  • **/*Test.java - includes all of its subdirectories and all Java filenames that end with "Test".
  • **/*Tests.java - includes all of its subdirectories and all Java filenames that end with "Tests".
  • **/*TestCase.java - includes all of its subdirectories and all Java filenames that end with "TestCase".

If the test classes do not follow any of these naming conventions, then configure Surefire Plugin and specify the tests you want to include.

See Maven Surefire plugin documentation


(i) It is safer to exclude the integration tests patterns as well, depending on your own development setup:

  • **/*TestAPI.java
  • **/*IntegrationTest.java


Results are saved in each Maven module, in XML and TXT formats: $module/target/surefire-reports/*


    <properties>
        <!-- Set the reporting settings -->
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>


    <build>
        <plugins>
            <!-- To run UNIT tests and generate execution reports. These reports are required for SonarQube -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
                <configuration>
                    <excludes>
                        <exclude>**/*TestAPI</exclude>
                        <exclude>**/*IntegrationTest</exclude>
                    </excludes>
                </configuration>
            </plugin>

            ...
        </plugins>
    </build>



Failsafe INTEGRATION tests

Failsafe will process the Integration Tests = tests that should use Spring test / black box tests / etc.


By default, the Failsafe Plugin will automatically include all test classes with the following wildcard patterns:

  • **/IT*.java - includes all of its subdirectories and all Java filenames that start with "IT".
  • **/*IT.java - includes all of its subdirectories and all Java filenames that end with "IT".
  • **/*ITCase.java - includes all of its subdirectories and all Java filenames that end with "ITCase".

If the test classes do not follow any of these naming conventions, then configure Failsafe Plugin and specify the tests you want to include.


Results are saved in each Maven module:

  • Results are saved in XML and TXT formats in $module/target/failsafe-reports/*
  • Results are saved in BINARY (jacoco format) in $module/target/jacoco-it.exec


Notes:

  • No tests will be run if mvn clean install -DskipIntegrationTests
  • The exclusion | inclusion list is something that depends on your own situation


    <properties>
        <!-- Set the reporting settings -->
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>


    <build>
        <plugins>

            <!-- To run INTEGRATION tests and generate execution reports. These reports are required for SonarQube -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>2.22.1</version>
                <configuration>
                    <includes>
                        <include>**/*TestAPI</include>
                        <include>**/*IntegrationTest</include>
                    </includes>
                </configuration>
                <executions>
                    <execution>
                        <id>default-integration-test</id>
                        <goals>
                            <goal>integration-test</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            ...
        </plugins>
    </build>



Jacoco code coverage

Jacoco is a plugin that will:

  1. Parse SUREFIRE and FAILSAFE results
  2. Extract and generate code coverage reports (XML format)
  3. Convert the XML into binaries (*.exec) for SonarQube


Jacoco will generate the following binaries reports:

  • 1 unit test report per project, for every module: $parent/$module/target/jacoco.exec
  • 1 centralized integration test report: $parent/target/jacoco-it.exec


    <properties>
        <!-- Project configuration -->
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <!-- Sonar -->
        <!-- Tell sonar where to look for the binaries coverage files. Property inherited by submodules -->
        <sonar.junit.reportPaths>${project.build.directory}/surefire-reports</sonar.junit.reportPaths>
        <sonar.jacoco.reportPath>${project.build.directory}/jacoco.exec</sonar.jacoco.reportPath>
        <sonar.coverage.jacoco.xmlReportPaths>${project.build.directory}/site/jacoco/jacoco.xml</sonar.coverage.jacoco.xmlReportPaths>
        <!-- Integration tests -->
        <sonar.jacoco.itReportPath>${project.build.directory}/jacoco-it.exec</sonar.jacoco.itReportPath>
    </properties>


    <build>
        <plugins>
            ... 

            <!-- JACOCO test coverage plugin.
                 Use it to compile SUREFIRE (unit tests) and FAILSAFE (integration tests) reports for SonarQube
                 (i) attach that plugin to Maven TEST phase
                 Reports are generated in "${project.build.directory}/site/jacoco/*" by default
                 Good documentations:
                    https://wiki.onap.org/display/DW/Implementing+Code+Coverage
                    https://www.devcon5.ch/en/blog/2015/05/29/multi-module-integration-test-coverage-sonar-jacoco/
                    https://www.eclemma.org/jacoco/trunk/doc/maven.html
                 -->
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>0.8.3</version>
                <configuration>
                    <append>true</append>
                    <!-- Use offline bytecode (with powermock changes) -->
                    <excludes>
                        <exclude>*</exclude>
                    </excludes>
                </configuration>
                <executions>
                    <!-- Support for PowerMock tests -->
                    <!-- See https://www.igorkromin.net/index.php/2018/03/06/quick-look-at-jacoco-vs-cobertura-performance-and-coverage-results/ -->
                    <execution>
                        <id>jacoco-instrument</id>
                        <goals>
                            <goal>instrument</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>jacoco-restore-instrumented-classes</id>
                        <goals>
                            <goal>restore-instrumented-classes</goal>
                        </goals>
                    </execution>

                    <!-- ## UNIT TESTS ## -->
                    <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven SUREFIRE plugin is executed. -->
                    <execution>
                        <id>pre-unit-tests</id>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                    </execution>
                    <!-- Create reports -->
                    <execution>
                        <id>report-unit-tests</id>
                        <goals>
                            <goal>report</goal>
                        </goals>
                    </execution>
                    <!-- ## INTEGRATION TESTS ## -->
                    <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven FAILSAFE plugin is executed. -->
                    <execution>
                        <id>pre-integration-tests</id>
                        <goals>
                            <goal>prepare-agent-integration</goal>
                        </goals>
                        <configuration>
                            <!-- Only 1 destination file to aggregate ALL integration tests reports -->
                            <!-- the "session.executionRootDirectory" = parent folder that is being build by Jenkins -->
                            <destFile>${session.executionRootDirectory}/target/jacoco-it.exec</destFile>
                            <append>true</append>
                        </configuration>
                    </execution>
                    <!-- Create reports -->
                    <execution>
                        <id>report-integration-tests</id>
                        <goals>
                            <goal>report-integration</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

        </plugins>
    </build>



SonarQube plugin

SonarQube released a maven plugin to work with SONAR. This plugin already includes most of the configuration and default behavior is correct.

  • Sonar will read Jacoco "exec" binaries files
  • Sonar will process the Surefire & Failsafe reports


    <properties>
        <!-- Sonar -->
        <sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
        <sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
        <sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
    </properties>


    <build>
        <plugins>
            ..

            <!-- SonarQube engine -->
            <plugin>
                <groupId>org.sonarsource.scanner.maven</groupId>
                <artifactId>sonar-maven-plugin</artifactId>
                <!-- Do not forget to change version in JenkinsFile as well -->
                <version>3.6.0.1398</version>
            </plugin>
        </plugins>
    </build>



Maven complete POM example

This is how the parent POM should look like to use SonarQube with correct coverage in dedicated PROFILE.

To execute: mvn clean install -Pquality_control


    <properties>
        <!-- Project configuration -->
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        ...  
        <!-- jUnit -->
        <dependency>
             <groupId>junit</groupId>
             <artifactId>junit</artifactId>
             <version>${junit.version}</version>
             <scope>test</scope>
        </dependency>      
    </dependencies>

    <profiles>
        <profile>
            <id>quality_control</id>
            <activation>
                <activeByDefault>true</activeByDefault>
            </activation>
            <properties>
                <!-- SonarQube and Jacoco tests reports -->
                <surefire.version>2.22.2</surefire.version>
                <jacoco.version>0.8.3</jacoco.version>
                <!-- Sonar global configuration -->
                <sonar.language>java</sonar.language>
                <sonar.sourceEncoding>${project.reporting.outputEncoding}</sonar.sourceEncoding>
                <sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin>
                <sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
                <sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
                <!-- Dependencies checks (OWASP) reports -->
                <!-- sonar.dependencyCheck.reportPath>${project.build.directory}/dependency-check-report.xml</sonar.dependencyCheck.reportPath -->
                <!-- Tell sonar where to look for the UNIT coverage files. Property inherited by submodules -->
                <sonar.junit.reportPaths>${project.build.directory}/surefire-reports</sonar.junit.reportPaths>
                <sonar.jacoco.reportPath>${project.build.directory}/jacoco.exec</sonar.jacoco.reportPath>
                <sonar.coverage.jacoco.xmlReportPaths>${project.build.directory}/site/jacoco/jacoco.xml</sonar.coverage.jacoco.xmlReportPaths>
                <!-- Integration tests -->
                <sonar.jacoco.itReportPath>${project.build.directory}/jacoco-it.exec</sonar.jacoco.itReportPath>
            </properties>

            <!-- To generate units and integrations test reports -->
            <build>
                <plugins>
                    <!-- To run UNIT tests and generate execution reports. These reports are required for SonarQube -->
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-surefire-plugin</artifactId>
                        <version>${surefire.version}</version>
                        <configuration>
                            <excludes>
                                <exclude>**/*TestAPI</exclude>
                                <exclude>**/*IntegrationTest</exclude>
                            </excludes>
                        </configuration>
                    </plugin>

                    <!-- To run INTEGRATION tests and generate execution reports. These reports are required for SonarQube -->
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-failsafe-plugin</artifactId>
                        <version>${surefire.version}</version>
                        <configuration>
                            <includes>
                                <include>**/*TestAPI</include>
                                <include>**/*IntegrationTest</include>
                            </includes>
                        </configuration>
                        <executions>
                            <execution>
                                <id>default-integration-test</id>
                                <goals>
                                    <goal>integration-test</goal>
                                </goals>
                            </execution>
                        </executions>
                    </plugin>

                    <!-- JACOCO test coverage plugin.
                         Use it to compile SUREFIRE (unit tests) and FAILSAFE (integration tests) reports for SonarQube
                         (i) attach that plugin to Maven TEST phase
                         Reports are generated in "${project.build.directory}/site/jacoco/*" by default
                         Good documentations:
                            https://wiki.onap.org/display/DW/Implementing+Code+Coverage
                            https://www.devcon5.ch/en/blog/2015/05/29/multi-module-integration-test-coverage-sonar-jacoco/
                            https://www.eclemma.org/jacoco/trunk/doc/maven.html
                         -->
                    <plugin>
                        <groupId>org.jacoco</groupId>
                        <artifactId>jacoco-maven-plugin</artifactId>
                        <version>${jacoco.version}</version>
                        <configuration>
                            <append>true</append>
                            <!-- Use offline bytecode (with powermock changes) -->
                            <excludes>
                                <exclude>*</exclude>
                            </excludes>
                        </configuration>
                        <executions>
                            <!-- Support for PowerMock tests -->
                            <!-- See https://www.igorkromin.net/index.php/2018/03/06/quick-look-at-jacoco-vs-cobertura-performance-and-coverage-results/ -->
                            <execution>
                                <id>jacoco-instrument</id>
                                <goals>
                                    <goal>instrument</goal>
                                </goals>
                            </execution>
                            <execution>
                                <id>jacoco-restore-instrumented-classes</id>
                                <goals>
                                    <goal>restore-instrumented-classes</goal>
                                </goals>
                            </execution>
                            <!-- ## UNIT TESTS ## -->
                            <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven SUREFIRE plugin is executed. -->
                            <execution>
                                <id>pre-unit-tests</id>
                                <goals>
                                    <goal>prepare-agent</goal>
                                </goals>
                            </execution>
                            <!-- Create reports -->
                            <execution>
                                <id>report-unit-tests</id>
                                <goals>
                                    <goal>report</goal>
                                </goals>
                            </execution>
                            <!-- ## INTEGRATION TESTS ## -->
                            <!-- Configure JaCoCo runtime agent. It is passed as VM argument when Maven FAILSAFE plugin is executed. -->
                            <execution>
                                <id>pre-integration-tests</id>
                                <goals>
                                    <goal>prepare-agent-integration</goal>
                                </goals>
                            </execution>
                            <!-- Create reports -->
                            <execution>
                                <id>report-integration-tests</id>
                                <goals>
                                    <goal>report-integration</goal>
                                </goals>
                            </execution>
                            <!-- ## MERGE ALL TESTS reports ## -->
                            <execution>
                                <id>merge</id>
                                <goals>
                                    <goal>merge</goal>
                                </goals>
                                <configuration>
                                    <!-- Only 1 destination file to aggregate ALL integration tests reports -->
                                    <!-- the "session.executionRootDirectory" = parent folder that is being build by Jenkins -->
                                    <destFile>${session.executionRootDirectory}/target/jacoco-it.exec</destFile>
                                    <fileSets>
                                        <fileSet implementation="org.apache.maven.shared.model.fileset.FileSet">
                                            <directory>${project.build.directory}</directory>
                                            <includes>
                                                <include>**/*.exec</include>
                                            </includes>
                                        </fileSet>
                                    </fileSets>
                                </configuration>
                            </execution>
                        </executions>
                    </plugin>

                    <!-- SonarQube engine -->
                    <plugin>
                        <groupId>org.sonarsource.scanner.maven</groupId>
                        <artifactId>sonar-maven-plugin</artifactId>
                        <!-- Do not forget to change version in JenkinsFile as well -->
                        <version>3.6.0.1398</version>
                    </plugin>
                </plugins>
            </build>
        </profile>

Jenkins

Jenkins plugins

To work with SonarQube, Jenkins requires the following plugins to be installed and enabled:


Actions:

  • Log into Jenkins as an administrator
  • Go to Manage Jenkin > Manage plugins


Jenkins settings # Maven Integration plugin

Jenkins settings # SonarQube scanner for Jenkins



Global configuration

  • Log into Jenkins as an administrator
  • Go to Manage Jenkins > Configure System
  • Scroll to the SonarQube servers section
  • Check Enable injection of SonarQube server configuration as build environment variables


Jenkins Global configuration # SonarQube



Job configuration

Now that you can generate these reports, especially the integration tests results, you need to configure Jenkins to use them!


Maven build (legacy)

This explain how to configure a Maven build (legacy) with SonarQube Runner.


In the Build Environment section > check Prepare SonarQube Scanner environment

Jenkins Build configuration #1


Then, set the Build command line:

Jenkins Build configuration #2


Adjust the SonarQube server URL + path to integration tests reports!

  • Maven build: clean install or clean deploy
  • Use SonarQube
    • Enable plugin: $SONAR_MAVEN_GOAL
    • Set SonarQube server URL: -Dsonar.host.url=http://localhost:9000/sonarqube
    • Give path of Integration tests: -Dsonar.jacoco.itReportPath=/var/lib/jenkins/workspace/Flow.e.r.s/target/jacoco-it.exec



JenkinsFile (Pipelines)

Requirements: have a local SonarQube instance configured in Jenkins, called "sonarqube"

pipeline {

    parameters {
        booleanParam(defaultValue: false, description: 'Enable quality control', name: 'qualityControlEnabled')
        booleanParam(defaultValue: false, description: 'Enable Maven dependencies checks (missing declaration / unused libraries / etc.)', name: 'mvnDependenciesChecksEnabled')
        booleanParam(defaultValue: false, description: 'Enable OWASP checks (search for known vulnerabilities in libraries)', name: 'owaspChecksEnabled')
    }


    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('MVN dependencies checks') {
            when {
                expression { params.mvnDependenciesChecksEnabled }
            }
            steps {
                withMaven(
                        maven: 'Maven 3.5.2',
                        mavenLocalRepo: '.repository'
                ) {
                    echo "MVN dependencies checks [unused / missing / bad scope / etc.]"
                    sh "mvn org.apache.maven.plugins:maven-dependency-plugin:analyze-report"
                }
            }
        }

        stage('OWASP vulnerabilities checks') {
            when {
                expression { params.owaspChecksEnabled }
            }
            steps {
                withMaven(
                        maven: 'Maven 3.5.2',
                        mavenLocalRepo: '.repository'
                ) {
                    echo "OWASP vulnerabilities checks [lib. with security issue(s)]"
                    sh "mvn org.owasp:dependency-check-maven:check"
                    // sh "mvn org.owasp:dependency-check-maven:aggregate"

                    // Publish report in Jenkins
                    dependencyCheckPublisher failedTotalHigh: '0', unstableTotalHigh: '1', failedTotalNormal: '2', unstableTotalNormal: '5'
                }
            }
        }

        stage('Build') {
            /*
            when {
                 //  optional: my condition(s)
            }
            */
            steps {
                withMaven(
                        maven: 'Maven 3.5.2',
                        mavenLocalRepo: '.repository'
                ){
                    script {
                        if (params.qualityControlEnabled) {
                            // On human request only
                            echo "Building with QUALITY controls from ${BRANCH_NAME} ..."
                            sh "mvn clean install -Pquality_control"
                        } else {
                            // Default case - for automatic builds
                            echo "Standard build"
                            sh "mvn clean install"
                        }
                    }
                }
            }
        }

        stage('SonarQube') {
            when {
                expression { params.qualityControlEnabled }
            }
            steps {
                withMaven(
                        maven: 'Maven 3.5.2',
                        mavenLocalRepo: '.repository'
                ){
                    withSonarQubeEnv('sonarqube') {
                        echo "Static code analysis with SonarQube runner"
                        // TODO ensure the version of the plugin matches the one in the PARENT pom
                        sh "mvn org.sonarsource.scanner.maven:sonar-maven-plugin:3.6.0.1398:sonar -Dsonar.projectName=\"my_project ($BRANCH_NAME)\" -Dsonar.projectKey=my_project:$BRANCH_NAME -Pquality_control"
                    }
                }
            }
        }
    }

    post {
        always {
            echo 'Cleaning workspace'
            deleteDir()
        }
    }


See SonarQube documentation

SonarQube

By default there will be 1 sonarqube' entry for every Jenkins artifactId that is build (parent name).


How to see the code coverage by module?

To have a quick glance of the project's modules:

  • SonarQube > project > Code

SonarQube view results #1


  • SonarQube > project > Measures' > Expand coverage > Conditions coverage

SonarQube view results #2


How to see measures by module?

By default SonarQube display measures for all the project's modules. This is sometimes not convenient... The old "code" dashboard has been removed, but you can still see the measure by sub-module with a little trick.


1. Select a module to analyze from the "code" menu

SonarQube select module, step 1


2. Notice the SonarQube URL and the selection parameter

SonarQube select module, step 2


3. Copy that &selected=.. parameter


4. Parse the value on the measures page

SonarQube select module, step 3


5. All good! You can browse the measures for that particular module.


SonarQube plugins

Go to SonarQube > Administration > Marketplace > "All"


I advised you to install:

  • Code smells
  • FindBugs
  • Git
  • JaCoCo (mandatory)
  • PMD


How to disable rule(s) in SonarQube

(i) This operation is only for logged user with administrators rights.


Quality profile

First of all you have to create a custom quality profile.

  • Go to Quality profiles
  • Click on the wheel of the quality profile you'd like to adjust > copy
  • Name the new profile > Click on the wheel top right > set as default


SonarQube # create quality profile


Disable / Enable rule for quality profile

  • Go to Rules
  • Search for the rule you want to modify
  • Click on the rule name

SonarQube # disable rule screen 1

  • Scroll down to the quality profile view
  • Disable rule for particular quality profile(s)

SonarQube # disable rule screen 2



Other

jUnit tests ordering

Since jUnit 4.11 you can set the order of your tests. If you have to use that it means something is probably wrong with your tests design!




References

This article is based on my daily work in VEHCO + European Parliament + LuxTrust.

This article is based on a lot of research and code browsing. Following articles are very good:

PowerMock support


Other article that are relevant for older versions of Sonar (versions < 6.x with Java 7)