TEZ-4693: Migrate from proprietary atlassian clover code coverage plugin to JaCoCo by Aggarwal-Raghav · Pull Request #465 · apache/tez

Aggarwal-Raghav · 2026-03-16T17:20:53Z

Code coverage using jacoco:

GO to tez-api/target/site/jacoco and run python -m http.server 8080 to see the report on http://localhost:8080/

Aggarwal-Raghav · 2026-03-16T17:22:13Z

tez-dag/pom.xml

        <configuration>
-          <argLine>${test.jvm.args}</argLine>
-          <environmentVariables>
-            <LOG_DIRS>${test.log.dir}</LOG_DIRS>


There is no need to explicit surefire plugins in child pom. Moved LOG_DIRS to parent pom and used <environmentVariables combine.children="append"> to ensure the argline and other env variables defined in parent pom under surefire plugins are available to child pom.

Aggarwal-Raghav · 2026-03-16T17:24:11Z

pom.xml

            <forkedProcessTimeoutInSeconds>900</forkedProcessTimeoutInSeconds>
            <testFailureIgnore>true</testFailureIgnore>
-            <argLine>-Xmx1024m -XX:+HeapDumpOnOutOfMemoryError</argLine>
-            <argLine>${test.jvm.args}</argLine>


Having 2 argline is a BUG. The test.jvm.args is overwriting the above argLine i.e. Xmx one
Screenshots:
Before:

After:

Aggarwal-Raghav · 2026-03-16T17:25:29Z

pom.xml

            <testFailureIgnore>true</testFailureIgnore>
-            <argLine>-Xmx1024m -XX:+HeapDumpOnOutOfMemoryError</argLine>
-            <argLine>${test.jvm.args}</argLine>
+            <argLine>@{argLine} -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:+EnableDynamicAgentLoading ${test.jvm.args}</argLine>


Added @{argLine} because of https://www.jacoco.org/jacoco/trunk/doc/prepare-agent-mojo.html
and -XX:+EnableDynamicAgentLoading because of jdk 21 + mockito (even though I have not seen the warning in tez) but adding it is shouldn't be a problem.

Aggarwal-Raghav · 2026-03-16T17:26:12Z

pom.xml

+                <configuration>
+                  <rules>
+                    <rule>
+                      <element>BUNDLE</element>


Inspired from: https://github.com/dnsbtchr/jacoco-maven-plugin-example/blob/master/pom.xml

tez-yetus · 2026-03-16T17:50:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 11s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ master Compile Tests _
+0 🆗	mvndep	2m 17s		Maven dependency ordering for branch
+1 💚	mvninstall	7m 40s		master passed
+1 💚	compile	2m 34s		master passed
+1 💚	javadoc	1m 50s		master passed
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 56s		the patch passed
+1 💚	codespell	0m 51s		No new issues.
+1 💚	compile	2m 36s		the patch passed
+1 💚	javac	2m 36s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	javadoc	1m 41s		the patch passed
			_ Other Tests _
+1 💚	unit	0m 30s		tez-dag in the patch passed.
+1 💚	unit	0m 24s		tez-tests in the patch passed.
+1 💚	unit	0m 19s		tez-ext-service-tests in the patch passed.
+1 💚	unit	2m 4s		root in the patch passed.
+1 💚	asflicense	1m 5s		The patch does not generate ASF License warnings.
		29m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-465/1/artifact/out/Dockerfile
GITHUB PR	#465
Optional Tests	dupname asflicense javac javadoc unit codespell detsecrets xmllint compile
uname	Linux 7ef166f4d872 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-465/src/.yetus/personality.sh
git revision	master / `b57f87e`
Default Java	Ubuntu-21.0.10+7-Ubuntu-124.04
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-465/1/testReport/
Max. process+thread count	117 (vs. ulimit of 5500)
modules	C: tez-dag tez-tests tez-ext-service-tests . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-465/1/console
versions	git=2.43.0 maven=3.8.7 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2026-03-17T14:04:47Z

@Aggarwal-Raghav : awesome contribution again! am I right to assume that the surefire argline related patches are not related to the introduction of the jacoco plugin? I'm struggling to find the connection: could you move argline fixes to a separate jira/patch?

Aggarwal-Raghav · 2026-03-17T14:18:45Z

@Aggarwal-Raghav : awesome contribution again! am I right to assume that the surefire argline related patches are not related to the introduction of the jacoco plugin? I'm struggling to find the connection: could you move argline fixes to a separate jira/patch?

Thanks @abstractdog , if you mean <argLine>@{argLine} then they are related. Please check

Without that the tez-dag tests

[INFO] --- jacoco:0.8.14:prepare-agent (jacoco-prepare-agent) @ tez-dag ---
[INFO] argLine set to -javaagent:/Users/raghav/.m2/repository/org/jacoco/org.jacoco.agent/0.8.14/org.jacoco.agent-0.8.14-runtime.jar=destfile=/Users/raghav/Desktop/apache/tez/master/tez-dag/target/jacoco.exec,excludes=**/generated/**
....
....
[INFO] Results:
[INFO]
[WARNING] Tests run: 562, Failures: 0, Errors: 0, Skipped: 9
[INFO]
[INFO]
[INFO] --- jacoco:0.8.14:report (jacoco-report) @ tez-dag ---
[INFO] Skipping JaCoCo execution due to missing execution data file.

The javaagent is not picked up

abstractdog · 2026-03-17T14:22:25Z

@Aggarwal-Raghav : awesome contribution again! am I right to assume that the surefire argline related patches are not related to the introduction of the jacoco plugin? I'm struggling to find the connection: could you move argline fixes to a separate jira/patch?

Thanks @abstractdog , if you mean <argLine>@{argLine} then they are related. Please check

https://www.jacoco.org/jacoco/trunk/doc/prepare-agent-mojo.html

https://maven.apache.org/surefire/maven-surefire-plugin/faq.html#late-property-evaluation

Without that the tez-dag tests
[INFO] --- jacoco:0.8.14:prepare-agent (jacoco-prepare-agent) @ tez-dag ---
[INFO] argLine set to -javaagent:/Users/raghav/.m2/repository/org/jacoco/org.jacoco.agent/0.8.14/org.jacoco.agent-0.8.14-runtime.jar=destfile=/Users/raghav/Desktop/apache/tez/master/tez-dag/target/jacoco.exec,excludes=**/generated/**
....
....
[INFO] Results:
[INFO]
[WARNING] Tests run: 562, Failures: 0, Errors: 0, Skipped: 9
[INFO]
[INFO]
[INFO] --- jacoco:0.8.14:report (jacoco-report) @ tez-dag ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
The javaagent is not picked up

I mean all the <environmentVariables combine.children="append"> and surefire plugin removal from submodules makes sense without the jacoco introduction (and a good improvement to pom.xmls in itself) , and only <argLine>@{argLine} -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:+EnableDynamicAgentLoading ${test.jvm.args}</argLine> is where the agent loading kicks in

Aggarwal-Raghav · 2026-03-17T14:25:18Z

yes, we can do that but then I have to add that {@argline} in 4 child pom's as well. Let me know if you want 2 JIRA, will make that change. Just confirm one more time.

Aggarwal-Raghav · 2026-03-17T16:47:17Z

Will rebase once #467 is merged

abstractdog · 2026-03-18T10:15:44Z

Will rebase once #467 is merged

makes sense, thanks! please update BUILDING.txt as well:

tez/BUILDING.txt

Lines 43 to 52 in 9725e38

    
            * Run clover                : mvn test -Pclover [-Dclover.license=${user.home}/clover.license] 
        
            * Run Rat                   : mvn apache-rat:check 
        
            * Build javadocs            : mvn javadoc:javadoc 
        
            * Build distribution        : mvn package[-Dhadoop.version=2.7.0] 
        
            * Visualize state machines  : mvn compile -Pvisualize -DskipTests=true 
        
           Build options: 
        
            * Use -Dpackage.format to create distributions with a format other than .tar.gz (mvn-assembly-plugin formats).  
        
            * Use -Dclover.license to specify the path to the clover license file

…gin to JaCoCo

abstractdog · 2026-03-19T08:50:15Z

BUILDING.txt

 * Install JAR in M2 cache   : mvn install
 * Deploy JAR to Maven repo  : mvn deploy
- * Run clover                : mvn test -Pclover [-Dclover.license=${user.home}/clover.license]
+ * Run jacoco                : mvn test -Pjacoco


I would appreciate a quick help here about where to find the jacoco report, othen than that, this looks good

you mean add that info in BUILDING.txt that it's present in target/site/jacoco/ and to render run
python3 -m http.server 8080

ah nevermind, it would break this list, it's fine this way
we can still add info later to a the apache webpage if we want to advertise this in Tez project

abstractdog · 2026-03-19T08:59:54Z

looks good to me, tried it in tez-api, +1, pending tests

tez-yetus · 2026-03-19T09:09:00Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 11s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ master Compile Tests _
+1 💚	mvninstall	9m 51s		master passed
+1 💚	compile	1m 45s		master passed
+1 💚	javadoc	1m 29s		master passed
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 9s		the patch passed
+1 💚	codespell	1m 14s		No new issues.
+1 💚	compile	1m 41s		the patch passed
+1 💚	javac	1m 42s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	javadoc	0m 48s		the patch passed
			_ Other Tests _
+1 💚	unit	2m 17s		root in the patch passed.
+1 💚	asflicense	0m 17s		The patch does not generate ASF License warnings.
		23m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-465/2/artifact/out/Dockerfile
GITHUB PR	#465
Optional Tests	dupname asflicense codespell detsecrets javac javadoc unit xmllint compile
uname	Linux e2357d259e1c 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-465/src/.yetus/personality.sh
git revision	master / `68e3457`
Default Java	Ubuntu-21.0.10+7-Ubuntu-124.04
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-465/2/testReport/
Max. process+thread count	104 (vs. ulimit of 5500)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-465/2/console
versions	git=2.43.0 maven=3.8.7 codespell=2.4.1
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

Aggarwal-Raghav commented Mar 16, 2026

View reviewed changes

Aggarwal-Raghav mentioned this pull request Mar 17, 2026

TEZ-4695: Consolidate surefire plugin config to remove redundant surefire blocks #467

Merged

TEZ-4693: Migrate from proprietary atlassian clover code coverage plu…

68e3457

…gin to JaCoCo

Aggarwal-Raghav force-pushed the TEZ-4693 branch from b57f87e to 68e3457 Compare March 19, 2026 08:44

abstractdog reviewed Mar 19, 2026

View reviewed changes

abstractdog self-requested a review March 19, 2026 08:59

abstractdog approved these changes Mar 19, 2026

View reviewed changes

abstractdog self-requested a review March 19, 2026 12:48

abstractdog approved these changes Mar 19, 2026

View reviewed changes

abstractdog merged commit 30932e9 into apache:master Mar 19, 2026
4 checks passed

Conversation

Aggarwal-Raghav commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aggarwal-Raghav Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

tez-yetus commented Mar 16, 2026

Uh oh!

abstractdog commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aggarwal-Raghav commented Mar 17, 2026

Uh oh!

abstractdog commented Mar 17, 2026

Uh oh!

Aggarwal-Raghav commented Mar 17, 2026

Uh oh!

Aggarwal-Raghav commented Mar 17, 2026

Uh oh!

abstractdog commented Mar 18, 2026

Uh oh!

abstractdog Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

abstractdog Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

abstractdog commented Mar 19, 2026

Uh oh!

tez-yetus commented Mar 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Aggarwal-Raghav commented Mar 16, 2026 •

edited

Loading

abstractdog commented Mar 17, 2026 •

edited

Loading