Allure Life Cycle Not Triggered When Running Tests via jar file

[santhosh0419]

Title:
Allure Results Not Generated When Running JAR File with -Dallure.results.directory Property

Description:
We are facing an issue where Allure test results are not being generated in the expected directory when running our test project as a standalone JAR file. The system property allure.results.directory is set via the -D argument at runtime, and when accessed within the tests using System.getProperty("allure.results.directory"), it returns the expected path. However, no Allure result files are actually created in the specified location.

Current Setup:

1. Project Build Process:

   • The project is built using Gradle and packaged into a fat JAR file containing all dependencies.
   • The tests are executed using the JUnit 5 platform.
   • Allure dependencies are included within the JAR.

2. Execution Command:
   The JAR is executed with the following command:

   java -Dallure.results.directory=C:\path\to\allure-results -jar test-automation-1.0-SNAPSHOT-all.jar

3. Code Snippet to Verify Property:
   Inside the test runner, the following code is used to verify the property value:

   System.out.println("Allure results directory: " + System.getProperty("allure.results.directory"));

   The correct path is printed, confirming that the system property is being set properly.

---

Issue Details:

• The tests execute successfully without any errors.
• The specified allure.results.directory path is correctly read within the test code.
• However, no Allure result files are generated in the specified directory.
• Running the tests directly via the IDE (without the JAR) generates results correctly in the specified directory.
  - Upon research it was found that the allure lifecycle is triggered by junit, but in this case since the run is triggered as java -jar the life cycle is not generated. How the Lifecycle can be setup?

---

Troubleshooting Steps Taken:

1. Validated Property Setting:

   • Printed the value of System.getProperty("allure.results.directory") to confirm it's correctly set at runtime.

2. Checked JAR Execution Behavior:

   • Ensured that the tests execute successfully with no runtime exceptions.
   • Verified if any Allure-related logs indicate issues.

3. Manual Directory Verification:

   • Created the allure-results directory manually before running the tests.
   • Tried setting the path with both relative and absolute values.

4. Build Configuration Checks:

   • Ensured the correct Allure dependencies are bundled in the JAR (allure-junit5, allure-commandline).
   • Verified test execution plugins in the build.gradle file.

5. Execution Order:

   • Confirmed the correct order of JVM arguments to ensure the -D property is passed before -jar.
   • Tried executing with different JVM versions.

6. Tried adding the Allure.step("Sample Test") statement explicitly:

• This statement properly runs from the Junit
• But when I run it in the jar I get the below exceptions

13:26:25.005 [ForkJoinPool-1-worker-2] ERROR io.qameta.allure.AllureLifecycle -- Could not start step: no test case running
13:26:25.009 [ForkJoinPool-1-worker-2] ERROR io.qameta.allure.AllureLifecycle -- Could not stop step: step with uuid 6540d59f-9993-4cd6-b251-234f94d4a369 not found

Request for Help:
We would appreciate insights or suggestions on:

1. Why Allure results are not being written to the specified directory when running from the JAR file.
2. Whether any additional configuration is required to ensure Allure can access the result directory correctly.
3. Any known issues related to running Allure with JUnit 5 in a fat JAR setup.

[santhosh0419]

@baev - I'm sorry for bothering you by tagging, but if you get a chance, could you please review this issue? It has been a blocker for a long time, and I haven't found any relevant answers in the community. Thank you.

[santhosh0419]

I found the root cause for the issue -
In a local JUnit5 environment, Allure-JUnit and Allure-RestAssured automatically handle the lifecycle setup. However, when running from a JAR, this automatic lifecycle management doesn't occur.
The key difference is the test execution context. In a local environment, JUnit provides this context automatically, but in a JAR execution, it's missing. This is why manual lifecycle management or a custom listener becomes necessary to ensure Allure reports are generated correctly.
To resolve this, I have to explicitly manage the Allure lifecycle, either through a custom listener or by manually initializing the lifecycle before test execution

[baev]

Most likely, the issue can be solved by setting junit.jupiter.extensions.autodetection.enabled to true; see https://junit.org/junit5/docs/current/user-guide/#extensions-registration-automatic-enabling

[santhosh0419]

@baev - Thank you for your suggestion! I tried setting junit.jupiter.extensions.autodetection.enabled=true as you recommended, but unfortunately, it’s still not working as expected. After setting it up, I explicitly added a few Allure statements using Allure.step() to test if it works. However, I'm encountering the following error during execution:

test-runner  | 06:45:22.679 [ForkJoinPool-1-worker-2] ERROR io.qameta.allure.AllureLifecycle -- Could not start step: no test case running
test-runner  | 06:45:22.682 [ForkJoinPool-1-worker-2] ERROR io.qameta.allure.AllureLifecycle -- Could not stop step: step with uuid c33d9c43-5b91-48ef-a11a-75a304042f1f not found

It seems like the Allure lifecycle is not being triggered properly. Specifically, the issue appears to be with how the Allure lifecycle interacts with the test execution when running as JAR file, as these errors indicate that Allure cannot find the corresponding test context.

I am trying to use Allure with RestAssured and AssertJ, but I can't seem to get the lifecycle working properly in this setup when running the automation suite from the JAR file. Is there something I might be missing in terms of initializing or linking the lifecycle for Allure, AssertJ, or RestAssured in this particular setup?

Sample Docker-compose.yml file:
services:
test-runner:
image: eclipse-temurin:21-jdk-alpine
container_name: test-runner
volumes:
- ./build/libs/automate-tests-1.0-SNAPSHOT-all.jar:/app/automate-tests-1.0-SNAPSHOT-all.jar
- ./allure-results:/app/allure-results
env_file:
- C:\Users<user>\TestAutomation\automate-tests.env
command: ["java", "-Dallure.results.directory=app/allure-results", "-jar", "/app/automate-tests-1.0-SNAPSHOT-all.jar", "--info"]

-Tried also adding the property 'junit.jupiter.extensions.autodetection.enabled=true' as JVM system property
command: ["java", "-Dallure.results.directory=app/allure-results", "-Djunit.jupiter.extensions.autodetection.enabled=true", "-jar", "/app/automate-tests-1.0-SNAPSHOT-all.jar", "--info"]

Test Runner:
public class TestRunner {
public static void main(String[] args) {
ConsoleLauncher.main(new String[]{
"--class-path", System.getProperty("java.class.path"),
"--scan-class-path",
});
}
}

junit-platform.properties:
junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.extensions.autodetection.enabled=true

This is how the Jar file is built:
tasks{
shadowJar{
archiveClassifier.set("all")
from(sourceSets.main.get().output)
from(sourceSets.test.get().output)
from(sourceSets.main.get().resources.srcDirs())
from(sourceSets.test.get().resources.srcDirs())
manifest {
attributes(
"Main-Class" to "org.tests.runner.TestRunner"
)
}
}
}

[baev]

I just checked the issue and everything works as expected. In my setup I used Maven to create artefacts, but it should make no difference:

1. Using maven-dependency-plugin download junit-platform-console-standalone and aspectjweaver

   <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-dependency-plugin</artifactId>
       <version>3.7.0</version>
       <executions>
           <execution>
               <id>copy</id>
               <phase>package</phase>
               <goals>
                   <goal>copy</goal>
               </goals>
           </execution>
       </executions>
       <configuration>
           <artifactItems>
               <artifactItem>
                   <groupId>org.junit.platform</groupId>
                   <artifactId>junit-platform-console-standalone</artifactId>
                   <version>1.11.3</version>
                   <type>jar</type>
                   <overWrite>true</overWrite>
                   <outputDirectory>${project.build.directory}</outputDirectory>
                   <destFileName>junit-platform-console-standalone.jar</destFileName>
               </artifactItem>
               <artifactItem>
                   <groupId>org.aspectj</groupId>
                   <artifactId>aspectjweaver</artifactId>
                   <version>${aspectj.version}</version>
                   <type>jar</type>
                   <overWrite>true</overWrite>
                   <outputDirectory>${project.build.directory}</outputDirectory>
                   <destFileName>aspectjweaver.jar</destFileName>
               </artifactItem>
           </artifactItems>
       </configuration>
   </plugin>

2. execute dependency:copy task to get target/aspectjweaver.jar and target/junit-platform-console-standalone.jar files
3. using maven-assembly-plugin create a jar with all dependencies

   <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-assembly-plugin</artifactId>
       <version>3.7.1</version>
       <configuration>
           <descriptors>
               <descriptor>testjar-with-dependencies.xml</descriptor>
           </descriptors>
       </configuration>
   </plugin>

   and testjar-with-dependencies.xml config file:

   <assembly xmlns="https://maven.apache.org/ASSEMBLY/2.2.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.2.0 https://maven.apache.org/xsd/assembly-2.2.0.xsd">
     <id>jar-with-dependencies</id>
     <formats>
       <format>jar</format>
     </formats>
     <includeBaseDirectory>false</includeBaseDirectory>
     <dependencySets>
       <dependencySet>
         <outputDirectory>/</outputDirectory>
         <useProjectArtifact>true</useProjectArtifact>
         <unpack>true</unpack>
         <scope>test</scope>
       </dependencySet>
     </dependencySets>
   </assembly>

4. execute assembly:single task to get target/junit5-java-maven-1.0-SNAPSHOT-jar-with-dependencies.jar jar with unpacked test dependencies
5. execute compiler:testCompile, resources:testResources and jar:test-jar to get junit5-java-maven-1.0-SNAPSHOT-tests.jar with compiled tests and test resources
6. then the following command to run tests:

   java \
   -cp target/junit-platform-console-standalone.jar \
   -javaagent:target/aspectjweaver.jar \
   org.junit.platform.console.ConsoleLauncher \
   execute \
   --fail-if-no-tests \
   -cp target/junit5-java-maven-1.0-SNAPSHOT-jar-with-dependencies.jar \
   -cp target/junit5-java-maven-1.0-SNAPSHOT-tests.jar \
   --scan-classpath

as the result, the tests are successfully run:

╷
├─ JUnit Platform Suite ✔
├─ JUnit Jupiter ✔
│  └─ SomeTest ✔
│     └─ name() ✔
└─ JUnit Vintage ✔

Test run finished after 145 ms
[         4 containers found      ]
[         0 containers skipped    ]
[         4 containers started    ]
[         0 containers aborted    ]
[         4 containers successful ]
[         0 containers failed     ]
[         1 tests found           ]
[         0 tests skipped         ]
[         1 tests started         ]
[         0 tests aborted         ]
[         1 tests successful      ]
[         0 tests failed          ]

and target/allure-results (as configured in src/test/resources/allure.properties file) folder is created with desired contents:

the steps are in place (both using Allure Runtime API and @Step annotation):

package com.example.junit5;

import io.qameta.allure.Allure;
import io.qameta.allure.Step;
import org.junit.jupiter.api.Test;

class SomeTest {
    @Test
    void name() {
        Allure.step("hi");
        annotatedStep();
    }

    @Step
    void annotatedStep() {
    }
}

[santhosh0419]

Thanks a lot @baev, I followed similar steps with Gradle, and it worked perfectly. I really appreciate the detailed setup you provided.