testng slow to start running tests

[Michael Ondrejko]

We are using testng with a large source base. With testng 6.3.2 and 6.4
there is a long pause before tests start running. It seems that this is the
time that testng is scanning classes for testng annotations. For our source
base, this requires 15+ minutes to complete. Unfortunately, our test
classes are in the same package structure as the product classes themselves
and it is not possible to change this organization. I have looked for a way
to limit that classes (and jars) that testng is scanning, but it seems to
pick up the test classes, everything in the classpath that matches the
package definition specified in testng.xml are scanned.

Is it possible to further limit this scanning or otherwise speed up the
startup time for testng.

[Cédric Beust ♔]

Hi Michael,

It's a bit hard to comment without any data, do you have the output of a
profiler to share?

Having said that, you are right that specifying packages is expensive,
consider specifying classes explicitly or only running your tests with a
minimal classpath.
--
Cédric

Post by Michael Ondrejko
We are using testng with a large source base. With testng 6.3.2 and 6.4
there is a long pause before tests start running. It seems that this is the
time that testng is scanning classes for testng annotations. For our source
base, this requires 15+ minutes to complete. Unfortunately, our test
classes are in the same package structure as the product classes themselves
and it is not possible to change this organization. I have looked for a way
to limit that classes (and jars) that testng is scanning, but it seems to
pick up the test classes, everything in the classpath that matches the
package definition specified in testng.xml are scanned.
Is it possible to further limit this scanning or otherwise speed up the
startup time for testng.
--
You received this message because you are subscribed to the Google Groups
"testng-dev" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/testng-dev/-/iVXlAbeoN3MJ.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/testng-dev?hl=en.

[Michael Ondrejko]

There are thousands of test classes, so specifying each individual test
class is not feasible. Previously, with testng 6.0.1m the project team used
an include sub-element in the package specification too filter specific
tests. For example,

<package name="com.company.*">
<include name=".*Test"/>
</package>

To process only classes that end in "Test" in the "com.company" package
hierarchy. However, this now results in all tests being excluded as the
include here only filters based upon package name. I tried to use wildcards
in the class element for testng.xml; however, class specification does not
seem to support wildcards.

Is there a way in testng.xml to specify a pattern for the classes to scan
for annotations within a package structure?

[Cédric Beust ♔]

Post by Michael Ondrejko
Is there a way in testng.xml to specify a pattern for the classes to scan
for annotations within a package structure?

This won't make a difference, the problem is that TestNG has to look up all
the classes and packages found on the class path, if only to apply that
filter.

But again, we're just guessing at this point until you run a profiler to
find out exactly where the performance problems are coming from.

[Alexandru Baxanean]

When using testng inside maven, is there any way to tell testng to scan for
classes only in test-classes directory ?
That's to avoid the useless scanning of hundreds of jar files and the
classes directory where source (not test) classes are located.

Post by Cédric Beust ♔

Post by Michael Ondrejko
Is there a way in testng.xml to specify a pattern for the classes to scan
for annotations within a package structure?

This won't make a difference, the problem is that TestNG has to look up
all the classes and packages found on the class path, if only to apply that
filter.
But again, we're just guessing at this point until you run a profiler to
find out exactly where the performance problems are coming from.
--
Cédric

[g***@msn.com]

We have 15 suite-file and use gradle to lunch the test, and we observed
that test class objects are created and then paused for ~15 minutes to
start running the actual test methods....very poor performance.

Post by Michael Ondrejko
We are using testng with a large source base. With testng 6.3.2 and 6.4
there is a long pause before tests start running. It seems that this is the
time that testng is scanning classes for testng annotations. For our source
base, this requires 15+ minutes to complete. Unfortunately, our test
classes are in the same package structure as the product classes themselves
and it is not possible to change this organization. I have looked for a way
to limit that classes (and jars) that testng is scanning, but it seems to
pick up the test classes, everything in the classpath that matches the
package definition specified in testng.xml are scanned.
Is it possible to further limit this scanning or otherwise speed up the
startup time for testng.

[Julien Herr]

Did you try recent releases like 6.11 or 6.12? Is it better?

Post by g***@msn.com
We have 15 suite-file and use gradle to lunch the test, and we observed
that test class objects are created and then paused for ~15 minutes to
start running the actual test methods....very poor performance.

Post by Michael Ondrejko
We are using testng with a large source base. With testng 6.3.2 and 6.4
there is a long pause before tests start running. It seems that this is the
time that testng is scanning classes for testng annotations. For our source
base, this requires 15+ minutes to complete. Unfortunately, our test
classes are in the same package structure as the product classes themselves
and it is not possible to change this organization. I have looked for a way
to limit that classes (and jars) that testng is scanning, but it seems to
pick up the test classes, everything in the classpath that matches the
package definition specified in testng.xml are scanned.
Is it possible to further limit this scanning or otherwise speed up the
startup time for testng.

--
You received this message because you are subscribed to the Google Groups
"testng-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/testng-dev.
For more options, visit https://groups.google.com/d/optout.