[testng-dev] How to run tests in parallel with constant thread count

[shankar KC]

Hi,
Our product is tweaked to support say 60 concurrent users.
Say I have 2 maven projects with 5 test suites. each having 10 tests. For
convenience lets say all tests except one each takes 1min to run. Long
running test takes 10 min to complete.
Lets assume i run these 2 projects in parallel in 2 jenkins job. Each job
runs 3 test suites in parallel with 10 tests in parallel( using
parallel=methods and thread-count =5)
Now when i start the tests during first minute we have 2 * 3 * 10. After
first min all tests in a suite completed execution except the long running
test. It means only 6 tests are running. Rest of the threads (2 * 3 * 9)
are idle and will not run remaining suites as current thread didnt finish
because of long running test. It cannot pick remaining 2 suites though they
are idle. How do we solve this and put a constant load on our server. Now
2nd batch of test suite execution happens only after 10 minutes. All the
threads were idle expect one for 9 minutes. This is a common issue in
product companies i think. But I am not sure how to tackle it properly
without wasting lot of time. How you all handle this at your organization?
Thanks
Shankar KC

[shankar KC]

Hi Cedric
Do you have any recommendations to run this and achieve constant load on
server?
Thank you
Shankar

Post by shankar KC
Hi,
Our product is tweaked to support say 60 concurrent users.
Say I have 2 maven projects with 5 test suites. each having 10 tests. For
convenience lets say all tests except one each takes 1min to run. Long
running test takes 10 min to complete.
Lets assume i run these 2 projects in parallel in 2 jenkins job. Each job
runs 3 test suites in parallel with 10 tests in parallel( using
parallel=methods and thread-count =5)
Now when i start the tests during first minute we have 2 * 3 * 10. After
first min all tests in a suite completed execution except the long running
test. It means only 6 tests are running. Rest of the threads (2 * 3 * 9)
are idle and will not run remaining suites as current thread didnt finish
because of long running test. It cannot pick remaining 2 suites though they
are idle. How do we solve this and put a constant load on our server. Now
2nd batch of test suite execution happens only after 10 minutes. All the
threads were idle expect one for 9 minutes. This is a common issue in
product companies i think. But I am not sure how to tackle it properly
without wasting lot of time. How you all handle this at your organization?
Thanks
Shankar KC
--
You received this message because you are subscribed to a topic in the
Google Groups "testng-dev" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/testng-dev/7NMM0UfMNds/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
Visit this group at https://groups.google.com/group/testng-dev.
For more options, visit https://groups.google.com/d/optout.

[shankar KC]

Hi All this is the response i got from testng users group.

=======================
Yes, and we parallelize tests as much as we reasonably can. But ultimately
we have the same issue you do, and just accept that we can't perfectly
balance all the test execution threads without a lot more work. And for us,
at least, that extra work isn't worth it because all it would gain is a
small improvement in test run time. So we do other things to improve test
run time - running tests of slow changing features less often, running slow
tests less often, etc.
=======================
Is there any better idea to handle this? After first minute most of the
threads are idle thought there are tests to be executed. We have thousands
of tests to run and this behavior slowing down our execution. Please let me
know your thoughts on problems like this.
Thanks
Shankar KC

Post by shankar KC
Hi Cedric
Do you have any recommendations to run this and achieve constant load on
server?
Thank you
Shankar

Post by shankar KC
Hi,
Our product is tweaked to support say 60 concurrent users.
Say I have 2 maven projects with 5 test suites. each having 10 tests. For
convenience lets say all tests except one each takes 1min to run. Long
running test takes 10 min to complete.
Lets assume i run these 2 projects in parallel in 2 jenkins job. Each job
runs 3 test suites in parallel with 10 tests in parallel( using
parallel=methods and thread-count =5)
Now when i start the tests during first minute we have 2 * 3 * 10. After
first min all tests in a suite completed execution except the long running
test. It means only 6 tests are running. Rest of the threads (2 * 3 * 9)
are idle and will not run remaining suites as current thread didnt finish
because of long running test. It cannot pick remaining 2 suites though they
are idle. How do we solve this and put a constant load on our server. Now
2nd batch of test suite execution happens only after 10 minutes. All the
threads were idle expect one for 9 minutes. This is a common issue in
product companies i think. But I am not sure how to tackle it properly
without wasting lot of time. How you all handle this at your organization?
Thanks
Shankar KC
--
You received this message because you are subscribed to a topic in the
Google Groups "testng-dev" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/testng-dev/7NMM0UfMNds/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
Visit this group at https://groups.google.com/group/testng-dev.
For more options, visit https://groups.google.com/d/optout.