[testng-dev] Possibility to retry tests until FAILED (not success status)

[Katya L]

We need to have a possibility to run tests until test will be failed
(because we have flaky tests and want to run a test until it failed and
then investigate: read logs etc) Invocation count is not a workaround.
1st problem: hardcoded FAILURE test status in condition for retry:
boolean willRetry = retryAnalyzer != null && status == ITestResult.FAILURE
&& failure.instances != null && retryAnalyzer.retry(testResult);

In old TestNG versions we had a chance to avoid it by the following way:

class RetryUntilFailed : IRetryAnalyzer {

companion object {
private var retryCount = 0
private var maxRetryCount = 1000
}

override fun retry(result: ITestResult?): Boolean {
if (result!!.getAttribute("RETRY") == "NO")
return false
else if (result.getAttribute("RETRY") == "YES" && retryCount < maxRetryCount) {
retryCount++
return true
}
return false
}
}

class RetryUntilFailedListener : IInvokedMethodListener {
override fun beforeInvocation(method: IInvokedMethod?, testResult: ITestResult?) {

}

override fun afterInvocation(method: IInvokedMethod?, testResult: ITestResult?) {
if (method!!.isTestMethod) {
if (testResult!!.status == ITestResult.SUCCESS) {
println("SUCCESS. NEED TO RETRY")
testResult.status = ITestResult.FAILURE //because retry works only if (testResult.getStatus() == ITestResult.FAILURE)
testResult.setAttribute("RETRY", "YES")
} else if (testResult.status == ITestResult.FAILURE)
testResult.setAttribute("RETRY", "NO")
}
}

}



But in the latest TestNG it doesn't work because handleInvocationResults()
is invoked before afterInvocation() and no chance to override test status
before retrying logic

finally {
...
handleInvocationResults(tm, results, expectedExceptionClasses, failureContext);
...
// Run invokedMethodListeners after updating TestResult
runInvokedMethodListeners(AFTER_INVOCATION, invokedMethod, testResult);
...
}



Any ideas?
Thanks in advance

[Julien Herr]

Hi,

I'm not sure the regression is so recent. Could you tell us which is the
last version working as you expect?
And I can't remember if the new behavior is expected or not.

Could you open an issue on GitHub? It is easier to follow issue there.

Julien

Post by Katya L
We need to have a possibility to run tests until test will be failed
(because we have flaky tests and want to run a test until it failed and
then investigate: read logs etc) Invocation count is not a workaround.
boolean willRetry = retryAnalyzer != null && status == ITestResult.FAILURE
&& failure.instances != null && retryAnalyzer.retry(testResult);
class RetryUntilFailed : IRetryAnalyzer {
companion object {
private var retryCount = 0
private var maxRetryCount = 1000
}
override fun retry(result: ITestResult?): Boolean {
if (result!!.getAttribute("RETRY") == "NO")
return false
else if (result.getAttribute("RETRY") == "YES" && retryCount < maxRetryCount) {
retryCount++
return true
}
return false
}
}
class RetryUntilFailedListener : IInvokedMethodListener {
override fun beforeInvocation(method: IInvokedMethod?, testResult: ITestResult?) {
}
override fun afterInvocation(method: IInvokedMethod?, testResult: ITestResult?) {
if (method!!.isTestMethod) {
if (testResult!!.status == ITestResult.SUCCESS) {
println("SUCCESS. NEED TO RETRY")
testResult.status = ITestResult.FAILURE //because retry works only if (testResult.getStatus() == ITestResult.FAILURE)
testResult.setAttribute("RETRY", "YES")
} else if (testResult.status == ITestResult.FAILURE)
testResult.setAttribute("RETRY", "NO")
}
}
}
But in the latest TestNG it doesn't work because handleInvocationResults()
is invoked before afterInvocation() and no chance to override test status
before retrying logic
finally {
...
handleInvocationResults(tm, results, expectedExceptionClasses, failureContext);
...
// Run invokedMethodListeners after updating TestResult
runInvokedMethodListeners(AFTER_INVOCATION, invokedMethod, testResult);
...
}
Any ideas?
Thanks in advance