When we run our test suite, sometime few test cases may fail because of various reasons like locator has been changed, because of defect or may be due to slow response of the system at some times.
Test cases failed because of slow response due to internet issue, should not fail actually, so we need to retry those test cases. Now the question is, how can we run the test cases which are failed.
Let us talk about how can we execute failed test cases using testng in selenium Webdriver.
Two approaches
1. Using testng-failed.xml
2. Using IRetryAnalyser
Let us talk one by one.
How to execute failed test cases using testng-failed.xml ?
Every time tests fail in a suite, TestNG creates a file called testng-failed.xml in the output directory. This XML file contains the necessary information to rerun only these methods that failed, allowing you to quickly reproduce the failures without having to run the entirety of your tests.
If you want to rerun the test exactly after the failure you need to call the method that failed. You can get that method name from ITestResult object.
If you want to rerun all the failed test cases together, then you can give the testng-failed.xml as input xml after the first execution.
Steps –
1-If your test cases are failing then once all test suite completed then you have to refresh your project. Right click on project > Click on refresh or Select project and press f5.
2-Check test-output folder, at last, you will get testng-failed.xml
3- Now simply run testng-failed.xml.
That’s done, it will execute your failed test cases
How to execute failed test cases using IRetryAnalyser?
Another way to run failed test cases using testng is, using IRetryAnalyser interface. Again this is very simple to implement.
How retryAnalizer works is, we define maximum retry count (how many times you wish to retry failed test case).
If the test case is failing, it shows as Skipped, and in the last try, it shows as test case failed. (if it continuous to fail till last retry count).
Let us create class for defining retry. Implement IRetryAnalyser.
package retrypackage;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
public class RetryClass implements IRetryAnalyzer {
private int count = 1;
private static int maxTry = 3;
@Override
public boolean retry(ITestResult result) {
if(count<maxTry){
count++;
return true;
}
return false;
}
}
Using retryAnalyzer attribute in the @Test annotation
Next step is to associate your test cases with IRetryAnalyzer. In order to do this, you need to use the method below.
@Test(retryAnalyzer = Retry.class)
public void testCase() {
}
Now, create a sample test case, I am failing the test case purposely, so that we can observe the RetryAnalyzer functionality.
Below is sample test case –
package retrypackage;
import org.testng.Assert;
import org.testng.annotations.Test;
public class TestCase {
@Test(retryAnalyzer = RetryClass.class)
public void test(){
Assert.assertTrue(false);
}
}
Run the above test case as TestNG Test.
Below console Output shows, that test case as attempted to run 3 times as specified in retry class.
Console Output –
[TestNG] Running:
C:UsersPrakashAppDataLocalTemptestng-eclipse-1379819527testng-customsuite.xml
FAILED: test
java.lang.AssertionError: expected [true] but found [false]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertTrue(Assert.java:42)
at org.testng.Assert.assertTrue(Assert.java:52)
at retrypackage.TestCase.test(TestCase.java:10)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:639)
at org.testng.internal.Invoker.retryFailed(Invoker.java:963)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1142)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
at org.testng.TestRunner.privateRun(TestRunner.java:774)
at org.testng.TestRunner.run(TestRunner.java:624)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:359)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:354)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:312)
at org.testng.SuiteRunner.run(SuiteRunner.java:261)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1215)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
at org.testng.TestNG.run(TestNG.java:1048)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:132)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:230)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:76)
SKIPPED: test
java.lang.AssertionError: expected [true] but found [false]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertTrue(Assert.java:42)
at org.testng.Assert.assertTrue(Assert.java:52)
at retrypackage.TestCase.test(TestCase.java:10)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:639)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:816)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1124)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
at org.testng.TestRunner.privateRun(TestRunner.java:774)
at org.testng.TestRunner.run(TestRunner.java:624)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:359)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:354)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:312)
at org.testng.SuiteRunner.run(SuiteRunner.java:261)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1215)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
at org.testng.TestNG.run(TestNG.java:1048)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:132)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:230)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:76)
SKIPPED: test
java.lang.AssertionError: expected [true] but found [false]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:496)
at org.testng.Assert.assertTrue(Assert.java:42)
at org.testng.Assert.assertTrue(Assert.java:52)
at retrypackage.TestCase.test(TestCase.java:10)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:639)
at org.testng.internal.Invoker.retryFailed(Invoker.java:963)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1142)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
at org.testng.TestRunner.privateRun(TestRunner.java:774)
at org.testng.TestRunner.run(TestRunner.java:624)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:359)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:354)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:312)
at org.testng.SuiteRunner.run(SuiteRunner.java:261)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1215)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
at org.testng.TestNG.run(TestNG.java:1048)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:132)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:230)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:76)
===============================================
Default test
Tests run: 3, Failures: 1, Skips: 2
===============================================
===============================================
Default suite
Total tests run: 3, Failures: 1, Skips: 2
===============================================
[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@880ec60: 100 ms
[TestNG] Time taken by org.testng.reporters.EmailableReporter2@67b64c45: 15 ms
[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 20 ms
[TestNG] Time taken by org.testng.reporters.jq.Main@39ba5a14: 77 ms
[TestNG] Time taken by org.testng.reporters.XMLReporter@17f6480: 8 ms
[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@35f983a6: 11 ms