Add retrying of failed tests by default to reduce flaky tests (#5461)

Whenever a test fails we retry it five times and then decide the
overall result based on that. If three or more of the five retries
fail too then the overall result is FAIL, two or lower means PASS.

We're modifying nunitlite here so this only works for those tests
right now, we'd need to add it to the other test runners too.

diff --git a/external/nunit-lite b/external/nunit-lite
index 690603bea98aae69fca9a65130d88591bc6cabee..5b29e610f4ac95afcb8108fdd6d0bb1936f6cc83 160000 (submodule)
--- a/external/nunit-lite
+++ b/external/nunit-lite
@@ -1 +1 @@
-Subproject commit 690603bea98aae69fca9a65130d88591bc6cabee
+Subproject commit 5b29e610f4ac95afcb8108fdd6d0bb1936f6cc83

diff --git a/mcs/tools/nunit-lite/NUnitLite/nunitlite.dll.sources b/mcs/tools/nunit-lite/NUnitLite/nunitlite.dll.sources
index edf8c1283791bde051589c5199693cb6ad1014ba..c581a0c277dfa37def4d2823e3f03e023aefa09a 100644 (file)
--- a/mcs/tools/nunit-lite/NUnitLite/nunitlite.dll.sources
+++ b/mcs/tools/nunit-lite/NUnitLite/nunitlite.dll.sources
@@ -184,6 +184,7 @@
 ../../../../external/nunit-lite/NUnitLite-1.0.0/src/framework/Internal/Commands/CommandStage.cs
 ../../../../external/nunit-lite/NUnitLite-1.0.0/src/framework/Internal/Commands/DelegatingTestCommand.cs
 ../../../../external/nunit-lite/NUnitLite-1.0.0/src/framework/Internal/Commands/ExpectedExceptionCommand.cs
+../../../../external/nunit-lite/NUnitLite-1.0.0/src/framework/Internal/Commands/FlakyTestRetriesCommand.cs
 ../../../../external/nunit-lite/NUnitLite-1.0.0/src/framework/Internal/Commands/ICommandDecorator.cs
 ../../../../external/nunit-lite/NUnitLite-1.0.0/src/framework/Internal/Commands/MaxTimeCommand.cs
 ../../../../external/nunit-lite/NUnitLite-1.0.0/src/framework/Internal/Commands/OneTimeSetUpCommand.cs