Open
Description
Since #3900 I have observed hudson.plugins.ec2.EC2RetentionStrategyTest.testRetentionDespiteIdleWithMinimumInstanceActiveTimeRange
failing multiple times in CI (but not locally) with
java.lang.AssertionError: expected:<2> but was:<3>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at hudson.plugins.ec2.EC2RetentionStrategyTest.testRetentionDespiteIdleWithMinimumInstanceActiveTimeRange(EC2RetentionStrategyTest.java:612)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:706)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.lang.Thread.run(Thread.java:1583)
We run BOM tests with surefire.rerunFailingTestsCount=1
to retry flaky tests one time, which prevents mildly flaky tests from creating too much noise but is ineffective against very flaky tests. The fact that it is ineffective here indicates this is a very flaky test.
We should look into making the test less flaky or exclude it if we cannot figure out how.
Activity