Running a Test Multiple Times in Spock

Yesterday I encountered a flakey test in some code I had written. It would only fail every now and then, which made it very tough to figure out why it was failing. I wanted to run the test multiple times to see how often it failed and help me narrow down a cause.

I am using Spock for my tests, so I did this using the below:

where:
i << (1..100)

The where block in Spock allows you to outline different scenarios and use those in your test via a variable. In this case I am defining 100 scenarios where i is a number starting at 1, and looping all the way to 100.

I added println(i) in my given block to print to the console which test was being run. Then to view each test result seperately I add the @Unroll annotation to my test. For example the test would look like this:

@Unroll
def "flakey test"() {
  given:
  println(i)
  setupTest()

  when:
  def result = myClass.doSomething()

  then:
  result

  where:
  i << (1..100)
}

Using this I was able to see that my test was failing around 10% of the time on average.

I added some prints at key points in my code to see if there were any differences in the successful runs and failing runs. I narrowed the issue down to my database and the information coming back from it.

Using this information I was able to figure out what the issue was and apply a fix. To check my fix had worked, I ran the test 1000 times to make sure it wasn’t just less flakey.

You can find the list of all of my Knowledge Sharing posts here.