Mitigating Flaky Tests

February 28, 2020 - 11:15am to 12:15pm
August Shi
University of Illinois
3-180 Keller Hall
Mats Heimdahl

Abstract: Software is an important part of our society, and we depend on software developers to implement and maintain the quality of software. In turn, developers rely on regression testing, the practice of running tests after every software change, to check that they do not introduce bugs as they make changes. However, regression testing is plagued by flaky tests that non-deterministically pass and fail independently of software changes. The outcomes of flaky tests often mislead developers about the validity of their changes. Flaky tests are also quite prevalent, e.g., Google once reported 16% of their 4.2 million tests are flaky and that 84% of their changes with test failures actually involve a flaky test. To mitigate the negative effects of flaky tests, I have developed several techniques to automatically fix and detect different types of flaky tests. My techniques have fixed and detected hundreds of flaky tests in open-source software. This talk will focus on my work in fixing and detecting a specific type of flaky tests, namely order-dependent flaky tests, which pass when run in one order of the test suite but fail when run in a different order. First, I will present my technique, iFixFlakies, for automatically fixing order-dependent flaky tests. Next, I will present my technique, PolDet, for proactively detecting root causes that lead to order-dependent flaky tests before such tests can lead to flaky test failures.


Bio: August Shi is a PhD candidate in Computer Science at the University of Illinois at Urbana-Champaign. August works with Professor Darko Marinov in the area of Software Engineering, with a focus on Software Testing. His research aims to improve developer productivity by making regression testing (1) more reliable with respect to flaky tests and (2) faster without loss in quality of testing. August has published in ICSE, ESEC/FSE, ASE, ISSTA, OOPSLA, ICST, and ISSRE, and his work on improving regression testing received an ACM SIGSOFT Distinguished Paper Award at ICSE 2017. More information is available on his web page:


*To sign up for an appointment to meet with August Shi, please pay a visit to the following link: Appointment Sign Up Sheet