Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub

by Beller, Moritz and Gousios, Georgios and Zaidman, Andy

You can get a pre-print version from here.

Abstract

Continuous Integration (CI) has become a best practice of modern software development. Yet, at present, we have a shortfall of insight into the testing practices that are common in CI-based software development. In particular, we seek quantifiable evidence on how central testing is to the CI process, how strongly the project language influences testing, whether different integration environ- ments are valuable and if testing on the CI can serve as a surrogate to local testing in the IDE. In an analysis of 2,640,825 Java and Ruby builds on Travis CI, we find that testing is the single most important reason why builds fail. Moreover, the programming lan- guage has a strong influence on both the number of executed tests, their run time, and proneness to fail. The use of multiple integra- tion environments leads to 10% more failures being caught at build time. However, testing on Travis CI does not seem an adequate surrogate for running tests locally in the IDE. To further research on Travis CI with GitHub, we introduce Travistorrent.

Bibtex record

@inproceedings{BGZ17,
  author = {Beller, Moritz and Gousios, Georgios and Zaidman, Andy},
  title = {Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub},
  booktitle = {Proceedings of the 14th Working Conference on Mining Software Repositories},
  series = {MSR'17},
  year = {2017},
  publisher = {IEEE},
  location = {Buenos Aires, Argentina},
  url = {/pub/tests-broke-build-explorative-analysis-travis-ci-github.pdf}
}

The paper