In this course, we will learn

We will also learn

  • How to read papers (this lecture!)
  • How to do systematic literature reviews
  • How to write about papers

Course Teacher

Georgios Gousios

Georgios Gousios,


Date Week Lecture Topic Lecturer
3/9 1 1 Course introduction, Quantitative methods in Software Engineering MB
5/9 1 2 Discussion groups GG
10/9 2 1 Process Analytics AR / GG
12/9 2 2 Testing Analytics, Testing Paper students, MB
17/9 3 1 Build Analytics, Building Paper students, MB
19/9 3 2 Bug Prediction students, MB
24/9 4 1 Software Ecosystem Analytics JH

Schedule (2)

Date Week Lecture Topic Lecturer
26/9 4 2 Release Engineering Analytics students, AR
1/10 5 1 Results: Survey on software analytics students
3/10 5 2 Code review students, GG
8/10 6 1 Runtime and Performance Analytics, Analytics at work: Adyen Mauricio Aniche, MK
10/10 6 2 App store analytics students, MK
15/10 7 1 Analytics at work: ING Hennie Huijgens
17/10 7 2 Results: Replication project results students

How will we study

The course is a seminar. Therefore:

Course deliverables: The survey

We will work together on a survey of modern software analytics. The topics we will cover are:

Level Topics
Product source code, repository, builds, static analysis, ecosystems
Process PR/Code reviews, release engineering, communities, agile, testing etc
Runtime logs, crashes, security, performance
Community app stores, social media, user acceptance

All topics to work on correspond to the contents of the project’s TOC.

All topics have an instructor attached to them. You can find the responsible instructor on the course’s home page

Working on the survey (1)

Working on the survey (2)

The survey is a Bookdown document (template), available on GitHub. Fork it and file PRs. Every group works on one chapter (one file).

How to read a paper

Keshav, Srinivasan. “How to read a paper.” ACM SIGCOMM Computer Communication Review 37.3 (2007): 83-84.

D Read the above paper. After 10 minutes, we will discuss it.

How to get a paper

Q How much does one article cost?

We use Google Scholar to retrieve metrics. There is other tools such as Scopus. Even the IEEE Xplore Digital Library gives metrics.

D Use the different services to derive number of reads, number of citations, and the BibTeX entry for: Beller, Moritz, Georgios Gousios, and Andy Zaidman. “How (much) do developers test?.” Software Engineering (ICSE), 2015 IEEE/ACM 37th IEEE International Conference on. Vol. 2. IEEE, 2015. [beller2015much]

Course deliverables: Paper Replication

Replication: Redoing a published study following exactly the same research protocol, using the same (exact) or different subjects.

Replication is often touted, seldom practiced. To change this, we will replicate existing studies, as follows:

How to work

A deliverable is either a document and/or source code.

How to review:

Preparation for classes comprises reading a set of papers for the lecture topic (usually 2-3).


The final course grade will be calculated as:

All deliverables will be peer-reviewed and cross-graded by 2 other teams. The peer-review grade is 50% of the final grade per grade item. The results add up to 110%.

TODO for today

Slide symbols

Lecture notes


B. Kitchenham, “Procedures for performing systematic reviews,” Keele, UK, Keele University, vol. 33, no. 2004, pp. 1–26, 2004.
B. Kitchenham, O. P. Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman, “Systematic literature reviews in software engineering–a systematic literature review,” Information and software technology, vol. 51, no. 1, pp. 7–15, 2009.