Project Title : Improving diffoscope tool and reproducibility of Debian packages

• Project details
• Mentor and Co-Mentor
• Project Discussion
  • Introduction to Reproducible Builds in Debian
  • Application Review Period
  • Community Bonding Period
• Project Implementation
  • Coding Period
• Challenges and Work Left
• Future work
• Acknowledgement

Project details

This project aims to improve diffoscope tool and fix Debian packages which are unreproducible in Reproducible builds testing framework. diffoscope recursively unpack archives of many kinds and transform various binary formats into more human readable form to compare them. As a part of this project I worked on argument completion feature and ignoring .buildinfo feature. This project is a part of Reproducible Builds effort

• Links:
• Reproducible Builds
• Reproducible Builds Testing framework
• Diffoscope

Mentor and Co-Mentor

• Jérémy Bobbio (Lunar) : Mentor
• Reiner Herrmann (deki) : Co-Mentor
• Holger Levsen (h01ger) : Co-Mentor
• Mattia Rizzolo (mapreri) : Co-Mentor

Project Discussion

• Introduction to Reproducible Builds in Debian

  First time I came to know about Reproducible Builds was during Debconf

• I started to get involve from the start of March 2016. At the beginning Lunar suggested me to watch the talks given on Reproducible Builds wiki. I read documentation on Reproducible Builds site and started to participate in IRC discussions on #debian-reproducible.

• Application Review Period

  During proposal discussion period we discussed the areas where work needs to be done. I wrote the proposal and got it reviewed by community on the mailing list. Simultaneously, I worked on bug #818111 and submitted patch for same. That not only helped me to understand the concept of Reproducible Builds but also helped me to setup testing environment required to check the reproducibility of Debian packages.

• Community Bonding Period

  During community bonding period I studied the codebase of diffoscope and also spent enough amount of time for learning Python3 metaprogramming and other OOP concepts. We also discussed more about hiding differences and options for same. I couldn’t finish my project research work during this period since I had exams in May 2016 and it consumed almost half of community bonding period and week 1 of coding period.

Project Implementation

• Coding Period

  I continued my research regarding behaviour of diffoscope and Lunar suggested me to write specification for hiding options. I spent most of the time researching differences present on https://tests.reproducible-builds.org/debian/reproducible.html. Based on the work done, I came up with specification called Hide Profile for allowing diffoscope to hide different profiles. Link to the specification:
  • https://wiki.debian.org/ReproducibleBuilds/HideProfilesSpecification

  Later, I started to work on wish-list feature bug #826711. Links to commits:

  • https://anonscm.debian.org/cgit/reproducible/diffoscope.git/commit/?id=d733cb47903c54a3c4a699131e1b7bde3e492bbf

  • https://anonscm.debian.org/cgit/reproducible/diffoscope.git/commit/?id=77dfd8d3e5e2e5d8ba31e606ef36cbd8f2315743

  • https://anonscm.debian.org/cgit/reproducible/diffoscope.git/commit/?id=c15e6b5f74ae92462dee226a699e120d7017effb

    Above changes added argument completion feature to diffoscope and now you can press tab for option completion.

    • Build different Debian packages using prebuilder to study the output of diffoscope in different cases.

      Link : https://github.com/satyamz/buildinfo-cases/commits?author=satyamz

    • Collected information regarding how other Free softwares implements ignore options/interfaces.

      Link: https://gist.github.com/satyamz/77d86e1fc78d104c784c0238a3a7c0e3

    • Implementation of Ignoring .buildinfo file while comparing .changes files use case.

      • https://anonscm.debian.org/cgit/reproducible/diffoscope.git/commit/?id=c61f41c5128094508d6fff8b746cecc3841f6b27

Challenges and Work Left

• To understand the main purpose of diffoscope in the context of Reproducible Builds. I had to go through complete Reproducible Builds project. It consumed significant amount of time to understand what Reproducible Builds is, why it’s necessary important for Free software to build reproducibly. Diffoscope is the last tool in Reproducible Builds toolchain. It was a big challenge for me to understand whole process and objective of diffoscope.
• Work Left:
  • Implement important use cases given in the Hide Profiles Specification.

Future work

• Based on the research work and implementation done during Coding Period make diffoscope better and enhance ignoring capabilities of diffoscope.
• Improve the parallel processing feature of diffoscope. This particular problem is hard to understand and implement.
• Make diffoscope better by solving exsting bugs.

Acknowledgement

I would like to express my deepest gratitude to Lunar for mentoring me throughout Google Summer of Code program and for being cool. Lunar’s deep knowledge regarding diffoscope and Python skills helped me a lot throughout the project and we literally had great discussions. I would also like to thank Debaian community and Google for giving me this opportunity. Special thanks to Reproducible Builds folks for all the guidance!