I attended GopherCon India 2017, there was a talk on “Package Oriented Design In Go” by William Kennedy. In that talk, William explained some really important and thoughtful design principles which We can apply in our day to day life, while writing Go. Hence, I wanted to apply these design philosophies to the Go projects in which I have been working on as a part of OpenEBS project. I learnt a good practice of having a Go Kit project at the organization level from William’s talk.
The Kit project in Go is the common project containing all the standard libraries or packages used across all the Go projects in the organization. Packages in the Kit project should follow design philosophies.
Sometimes, We write same Go packages again and again to do the same task at different levels in the different Go projects under the same organization. For example, we write custom logger package in the different Go projects. If the custom logger package is same across the organization and can be reused by simply importing it, then this custom logger package is the perfect fit for Kit project. You can sense how much time and cost it will save for us when we have a Kit project.
Maya is the kit project in the progress. I will walk through our journey of creating a Kit project called maya for OpenEBS organization from existing Go projects. At OpenEBS, as a open source and growing Go project, We value Go principles and We try hard to leverage Go’s offerings. Maya is the Kit project for the Application projects like maya-apiserver, maya-storage-bot etc. Maya contains all the kubernetes & nomad API’s, common utilities etc. needed for development of maya-apiserver and maya-storage-bot. In the near future, we are trying to push our custom libraries to maya. So that, it will become a promising Go kit project for OpenEBS community.
We have specifically followed the package oriented design principles in Go to create maya as a kit project.
Orchprovider : orchprovider
contains packages of different orchestrators such as kubernetes and nomad.
types: types
provides all the generic types related to orchestrator.
pkg: pkg
contains packages like nethelper, util etc.
volumes: volumes
contain packages related to volume provisioner and profiles.
Packages in the Kit project should be purposeful. These packages must provide, not contain. In maya, we have packages like types, orchprovider, volumes etc. name of these packages suggests the functionality provided by them.
Portability is important factor for packages in kit project. Hence, we are making maya in such a way that it will be easy to import and use in any Go project. Packages in the Maya are not single point of dependency and all the packages are independent of each other.
├── buildscripts
│ └── docker
├── command
├── docs
├── example
├── scripts
│ └── config
└── templates
├── buildscripts
│ └── docker
├── cmd
├── docs
├── lib
│ ├── api
│ │ └── v1
│ ├── artifacts
│ │ ├── docker
│ │ ├── docker-compose
│ │ └── k8s
│ ├── config
│ ├── flaghelper
│ ├── loghelper
│ ├── mockit
│ │ └── etcmayaserver
│ ├── nethelper
│ ├── orchprovider
│ │ ├── k8s
│ │ └── nomad
│ ├── profile
│ │ ├── orchprovider
│ │ └── volumeprovisioner
│ ├── server
│ ├── util
│ └── volumeprovisioner
│ └── jiva
└── proposals
├── buildscripts
│ └── docker
├── command
├── docs
├── example
├── orchprovider
│ ├── k8s
│ │ └── v1
│ └── nomad
│ └── v1
├── pkg
│ ├── nethelper
│ └── util
├── scripts
│ └── config
├── templates
├── types
│ └── v1
│ └── profile
│ └── orchestrator
└── volumes
├── profile
│ └── volumeprovisioner
└── provisioner
└── jiva
├── buildscripts
│ └── docker
├── cmd
├── docs
├── lib
│ ├── artifacts
│ │ ├── docker
│ │ ├── docker-compose
│ │ └── k8s
│ ├── config
│ ├── flaghelper
│ ├── loghelper
│ └── server
└── proposals
Maya-apiserver uses maya as a Kit project. Maya-apiserver exposes OpenEBS operations in form of REST APIs. This allows multiple clients e.g. volume related plugins to consume OpenEBS storage operations exposed by Maya API server.
Maya-apiserver has been designed keeping storage inside containers in mind. In other words, containerized storage. This leads to maya-apiserver using best of breed container orchestrators. Hence maya-apiserver abstracts the container related orchestration needs in form of orchprovider namespace. Therefore,to achieve above need maya-api server uses orchprovider package provided by Maya.
There can be multiple persistent volume provisioners e.g. jiva & cstor which has the details of workings of its volume. maya-apiserver tries to abstract these into volumes operations whose specifics will reside in individual namespaces i.e. jiva, cstor, etc. Hence, maya-apiserver will make use of volumes package from Maya to satisfy above requirements.
Go Kit project should contain packages which are usable, purposeful and portable. Go Kit projects will improve the efficiency of the organization at both human and code level.
]]>In week 4, I mostly worked on designing an interfaces and tackling different issues related to argument completion feature of diffoscope and in week 5 I worked on adding hiding .buildinfo from .changes files.
.buildinfo
files are given but I needed diffoscope outputs for .changes
files. Hence, I had to build packages locally using our experimental tool chain. My goal was to generate different outputs and to see how I can hide .buildinfo
files from .changes
.I updated argument completion patch as per suggestions given by Paul Wise (pabs). Patch has been reviewed by Mattia Rizzolo, Holger Levsen and merged by Reiner Herrmann (deki) into diffoscope master. This patch closes #826711. Thanks all for support.
For Ignore .buildinfo
files when comparing .changes
files, we finally decided to enable this by default and without having any command line option to hide.
Last week I researched more on .changes
and .buildinfo
files. After getting guidelines from Lunar I was able to understand the need of this feature. I am in the middle of implementation of this particular problem.
hiding .buildinfo from .changes
I am thankful to Shirish Agarwal for helping me through visa process. But, unfortunately I won’t get visa till 5th July. So I don’t think, I would make it to debconf this year. I will certainly attend Debconf 2017. Good news for me is I have passed mid-term evaluations of Google Summer of Code 2016. I will continue my work to improve Debian. Even, I have post GSoC plans ready for Debian project ;)
Have a nice day :)
]]>In last 10 days, I had build different Debain packages at my own using prebuilder to experience the reproducibility issues. I am thankful to deki and Lunar for suggesting me to do that task. Based on this experience, I managed to find more use cases for –hide=profiles specification.
I also researched differences of different unreproducible Debian packages on http://tests.reproducible-builds.org . There are many packages available for examination.
In brief I did following tasks:
argcomplete
python module and had some hands on experience with module. Purpose of doing this was to add argument completion feature to Diffoscope. pabs had filed bug report for this #826711. I am implementing this feature and discussing issues with pabs as well as researching diffs side-by-side to generate more use cases. Here, Thanks to pabs for guidance and support :)Upcoming week will be an important as well as fun week because I will be implementing the use cases. Right now, I am currently looking at different softwares which ignores stuff and taking notes of it. So that, it will help me during implementing solution of use cases. I am also looking forward to feedback from community on use cases and CLI interfaces. Have a great week :)
]]>I started working from 28 May as the exams ended. This report is bit late. I started work on –hide=profiles flag. Till now, I did/doing following tasks:
Lunar suggested me to create the specification for –hide=profiles. It can be found here https://wiki.debian.org/ReproducibleBuilds/HideProfilesSpecification
General purpose of addition of –hide=profiles to diffoscope is because we want to provide an alternative for tool like OBS pkg-diff. We want to increase userbase of diffoscope and thus number of contributors. Hopefully, I will finish this part as soon as possible.
We are still discussing, how things should be done! Right now, I am researching diff of unreproducible packages and trying to develop use cases. I will update the HideProfilesSpecification wiki with my findings. Lunar has given me a proper directions and community is very much helpful. :)
My goal for an upcoming week is to search for packages which are unreproducible and generate use cases for diffoscope. This week I will also try to fix the reproducibility of Debian packages.
I am loving, contributing to Debian and free software and it’s a great learning experience. Many things to come in upcoming time :)
]]>I am Satyam Zode I am a final year Computer Science student (Satyam_z on IRC). I live in Pune, India (GMT +5:30). I am pursuing my undergraduate degree in Computer Engineering from Pune Institute of Computer Technology, Pune. I have been programming for the past 4 years. I am well versed in C/C++, Python3, and Golang. My Alioth and Github handles are satyamz-guest and satyamz respectively. I have been using GNU/Linux and free software from last four years. I am an open source enthusiast and I have been following Hacker culture since past three years.
I am glad that I have got an opportunity to contribute to the Debian Project via Google Summer of Code 2016. I am accepted for project Improving diffoscope tool and reproducibility of Debian packages. This Summer and beyond I will be working with Debian Reproducible Builds team to improve the debbugging tool called Diffoscope (previously known as debbindiff). Thanks a bunch to Debian community, Lunar, Holger Levsen, Reiner Herrmann, Mattia Rizzolo and reproducible-builds folks for giving me this opportunity. Here is my GSoC'16 Proposal. And Yay! It really feels great.
I will be working on Diffoscope tool which is debbugging tool developed under reproducible-builds effort. Basically, Diffoscope compares two files and shows the difference in text and html format. Diffoscope is mainly developed to compare two Debian packages which may consist of binary files, tar files, text files etc. Diffoscope helps to identify difference between two Debian packages with respect to timestamps, file ordering etc. Diffoscope will try to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human readable form to compare them. It can compare two tarballs, ISO images, Debian packages or PDF just as easily. Diffoscope helps to identify the reproduciblity of Debian packages. During this summer I will be improving Diffoscope. I will be mainly working on:
My next blog post will be regarding community bonding. Thanks for reading :)
]]>