I just did some Debian package review in a somewhat unusual way, and I
wanted to share that. I'm hoping other Debian developers (and other free
software contributors) that need to review others' contributions can
learn something from this, and that I can use this blog post as a way to
find out if other people are doing something similar.
It was pretty exciting! At the end of it, I joined #debian-mentors to talk
about how my cool process. Someone summarized it very accurately:
<sney> it almost sounds like you're working to replace yourself with automation
Context about alpine in Debian
(Skip to "Package review, with automation" if you're familiar with Debian.)
I'm the maintainer of alpine in Debian. There are quite a few problems
with the alpine package in Debian right now, the biggest of which are:
- We're one version behind -- 2.11 is the latest available, but 2.10 is the newest that we have in Debian.
- The packaging uses a decreasingly-popular packaging helper, cdbs, about which I happen to know less than the dh-style helper (aka dh7).
- There are lots of bugs filed, and I don't respond in a timely fashion.
This doesn't change my deep love for alpine -- I've had that for about
half my life now, and so far, I don't see it going away.
A month or so ago, I got a friendly private message from Unit193, saying
he had converted the package to the dh style, and also packaged the
newer version. They wanted to know if they should clean this up into
something high-enough quality to land in Debian.
(In Debian, we have a common situation where enthusiastic users update
or create a new package, and aren't yet Debian Developers, so they don't
have permission to upload that directly to the "Debian archive", which
is the Debian equivalent of git master. Package "sponsorship" is how we
handle that -- a Debian Developer reviews the package, makes sure it is
of high quality, and uploads it to the Debian archive along with the
Debian Developer's OpenPGP signature, so the archive processing tools
know to trust it.)
On Friday evening, I had a spare moment, so I sent a private message to
Unit193 apologizing for not getting back to them in a reasonable amount
of time. Having another person help maintain is a pretty exciting
prospect, and I wanted to treat that enthusiasm with the respect it
deserves, or at least apologize when I haven't. I was surprised to see a
reply within a few minutes. At that point, I thought: I wasn't
planning on doing any package review this weekend, but if they're
online and I'm online... might as well!
Package review, with automation
Unit193 and I popped into ##alpine on irc.freenode.net, and I started
reading through their packaging changes, asking questions. As I asked
questions, I wondered -- how will I know if they are going to fix the
issues I'm raising?
Luckily, Unit193 wanted to use git to track the packaging, and we
settled on using git-buildpackage, a tool that was fairly new to both of
us. I thought, I might as well have some executable documentation so I
don't forget how to use it. ("Executable documentation" is Asheesh-speak
for a shell script.)
One thing I knew was that I'd have to test the package in a pbuilder, or
other pristine build environment. But all I had on me at the moment was
my work laptop, which didn't have one set up. Then I had a bright idea:
I could use Travis-CI, a public continuous
integration service, to check Unit193's packaging. If I had any
concerns, I could add them to the shell script and then point at the
build log and say, "This needs to be fixed." Then there would be great
clarity about the problems.
Some wonderful things about Travis-CI:
- They give you root access on an Ubuntu Precise (10.04) virtual machine.
- Their build hosts are well-connected to the Internet, which means fast downloads in e.g. pbuilder.
- They will let you run a build for up to 50 minutes, for free.
- Build just means "command" or "set of commands," so you can just write a shell script and they will run it.
- Travis-CI will watch a github.com repository, if you like. This means you can 'git commit --allow-empty' then 'git push' and ask it to re-run your script.
Since Unit193's packaging was in git (but not on github), I created a
git repo containing the same contents, where I would experiment with
fixes for packaging problems I found. It'd be up to Unit193 to fix the
problems in the Alioth packaging. This way, I would be providing advice,
and Unit193 would have an opportunity to ask questions, so it would be more
like mentorship and less like me fixing things.
We did a few rounds of feedback this way, and got the packaging to higher
and higher quality. Every time Unit193 made a fix and pushed it, I would
re-run the auto-build, and see if the problems I spotted had gone away.
While the auto-build runs, I can focus on conversing with my mentee
about problems or just generally chatting. Chatting is valuable
community-building! It's extremely nice that I can do that while waiting
on the build, knowing that I don't have to read it carefully -- I can
just wait a few minutes, then see if it's done, and see if it's red or
green. Having the mentee around while I'm reviewing it means that I can
use the time I'm waiting on builds as fun free software social time.
(Contrast this with asynchronous review, where, all alone, I would wait
for a build to finish, then write up an email at the end of it all.)
This kind of mentorship + chatting was spread out over Friday night,
Saturday night, and Sunday morning. By the end of it, we had a superb
package that I'm excited to sign and push into Debian when I'm next near
my OpenPGP key.
Implementation details
You can see the final shell script here:
And you can see the various builds here:
The shell script:
- Alternates between the Alioth packaging vs. my fork of it. (This way, I can test packaging changes/suggestions.)
- Disables ccache in pbuilder, due to a permissions problem with ccache/pbuilder/travis-ci, and I didn't need ccache anyway.
- Handles 'git dch' slightly wrong. I need to figure that out.
- Optionally passes --git-ignore-new to git-buildpackage, which was required initially, but should not be required by the time the package is ready. (This is an example of a thing I might forget to remark upon to my mentee.)
- Plays games with git branches so that git-buildpackage on Travis-CI can find the pristine-tar branch.
- Tries to use cdn.debian.net as its mirror, but on Saturday ran into problems with whicever mirror that is, so it falls back to mirror.mit.edu in case that fails.
- Contains a GPG homedir, and imports the Debian archive key, so that it can get past Ubuntu-Debian pbuilder trust issues.
I also had a local shell script that would run, effectively:
- git commit --allow-empty -m 'Trigger build'
- git push
This was needed since I was basically using Travis-CI as remote shell
service -- moreover, the scripts Travis-CI runs are in a different repo
(travis-debcheck)
than the software I'm actually testing (collab-maint/alpine.git).
Unit193 and I had a technical disagreement at one point, and I realized
that rather than discuss it, I could just ask Travis-CI to test which
one of us was right. At one point in the revisions, the binary package
build failed to build on Precise Pangolin (the Ubuntu release that the
Travis-CI worker is running), and Unit193 said that it was probably due
to a problem with building on Ubuntu. So I added a configuration option
to build just the source package in Ubuntu, keeping the binary package test-build within the Debian sid pbuilder, although I believed
that there was actually a problem with the packaging. This way, I could
modify the script so that I could demonstrate the problem could be
reproduced in a sid pbuilder. Of course, by the time I got that far,
Unit193 had figured out that it was indeed a packaging bug.
I also created an option to SKIP_PBUILDER; initially, I wanted to get
quick automated feedback on the quality of the source package without
waiting for pbuilder to create the chroot and for the test build to
happen.
You might notice that the script is not very secure -- Niels Thykier
already did! That's fine by me; it's only Travis-CI's machines that
could be worsened by that insecurity, and really, they already gave me a
root shell with no password. (This might sound dismissive of Travis-CI
-- I don't mean it to be! I just mean that their security model already
presumably involves throwing away the environment in which my code is
executing, and I enjoy taking advantage of that.)
Since the Travis virtual machine is Ubuntu, and we want to run the
latest version of lintian (a Debian packaging "lint" checker), we run
lintian within the Debian sid pbuilder. To do that, I use the glorious
"B90lintian" sample pbuilder hook script, which comes bundled with
pbuilder in /usr/share/doc/pbuilder/.
The full build, which includes creating a sid pbuilder from scratch,
takes merely 7-10 minutes. I personally find this kind of shockingly
speedy -- in 2005, when I first got involved, doing a pbuilder build
seemed like it would take forever. Now, a random free shell service on
the Internet will create a pbuilder, and do a test build within it, in
about 5 minutes.
Package review, without automation
I've done package review for other mentees in the past. I tend to do it
in a very bursty fashion -- one weekend day or one weeknight I decide
sure, it's a good day to read Debian packages and provide feedback.
Usually we do it asynchronously on the following protocol:
- I dig up an email from someone who needed review.
- I read through the packaging files, doing a variety of checks as they occur to me.
- If I find problems, I write an email about them to the mentee. If not, success! I sign and upload the package.
There are some problems with the above:
- The burstiness means that if someone fixes the issues, I might not have time to re-review for another month or longer.
- The absence of an exhaustive list of things to look for means that I could fail to provide that feedback in the first round of review, leading to a longer wait time.
- The person receiving the email might not understand my comments, which could interact really badly with the burstiness.
I did this for Simon Fondrie-Teitler's python-pypump package recently.
We followed the above protocol. I wrote a long email to Simon, where I
remarked on various good and bad points of the packaging. It was part
commentary, part terminal transcript -- I use the terminal transcripts
to explain what I mean. This is part of the email I sent:
I got an error in the man page generation phase -- because at
build-time, I don't have requests-oauthlib:
make[2]: Leaving directory `/tmp/python-pypump-0.5-1+dfsg/docs'
help2man --no-info \
-n 'sets up an environment and oauth tokens and allows for interactive testing' \
--version-string=0.5.1 /tmp/python-pypump-0.5-1+dfsg/pypump-shell > /tmp/python-pypump-0.5-1+dfsg/debian/pypump-shell.1
help2man: can't get `--help' info from /tmp/python-pypump-0.5-1+dfsg/pypump-shell
Try `--no-discard-stderr' if option outputs to stderr
make[1]: *** [override_dh_auto_build] Error 1
This seems to be because:
➜ python-pypump-0.5-1+dfsg ./pypump-shell
Traceback (most recent call last):
File "./pypump-shell", line 26, in <module>
from pypump import PyPump, Client
File "/tmp/python-pypump-0.5-1+dfsg/pypump/__init__.py", line 19, in <module>
from pypump.pypump import PyPump, WebPump
File "/tmp/python-pypump-0.5-1+dfsg/pypump/pypump.py", line 28, in <module>
from six.moves.urllib import parse
ImportError: No module named urllib
$ ./pypump-shell
Traceback (most recent call last):
File "./pypump-shell", line 26, in <module>
from pypump import PyPump, Client
File "/tmp/python-pypump-0.5-1+dfsg/pypump/__init__.py", line 19, in <module>
from pypump.pypump import PyPump, WebPump
File "/tmp/python-pypump-0.5-1+dfsg/pypump/pypump.py", line 29, in <module>
from requests_oauthlib import OAuth1
ImportError: No module named requests_oauthlib
The deeper problem was a missing build-dependency, and I explained that
in my email. But the meta problem is that Simon didn't try building
this in a pbuilder, or otherwise clean build environment.
Simon fixed these problems, and submitted a fresh package to
Paul Tagliamonte and myself. It happened to
have some typos in the names of the new build dependencies. Paul
reviewed the fixed package, noticed the typos, fixed them, and uploaded
it. Simon had forgotten to do a test build the second time, too, which
is an understandable human failure. There was a two-day delay between
Simon's fixed resubmission, and Paul signing+uploading the fixed result.
In a pedagogical sense, there's something disappointing about that
exchange for me: Paul fixed an error Simon introduced, so we're not
teaching Simon to take total responsibility for his packages in Debian,
nor to understand the Debian system as well as he could. (Luckily, I
think Simon already understands the importance of taking responsibility!
In this case, it's just a hypothetical in this case.)
For the future
The next time I review a package, I'm going to try to do something
similar to my Travis-CI hack. It would be nice to have the do.sh script
be a little more abstract; I imagine that as I try to use it for a
different package, I'll discover the right abstractions.
I'd love it if Travis-CI did not require the git repositories to be on
GitHub. I'd also like if the .travis.yml file could be in a different
path. If so, we could create debian/travis-configuration (or something)
and keep the packaging files nice and separate from the upstream source.
I'd also love to hear about other people's feedback. Are you doing
something similar? Do you want to be? Would you have done some of this
differently? Leave a comment here, or ping me (paulproteus) on #debian-mentors
on irc.debian.org (aka irc.oftc.net).
I'll leave you with some conversation from #debian-mentors:
<paulproteus> The automation here, I think, is really interesting.
<paulproteus> What I really want is for mentees to show up to me and say "Here is my package + build log with pbuilder, can you sponsor it please?"
<Unit193> Oooooh!
-*- Unit193 gets ideas.
<paulproteus> Although the irony is that I actually like the community-building and relationship-building nature of having these things be conversations.
<bremner> how will this brave new world cope with licensing issues?
<paulproteus> bremner: It's not a replacement for actual review, just a tool-assist.
<paulproteus> bremner: You might be relieved to know that much of Unit193's and my back and forth related to get-orig-source and licensing. (-:
<bremner> I didn't doubt you ;).
<paulproteus> If necessary I can just be a highly productive reviewer, but I would prefer to figure out some way that I can get other non-paulproteus people to get a similar assist.
<paulproteus> I think the current blocker is "omg travis why are you bound to githubbbbbbbb" which is a reasonable concern.