Spacewalk: building a patch culture

People behave as they see others behave. If people on a mailing list are civil, then newcomers will try to be civil. If people on a mailing list are snippy, then newcomers will feel justified being snippy.

And if people submit patches to a mailing list, and those patches are treated well by the engineers who receive them, then newbies will be more comfortable submitting patches to that list.

In the past few months, we have seen two newcomers to the Spacewalk project: Joshua Roys of Georgia Tech Research Institute, and Maxim Burgerhout, sysadmin from parts unknown. (I hope they won’t mind being discussed in so public a way.) Let’s see how they joined up.

Maxim Burgerhout’s very first post to spacewalk-devel list, in April, was a patch (0). It was a simple patch to, which is part of Spacewalk’s built-in configuration management. Honestly, so many people use puppet and cfengine these days, I don’t expect many people to be using Spacewalk for configuration management. But its one advantage is that it is quite well-integrated, so it’s worth a look. Maxim is clearly a power user of the rhncfg functionality, but hacking code is only an occasional thing for him. In his words, “I usually spend my time being an sysadmin, so I hope I’m doing this ‘sending patch’ stuff the right way. ;-)” Which implies that this was Maxim’s first-ever patch sent to a mailing list! W00t! Since then he has submitted another couple of patches, and is thus a member in good standing of the Spacewalk team.

Joshua Roys, from Georgia Tech Research Institute, made his initial post to spacewalk-devel list in March 2009 with a simple patch fixing bootstrap scripts to retrieve a server cert using SSL (1). It wasn’t quite the right idea, as Jan Pazdziora gently pointed out, due to a (very common) chicken-and-egg problem with the cert.

Then in April, Joshua dug into a tricky issue with oracle-instantclient-selinux, for which Jan publicly thanked him (2).

On May 28th, Joshua posted the largest patch ever offered by a volunteer contributor to Spacewalk (3) — a massive undertaking, in Joshua’s words, “to make the life of government Information Assurance teams easier on Linux.” This patch, essentially a complete new feature, also came with detailed descriptions of the new functionality, including screen shots (4). He has followed it up with a few patches since, expanding the initial functionality.

Spacewalk is a daunting codebase. I know, because I was one of the original engineering managers for that codebase, and trying to dig back into it after four years is painful. Still, dig into it I shall, in the hopes of making it a little more accessible for the next guy.

Which makes it all the more impressive that Joshua was able to make such invasive changes to such a huge codebase. Why did he do it? Simple. He needed the functionality, so he wrote it, and he needed other people to support it, so he shared it.

I would argue that creating a strong “patch culture”, which the Spacewalk engineers did from the day they launched, is part of what is making such successes in Spacewalk possible now.


Spacewalk: building a patch culture

An update on Professor Quest.

It’s been a while since I’ve posted anything substantive about what I’m doing these days, and given the week I’ve just had, I figured this was as good a time as any to catch folks up. Those of you that are interested, anyway.

Today I’ll focus on our shared quixotic attempt to change the way that computer science education works.

This is one of my big goals for the year, both personally and professionally. It all started with a simple premise: wouldn’t it be great if computer science programs around the world could learn computer science by actually *doing* computer science, and helping out the world in the process? Pretty cool idea, right? And also rather obvious, it would seem.

I quickly discovered that it wasn’t as obvious as all that. A big chunk of the last year of my life has been spent learning all about how kids study computer science in college.

In short: I’m not a fan.

On Thursday and Friday of last week, I was at Drexel University in Philadelphia, at the Softhum workshop — an NSF-funded workshop put together by the folks responsible for The “H” stands for Humanitarian; the mission of the HFOSS organization is to get kids excited in computer science by giving them project-based learning opportunities in computer science and engineering that have a humanitarian purpose. Learn something, help your fellow man. A simple and awesome idea. Softhum gathered together about 25 professors from around the country, large schools and small schools, all interested in learning more about putting their students to work in the open source world. I was there, along with Frank Hecker from Mozilla, to provide the viewpoint of open source projects. and are two organizations that share a very similar agenda, and I expect that they will be greatly complementary. focuses on identifying and eliminating the barriers that prevent professors from participating in open source; provides a very specific context in which to put these principles to work. I’m happy to be part of both, and I was delighted to see the people behind Softhum (Ralph Morelli, Heidi Ellis, Greg Hislop) pushing its participants towards

I was gratified to see that one of the biggest questions shared by professors was, “how can I teach open source development practices to my students if I have not myself participated in open source?” It makes me believe that we’re on the right track with POSSE, and with The Mel now driving much of the day-to-day work of making the event happen, and Chris and Humph having the teaching end well in hand, I feel confident that at the conclusion of POSSE in late July, I will be standing high atop the Red Hat parking deck in my flight suit, proudly declaring Mission Accomplished, with a steely gaze in my eye and a spring in my step. Ah, what a fine day that will be. And then, of course, we get to figure out how to make that program scale from seven professors to seven hundred — but one step at a time.

The other big question, though, was a question for which I, personally, am not yet satisfied that there’s A Good Answer For. Of the many unanswered questions that remain, it is this particular unanswered question that will, I think, mark the difference between A Good Idea and A Movement.

That question: “what are the small open source projects that my students can work on, right now, and where can I find them?”

Mozilla is definitely working hard on answering this problem. Which is not surprising; they’ve been out front on this stuff for a while. They’ve instituted the student-project tag for their bugzilla instance, and to date, there are 80 issues in their bugzilla tagged this way. Maybe not perfect — but for a professor who’s considering throwing kids at the Mozilla project, it’s *incredibly* useful. That, combined with the mentoring page for Mozilla at TOS, gives a professor a pretty clear idea of how to get their students involved in the Mozilla project.

We’re certainly close to that in Fedora-land, and with Chris Tyler’s help, I expect that we’ll get closer in a hurry. We need our own project mentoring page on TOS, and that should be pretty straightforward — but figuring out what Fedora projects are suitable for student development may be somewhat more challenging. Looking at the status quo, I see that we’ve got a great list of unaccepted feature requests, which are categorized as “FeaturePageIncomplete” on the Fedora wiki, listed on this single page. (gregdek’s feature request #1: fix mediawiki to put a horizontal scroll bar on this page so you can actually *see* all of them.) There’s very little indication about which of these projects might be suitable for student developers, though, and that’s going to be really important. I’d love to hear recommendations about how to do that.

Anyway, we’re getting closer all the time. Once we’re satisfied that Fedora is Doing It Right, we get to work on all the other projects out there.

Good stuff. I’m excited by our rapid progress in this area in the past few months, and I enjoyed making lots of new friends in Philadelphia. Maybe next summer we’ll hold a POSSE there at Drexel; Greg Hislop seemed to like the idea and was ready to volunteer the space. I hope things go well enough that we can take him up on his offer. Philly’s a great college town.

An update on Professor Quest.

Waterloo — couldn’t escape if I wanted to

I visited the University of Waterloo last week to keynote a small conference at their Accelerator Centre, which is their technology incubator. The topic of the conference: “open or closed, which should you choose with your new project?” I posed my keynote as a set of questions that every software innovator should ask themselves when making this decision — and, of course, the questions I chose were quite leading. 🙂

Sometimes I’ve got the mojo and sometimes I don’t, and for this speech, on this day, I felt like I had the mojo. It was a good day. It was a great crowd, very engaged. Mad props to Chris Tyler of Seneca for attending with me, and ferrying me to and from YYZ, and all of the great discussions that we had on those long rides. (And also for the roasted chicken at Swiss Chalet. Yum.)

But the *real* reason I wanted to attend was because one of the hosts of the event was C4, a technology transfer organization counting ten Canadian members, university and industry, from Southwest Ontario. When pushing the open source agenda at the university level, one of the many hurdles we hit repeatedly (and particularly in the US) is the intransigence of TT folks, and their inability to appreciate, or even understand, the value of open source software. After the conference concluded, I had the opportunity to sit down at lunch with a roomful of open source friendly TT professionals, to pick their brains about why, exactly, it’s so hard for TT folks to “get” open source, and what we can do to help change their minds.

Following are some of the key insights I gleaned about the business of TT from this discussion — and if I’ve misunderstood, please feel free to comment, since I’m still new to all this.

* TT orgs differ by mission largely by how the university itself treats IP. In universities where IP is institutionally-owned — which describes the vast majority of large American universities — the mission of TT orgs tends strongly to be generating revenue for the institution. In orgs in which the IP is researcher-owned, which describes a significant percentage of Canadian universities, near as I can tell, and a handful of US schools — the goal of the TT office is much more a facilitator to help the researcher get that technology out into the world, in whatever way the researcher sees fit.

* Most TT folks aren’t generally good at software; it’s not their sweet spot. They are more accustomed to dealing with technology from industries that involve the creation of physical goods. Therefore, their first question is *always* “what defensible IP have you created?” It’s not just that TT orgs are uncomfortable with open source software; frequently, they are uncomfortable with *any* software.

* In those cases where TT is seen as a money-maker, the role of the TT office is very much like the role of a venture capitalist. Unfortunately, the savvy that some VCs have picked up when it comes to dealing with open source has not yet rubbed off on TT folks — and let’s face it, TT orgs doesn’t have the wherewithal, the time, or the inclination to gain that degree of knowledge. VCs look at a whole range of variables in assessing the potential success of a project. TT orgs look at the one thing they can most easily quantify and protect: “what’s the IP?” It might be worthwhile to hook up the smart VCs who “get” open source with some of the larger TT orgs who don’t. Seems like it could be a win-win: VCs get an inside track on cool technology; TT orgs get a more reliable set of indicators about what makes open source profitable.

* Because TT is seen as a revenue stream, university policies make it difficult/impossible to go around them. TT orgs don’t have to “kill” open source projects, and generally they don’t; they just never give permission. Allow me to explain by way of a brief skit: “Hi, Dr. Doom, can I release this work from my thesis as an open source project? Yes, you say, I just have to get clearance through the TT office? Okay, great! Hello, TT office? I’ve got this software project I’d like to release as an open source project. You’ll look it over and get back to me? Great! When can I expect to hear back from you? Hello? Hello?” (Exeunt omnes, laughing.)

* Some professors just release their work anyway, and the ones who do often use the guise of “publication” to do so. Which reinforces the importance of creating a journal for this purpose; not only will it help solve the publish-or-perish issue, it may also help with the TT issue. I’m glad that it’s one of the key focuses (foci?) of the TOS group.

* There is *zero* information for TT orgs to look at when it comes to open source. This was the message that the C4 folks left me with: how will TT folks learn if someone doesn’t produce guidelines for them to follow? The C4 folks vowed to work with us on this problem, if we could help give them some direction. I hope we can figure out how to take them up on that offer.

A very useful meeting for me; I learned a lot. And Chabriol, thank you so much for my speaker’s gift. The dogs absolutely loved the treats. Definitely the most thoughtful gift I’ve ever received as a speaker.

Waterloo — couldn’t escape if I wanted to

4th grade math: an update

As with all new projects, uptake is slow but steady. The mailing list traffic looks pretty good for its first month of life; we’ve got a few people who are working on various math-related activities; there are nibbles of interest.

Pulling all of this interest into a coordinated effort will be a continually ongoing exercise.

The next real milestone, I think, is to really flesh out the Math4 teams. Putting together a few strong teams that consist of at least one developer and one teacher will really help move us forward.

Another important goal for me personally is to ramp up my recruiting efforts — which means refining the pitch and finding as many opportunities as possible to make the pitch to interested people. There are only so many hours in the day, and I’m pulled in a ton of different directions, but pitching the vision is probably the most important activity I can be engaged in right now. Which means that my time for hacking Mongo is somewhat curtailed. Alas. I suck as a coder anyway. 🙂

Anyway. There’s lots of efforts out there to “help kids” using free software. I sincerely believe that helping to fill out the 4th grade math curriculum is one of the most immediately actionable things we can do as a community. Tell your friends — especially your teacher friends. Join the fight.

4th grade math: an update

Still looking for professors: tell your friends!

Red Hat wants professors to participate in open source projects for a week this summer, and we’re willing to put our money where our mouth is.

We are accepting applications for this program right now. There have been some nibbles, but no formal applications yet.

The deadline for professors to apply is Friday, April 3rd, 2009. If you know any professors who would like to participate in an open source project for a week this summer, please tell them about this program. And if you have any questions, feel free to ask me.

That is all. 🙂

Still looking for professors: tell your friends!

Hello, friends from TeachingOpenSource!

Chris Tyler, you are the man.

Now we’ve got a great bit of infrastructure for educators who are trying to teach open source to their students. What are we going to do with it?

Funny, I just spent a day with friends in Chattanooga at SIGSCE. A room full of professors and industry folks, all of whom care about exposing students to open source. We had some amazing discussions. My agenda was to find out why, exactly, more professors aren’t already engaging their students in open source. Some big obstacles, and some ideas about what we might be able to do about them:

1. Tenure. Wow, is this a big one. The way tenure works at universities really makes it difficult for junior professors to engage in any open source activities at all, and I’ve heard enough excited professors hammer this point, again and again, over and over, that it’s starting to sink in. We really need to figure out how to help profs solve this problem.

Some ideas here, briefly. One, create an academic-oriented open source conference with peer-reviewed, juried papers. Two, create an academic-oriented open source journal with peer-reviewed, juried papers. Three, work to convince professors that “peers” in computer science don’t have to be professors — that when Alan Cox says “yes, this paper at OLS is good,” it should have as much clout as the opinion of any professor. Four, focus on bringing research money to the actual creation of open source projects; nothing sells tenure boards like cash in the university’s hands.

2. Having bite-sized projects. This is a huge problem. Every time a professor thinks “gee, I’d like to find a project suitable to have my kids work on” and then goes to Sourceforge to find this project, a kitten dies and a professor loses his/her mind.

This is actually a very broad problem. Every sizeable open source project faces this problem — the problem of identifying on-ramp mini-projects that (a) are not so critical that they will harm the project if they Do It Wrong, (b) are small enough to be easily handled in several weekends of hard work, and (c) have artifacts like, oh, use cases — artifacts that professors need because they’re trying to teach this stuff.

Google Summer of Code does the open source community the great favor of *forcing* open source projects to articulate these kinds of mini-projects. For every project that Google approves, they turn down two. (More or less.) Figuring out how to leverage this work so that professors can pick up the projects that Google leaves on the table could be *tremendously* useful — and Leslie is thinking about it, so that’s good. 🙂

3. Having course materials. Lots of professors are doing lots of good stuff, but it’s spread out all over creation. Identifying all the CC-licensed CS curriculum stuff in the world and aggregating it somewhere would be great. Like, say, at That would be awesome.

So. A productive couple of days, and lots of good ideas. Now comes the hard work of turning these ideas into reality — but the establishment of is an important step down this road.

Hello, friends from TeachingOpenSource!