Dook-a-lyptus!

Loving the work that the RENCI folks at Duke are doing on top of Eucalyptus. They’ve got a set of patches that sit atop Eucalyptus proper, and they call their patches “Neuca”. I lol’d when I found that out.  It rolls swimmingly off the tongue.

We’ve seen quite a few of these kinds of projects.  It’s a key indicator of success that people are building this functionality on top of our base.

Our ability to incorporate these kinds of patches directly into mainline will be a key indicator of our maturity as an open source company going forward.

As I discussed at SCALE 10x this past weekend, I believe that our current contributor agreement needs an update.  That work will take some time, and I can’t really say much about it yet — but the prospect of working more closely with RENCI and others provides strong motivation to Get It Right.  It’s a key challenge, and I’m exciting about tackling it head-on in the coming weeks.

(p.s. RAGE HATE SPELLING FAIL.  The likelihood of my typing “eucalytpus” is pretty much an even money bet.)

FUDCon, How I’ve Missed You

It’s always nice to visit family for a while, and see how the kids have grown.  FUDCon Blacksburg felt an awful lot like a family reunion — except one with a lot more learnin’ going on.  Just a small part of what I learned:

* ARM is coming.  Raspberry Pi is cool, and there’s way cooler down the road.  We’re working on our first little event kit that Eucalyptians will be able to use for demos at some point.  Right now it’s three laptops.  In a couple of years, it’s likely to be a laptop and a half-dozen itty bitty ARM systems.

* Talked with Seth and Smooge about a use case we’d been considering in Fedora-land for a long time: the “community cloud”, in which community members can basically dedicate machines to the Fedora cause.  It’s been a dream scenario for a long time — maybe we’ve got a shot at making it reality with Euca’s help.  Each contributed machine could become the equivalent of an “availability zone”, essentially.  Could be an interesting way of provisioning a lot of build systems in a pinch, and so on.  We’ll see how things shake out.

* Silvereye, a project to simplify Euca installation, is coming along.  I installed my first ever Eucalytpus system this weekend using Silvereye 0.01 and Euca 2.0.3, and I’m basically the equivalent of a trained chimpanzee.  We’ve got some improvements to make, but I’m convinced that we’re on the right path.  (Silvereye == Silver I == AgI == Silver Iodide == Cloud Seeder.)

* I love the ideas behind Boxgrinder, and Marek’s presentation reinforced that lovin’ feeling.  I think we should absolutely be pimping Boxgrinder as one of the primary tools to build Euca images.  Poke to Marek to make slide deck public plz. 

* Thanks to John Mark’s excellent Gluster presentation, I learned why Eucalyptus calls its storage mechanism “Walrus”.  I am clearly late to this understanding.

Good times, good times.  Looking forward to the next one, whenever and wherever that may be.

Why the Fedora ISV SIG never caught fire

Here’s a list of popular open source products that cannot currently be found in Fedora repos:

• Zimbra
• JasperSoft
• SugarCRM
• Alfresco
• Magento
• Eucalyptus
• JBoss

Once upon a time, it was part of my job to help these kinds of companies to work more closely with Fedora. We created the ISV SIG for this purpose. Karsten and I would go to trade shows and meet with various open source vendors, and we’d talk with them at length about the great benefit of leveraging the Fedora install base, and the power of “yum install YourCoolProduct”, and the general usefulness of building an ISV packaging community, and they’d nod and smile, and then we’d have a follow-up meeting or two to discuss the ins and outs of being in a distro. And then… well, nothing much would happen.

Now, as it turns out, I’m in a position to appreciate, and articulate, these issues from the ISV’s perspective.

What do the applications listed above have in common? A couple of key points.

Point One: they are all sponsored by companies, who use the open source projects as a base from which to build proprietary products.

Point Two: they all tend to be the primary application running on their machine — in other words, they are appliance stacks — and they need to limit variance in those stacks to help guarantee a good experience for their users.

It’s easy to claim, and many do, that these projects aren’t in Fedora (or Ubuntu, for that matter) because of Point One. In truth, Point Two is *way* more important.

There’s a great page on the Fedora Wiki that does a good job of discussing the potential gains and losses of putting your ISV application into Fedora. I’m going to go through those gains and losses, and share my opinions of them, now that I’m on the other side of the fenceline.

[GAIN] Reduced maintenance burden for all dependencies that are already packaged in Fedora: no need to ship security updates for those components.

This is a good potential gain, but note that it does not require the ISV to be *in* the distro to get this gain. It’s entirely possible to package *on top of* the distro, track the distro closely, and get all of these maintenance gains, without incurring the high cost of pushing packages into the distro and maintaining them. I suspect that this is precisely what many companies choose to do.

[GAIN] Code auditability: the Fedora packaging processes ensure that all code is described by metadata (i.e., spec files). The packaging tools allow this data to be queried in informative ways. ISVs don’t necessarily track this data otherwise.

Also true, but again, note that it’s possible to build RPMs and get the same advantages without putting those RPMs into the distro. There are two separate costs here: there’s the cost of building an RPM, which is comparatively low if you’ve got the source and an experienced packager at your disposal — but then there’s the cost of pushing the RPM into the distro and following the distro’s rigorous rules around versioning and namespacing and supportability, which is a *much* higher cost for the ISV. The gain from that additional cost must therefore be demonstrably compelling.

[GAIN] Availability of package-specific expertise: ISVs can consult other packagers about the upsteams of their dependencies. Each Fedora package maintainer acts as a known point of contact for their package’s upstream project.

This is very much a potential gain, if it’s true. But what happens when most of the packages aren’t yet in Fedora? This is especially problematic in the Java world, where there are tons and tons and tons of jar files that are not “packaged” as such in Fedora, but are still perfectly useful to the Java community in jar form. If the distro packaging expertise for a particular jar doesn’t yet exist yet, then the company who pushes the packages into the distro must take on the initial cost of becoming that expert. It’s definitely true that this expertise can be shared over time, and also true that such shared expertise is a long-term win — but the upfront cost is high, especially for a small company that has lots of competing priorities.

[GAIN] The trust of Fedora users: ISV products packaged in the Fedora way will be more warmly-received by Fedora users than standalone GNU/Linux binaries.

Citation needed. I mean, yes, I believe this too, but it’s a gain that’s difficult to quantify. The real benefit we’re trying to claim here is that “yum install foo” is a simpler and awesomer experience — and it is. But the difference between “yum install foo” and “wget foo-installer | sh”, which adds the ISV’s yum repo and gpg key and then kicks off “yum install foo”, is not really that great.

[GAIN] Stability on Fedora: standalone binaries break frequently because Fedora is such a fast-moving target. Built-from-source packages have proven much more stable, since incompatilities are caught during mass rebuilds.

This is a bit of a tautology. It’s essentially arguing that your ISV packages will build better with Fedora because you’re working to make them build better with Fedora. Which is true, but again, can be true by building *on top of* Fedora and not *in* Fedora.  And it also only addresses build time failures, which, for an application, are failures that you’re likely to find immediately anyway if you’re doing proper build/test integration internally.

[GAIN] Bug triaging: Fedora users report bugs to Red Hat Bugzilla first; the package maintainer decides if it’s a packaging bug or an upstream bug. If it’s an upstream bug the packager will ideally create a minimal test case and send it to the upstream maintainers.

This is a strong *potential* gain, if the package maintainer is a trusted and responsible member of the community. But what if the package maintainer is an employee of the company, as is usually the case? It’s not a gain at all.  And what if the package maintainer also maintains 20 other packages, and isn’t particularly responsive?  Then it’s a net loss.

[LOSS] Binary dependency predictability: dependency updates may mean that the deployed set of components is not the same set of binaries the ISV tested during their release process.

Bingo!  No more calls, please — we have a winner.

Here’s the thing: an ISV does not have the luxury of dealing with variance. We’re dealing with tons of bugs, every day, because we’re young companies, pushing as hard and as fast as we can to make our software experience better. When we’re trying to kill a crazy bug for users/customers, the first order of business is to reduce the uncertainties, and the easiest way to do that is to be *very* specific about configurations. This is especially true as the software increases in complexity.

We can assume high competence and good faith on the part of community maintainers, and still be relatively certain that those good actors will make changes, for good reasons, that will damage the ISV’s application stack in unpredictable and important ways. Software is mean-spirited like that.

This could, in theory, be mitigated by keeping multiple versions of things, and having better mechanisms for tracking those versions. This is something that Red Hat Network customers wanted for years, and finally got — the ability to install a very specific package manifest that is not “all latest packages”, but “these specific package versions”.  But Linux distros don’t work that way, for good or ill.

In theory, everyone should always be running the latest version of things. In practice, that can be very difficult — and it can be *especially* difficult for the ISV when multiple distros have different notions of what the latest version is, and *exceptionally* difficult when those package manifests can change without warning, and outside of your control.

Maintaining a functioning product in multiple cutting-edge distros, with different release cycles and different dependencies, requires a serious, serious commitment to continuous integration and testing. I believe that Eucalyptus has a better process for this than most — and still it will be a tremendous challenge for us to keep up with two different fast-moving distros in Fedora and Ubuntu.

[LOSS] Unity with Windows release process: someone on the ISV’s team will need to be a Fedora contributor or they will need to recruit an external packager.

You can replace “Unity with Windows release process” with “Unity with Ubuntu release process” and the problem is the same. There are huge differences, of course, between a Windows release process and a Linux release process — but even staying in the Linux world, there’s a considerable difference between the Ubuntu release process and the Fedora release process, and expertise in the one in no way guarantees success in the other.

[LOSS] Ability to customize dependencies arbitrarily: there are rare cases where Fedora ships different versions of the same component for compatibility but in general this is strongly discouraged; custom patches should be sent upstream or eliminated by patching the product’s code to not require them.

Absolutely.

[LOSS] Download counting/tracking: if an ISV provides a tar-based distribution from their website, they can track counts and/or emails. This may be important for their marketing department.

Ayup.

* * * * *

It looks pretty grim in the end, doesn’t it? Well, it’s not as dark as all that. There are legitimate ways for the committed ISV to bridge the gaps over time:

1. Commit to building RPMs (and dpkgs), from source, the right way, for the ISV product, and making those source packages available to whomever wants them. There are legitimate reasons for an open source company to do this, and it’s a necessary precondition to being in the distros anyway.

2. Release their Linux versions as add-on yum/dpkg repos.  Of course, this also means being able to supersede/obsolete distro packages with foo packages, but this is easily done by maintaining separate namespaces.

3. Continue to work with other ISV vendors on packaging best practices at every opportunity, even if those packages don’t immediately end up in the distro.

4. Explore development builds that depend on the latest packages, available from wherever. One of the great advantages of Fedora, and other fast-moving distros, is that they do a great job of managing the future. We don’t want to live in the future, but we certainly want to have our eye on it, and that’s a great reason to continue to *try* be in Fedora — but we also need to make it clear to potential users that the future and the present don’t always see eye-to-eye, and that can be difficult messaging to convey.

The truth of the matter is that not every user understands the intricacies of the open source development model, and most ISVs in a competitive market get one shot to connect with their potential customers. One. Which means that the ISVs are going to do everything they possibly can to make sure that they’ve got control over how that experience goes, at the lowest possible development cost.

Fedora can afford to live right on the bleeding edge because they’ve got CentOS/RHEL to fall back on. Not everyone has that luxury.

(p.s. looking forward to talking more about this at FUDCon.  Also: the drinking.)

Coworking in the Bull City

Eucalyptus is now a proud sponsor of Bull City Coworking. Various folks have tried to get a coworking operation up and running in Durham in the last little while; props to Robert Petrusz and the gang for actually getting it done.

Both Andy Grimm and I are Eucalyptus employees who live and work in Durham.  Now we have a space to hang out in, which is handy, because while working from home has its advantages, if you do it *every single day of your life* it can get old in a hurry.  So here were are, in East Downtown Durham, a stone’s throw from the best Cuban food in the Triangle.  The space is spare, but growing.  I CAN HAZ WHITEBOARDS, which is awesome.

So, Durhamites.  If you want to get out of your stuffy home office and come hang out for the day sometime, and especially if you want to talk about open source cloud awesomeness, ping me and I’ll set you up with a day pass.  Y’all come, hear?

Building Community QA Tools for AWS

One of the reasons that Eucalyptus works as comparatively well as it does is because the QA behind the scenes is tremendous. There’s a lot of resources behind the scenes, constantly running tests against various Eucalyptus builds, to make sure that behaviors are stable.

Still, it’s impossible to have too much QA.  You can always use more QA, and you can always use more tools for QA.  Which is why the eutester framework is exciting to me: it’s the first legitimate opportunity to involve Eucalyptus users in a meaningful testing process.

Basically, it’s a testing framework that harnesses euca2ools+boto+paramiko (ssh2 for python) to run automated tests against Eucalyptus instances — or, with minimal modification, AWS instances.  It’s dead simple, and over time I believe will prove to be incredibly extensible.

Here’s a good example of an early bit of code.  You don’t really even need to know Python to follow it.

So if you’re running a Eucalyptus instance, take Eutester for a spin. Swing by IRC and let us know if it works for you, and if you’d like to see some use cases added.  If you can write them yourself (and it’s not rocket science), even better.

More info about the Eutester project here.

Mitch wrote the book on AWS tools.

No, seriously, he did.

Actually, Mitch Garnaat has written a bunch of stuff. He wrote boto, the excellent Python library that talks to AWS-based backends, and on top of that he wrote euca2ools, which is the free software client of choice worldwide for sending management commands to AWS, and to Eucalyptus, and to OpenStack Nova (well, the parts that have been written yet, anyway.)

Now he’s written an actual book about how to manage your AWS instances with Python and Boto.

I would argue that Mitch Garnaat is the authoritative voice on AWS command line management tools, and this book cements the place that he has already claimed for himself.

Well done, Mitch. Well done.

*slow clap*

Recent and upcoming doings in Euca-land

Busy times with Eucalyptus.

A couple of meetings tomorrow in Eucalyptus-land, aka irc.freenode.net #eucalyptus:

Meeting the first. 10am Pacific US time. The topic: EucaTV. The long-term goal: to figure out how to populate the Eucalyptus Education channel with awesome HOWTOs that explain how to run a Eucalyptus cloud. Good times.

Meeting the second. 11am Pacific US time. The topic: CLOUD MACHINE. The long-term goal: to build the awesomest possible version of Eucalyptus that runs right off of USB stick. We’ve already got Fast Start, but we’re going to take it to ANOTHER LEVEL.  (Precise level yet to be determined.)  Click, click, buzz, buzz, completely functional private cloud in minutes. Good times.

Also of note: the new Eucalyptus community mailing list, where everything Eucalyptus will be ultimately be reported upon, debated, endlessly discussed, worked and reworked and reworked some more. All right there in public. Come see some sausage being made!

See you at either one meeting or the other meeting, or maybe even at both meetings.  And also on the mailing list.  Good, good times.

The Wikipedia Code Challenge is live.

And I’m beat.  :)

In addition to working with the nice folks at Eucalyptus, I’ve also been working with the nice folks at the Wikimedia Foundation to launch the October 2011 Coding Challenge.  We’ve been working through the various aspects: logistics, the challenge areas themselves, and of course legal.

We launched about 10 minutes ago, and we have 10 signups for the challenge, and we’ve received 2 emails from people with questions.  That means we can expect a run rate of… yeah, I’m not gonna do that math.

That’s the power of putting a banner ad on a site that gets 10 million hits a day.

So check it out.  And if you want to write some code to help Wikipedia out (and some of the challenges are pretty fun and pretty cool), feel free to sign right up.

The contest is open until November 7th at 23:59 UTC.  Then the judging begins.  I suspect it’s going to be a very long few weeks — but it’s been an awesome ride so far.

The heady scent of Eucalyptus

I’ve long admired the folks at Eucalyptus. I remember in the early days of the Fedora Cloud SIG, the earliest people from the community who showed up to participate were Graziano Obertelli and Garrett Holmstrom. They were among the first non-Redhatters to do heavy lifting as we tried to get that SIG off the ground.

Now here we are, twenty months (-ish) later. The world moves fast, and hot new technologies race forward. Cloud is now at the epicenter of the open source world, and a lot of new entrants are moving quickly. It’s awesome that the open source model continues to crush. The old saw that “open source is for duplication and not innovation” is exposed even more clearly in the cloud world as the lie it’s always been.

One of the things I’ve learned about open source, though, is that sharing the source code isn’t nearly enough. You’ve got to be out there talking about what you do, and inviting people to join you. You’ve got to be working in the open. You’ve got to share: both the good stuff and the not-so-good stuff. You’ve got to be a little bit vulnerable. It’s not easy.

Eucalyptus has been consistently excellent at turning out great open source software for driving private clouds. They’ve been less great at doing the other open source stuff — not because they’ve been unwilling, but because they’ve been sprinting so hard, for so long, serving their customers and growing their business.

It’s a problem with which I am intimately familiar.

There’s great things happening in Euca-land, but most Eucalyptoids are too busy building the future to talk about what they’re doing. So I’m going to help them.

It’s going to be fun to be back in the open source community again. See you on the lazyweb.

The Community of One

I had the good fortune of being present for Jono Bacon’s excellent community session at OSCON a few weeks ago. (The 40 minute version, not the 15 minute keynote version with which he was wholly dissatisfied.) As the open source model matures, community management as a discipline is maturing with it, and Jono did an excellent job of bringing together some of his own best practices in a concise and useful presentation. He promised to have his deck uploaded soon, so as soon as he does, I’ll link to it.  (Jono: hint hint.)

One of the most interesting slides in that deck, and for me one of the most personally resonant, was a page that arrayed all of the potential responsibilities of the community manager — a grid of words that took up the entire page. Words like growth and transparency and process and conduct and governance and buzz, and lots more besides. An intimidating array of words, really.

As we chatted afterwards, the would-be founder of a small niche project talked about the pains of getting people to engage in his project, and how he could possibly keep track of all of those community tasks while he was busy slinging code, and in that huge array of potential responsibilities, where should he start? How should he grow beyond a Community of One?

It was a great question, and it set me to thinking for days afterwards.

At first I thought, “it’s most important to articulate your needs.” And it’s true: articulating your needs is very important. People can’t help you if they don’t know what you need, and mostly, people don’t have time to guess. So it’s very important to have, at the very least, a visible list of “things I’ll be doing next.” A good, updated TODO somewhere prominent in the repo, perhaps.

But what if no one is looking at that TODO?

So then I thought, “it’s most important to be vocal.” And it’s true: being vocal is very important. People won’t follow you if they can’t find you, and the only way people can find you is if you talk about what you’re doing. Which doesn’t come naturally to a lot of people. So it’s very important to blog about what you do — to celebrate your victories and ruminate on your challenges.

But what if no one is interested in your victories and your challenges?

Which brought me, finally, to this:

It’s most important to be solving a problem that really matters to other people.

The oft-quoted chestnut about open source development is that it’s all about developers “scratching their own itch,” and of course this is absolutely true — but community only happens when other people have that same itch.

Maybe that’s not the most encouraging advice — “gee, you should have better ideas!” — but I think it really is at the heart of successful communities. Successful communities are passionately engaged in the pursuit of the same goal. If that goal isn’t an audacious goal, with some power to stir the blood, it’s just harder to engage people in its pursuit.

Which means that you should seek out other people who might share your problem, or even a problem like it, and you should talk to them about your ideas.  Such a simple thing — and so hard to do.

It’s hard, of course, because we cherish our own ideas.  It’s hard to listen to people shoot holes in those ideas. And they will — will they ever. Even with legitimately good ideas, you will always find people who will say “no”, often quite stridently. That’s okay. Remember that your goal is to find the people who say “yes”, or “maybe”, and if you can find even one or two people who passionately agree with your idea, that’s enough to justify continuing towards your grand solution.

And if you don’t find those one or two people? Well, it is possible that you’re way ahead of your time, and that you should continue to track your idea relentlessly — but I myself, in similar circumstances, have often found it useful to ruminate on the words of the great Richard Feynman:

“The first principle is that you must not fool yourself — and you are the easiest person to fool.”