A big advantage to following AWS…

…is that any time a user runs into problems figuring out how to do something with Eucalyptus, it’s quite likely that the corresponding AWS procedure, as documented by the AWS community, will “just work”.

Example: growing an EBS volume.  The commands listed here:

http://blog.edoceo.com/2009/02/amazon-ebs-how-to-grow-storage.htm

…basically work by switching ec2-tools and euca2ools.  It’s nice to have that kind of knowledge base to fall back on, even as a starting point that may need to change subtly for euca-specific cases.

Euca School, first day of class!

I’m excited for our first #eucalyptus-classroom session on freenode on Monday (noon Eastern US time). Expect it to be the first of many.

The topic for this first meeting is Eutester, which is the new Eucalyptus harness for writing test cases.  This is A Big Deal, since Eutester is based on boto and paramiko, which are also compatible with AWS.  For users who are planning to script interactions between Eucalyptus and AWS, Eutester will be absolutely invaluable.  We’re also going to be using it to write the entire nextgen Eucalyptus testing suite (which is already immense btw.)

Be sure to take a look at Vic’s blog post about setting up the Eutester environment.

Hope to see you there.

Coming Soon: Eucalyptus 3.next

The first release of Eucalyptus 3 is out, and it’s time to get ready for Eucalyptus 3.1.

There will be a feature listing for Eucalyptus 3.1 on our website somewhere at some point, but that feature list won’t really tell the story.  Here’s the story: Euca 3.1 is the point at which we return to developing code the open source way.

With Eucalyptus 2, we introduced our Enterprise Edition (EE).  That code was a fork of the open source edition, and included proprietary features that were not available in the open source release.  What was the nature of these proprietary features?  Basically, they were hooks to allow Eucalyptus to talk to enterprise-y things, like Big Proprietary SANs and Big Proprietary Virtual Machine Managers. These are the kinds of hooks that large enterprises, who use a combination of open source and proprietary technologies, badly need — in fact, they are hard requirements for these kinds of environments.

The ideal way to add these hooks would have been to add them as well-isolated modules — but in the fast-moving startup world, the ideal way is not always possible.  The decision was made to manage the two separate codebases side by side, with the good intention that when a feature would go into one version, it would be added into the other at the same time.

We all know the old saying about good intentions, don’t we?  🙂

Looking back, it’s probably not accurate to say that the decision was a “mistake”, per se, but it’s definitely true that the open source version suffered.  In retrospect, it was inevitable; whenever a hard choice needed to be made about the allocation of scarce resources, the choice was always, always, always to solve the customer’s problem.  So the subscription version was patched and tested, while the open source version atrophied, with the lack of commit activity leading many observers to conclude that Eucalyptus Was Dying.  Which was completely wrong, of course; the customer base grew the whole time, and the company grew with it.  The new version of the product marched along, with a shared understanding among Eucalyptoids that when the time came, the codebase would be rebuilt the right way.  The proprietary hooks would be pulled out into modules.  Open source would never again be treated as a second-class citizen.

It’s been a hard and frustrating wait for a company that views itself as open source to the bone.

So, starting with Eucalyptus 3.1, the two trees will be joined into a single tree, with a handful of modules available to subscription customers only.

The road has been long.  As the engineers pushed tirelessly towards the GA of Eucalyptus 3.0, Andy took a machete to mainline and created a parallel branch in which the proprietary bits were unceremoniously hacked out.  It wasn’t necessarily pretty, but it worked.  He also made sure that the bits would sync to this new branch on a nightly basis.  Graziano then set it up so that the new branch would be pulled into Launchpad, where we called it “devel”. Now the time has come to turn our attention to fixing that code in earnest.  We recently renamed the branch 3.1, and set it up as the mainline for the next release.  Go check it out.

What comes next?  Glad you asked!

1. We’re cleaning up the migration from MySQL to Postgres.  We decided to move to Postgres because we believe that this will make it easier to redistribute to the open source community.  The Postgres bits have been there for a while; we just turned them on when we branched, and everything seems to work so far, but there’s always a bit of housekeeping to do with such changes.

2. The entire codebase will be reexamined for proprietary bits that may have been left in the 3.1 tree.  This will require some refactoring to ensure a separation that is truly clean.  We will then make those modules available to our subscription customers only.

3. QA, QA, QA.  We will put the 3.1 release through the same rigorous QA cycle that we’ve previously reserved for the subscription version.  We’ll also be working on opening up more and more of this testing code, so that people can build their own QA tools for testing their Eucalyptus cloud (and EC2 as well).  See the eutester project for more info.

4. We will release the documentation under a Creative Commons license.  The documentation for Eucalyptus 3 is available right now, but we’ve still got some work to do on trademarks and licensing questions.  Once we do, we’ll make the DocBook source files available, and we may even kick off our first translation projects.  See the documentation project for more info.

5. We will make a lot of engineering decisions about how to open up the development process.  Issue tracking, revision control, patch management, feature process, bug triaging, release management, distro packaging — we’ll make key decisions about all of these issues in the coming weeks, and we’ll be sharing them with the Eucalyptus community every step of the way.

So when will the 3.1 release happen?  Tough to say.  Once we start putting the code through QA, we will have a good idea of how functional the code is; from what we know now, it’s certainly good enough to try out in non-production environments, with a reasonable expectation that things will mostly Just Work.  But the difference between that state, and the desired state of Open Source Goodness, will take a little while to bridge.

It’s an exciting time.  I’m proud of what we’re accomplishing in a short timeframe, and the future looks very bright.

Building Community QA Tools for AWS

One of the reasons that Eucalyptus works as comparatively well as it does is because the QA behind the scenes is tremendous. There’s a lot of resources behind the scenes, constantly running tests against various Eucalyptus builds, to make sure that behaviors are stable.

Still, it’s impossible to have too much QA.  You can always use more QA, and you can always use more tools for QA.  Which is why the eutester framework is exciting to me: it’s the first legitimate opportunity to involve Eucalyptus users in a meaningful testing process.

Basically, it’s a testing framework that harnesses euca2ools+boto+paramiko (ssh2 for python) to run automated tests against Eucalyptus instances — or, with minimal modification, AWS instances.  It’s dead simple, and over time I believe will prove to be incredibly extensible.

Here’s a good example of an early bit of code.  You don’t really even need to know Python to follow it.

So if you’re running a Eucalyptus instance, take Eutester for a spin. Swing by IRC and let us know if it works for you, and if you’d like to see some use cases added.  If you can write them yourself (and it’s not rocket science), even better.

More info about the Eutester project here.

Mitch wrote the book on AWS tools.

No, seriously, he did.

Actually, Mitch Garnaat has written a bunch of stuff. He wrote boto, the excellent Python library that talks to AWS-based backends, and on top of that he wrote euca2ools, which is the free software client of choice worldwide for sending management commands to AWS, and to Eucalyptus, and to OpenStack Nova (well, the parts that have been written yet, anyway.)

Now he’s written an actual book about how to manage your AWS instances with Python and Boto.

I would argue that Mitch Garnaat is the authoritative voice on AWS command line management tools, and this book cements the place that he has already claimed for himself.

Well done, Mitch. Well done.

*slow clap*