Cloud Horror Stories at OSCON

Running a private cloud can be like being the victim in a horror movie. You stand up the cloud and everything’s just fine. You get some workloads running. You’re scaling up. You’re scaling down. All is well. And then the spooky music starts, and little things start going wrong — and then suddenly the chairs are spinning in the air and you’re running for your life.

If you’re in Portland for OSCON on Monday night, come on by the Cloud Horror Stories birds-of-a-feather session. We’ll have people there with scary stories to tell, so if you’re interested in learning how things can go terrible wrong in cloud-land, come on by. Even better — if you’ve got a story of your own to share, we’d love to hear you tell it, preferably in your spookiest voice.

See you around the campfire. BOOOOOO!

(And no, we’re not actually going to start a campfire in the convention center. Definitely, almost certainly, maybe not. We can do that flashlight-under-the-chin thing, anyway.)

Cloud Horror Stories at OSCON

AMI-to-EMI: ACHIEVEMENT UNLOCKED!

The ami2emi project has been moving along. As in, it actually works for a number of cases now. Configure cloud parameters for AWS and Euca, run a script, and boom: your chosen AMIs are brought to life on your Euca cloud. When it works, it Just Works. It’s cool. 🙂

Note that these cases do *not* yet cover configuration of the applications themselves — just the images.  The bits might still need to be twiddled to get the apps working properly, but all of the actual bits are successfully transferred into the new image, and the new image successfully spins up instances, and you can ssh to them and everything.

The AMIs that can currently be auto-slurped into Eucalyptus successfully share certain characteristics:

1. They carry their own kernels inside the image. On the AWS side, that means they’re linked against the stock pv-grub kernels, and we link them similarly to the kexec-loader kernels on the Euca side.

2. They’re instance-store images, rather than EBS-based images. At least so far.

You can find the list of currently tested images here. That list will expand rapidly as we have time to run more test cases.

Note the large number of Bitnami instances in this list. That must be because they’re Crazy Awesome.

So, give it a whirl. Patches exceptionally welcome, since it’s heinous bash scripting and I can use all the help I can get. At least I document my code, sorta. 🙂

AMI-to-EMI: ACHIEVEMENT UNLOCKED!

Converting AMIs to EMIs

One of the most common questions I’ve heard asked from Eucalyptus users is this one: “how easy is it to convert an AMI to an EMI?” And the answer is: not as easy as it should be.

We’ve got some process guidelines on our wiki, thanks to Tim Gerla — but we should have the ability to do much of this automagically.  So that’s the tool I’ve started work on.  In the spirit of “release early release often”, I’ve uploaded a very early iteration of this tool.  Find the brand new Github repo here.

It’s quite a ways yet from being prime-time, but it’s allowed me to do quite a bit of testing.  Some assumptions I’ve started with:

* First, I pulled a list of all public AMIs on us-east-1.  There were about 20,000 public images available there as of mid-November.

* Then I selected the subset of AMIs that were built with a PV-GRUB kernel, and I’m importing them to an instance of Eucalyptus running the kexec-loader kernel.  In both cases, the AKI/EKI is just a bootstrapping mechanism that then hands control over to the image’s own kernel, so we shouldn’t get caught up by kernel incompatibilities.  Using only the subset of kernels with PV-GRUB AKIs leaves us with about 7000, and picking one particular AKI (aki-825ea7eb) gives us about 1700.

* From there, we’re working with individual distros, and there will be idiosyncracies between the various distros out there, so it makes sense to pick one and go with it.  There’s a lot of Ubuntu out there on AWS, so I just grepped on AMIs with “ubuntu” in the name.  That’s 1067 images in my dataset.

* Now that we’ve got a reasonable set of images to examine, here’s the process that the scripts walk through.  For each AMI, we start an AWS instance, ssh to that instance, install euca2ools if they aren’t already installed, scp Euca credentials to the instance, and bundle the instance to the specified Euca cloud.  Then we fire up the resultant EMI on the Eucalyptus side and see if we can ssh in.  And we bail with appropriate error messages at various places along the line.

I’ve run through a few hundred images at this point.  Not one of them has been completely successful from start to finish.  About 10% so far have yielded a bundled EMI that boots and yields a Eucalyptus instance in a running state.  Can’t ssh into any of them yet, though.

The good news is that the failures are all quite specific and predictable, and the next steps are clear.  Do a better job of guessing login IDs.  Figure out why fstabs fail.  Look for rogue kernel modules.  Make sure we’re doing key injection properly on the Euca end.  Comb through the results of euca-get-console-output and look for patterns.  The big win is having tools that allow us to do that work in minutes, rather than in hours or days. Every step gets us closer to the goal of fully automated conversions on the fly.

Oh, and an apology: a lot of this should probably have been written using Eutester, instead of as a bunch of shell scripting.  The fact is, I’m a terrible hack.  But I’m just leading the charge temporarily; when I’ve figured out the basics, the real coders can swoop in and do things the right way.  In the meantime… patches welcome.

Converting AMIs to EMIs

Shoes for the Cobbler’s kids

We’re big fans of the Cobbler project here at Eucalyptus. We think it’s the best tool in the open source world for bare metal provisioning.  We’ve invested in a gigantic QA environment for continual integration testing, and Cobbler is one of the linchpins of that environment. It’s the kind of tool that’s best appreciated by sysadmins who deal with *a lot* of systems.

I’m sort of attached to Cobbler personally, since I watched it grow out of Red Hat’s Emerging Technologies team several years ago. Now it’s grown past its Red Hat roots to become a truly independent project — and independent projects need support from time to time.

The Cobbler folks have set up an Indiegogo campaign to raise some funds for some much-needed infrastructure, and as proud Cobbler users, we are proud to help them out. Their goal is to raise $4000, and Eucalyptus will match every donation, dollar for dollar, until they reach their goal.

If you’ve used Cobbler and it’s helped you do your job, pitch in. The campaign is running for two more weeks; let’s help put them over the top.

Shoes for the Cobbler’s kids

Step two: put your cloud in that box!

(I’m sure you all know that step one is “cut a hole in the box”.)

We’ve been continually working to improve the install process of Eucalyptus over the past few months.   In particular, we’ve been working on a project that we call Silvereye.  Our most recent goal: make it trivial to install a fully-running Eucalyptus cloud on a single machine.

A cloud on one machine?  Why bother?  Well, lots of reasons, actually.  The biggest: the developer workstation.  If you’re hacking on Eucalyptus, it’s pretty awesome to have Eucalyptus on a single system that you tear down and rebuild in 15 minutes.

Anyway: mission accomplished.  Go to our Silvereye downloads directory and get the latest build (right now it’s silvereye_centos6_20121004.iso).  Burn it to DVD, boot your target system, and choose the “Cloud-in-a-box” option from the Centos-based installer.  Answer some simple questions.  Boom, in 15 minutes you’ve got a cloud-in-a-box!

(Note #1: a helpful README can be found in the Github repo for Silvereye: github.com/eucalyptus/silvereye.)

(Note #2: in the cloud-in-a-box config, when you log in as root for the post-install config, it’ll say “hey, do you want to install the frontend now?”  Answer yes.  It automatically installs the node controller for you.)

(Note #3: Silvereye is not supported. At all. If you use it, there are ABSOLUTELY NO GUARANTEES that it won’t burn down your house, steal your pickup truck, or throw your mother into a wood-chipper.)

Silvereye is mostly the work of sysadmin-par-excellence Bill Teachenor, based on the original Faststart installer written by David Kavanagh — but various folks are now working on it; Andy Grimm, Graziano Obertelli, and Andrew Hamilton have all been pushing the cloud-in-a-box on various distros, and Scott Moser of Canonical did some great proof-of-concept work on the UEC code. So thanks to all of them, and everyone else who’s played with it.

Give it a spin; it really is dead-easy.  We still need to round off a few corners before we can call it the official installer of record, but we’re quite close now.

Want that AWS-compatible cloud on your laptop?  Of course you do.  Now go get it.

Step two: put your cloud in that box!

A big advantage to following AWS…

…is that any time a user runs into problems figuring out how to do something with Eucalyptus, it’s quite likely that the corresponding AWS procedure, as documented by the AWS community, will “just work”.

Example: growing an EBS volume.  The commands listed here:

http://blog.edoceo.com/2009/02/amazon-ebs-how-to-grow-storage.htm

…basically work by switching ec2-tools and euca2ools.  It’s nice to have that kind of knowledge base to fall back on, even as a starting point that may need to change subtly for euca-specific cases.

A big advantage to following AWS…

Nurse Euca is here to help you.

“I never see what has been done; I only see what remains to be done.  –Marie Curie”

Automated installers are great. When they work, they work really well — but when they don’t, not only do they not work, but they bring great sadness to the hopeful user who trusted your automated installer.  Tragic!  Heartbreaking.

So why don’t automated installers work, when they don’t work?  In almost every case, it’s because there’s a condition your installer assumes that isn’t met.  And in this day and age, you don’t have just one installer for your software, you’ve got multiple potential install+config tools: multiple package managers, multiple configuration tools, multiple permutations of hardware, multiple permutations of hypervisor, multiple permutations of network topology.  Which means that you’d better do a *great* job of figuring out your environment before you try to lay down and configure your bits.

Enter Nurse Euca.  Nurse Euca will run before any install and take everyone’s temperature, offer an aspirin or a splint where needed — or will let you know if one of your requirements is Dead On Arrival.  (“I’m sorry, Doctor, but em1 appears to be in septic shock. I recommend against resuscitation.”)  (That’s totally gonna be an error message, btw.)

Awesome, right?  Well, it will be when we write it.  We’ve got bits and pieces of these kinds of checks in various places.  On Friday we will be having a hackfest to pull these threads together and get Nurse Euca jumpstarted.

Hey, here’s a mostly-empty Github repo!  By the end of Friday, it won’t be.

We’ll be on freenode, #eucalyptus-devel, at 7am Eastern US time.  Yes, it’s early; we’ve got some friends in the Old World who will be hugely helpful, so we’ve chosen the time to accommodate them.  See you then.

Nurse Euca is here to help you.

Next Euca hackfest: AppScale

Our next hackfest is this Friday, August 10th, from 11am to 2pm Pacific Time, in #eucalyptus-devel on freenode.  (If you show up at #eucalyptus or #eucalyptus-meeting, we will kindly direct you the right way.)  We will be working on integrating with AppScale.

You don’t know about AppScale, you say?  Well, you should.  AppScale is an open source platform for  Google App Engine apps.  The idea is that many applications designed to run on Google App Engine should “just work” with AppScale and your own cloud infrastructure.

There’s an Ubuntu-based AppScale image already built and ready to go for Eucalyptus; in the next couple of days, we’re going to get that running on our Eucalyptus Partner Cloud, and then we will see if we can get some of this App Engine code running on top of it.  By the end of the hackfest, we hope to have a few of these sample apps up and running, and filed away as Eucalyptus recipes.

If you want to join in the fun, just drop a line to the Eucalyptus community list and we’ll be happy to set you up with all the tools you’ll need.  See you on IRC.

Next Euca hackfest: AppScale

Today is the day.

Eucalyptus 3.1 is open for business.

No more artificial separation between Enterprise and Community.  No more frenzied checkins to the “enterprise edition” while the separate-but-equal “community version” atrophies.  No more working on new features behind closed doors for months on end.  No more wondering about what’s on the roadmap.  No more going weeks without any publicly visible check-ins.  No more.

Today is the day that we release Eucalyptus 3.1, and reassert our position as the world’s leading open source cloud software company.  With the emphasis on open source.  We’ve been working to get to this day for months, and now, the day has come.

For those who want to get started with the new bits immediately, the Faststart installer can be found here.  With two virt-capable laptops installed with Centos 6.2 minimal, you can have a private cloud running in 15 minutes if you follow the directions — and a few hours if you don’t.  🙂

Package repositories for the various distributions can be found here.

Source code is on Github. Here’s a look at all the recent commit activity.

Anyone who has questions can ask them here or here.

A list of all currently known bugs in 3.1 can be found here.

The list of features we’re currently scoping for 3.2 can be found here.

We have lots of other projects moving forward on Github as well.  Projects like Eutester for automated testing of Eucalyptus (and Amazon) instances, Recipes for automated deployments of Eucalytpus (and Amazon) instances, our nextgen installer Silvereye, and many others.

All of these projects are open to community participation and transparently managed.  We hold weekly meetings on IRC.  You can find the weekly meeting schedule here.  Minutes for all meetings for the past six months can be found here.

We’re also hiring.

“Build together. Run together. Manage together.”  That’s been the mantra for this release, and it speaks directly to the culture of our company.   If I learned anything at Red Hat, it’s that company culture matters.  It literally makes or breaks the company.  Especially in open source: either you’re an open source company, or you’re not.  We are deeply committed to the open source model, because we believe that it creates the best software, and we’re going to prove it.

The most exciting thing about today’s release, to me, is that we’re only getting started.  It’s been a long climb to get to this plateau. We’ve still got a lot of mountain yet to climb, though, and we’re looking forward to the challenge — but that can wait for another day. Maybe two.  Today is about appreciating where we’ve been, and enjoying the view.

Well done, Eucalyptians.  Well done.

Today is the day.

Eucalyptus 3.1 Beta Packages now available

We now have beta packages, along with installation instructions, available for CentOS/RHEL and Ubuntu.

Note: beta still means beta. We’re aiming for release candidates for Eucalyptus 3.1 within the next month or so.  Still, these packages are pretty stable for us so far, pass the majority of our ridiculous battery of QA tests, and are altogether suitable for a quick install to see what the fuss is all about.  And it’s a whole lot simpler than building from source.

As always, questions can be directed to mailing list or forums or IRC.

Eucalyptus 3.1 Beta Packages now available