Cloud Horror Stories at OSCON

Running a private cloud can be like being the victim in a horror movie. You stand up the cloud and everything’s just fine. You get some workloads running. You’re scaling up. You’re scaling down. All is well. And then the spooky music starts, and little things start going wrong — and then suddenly the chairs are spinning in the air and you’re running for your life.

If you’re in Portland for OSCON on Monday night, come on by the Cloud Horror Stories birds-of-a-feather session. We’ll have people there with scary stories to tell, so if you’re interested in learning how things can go terrible wrong in cloud-land, come on by. Even better — if you’ve got a story of your own to share, we’d love to hear you tell it, preferably in your spookiest voice.

See you around the campfire. BOOOOOO!

(And no, we’re not actually going to start a campfire in the convention center. Definitely, almost certainly, maybe not. We can do that flashlight-under-the-chin thing, anyway.)

Cloud Horror Stories at OSCON


The ami2emi project has been moving along. As in, it actually works for a number of cases now. Configure cloud parameters for AWS and Euca, run a script, and boom: your chosen AMIs are brought to life on your Euca cloud. When it works, it Just Works. It’s cool. 🙂

Note that these cases do *not* yet cover configuration of the applications themselves — just the images.  The bits might still need to be twiddled to get the apps working properly, but all of the actual bits are successfully transferred into the new image, and the new image successfully spins up instances, and you can ssh to them and everything.

The AMIs that can currently be auto-slurped into Eucalyptus successfully share certain characteristics:

1. They carry their own kernels inside the image. On the AWS side, that means they’re linked against the stock pv-grub kernels, and we link them similarly to the kexec-loader kernels on the Euca side.

2. They’re instance-store images, rather than EBS-based images. At least so far.

You can find the list of currently tested images here. That list will expand rapidly as we have time to run more test cases.

Note the large number of Bitnami instances in this list. That must be because they’re Crazy Awesome.

So, give it a whirl. Patches exceptionally welcome, since it’s heinous bash scripting and I can use all the help I can get. At least I document my code, sorta. 🙂


Converting AMIs to EMIs

One of the most common questions I’ve heard asked from Eucalyptus users is this one: “how easy is it to convert an AMI to an EMI?” And the answer is: not as easy as it should be.

We’ve got some process guidelines on our wiki, thanks to Tim Gerla — but we should have the ability to do much of this automagically.  So that’s the tool I’ve started work on.  In the spirit of “release early release often”, I’ve uploaded a very early iteration of this tool.  Find the brand new Github repo here.

It’s quite a ways yet from being prime-time, but it’s allowed me to do quite a bit of testing.  Some assumptions I’ve started with:

* First, I pulled a list of all public AMIs on us-east-1.  There were about 20,000 public images available there as of mid-November.

* Then I selected the subset of AMIs that were built with a PV-GRUB kernel, and I’m importing them to an instance of Eucalyptus running the kexec-loader kernel.  In both cases, the AKI/EKI is just a bootstrapping mechanism that then hands control over to the image’s own kernel, so we shouldn’t get caught up by kernel incompatibilities.  Using only the subset of kernels with PV-GRUB AKIs leaves us with about 7000, and picking one particular AKI (aki-825ea7eb) gives us about 1700.

* From there, we’re working with individual distros, and there will be idiosyncracies between the various distros out there, so it makes sense to pick one and go with it.  There’s a lot of Ubuntu out there on AWS, so I just grepped on AMIs with “ubuntu” in the name.  That’s 1067 images in my dataset.

* Now that we’ve got a reasonable set of images to examine, here’s the process that the scripts walk through.  For each AMI, we start an AWS instance, ssh to that instance, install euca2ools if they aren’t already installed, scp Euca credentials to the instance, and bundle the instance to the specified Euca cloud.  Then we fire up the resultant EMI on the Eucalyptus side and see if we can ssh in.  And we bail with appropriate error messages at various places along the line.

I’ve run through a few hundred images at this point.  Not one of them has been completely successful from start to finish.  About 10% so far have yielded a bundled EMI that boots and yields a Eucalyptus instance in a running state.  Can’t ssh into any of them yet, though.

The good news is that the failures are all quite specific and predictable, and the next steps are clear.  Do a better job of guessing login IDs.  Figure out why fstabs fail.  Look for rogue kernel modules.  Make sure we’re doing key injection properly on the Euca end.  Comb through the results of euca-get-console-output and look for patterns.  The big win is having tools that allow us to do that work in minutes, rather than in hours or days. Every step gets us closer to the goal of fully automated conversions on the fly.

Oh, and an apology: a lot of this should probably have been written using Eutester, instead of as a bunch of shell scripting.  The fact is, I’m a terrible hack.  But I’m just leading the charge temporarily; when I’ve figured out the basics, the real coders can swoop in and do things the right way.  In the meantime… patches welcome.

Converting AMIs to EMIs

Shoes for the Cobbler’s kids

We’re big fans of the Cobbler project here at Eucalyptus. We think it’s the best tool in the open source world for bare metal provisioning.  We’ve invested in a gigantic QA environment for continual integration testing, and Cobbler is one of the linchpins of that environment. It’s the kind of tool that’s best appreciated by sysadmins who deal with *a lot* of systems.

I’m sort of attached to Cobbler personally, since I watched it grow out of Red Hat’s Emerging Technologies team several years ago. Now it’s grown past its Red Hat roots to become a truly independent project — and independent projects need support from time to time.

The Cobbler folks have set up an Indiegogo campaign to raise some funds for some much-needed infrastructure, and as proud Cobbler users, we are proud to help them out. Their goal is to raise $4000, and Eucalyptus will match every donation, dollar for dollar, until they reach their goal.

If you’ve used Cobbler and it’s helped you do your job, pitch in. The campaign is running for two more weeks; let’s help put them over the top.

Shoes for the Cobbler’s kids

Step two: put your cloud in that box!

(I’m sure you all know that step one is “cut a hole in the box”.)

We’ve been continually working to improve the install process of Eucalyptus over the past few months.   In particular, we’ve been working on a project that we call Silvereye.  Our most recent goal: make it trivial to install a fully-running Eucalyptus cloud on a single machine.

A cloud on one machine?  Why bother?  Well, lots of reasons, actually.  The biggest: the developer workstation.  If you’re hacking on Eucalyptus, it’s pretty awesome to have Eucalyptus on a single system that you tear down and rebuild in 15 minutes.

Anyway: mission accomplished.  Go to our Silvereye downloads directory and get the latest build (right now it’s silvereye_centos6_20121004.iso).  Burn it to DVD, boot your target system, and choose the “Cloud-in-a-box” option from the Centos-based installer.  Answer some simple questions.  Boom, in 15 minutes you’ve got a cloud-in-a-box!

(Note #1: a helpful README can be found in the Github repo for Silvereye:

(Note #2: in the cloud-in-a-box config, when you log in as root for the post-install config, it’ll say “hey, do you want to install the frontend now?”  Answer yes.  It automatically installs the node controller for you.)

(Note #3: Silvereye is not supported. At all. If you use it, there are ABSOLUTELY NO GUARANTEES that it won’t burn down your house, steal your pickup truck, or throw your mother into a wood-chipper.)

Silvereye is mostly the work of sysadmin-par-excellence Bill Teachenor, based on the original Faststart installer written by David Kavanagh — but various folks are now working on it; Andy Grimm, Graziano Obertelli, and Andrew Hamilton have all been pushing the cloud-in-a-box on various distros, and Scott Moser of Canonical did some great proof-of-concept work on the UEC code. So thanks to all of them, and everyone else who’s played with it.

Give it a spin; it really is dead-easy.  We still need to round off a few corners before we can call it the official installer of record, but we’re quite close now.

Want that AWS-compatible cloud on your laptop?  Of course you do.  Now go get it.

Step two: put your cloud in that box!

A big advantage to following AWS…

…is that any time a user runs into problems figuring out how to do something with Eucalyptus, it’s quite likely that the corresponding AWS procedure, as documented by the AWS community, will “just work”.

Example: growing an EBS volume.  The commands listed here:

…basically work by switching ec2-tools and euca2ools.  It’s nice to have that kind of knowledge base to fall back on, even as a starting point that may need to change subtly for euca-specific cases.

A big advantage to following AWS…

Nurse Euca is here to help you.

“I never see what has been done; I only see what remains to be done.  –Marie Curie”

Automated installers are great. When they work, they work really well — but when they don’t, not only do they not work, but they bring great sadness to the hopeful user who trusted your automated installer.  Tragic!  Heartbreaking.

So why don’t automated installers work, when they don’t work?  In almost every case, it’s because there’s a condition your installer assumes that isn’t met.  And in this day and age, you don’t have just one installer for your software, you’ve got multiple potential install+config tools: multiple package managers, multiple configuration tools, multiple permutations of hardware, multiple permutations of hypervisor, multiple permutations of network topology.  Which means that you’d better do a *great* job of figuring out your environment before you try to lay down and configure your bits.

Enter Nurse Euca.  Nurse Euca will run before any install and take everyone’s temperature, offer an aspirin or a splint where needed — or will let you know if one of your requirements is Dead On Arrival.  (“I’m sorry, Doctor, but em1 appears to be in septic shock. I recommend against resuscitation.”)  (That’s totally gonna be an error message, btw.)

Awesome, right?  Well, it will be when we write it.  We’ve got bits and pieces of these kinds of checks in various places.  On Friday we will be having a hackfest to pull these threads together and get Nurse Euca jumpstarted.

Hey, here’s a mostly-empty Github repo!  By the end of Friday, it won’t be.

We’ll be on freenode, #eucalyptus-devel, at 7am Eastern US time.  Yes, it’s early; we’ve got some friends in the Old World who will be hugely helpful, so we’ve chosen the time to accommodate them.  See you then.

Nurse Euca is here to help you.