I don’t know how many people have actually met Vic Iglesias, our Quality Hobo. Here’s what he looks like in his natural habitat:
It’s really super important not to make Quality Hobo angry. Let me assure you from personal experience: no one wants that.
Here are some things that make Quality Hobo angry (and that you should, therefore, avoid):
- Stealing Quality Hobo’s cigarettes, electronic or otherwise.
- Remarking upon Quality Hobo’s resemblance to a certain celebrity.
- Wasting Quality Hobo’s time with questions about which test cases are most recent.
- Wasting Quality Hobo’s time by writing tests that don’t integrate into Eutester.
- Wasting Quality Hobo’s time with questions on how to build your QA environment.
- Wasting Quality Hobo’s time in any way at all.
Here’s what I’m saying: it was only a matter of time before our Hobo got mixed up with a Vagrant.
Vagrant is incredible, and Mitchell Hashimoto is incredible for creating it. It’s the sort of tool you try out for a particular reason, and then once you’ve used it, you find a million other reasons to use it. Last night, as I chased down a bug in Faststart, I got to try out Micro-QA, which is a mashup of Vagrant and Eutester and Jenkins and Ansible and all kinds of stuff, built on top of one of the standard Vagrant boxes for Centos 6.4.
Here’s the full instructions for getting a fully updated, complete testing environment for Eucalyptus installed on your laptop:
- Install Vagrant+Virtualbox on your laptop.
- Run “git clone” on the Micro-QA repo.
- Run “vagrant up”.
- Point your browser to http://localhost:8080.
Aaaaand you’re done.
Ease of automated testing is one of those force multipliers that doesn’t seem super-exciting, but really is amazingly super-exciting. Because here’s the thing: When the cost of testing is higher than basically zero, people don’t bother with it – or, they do a really half-assed job of it and then say “oh yeah, I totally tested that.” Which is waaaaaay worse.
In Micro-QA, we have a tool that brings the cost of automated QA to near-zero. And not just for QA folks: it’s a tool that can be used by our QA team, our engineering team, our support team, our customers, and our community, all with comparatively little knowledge required. It’s a huge win for us.
And here’s the kicker: it’s not only a QA tool; it’s also a great tool for hybrid cloud diagnostics. Set up your Euca environment; set up your AWS environment; bake in some tests that run against each; run them on a regular basis; scream when something breaks. It’s kinda sorta magic.
Anyway. If you’ve got a Euca install, go get Micro-QA running on your laptop, bring it up in your browser, pick some test cases to run, drop the contents of your eucarc file into your test case, and run it. If it breaks, ping us on Freenode (#eucalyptus-qa) and let us know what broke.
Vic announced Micro-QA less than six months ago. It was cool at the time, but now it’s way past cool. I’m really impressed by how far it’s come in such a short time. It’s like I’m living in the future.
(For the record: Vic is really a super sweetheart of a guy, and I don’t think he actually lives under a bridge at present. I think he may be living Between Two Ferns, though.)
A lot of people have been visiting our table in the OSCON Hack Zone — mostly because of the presence of our Little Black Boxes.
The common question we’ve heard: “where did you guys *get* those things?”
INORITE? They are *totally* cute. We bought the parts and assembled them ourselves. They are now Standard Issue to all new Eucalyptus engineers; a short stack of three gives any engineer enough firepower to do serious development and testing on the whole Eucalyptus stack.
Here’s the parts list from Amazon.com, courtesy of the talented and ruggedly handsome @zacharyjhill:
The main housing unit is an Intel NUC, about 4″ by 4″ by 2″. The SSD is available in different sizes; ours is 128GB. With some of the boxes, we only use 8GB of RAM and with others we use 16GB. We also like to have wireless, though it’s not required – and don’t forget the cheapo power cable.
Building a fully functional 3-node Eucalyptus developer cloud: $1500.
Having an entire AWS-compatible cloud the approximate size of a coffee cup: priceless.
Awesome USB pens with complete virtual Eucalyptus cloud not included — those you’ll have to get from us at OSCON. Come see us in the Hack Zone. ;)
Running a private cloud can be like being the victim in a horror movie. You stand up the cloud and everything’s just fine. You get some workloads running. You’re scaling up. You’re scaling down. All is well. And then the spooky music starts, and little things start going wrong — and then suddenly the chairs are spinning in the air and you’re running for your life.
If you’re in Portland for OSCON on Monday night, come on by the Cloud Horror Stories birds-of-a-feather session. We’ll have people there with scary stories to tell, so if you’re interested in learning how things can go terrible wrong in cloud-land, come on by. Even better — if you’ve got a story of your own to share, we’d love to hear you tell it, preferably in your spookiest voice.
See you around the campfire. BOOOOOO!
(And no, we’re not actually going to start a campfire in the convention center. Definitely, almost certainly, maybe not. We can do that flashlight-under-the-chin thing, anyway.)
I first became aware of Seth Vidal years ago. I didn’t know him at all; I knew him only from his work, and from that work I surmised that he was Not My Friend.
I was working for Red Hat, you see — and Seth, at the time, was not.
People think of Red Hat now as this hugely successful software behemoth, but I can assure you that it wasn’t always that way. There was a time, not so long ago, when Red Hat could only dream of making the kind of money they’re making now. Not that there weren’t huge expectations: after the initial IPO in August 1999, the stock price went up to a ridiculous 132 dollars per share. When I joined in February of 2001, the share price was at a more realistic 6.44. Not long after that, it dropped to 3.12. I couldn’t help but feel a little bit responsible.
I was part of the Red Hat Network team. The idea, in a nutshell, was that we would encourage users to download Red Hat Linux for free, all they liked — but if they wanted timely software updates, we would force them to pay. Muwahahaha! RHN was the mechanism to deliver those updates, and the RHN server would be proprietary software. For me, this was an uncomfortable compromise; I had hoped to come to Red Hat to work on open source software, so it was painful to be a part of the only team in the company that was writing proprietary software. Matthew Szulik, the CEO, would meet with us every quarter to remind us how critical our work was. We were the business engine that would fuel the company, he told us. For a time, it was almost true.
Meanwhile, Seth was busily working on a rewrite of a little tool called Yup. Yup was the update tool for Yellow Dog Linux, and Seth decided to rewrite it to work with Red Hat Linux. He called this new tool Yum (Yellow Dog Updater, Modified). He made it primarily because he himself needed it. And because it was such a useful little tool, other people started using it. A lot of other people. Wow, just a whole lot of people started using Yum. It was far simpler than RHN, and for most users, it was better — or at least good enough. And it was, of course, Free Software.
Today, bits of Yum-related source code can be found in nearly all of the software packaging that Red Hat does — and that includes Spacewalk, the open source descendent of RHN. Open source is especially powerful when it’s commoditizing away the value proposition of proprietary software, and boy, did Yum ever do that. Yum is great software, sure — but to me, Seth’s truly lasting professional legacy is that he taught mighty Red Hat a humbling lesson about open source. And having well learned that lesson, Red Hat proceeds now to teach it to everyone else.
I myself was on the pointy end of that lesson, and it was one of the reasons that I left the RHN team to focus on the nascent Fedora community in 2004. One of my new duties was to write for the late, great Red Hat Magazine; one of my first assignments was to write an article about the resounding successes of the Fedora Project. Resounding Successes, don’t you know! Exciting! So I sent some emails to some of my new friends in Fedora-land, asking for their ideas for this article.
I often wish I still had my old Red Hat emails lying around, because I’m sure Seth’s response to this particular email would have made for great reading. I do remember the gist, which was this: “dude, you do *not* want me talking about my opinions of the Fedora Project in public.” (He did reference Icon Ryabitsev’s excellent mock IRC chat, which is required historical reading for anyone curious about the early days of the Fedora project.)
So anyway, I asked Seth if he’d be willing to talk to me, in private, off the record, about how we might be able to improve Fedora to a point at which he *would* be happy to talk about it in public. He agreed.
That was when I really started getting to know Seth Vidal. Some of my fondest personal and professional memories come from the time that followed. Now that all seems like — sadly, is — a lifetime ago.
* * * * *
I saw Seth two nights before he died.
The American Dance Festival is a centerpiece of summer life in Durham. Performances happen just about every night at various venues across Durham, for a couple of months. ADF is a Durham institution, and we’re very proud of it here. My wife bought us front-row tickets to see an Argentinian aerial dance troupe at the Durham Performing Arts Center last Saturday night. When we arrived, four seats down from us sat Seth and his companion Eunice.
It was how I usually saw Seth, after I left Red Hat: just hanging around Durham, while we were both doing Durham Things. Seth was not only a linchpin of the global open source community; he was a linchpin of of the local, real world community that he and I shared. Seeing him at an ADF performance, or the Durham farmer’s market, or at a local restaurant like Toast, or Parker and Otis, was always a treat, but never a surprise: “oh, yeah,” I would think, “sure Seth is here.” Or if I didn’t see him, I would hear about him from mutual friends: “hey, Seth was here the other day.” He was a presence in Durham. It seemed like everyone knew him, or at least knew of him — people who knew little or nothing about his open source life.
So we chatted that night, crowded against the stage. He asked me how I was doing. For those of us who knew Seth well, he had a very particular way of asking that question, that you can hear in your head even now. He asked the question with concern and purpose. For him, that question was never a throwaway.
I told him I was doing well. We talked about the great seats. He told me that he and Eunice had bought ADF season tickets, because of course they had; there’s not a more Durham thing you can do. I wanted to chat more, because I hadn’t seen him in a couple of months — but I was in people’s way. We agreed that we’d definitely catch up at the upcoming Flock conference, if not sooner. I hurried to take my seat.
The performance was great, but the house was crowded and hot, so when the show ended, my wife and I hurried out without saying our goodbyes. Not even a look or a wave back — because hey, I knew that I’d be seeing him soon enough anyway, right? Right?
* * * * *
My personality flaws have always been magnified in the presence of cyclists.
At my worst, I am impatient, easily annoyed, and impulsive. When I’m in a bad mood, I’m exactly the kind of guy who will sulk behind a pack of cyclists and then speed around them at the first available opportunity. I know a lot of cyclists, and have nothing but respect for them in the abstract — but in the real world, whenever cyclists take up space in my precious roadway, especially when I’m in a hurry, I often find myself fighting the urge to act like an asshole.
I wish I could say that I have nothing at all in common with the guy who ran Seth into the ditch on Monday night. I wish I could say, honestly, that I couldn’t possibly imagine myself in his position. How I would love to be able to say that.
I drove around Durham on Tuesday for much of the day, just thinking. Mostly I drove around Watts-Hillandale, the neighborhood where Seth lived. In particular, I must have driven up and down the length of Hillandale Road a dozen times. It’s a neighborhood street, but it’s also a thoroughfare, which means that people speed along it all the time. People who are in a hurry to get to someplace else. People like me.
It’s funny, how invisible bike lanes and “Share the Road” signs are, until you have reason to notice them — and then you notice them everywhere. For instance: the big yellow “Share the Road” sign on the 1900 block of Hillandale Road. That sign is big. All those signs are big. You can’t miss them. How can you miss them?
I fantasized briefly about altering that sign, and every other sign in Durham, to read “Share the Fucking Road”, so that for a few days, everyone might actually take a good look at them, and be Enlightened. Except the responsible side of my brain pointed out that it would be an angry and ultimately futile gesture, and that I would probably do well to stop brooding, and get back to my life, and just fix my own self.
So that last thing — fix my own self, slow down, be patient — that’s a thing I can work on. But stop brooding? Get back to my life? Not quite sure how to do those things just yet.
Bye Seth. I’ll save a chair at Toast for you.
The ami2emi project has been moving along. As in, it actually works for a number of cases now. Configure cloud parameters for AWS and Euca, run a script, and boom: your chosen AMIs are brought to life on your Euca cloud. When it works, it Just Works. It’s cool. :)
Note that these cases do *not* yet cover configuration of the applications themselves — just the images. The bits might still need to be twiddled to get the apps working properly, but all of the actual bits are successfully transferred into the new image, and the new image successfully spins up instances, and you can ssh to them and everything.
The AMIs that can currently be auto-slurped into Eucalyptus successfully share certain characteristics:
1. They carry their own kernels inside the image. On the AWS side, that means they’re linked against the stock pv-grub kernels, and we link them similarly to the kexec-loader kernels on the Euca side.
2. They’re instance-store images, rather than EBS-based images. At least so far.
You can find the list of currently tested images here. That list will expand rapidly as we have time to run more test cases.
Note the large number of Bitnami instances in this list. That must be because they’re Crazy Awesome.
So, give it a whirl. Patches exceptionally welcome, since it’s heinous bash scripting and I can use all the help I can get. At least I document my code, sorta. :)
One of the most common questions I’ve heard asked from Eucalyptus users is this one: “how easy is it to convert an AMI to an EMI?” And the answer is: not as easy as it should be.
We’ve got some process guidelines on our wiki, thanks to Tim Gerla — but we should have the ability to do much of this automagically. So that’s the tool I’ve started work on. In the spirit of “release early release often”, I’ve uploaded a very early iteration of this tool. Find the brand new Github repo here.
It’s quite a ways yet from being prime-time, but it’s allowed me to do quite a bit of testing. Some assumptions I’ve started with:
* First, I pulled a list of all public AMIs on us-east-1. There were about 20,000 public images available there as of mid-November.
* Then I selected the subset of AMIs that were built with a PV-GRUB kernel, and I’m importing them to an instance of Eucalyptus running the kexec-loader kernel. In both cases, the AKI/EKI is just a bootstrapping mechanism that then hands control over to the image’s own kernel, so we shouldn’t get caught up by kernel incompatibilities. Using only the subset of kernels with PV-GRUB AKIs leaves us with about 7000, and picking one particular AKI (aki-825ea7eb) gives us about 1700.
* From there, we’re working with individual distros, and there will be idiosyncracies between the various distros out there, so it makes sense to pick one and go with it. There’s a lot of Ubuntu out there on AWS, so I just grepped on AMIs with “ubuntu” in the name. That’s 1067 images in my dataset.
* Now that we’ve got a reasonable set of images to examine, here’s the process that the scripts walk through. For each AMI, we start an AWS instance, ssh to that instance, install euca2ools if they aren’t already installed, scp Euca credentials to the instance, and bundle the instance to the specified Euca cloud. Then we fire up the resultant EMI on the Eucalyptus side and see if we can ssh in. And we bail with appropriate error messages at various places along the line.
I’ve run through a few hundred images at this point. Not one of them has been completely successful from start to finish. About 10% so far have yielded a bundled EMI that boots and yields a Eucalyptus instance in a running state. Can’t ssh into any of them yet, though.
The good news is that the failures are all quite specific and predictable, and the next steps are clear. Do a better job of guessing login IDs. Figure out why fstabs fail. Look for rogue kernel modules. Make sure we’re doing key injection properly on the Euca end. Comb through the results of euca-get-console-output and look for patterns. The big win is having tools that allow us to do that work in minutes, rather than in hours or days. Every step gets us closer to the goal of fully automated conversions on the fly.
Oh, and an apology: a lot of this should probably have been written using Eutester, instead of as a bunch of shell scripting. The fact is, I’m a terrible hack. But I’m just leading the charge temporarily; when I’ve figured out the basics, the real coders can swoop in and do things the right way. In the meantime… patches welcome.
We’re big fans of the Cobbler project here at Eucalyptus. We think it’s the best tool in the open source world for bare metal provisioning. We’ve invested in a gigantic QA environment for continual integration testing, and Cobbler is one of the linchpins of that environment. It’s the kind of tool that’s best appreciated by sysadmins who deal with *a lot* of systems.
I’m sort of attached to Cobbler personally, since I watched it grow out of Red Hat’s Emerging Technologies team several years ago. Now it’s grown past its Red Hat roots to become a truly independent project — and independent projects need support from time to time.
The Cobbler folks have set up an Indiegogo campaign to raise some funds for some much-needed infrastructure, and as proud Cobbler users, we are proud to help them out. Their goal is to raise $4000, and Eucalyptus will match every donation, dollar for dollar, until they reach their goal.
If you’ve used Cobbler and it’s helped you do your job, pitch in. The campaign is running for two more weeks; let’s help put them over the top.
Today we officially launch the next generation of FastStart, the quick deployment solution for Eucalyptus. We think it’s a pretty dramatic improvement to our previous version, and it’s certainly the easiest way to stand up your own AWS-compatible private cloud.
And while I have you, I’d like to shout out to the guy who made most of this happen: a guy named Bill Teachenor. When you use FastStart today and discover that it’s totally awesome, come by #eucalyptus and say thanks to bteachenor for all his hard work on the Silvereye project, the codebase upon which the new FastStart is based. There were plenty of other folks who helped — but Bill was the one who took the ball.
Open source is powerful because you don’t need anyone’s permission to make it better. You just need time, belief, determination, and a bit of skill in the right places. Bill looked at FastStart with the eyes of an experienced sysadmin, picked out a whole bunch of places where we could do stuff better, and led the way. When you write good code that does useful stuff, people will follow. Rough consensus and working code: it’s what drives the open source world.
So here’s to Bill, and all the folks who say “I can make this better” and then commit code at 2am to prove it.
(I’m sure you all know that step one is “cut a hole in the box”.)
We’ve been continually working to improve the install process of Eucalyptus over the past few months. In particular, we’ve been working on a project that we call Silvereye. Our most recent goal: make it trivial to install a fully-running Eucalyptus cloud on a single machine.
A cloud on one machine? Why bother? Well, lots of reasons, actually. The biggest: the developer workstation. If you’re hacking on Eucalyptus, it’s pretty awesome to have Eucalyptus on a single system that you tear down and rebuild in 15 minutes.
Anyway: mission accomplished. Go to our Silvereye downloads directory and get the latest build (right now it’s silvereye_centos6_20121004.iso). Burn it to DVD, boot your target system, and choose the “Cloud-in-a-box” option from the Centos-based installer. Answer some simple questions. Boom, in 15 minutes you’ve got a cloud-in-a-box!
(Note #1: a helpful README can be found in the Github repo for Silvereye: github.com/eucalyptus/silvereye.)
(Note #2: in the cloud-in-a-box config, when you log in as root for the post-install config, it’ll say “hey, do you want to install the frontend now?” Answer yes. It automatically installs the node controller for you.)
(Note #3: Silvereye is not supported. At all. If you use it, there are ABSOLUTELY NO GUARANTEES that it won’t burn down your house, steal your pickup truck, or throw your mother into a wood-chipper.)
Silvereye is mostly the work of sysadmin-par-excellence Bill Teachenor, based on the original Faststart installer written by David Kavanagh — but various folks are now working on it; Andy Grimm, Graziano Obertelli, and Andrew Hamilton have all been pushing the cloud-in-a-box on various distros, and Scott Moser of Canonical did some great proof-of-concept work on the UEC code. So thanks to all of them, and everyone else who’s played with it.
Give it a spin; it really is dead-easy. We still need to round off a few corners before we can call it the official installer of record, but we’re quite close now.
Want that AWS-compatible cloud on your laptop? Of course you do. Now go get it.
Here’s the thing about running your own cloud infrastructure: once you make the decision to rely on it, then it had better work. The whole thing. Every part of it. Under heavy load. All the time.
Obvious, right? But it bears repeating. When you decide to make the move to doing things The Cloud Way, you are placing a gigantic bet on your infrastructure layer — and that bet is placed not only on the Cloud As A Whole, but on every individual component that comprises that cloud. In the open source world, these are frequently components that you didn’t write and do not control. I can assure you that customers don’t care in the least.
At Eucalyptus, we have smart and demanding customers, with extremely high expectations. They are not content with assurances that things will be production-ready at some magical release point in the future. They don’t care whether the bugs are in the cloud controller code, or the node controller code, or in libvirt, or in the kernel. They are using Eucalyptus at extreme scale, right now, to solve extreme business problems, right now. Which means that when their cloud breaks, they expect fixes right now — and if that means libvirt patches or kernel patches, that’s what it means. That’s why they give us all that nice money. That’s why customers pay us for free software.
Our customers try to squeeze every ounce of performance out of their machines; that’s part of the point of having a cloud, after all. And when the virtualization technologies we depend upon experience heavy load over a long period of time, we see some crazy things. Like segfaults in libvirtd, for instance. Or libvirt handlers that suddenly and inexplicably lose their mind. Or other weird occurrences that might lead one to believe that libvirt isn’t quite as thread safe as advertised. These failures may only occur at times of very high load, and they may not happen often — but they do happen. And when they happen, we have to handle them. The 3.1.2 release is the result of many hours of hard work by our engineers to find and fix these issues.
It’s a challenge and a privilege to serve customers like this. At times it can put incredible stress on the entire organization — and it’s at precisely these times when we are at our very best. Watching great engineers solve critical problems under pressure is a lot like watching great athletes at the end of a big game — and when they win, it’s just as exhilarating. These engineers are at the heart of what we do. Compared to them, I’m just selling tickets and fetching Gatorade.
It’s not that hard to put together a bunch of components and call it a cloud. But making a cloud bulletproof? That’s hard. And that, friends, is where we are the best in the world.