Modifying an existing cloud image

We recently updated emis.eucalyptus.com to point to official cloud images created by Linux distribution providers themselves rather than creating sample one's ourselves. While this goes a long way toward making it easy to grab a stable, up to date image, there are still cases where you might need to update software that exists on the image already or add more software of your own.

For most images, including all of those referenced on emis.eucalyptus.com, this is a fairly straightforward process.  Once you have downloaded an image, just perform the following steps:

Decompress the image, if necessary. The Fedora image, for instance, needs to be decompressed using xunzip/xz

Convert the image to RAW format, if necessary.  If your image ends in .img or .qcow2, it is likely in QCow2 format and needs to be converted using qemu-img from qemu-utils:

qemu-img convert -0 raw disk.img disk.raw

Add the raw disk image using kpartx:

kpartx -a -v disk.raw

Mount the first partition of the disk image:

mount /dev/mapper/loop0p1 /mnt

chroot into the mounted volume:

chroot /mnt

Once inside the chroot, you may update and/or add any software you need.  For Debian/Ubuntu images, you might want to run apt-get update followed by apt-get upgrade to ensure the latest versions of system packages in the image.  For Fedora/CentOS images, run yum upgrade.  Once finished with your changes, exit the chroot environment:

exit
umount /mnt


You should now have an updated image ready to push to your Eucalyptus cloud.  To do this, simple follow the instructions provided on http://emis.eucalyptus.com.

Importing an HVM image from EC2

I have recently taken a new role at Sales Engineer at Eucalyptus and one time consuming issue I keep running into with clients is image management - specifically being able to import an EBS backed image from EC2 into Eucalyptus.

The merits of this endeavor are open for debate: I would much prefer hybrid users use separate images on different clouds and simply use cloud-init scripts, or a similar mechanism, to pull up the same application stack on each of them.  That being said, customers want it, so we try to make it as easy as possible for them.

Another Sales Engineer here at Eucalyptus wrote a great guide on how to do this with HVM images on EC2:

https://github.com/eucalyptus/eucalyptus/wiki/Import-an-HVM-AMI

Important note: When using an EBS image in Eucalyptus, Eucalyptus assumes it to be an HVM (Hardware Virtual Machine) image.

I decided to take this a step further and automate it with a simple shell script.  You can grab it here:

https://github.com/iamfuzz/euca-image-tools/blob/master/fetch-hvm-image.sh

Please note that this isn't an officially supported tool and hasn't been tested by anyone other than yours truly. That being said, to test it out, the following pre-requisites must be met:

* An AWS account
* Obtaining the AMI ID of the HVM image you'd like to import
* An instance store AMI in your account to be used as a scratch instance with root access (some Ubuntu images may not work as they use user ubuntu instead)
* Enough disk space in the TEMPDIR on your local machine to store the image
* An ssh key setup on AWS and downloaded to your home directory as <key-name>.priv

Once you have downloaded the script, make sure you have your AWS secret key and access key exported in your shell environment, and then run it as follows:

./fetch-hvm-image ami-XXXXXXXX key-name [ami-XXXXXXXX]

The first AMI is the HVM AMI and is required.  The second AMI argument is optional and if left unspecified, the script will use the first instance-store AMI in your account that it finds.

If the script finishes successfully, it will display the image location in your TEMPDIR.  From there you can import it manually into your Eucalyptus setup or use the new, unofficial version of eustore mentioned in my previous blog post to do it for you.

Please note that many images will require modifications before importation.  For instance, the official SuSe images have startup scripts in them specific to AWS that can cause boot problems in Euca an must be disabled.  If you run into problems, feel free to post about it here, and I will do my best to help you out.

Enjoy!

eustore now supports bootable EBS images

Those of you familiar with Eucalyptus are likely already familiar with the eustore client, which is a utility included with the popular euca2ools.  It allows for the automatic fetching, uploading, and registering of instance-store based images in your Eucalyptus cloud.

It pulls its meta-data and images from http://emis.eucalyptus.com but currently filters on only our most tested images which include Fedora 16/17, Centos 5/6, and OpenSuSe 12.2.  Unfortunately, it provides no support for bootable EBS images.

However, as of today, you can beta test a new version of euca2ools that does include the ability for eustore to fetch, upload, and register your bootable EMS image:

yum install http://downloads.eucalyptus.com/testing/euca2ools/euca2ools-2.1.9999-0.0.1.20130315gitb1a30e90.el6.noarch.rpm

The new functionality is contained within the utility eustore-install-image.  To use it, you will need a bootable EBS image available on an accessible http server.  From there, you can simply point eustore-install-image to it and watch it work:

eustore-install-image -u <http://path.to/imagename.img> -v <image_size_in_GB> -k <eucalyptus_ssh_key>

You must have your SSH key located in your home directory and it should be named <keyname>.priv.  The utility assumes that you are passing it an unpacked EBS image bootable from /dev/sda1.

Please note that this has only been tested internally on a limited number of images and all feedback is not only welcomed but encouraged.  Enjoy!

Some of you may have heard that Eucalyptus no longer supports Debian or Ubuntu.
While it is true that given our current customer base and target customer base
that we have decided to support only CentOS and RHEL from a contractual
standpoint with our customers, it does not mean we have abandoned our
community efforts behind Debian and Ubuntu.

In fact, the opposite is true.  There was a long period of time where Eucalyptus
was not available in Debian and/or Ubuntu or only very old versions were
available.  Today, Eucalyptus 3.1 is available in Debian Unstable (we narrowly
missed the deadline to be included in Wheezy) and in Ubuntu Precise via the
Canonical Partner Repository.  Thanks goes out to James Page and others who made
that happen in Ubuntu.

The reason Eucalyptus 3.2 has not been made available in Debian is simply
because changes made to Eucalyptus require a new version of the Google Guava
libraries.  The Debian Java Team has already packaged the needed version and
uploaded it to Debian Experimental.  As soon as they deem it worthy for inclusion
in Unstable, we will look at updating Eucalyptus there as well.

We likely won't be making  any further release via the Canonical Partner
Repository as future versions of Ubuntu will have the Eucalyptus packages in
Ubuntu's Universe Repository.

For those of you who would like to build Debian Packages for Eucalyptus
from one of our development branches - in case you need one of the new features
it provides or perhaps just because you like to live on the bleeding edge - that
too is possible.

Building your own Debian Packages

Before building your own Debian packages, you will need to have the following:

 * A basic knowledge of git an bzr
 * The packages git, bzr, dpkg-dev, devscripts, pbuilder, and aptitude installed
 * A git clone of eucalyptus from github
 * The debian directory containing the proper files for building a package

Here is an example of grabbing the Eucalyptus source code and the Debian build
files, and building the Debian packages that make up Eucalyptus:

apt-get update
apt-get insall dpkg-dev devscripts
git clone git://github.com/eucalyptus/eucalyptus.git
bzr branch lp:~brian-thomason/+junk/eucalyptus-sid
cd eucalyptus
git checkout maint/3.2/testing
cp -a eucalyptus-sid/debian ./
dch -v 3.2.1-0 Experimental Build
sudo /usr/lib/pbuilder/pbuilder-satisfydepends
*You will need to comment out the following line:

import org.rhq.helpers.pluginAnnotations.agent.Units;

in the file:

clc/modules/storage-controller/src/main/java/edu/ucsb/eucalyptus/cloud/ws/BlockStorage.java

This will be fixed in the repository itself soon.

Amd finally, build the packages:

dpkg-buildpackage -b

This will build packages from the 3.2 maintenance branch, which will soon be
frozen and released as bugfix release 3.2.1.  If you would like to build from
the next feature release branch, replace 'maint/3.2/testing' with 'testing' in
the above step.

After the Eucalyptus packages have been built, you may then install the DEB
packages using:

dpkg -i ../*.deb

IMPORTANT NOTE:

These builds of Eucalyptus will require a newer version of libguava-java than is
available in Debian Unstable. Fortunately though, this package is
available in Debian Experimental and can be fetched here:

http://ftp.us.debian.org/debian/pool/main/g/guava-libraries/libguava-java_13.0.1-2_all.deb

You can fetch it with wget and install it using dpkg as well.

Once installed, you may follow the install guide loaded here for instructions on
how to setup your newly installed cloud:

http://www.eucalyptus.com/docs/3.2/ig

In the coming weeks I will expand upon this topic here by:

 * Explaining how to create an apt repo for each build you make and hosting
that static repo in S3/Walrus
 * Providing a script to automate much of the process

Best regards,
Brian Thomason
Engineer
Eucalyptus Systems, Inc.

My First Blog Post! - Oh and a Eucalyptus 3.1 Debian ISO

After 15+ years of being online, I think it's finally time that I started a blog.  I would have done it sooner, but I just didn't really have anything useful to share - although that doesn't seem to stop the millions in twitter land.

Since it is my first ever blog post, I think it best to give a bit of background on myself, but I will try not to bore you in the process.

My name is Brian Thomason, and I work for Eucalyptus Systems, Inc., the creators of the private Cloud, as a Release Engineer.  I will soon be transitioning into the role of Partner Engineer there and will be sure to keep you abreast of what's going on there with ramblings here from time to time.

I started my career 10 years ago at Lindows (which became Linspire) doing Debian packaging work.  While I cut my teeth on Linux using Mandrake 7.1, I later fell in love with Debian due to its package management system, and more specifically its dependency resolution tool, apt.  Lindows was trying to take Debian to the next level on the consumer Desktop by creating a user interface for apt dubbed CNR  (Click-N-Run)  In a world that was then dominated by off-the-shelf software on CD-ROM, Lindows saw the vision of digitally distributed software, and that was a vision I bought into eagerly.  Apparently it wasn't a bad idea, as Apple's App Store has shown.  Unfortunately, Lindows started going downhill financially and eventually shut its doors.

From there, I moved on to Canonical, the company behind the popular Linux distribution Ubuntu.  For my first few years there I worked on Debian packaging of third party software just as I had done before at Lindows.  During my last year there I was transitioned to the Systems Integration team where I worked on taking large software projects and integrating them into Ubuntu. (glorified terminology for more Debian packaging work) This is when i got my first taste of the "Cloud", working on projects such as CloudFoundry and OpenStack.

Ironically, upon my return from an OpenStack sprint (one-week jam session) at Canonical, I found a message from a Eucalyptus recruiter waiting in my LinkedIn inbox asking if I was looking for new opportunities.

So here I am.  You can guess what Eucalyptus hired me for - yes, aside from my dashing good looks and charming wit, it was for my experience with packaging various types of software for Debian/Ubuntu.

All of that is what has led me to you today.  Thanks to much help from the Debian Eucalyptus Team, Eucalyptus 3.1 is now available in Debian Unstable (Sid).  Sadly, it made it in just after the Wheezy freeze so won't likely make it into that release.

However, if you' like to give it a try, I have created an installer ISO image based off of the Debian Mini ISO that will allow you to easily install Eucalyptus atop Debian Sid on your amd64 system.  Please note that the mini ISO requires that you have an internet connection to perform the install as only a very minimal set of packages is included on the ISO itself - hence, Mini.

Without further ado, here it is:

https://docs.google.com/open?id=0B1CYTrJvmzcTTWlLeGtPLWQwWVk

Please comment if you have issues/questions.  For those not familiar with Eucalyptus, I will be posting a set of configuration instructions here very soon.

Thank you for reading, and enjoy!