Recently in System Administration Category

The XEN hypervisor uses big files (a couple of gigabytes) as filesystem images for virtual machines. Unlike other virtualisation solutions XEN does not impose its own internal structure on the image file. The big file simply has to contain an ordinary ext3 filesystem and, optionally, a partition table just as if it were a real hard disk. The ability to use big files as hard disks comes in handy if you are running short of space on your main hard disk. With an external hard disk you should be well prepared to run a number of virtual machines as big files. However, having the filesystem of a virtual machine in a big file raises the question of how to boot the virtual machine. Essentially there are two options to do that:

1. Provide the VM's kernel and the init-ramdisk, which are usually stored inside the filesystem (in the /boot directory), as separate files together with the big file, and modify the VM's configuration to use them.

2. leave the kernel and the init-ramdisk in the big file and provide a working boot sector that accesses the kernel inside the big file, using the native XEN pygrub bootloader to start the virtual machine.

Both options require that the big file must be associated with a real, special device file (i.e /dev/loop0) in order to create a filesystem on the big file. While for the first option it is sufficient to simply connect the big file with the loop device, using the "losetup /dev/loop0 bigfile" command, the second option is much more complex, as the big file has to be partitioned like an ordinary hard disk before the filesystem can be created.

For the rest of this article we will focus on the second option which is much more appealing as everything is kept inside the big file. I will show you how exactly the big file is turned into a virtual hard disk and how you can access and modify the information stored in the virtual machine's own filesystem.

Getting Partitions And Filesystem Sizes Sorted

Our journey through the big file's internal structure naturally begins with the creation of the big file.


As a second step we use this chunk of 4141875200 bytes to act as a hard disk and try to partition the bigfile as usual:


As expected, the fdisk program throws a number of error messages at us, because we have given a big file instead of a real hard disk to the program. But let's see how the fdisk program recognizes our new hard disk in detail
.
Obviously there is no partition table yet, but the program assumes that the big file represents a hard disk with 255 heads and 63 sectors of 512 bytes data each. Every cylinder of our virtual hard disk is made of 255 x 63 x 512 bytes = 8225280 bytes which represents the units in which we can chop the hard disk space into partitions now. All in all there are 503 cylinders in our virtual hard disk which makes a total of 503 x 8225280 bytes = 4137315840 bytes to spend on partitions.

But wait, didn't we create 4141875200 bytes in the first place? That's 4559360 bytes less than what we had originally. Well, this loss is due to the fact that for the 504th cylinder we'd need 8225280 bytes which we don't have, so this loss is inevitable. But the important consequence of this reduction of space is that we cannot create a filesystem on the whole bunch of data we supplied. At the moment the size of our filesystem is not determined at all. The next step is to create a new primary partition inside our big file using all the space we have:


After having written the partition table to the big file, have you checked for the new device file /dev/loop0p1? Don't worry, it does not exist! Adding p1 to the disk label is fdisk's way to denote partitions, that does not mean that you'll find such a thing in the /dev directory.

Poking Inside The Big File

From the partition table you can see that 4040316 blocks have been allocated for the new partition. With each block storing 1024 bytes we now know our first partition size, it's 4040316 x 1024 bytes = 4137283584 bytes. This is another number we never saw before! After having written off some 4.5 megabytes because we cannot use half a cylinder, we now face another loss of exactly 4137315840 - 4137283584 = 32256 bytes.

Of course these 32256 bytes at the beginning of the big file are there for a purpose, which is to store the partition table. Our first partition begins right after this amount of data, at an offset of 32256 inside the big file. The amount of 32256 bytes results from the fact that one track (63 sectors of 512 bytes for one head) are put away for the partition table. Now it's time to use a second loop device (/dev/loop1) to poke inside the big file at exactly the point where our first partition begins and create a new filesystem there:


It's essential that we supply the number of blocks as a parameter to the mkfs command to ensure, that our new filesystem on the first partition fits exactly in the space we have allocated. Without this parameter our filesystem would become too big, as the 4.5 megabytes after the first partition would be used for the filesystem too, and when the virtual machine is going to use the filesystem its actual size would conflict with the numbers in the partition table. Either the partition table or the filesystem's superblock is lying, which will cause distress for the virtual machine that expects a consistent filesystem to operate.

Writing The Master Boot Record

You can fill up the filesystem with whatever carefully selected quality open source software you can find on the planet, but in the end we need to write the new virtual disk's master boot record to boot the jewel. There is one step of preparation to be done before we can use the grub shell to write the MBR. We have to make a symbolic link named /dev/loop to the device that points to the master boot record, that is to the beginning of the big file, /dev/loop0 in the example above.


Now your spick-and-span virtual hard disk is ready to boot.

Current Phishing Attacks

In early October when news came up that quite a lot of Hotmail and Gmail accounts had been compromised by phishers who tricked users into using a faked login page the general advice for scared users was: "change your email password immediately". Although nobody really knew how the email passwords published on the internet were gathered, it became common knowledge that only those users who had used the phishing sites to log into their email accounts were vulnerable. Everybody else whose account was not blocked by the email providers started to feel safe again.

The fundamental problem with email is that access to the mail account is only protected by a password, and that every time a user fetches his email from the internet service provider (ISP), his password is transmitted to the mail server in clear text. To improve the situation email service providers are beginning to use opportunistic TLS, a method to encrypt traffic between mail servers. In this cases encryption is used if the mail server provides it, and in a perfect world mail would always travel encrypted to the user's mail server. But opportunistic TLS does not solve the user's problem as almost all email software he actually uses do rely on a password only to access an email account.

The Perfect Mail Server

In fact, what we have to do is to make sure that email is properly encrypted while the user fetches it from his email account, and that the mail server establishes a connection only if the person trying to get the mail is able to present more information than a password. This configuration would not only protect the email content and the password during transmission, it would as well ensure that someone only knowing the password would never be able to establish a connection to the mail server.

Fortunately this can be achieved without an additional burden on the user, because we only need a careful setup on the server side and provide the user with the additional information (a secret key) which has to be stored safely on the computer that initiates the mail transfer for her.

Setting Up A Secured Mail Domain

Let's start with the mail server configuration, which in my case is dovecot 1.0.7 on a CentOS server. I will show you how to configure dovecot to provide the mailbox of system users in a secure way. I assume that the firewall on the mail server is open for incoming mail on port 25 and that the mail server uses port 993 for outgoing secure IMAP traffic.

This is the essential part of the "dovecot.conf" file I use for secure IMAP access:
(I discuss the relevant parts of the configuration only)

First of all, by restricting the protocols to imaps and pop3s dovecot releases port 143 and 110 which could safely be blocked now. There are three certificate files which have to be in pem format. While the first file contains the public certificate for the mail server, the second must hold the unencrypted secret key. The file permissions on the later have to be restricted to read access for the root user as the key is read by the dovecot process before dropping permissions to the dovecot user.

The third file does not only contain the RootCA's certificate but also a valid Certificate Revocation List in pem format appended to the RootCA certificate with a newline in between. Without the CRL in this file the user's certificate cannot be validated and the connection will not be established. It is extremely important to keep an eye on this CRL, as the time frame for replacing a CRL is usually only a few days long. If you miss to keep the CRL part up-to-date you'll risk that the perfect mail server setup expires in a few days, leaving your users with no access to their email. You can create long-living CRLs with the openssl crl command as you like, but bear in mind that the expiration date of your CRL does determine the ability of your users to get at their mail in the same way as does the expiration date of the user's certificate.

You may have noticed that in the authentication section everything is disabled except the use of two files which serve as a static user and password database. Only system users listed here are able to access their mailboxes. The two files "imap.passwd" and "imap.shadow" are exact copies of their system analogies, limited to the lines of valid email users. Just make sure that both are stored in a directory with the name of the email domain you use, which is shown as %d in the configuration file above.

Finally, the last line "ssl_require_client_cert = yes" determines that the mail server shuts down the connection if the user's email client is unable to present a client certificate, that the mail server can validate based on the content of the CA certificate file in the configuration.

By now we have made access to our mailboxes as difficult as possible, it's time to make it accessible for the user in a way that does not hurt. Please read on.

Supplementing The Mail Password With SSL Certificates

What the user needs is a certificate, a public key, which is signed by the RootCA used by the mail server and a corresponding secret key. And, of course, a software that uses this information to download the email from the user's mailbox to the local machine over an encrypted tunnel. With another careful setup on the client's computer this is automatically done by the fetchmail process that can be run via cron in certain intervals. So the users receive their mail without bothering about the encryption process at all. They can read their internet email just like they read their local email.

Fetchmail uses a config file ".fetchmailrc" which has to be protected carefully (read access for the root user only) as it contains all the user's email passwords. It looks like this:

For the client to establish the SSL connection it is not enough to present the public certificate to the mail server, it is also necessary to be able to use the secret key. And this is what the attacker who may have learned the email password will not posses and what will keep him out of our mailboxes.

If you have trouble creating those certificates, please drop me an email and I will see what I can do for you.

ISO Files

Most Linux distributions and a number of CD utilities like Clonezilla or GPARTED-Live come as iso-images, ready to be burned to a CD medium, to start an installation or running a live session for some useful purpose. But all these iso files are supposed to be booted inside an ordinary CD drive. What if you just don't have a CD burner at hand or if your CD drive is damaged or missing all together, because you are testing the shiny new motherboard without a CD drive?

Don't think you are stuck, there's help around the corner that will lead you into a habit of booting your iso files in a much more creative fashion.

"Stop for a moment, could I not simply put the iso file on a USB key using the dd command? Most modern BIOSes are perfectly capable of booting USB keys and surely there is a USB slot on your new motherboard you can use."

Of course you could do that, the iso file contains a valid iso9660 filesystem together with the bootloader for CDs. And the dd command will transfer the file to your USB key without adding any other structure. But, don't expect the new USB key to boot like a CD, because after you have changed your BIOS settings to allow booting from a USB hard disk, the BISO will expect the structure of a hard disk not that of an iso-filesystem. It will look for a MasterBootRecord (MBR) in sector 0 in the same way as if it would boot your ordinary hard disk. And an iso-filesystem doesn't have a MBR and a partition table where the BIOS expects it to be. So forget booting iso files directly.

"Ok, seems as if there is no way around putting a bootloader like grub on the USB key, and ... , wait, there is a file called iso9660_stage1_5 in the grub directory. It looks like grub has iso9660 support already. Can we use this to boot the iso file?"

Well, unfortunately no. The iso9660 support built into grub can be used to create a bootable CD with grub as the bootloader. Your file is being used when a new iso file containing your grub bootloader is created. What we need instead is the ability to read the kernel and the initramdisk inside the iso-filesystem at boot time when no filesystem is mounted.

"Sounds as if we need a loop device to get at the files inside the iso filesystem. Does grub support mounting a iso-filesystem using a loop device?"

The New Grub

Yes and no. The traditional grub (version 1) that everyone uses does not have this loopback support, but the grub development team has been working on grub2 for a number of years now, which has loopback support already. The new grub2 is still under development, so it's not yet ready to replace the legacy grub version 1, but grub2 is exactly what we need to boot our iso files directly.

A Word Of Caution

So we are about to prepare a USB key to boot iso files with the grub2 bootloader written into the key's master boot record. Writing MBRs on a Linux system usually raises blood pressure a bit, as we have to be extra cautious not to ruin our running Linux system. If you want to follow me from here, please make sure that you do not perform the grub2 installation on your company's production server and double-check everything before issuing commands, to be on the safe side. And for those of you, who hesitate to write MBRs at all, I have an offer for you at the end of this posting, read on.

Preparing The USB Boot Key

Like any other software grub2 can be installed using the distros repositories, at least with Fedora 11 that I use now. The installation will not replace our legacy grub software we are using to boot our computer, as the new files go into a directory "/boot/grub2" without changing anything of the current grub setup unless grub2 is deliberately used to overwrite the MBR. And this is the only thing we have to avoid. But for now, our objective is to get the grub2 software onto our USB key, writing the key's MBR will then be the crowning event before we start testing the key.

For preparing the boot key it is absolutely essential that you know how the Linux kernel recognizes your USB key. To find out, you can observe the log file "/var/log/messages" while you plug your key into the computer. For now I assume that you have one hard disk in your computer (/dev/sda) and your USB key will probably show up as /dev/sdb. First we re-partition the key and create two Linux partitions, a small one of 10 MBytes for the grub2 software and a larger one where all our iso files can be stored:

You can now mount the second partition and copy all the iso files you wish to boot into this location. Writing the MBR now requires that the first partition is mounted on a well know mount point like /mnt, because we will use this directory to write the grub2 software to the key with the following command:

Please double-check that your key is mounted and is called /dev/sdb before you use this command, which populates the directory /mnt/boot/grub2 with grub2 modules and writes the MBR into /dev/sdb.

As you may know there is one task still left before we can start testing our boot key, we need to create a menu list, which is called "/boot/grub2/grub.cfg" now. A number of changes have been made to the config file compared to the old grub, so let's have a look at the new configuration:

The most remarkable change is the numbering of partitions which has become more natural as the first partition now is (hd0,1), while hard disks still start with number 0. Then the old title entry is replaced by menuentry and loading the kernel is now done with the linux command. Apart from all these cosmetics we can now create a new device called (loop) that will replace a real partition like (hd0,2) to access files inside the iso-filesystem, once we have connected the iso file with this new device using the loopback command.

It is even possible to list the files that grub2 can see on the (loop) device using the grub2 command line:

Now it's time to check if the ISOLINUX menuentry works. Reboot and see your new boot key fire up the CentOS netinstall process. Voila.

Download Your Bootable USB Key

I have prepared an image of my boot key for you that you can download, if you want to create a boot key without the hassle of following the steps above. The small 32 MByte image file contains all the magic and can be copied to your USB key with the dd command. Please send me an email and I will give you a download link with the necessary instructions. Once you have your key working, you can delete and re-create the second partition to make space for all your live CD iso files on the key. All you need to edit is the config file.

That way you can carry all your distros on one key and boot them all from the grub2 menu without having to change the Linux system on your hard disk.

I know, you are one of those people who know that it is absolutely essential to make regular backups of your system's hard disk. There is no excuse not to make regular backups, but as we all know, we don't really do that. Or is it just me?

No, you have to admit that even though the reasons to make backups are indisputable, something strange always - no, not always, but many times - keeps you from doing the necessary, updating your backups and securely storing them away. Making backups is not fun, but that's not the reason why we don't do it regularly. I suspect that you as well as I simply don't have the right tool yet, to initiate an automatic process that boils down to getting a reliable backup complete and safe to restore, which does not require much attention while assembling the data for us.

Fortunately this situation is going to change dramatically for you, today. I've run into a backup solution that is just that, an automatic process that does not require user intervention when started to do the right thing. It's called CLONECILLA and is essentially a Live Linux system, optimised for backup and recovery.

Clonezilla - Your bootable backup CD

Clonezilla can be downloaded here as an ISO-file of 106 megabytes.. This file contains everything that makes up a bootable Linux system, so be careful to write this downloaded iso-file to your CD in raw-mode. Because it's of no use to have the iso-file on a CD in another file system, the iso-file is the file system and it is the only thing that has to be on the CD.

Did I mention that Clonezilla is a backup solution that covers all sorts of partition types, so that our windows-using friends could use it, too? Clonezilla has the ability to clone a complete hard disk no matter what kind of partition your disk comprises of. It even backs up the master boot record and the partition table, you will get a complete set of data from which all or any part will be recoverable when you need it. And all data is created as zipped files of max. 2 gigabytes size so that everything can be burned to DVDs, if needed.

Setting Clonezilla on the right track

As we are now heading for a complete backup of your laptop's hard disk, one thing is to be considered before we start, where will all the backup data be stored? Fortunately Clonezilla offers a number of possible storage media ranging from an additional (plugable) local disk, a SSH server or a SAMBA server to the traditional NFS server. I assume that you have a portable USB hard disk on which you will store the backup. Please ensure that the free space on this disk is about half of the size of what you are going to backup, and add a little bit of a buffer, too.

I have found that a second thing is important before you start letting Clonezilla create the backup, make sure that all partitions of your hard disk can be mounted. Best practice would be to perform a file system check on all partitions before starting the Clonezilla boot CD.

The following few steps are all you need to initiate a full backup of your laptop's hard disk and you will see that once you've selected a few things the rest is going in a jiffy.

Step 1	Boot your Clonezilla backup CD, select language and keymap, then select "Start Clonezilla"
Step 2	After selecting the backup media (local_dev) power on your USB disk and allow a few seconds for the system to recognize your new hard disk on which the backup will be stored. Clonezilla creates a directory on the USB disk derived from the date and time of the backup. You can choose where to store this directory on the disk. You can always switch to another terminal using <CTR><ALT>F2 to see that the USB disk is now mounted on /home/partimag. All files go into this directory.
Step 3	Select "savedisk" to ensure that the whole hard disk is backed up. When the partition information is read in Clonezilla will start the backup process automatically.
Step 4	Depending on the size of your hard disk, you can now relax and let Clonezilla compose your backup for you. Check the backup directory in the meantime to see it filling up with data.

The performance of creating the backup is very good, it took 47 minutes for a 30 gigabyte data partition on my three year old laptop. The whole disk (120 gigabyte) was ready after two and a half hours time leaving me a total of 66 gigabytes of compressed data.

Try Clonezilla today, it takes very little time to be secure. No excuses!