Provisioning of FreeBSD 9.0 jails based on ZFS clones

The FreeBSD jail is an operating system level virtualization technology. One common kernel is used by the host and the hosted containers. Hosted jails have restricted access to the resources of the system.

It’s not so complicated to create jails:

  • userland must be built at least once, later can be rebuilt, if the base system was upgraded
  • installed userland or parts of the userland for every jail
  • enabled and configured jails

ZFS will be used to deliver jails. A jail can be relative easy to clone from the snapshot of an example jail.
ZFS snapshots represent consistent filesystem states at the given moments. These snapshots are useful securing backups or creating filesystem clones.
Creating a snapshot takes a moment and cloning the snapshot takes an other one. ZFS clones are the writable instances of snapshots. The changes consume only space on disk, so the delta is stored. The common parts share the same data blocks and the modifications shall be written to new blocks. Create a snapshot before major modification and the rollback is relative simple.

2 ZFS dataset will be created on an existing ZFS pool – zroot in this case:

  • One dataset for the base userland of the template jail. This will be automatically mounted under /jail/template
  • A second one for skeleton of the jail specific data, config and additionally installed files. /jail/skeleton

Ports collection will be mounted in read only mode from the host for the whole system. Ports update will be easy. Jail related ports information will be stored on the cloned skeleton (see make.conf). Ports collection was installed for the host system:

portsnap fetch
portsnap extract
portsnap update

Creating the template of base userland:

zfs create -p zroot/jail/template
cd /usr/src
make world DESTDIR=/jail/template
make distribution DESTDIR=/jail/template

(The make world took near 2 hours on 3,3GHz Intel Dual Core E6800 with 4 GB of RAM and SATA2 mirror.)

Creating the skeleton for the additional data and configs:

zfs create zroot/jail/skeleton
mkdir -p /jail/skeleton/home /jail/skeleton/X11R6 /jail/skeleton/ports/distfiles /jail/skeleton/ports/packages
mv /jail/template/etc /jail/skeleton
mv /jail/template/tmp /jail/skeleton
mv /jail/template/root /jail/skeleton
chflags -R 0 /jail/template/var
mv /jail/template/var /jail/skeleton
mergemaster -t /jail/skeleton/var/tmp/temproot -D /jail/skeleton -i
cd /jail/skeleton
rm -R bin boot lib libexec mnt proc rescue sbin sys usr dev

Add the following Ports Collection related lines  /jail/skeleton/etc/make.conf

cd /jail/skeleton
echo WRKDIRPREFIX=/data/ports >
echo DISTDIR=/data/ports/distfiles >> etc/make.conf
echo PACKAGES=/data/ports/packages >> etc/make.conf

Preparing the template to refer into the cloned skeleton:

cd /jail/template
ln -s data/etc etc
ln -s data/home home
ln -s ../data/home usr/home
ln -s data/root root
rmdir usr/local
ln -s ../data/local usr/local
mkdir usr/ports
mkdir usr/src
ln -s ../data/X11R6 usr/X11R6
ln -s data/var var
ln -s data/tmp tmp

Create the master snapshots:

zfs snapshot zroot/jail/template@master
zfs snapshot zroot/jail/skeleton@master

Clone a new jail in two step:

zfs clone zroot/jail/template@master zroot/jail/1stjail
zfs clone zroot/jail/skeleton@master zroot/jail/1stjail/data

Configure the new jail in the rc.conf:

ifconfig_alc0_alias0=”inet 192.168.11.1 netmask 0xffffffff”
ifconfig_lo0_alias0=”inet 127.1.0.1 netmask 0xffffffff”

jail_enable=”YES”
jail_list=”1stjail”
jail_set_hostname_allow=”NO”

jail_1stjail_rootdir=”/jail/1stjail”
jail_1stjail_hostname=”domain.tld”
jail_1stjail_ip=”127.1.0.1″
jail_1stjail_procfs_enable=”NO”
jail_1stjail_devfs_enable=”YES”
jail_1stjail_mount_enable=”YES”
jail_1stjail_fstab=”/etc/fstab.1stjail”

The jail ip can be configured directly to the physical nic. I mainly use pf to do nat and port redirection for the jails. It’s an other story.

The /etc/fstab.1stjail file:

/usr/ports      /jail/1stjail/usr/ports       nullfs  ro              0       0
/usr/src      /jail/1stjail/usr/src       nullfs  ro              0       0

Starting the jail:

/etc/rc.d/jail start 1stjail
Configuring jails:.
Starting jails: 1stjail.
jls
JID  IP Address      Hostname                      Path
1  127.1.0.1       domain.tld                            /jail/1stjail

3 thoughts on “Provisioning of FreeBSD 9.0 jails based on ZFS clones”

  1. Hi,

    Thanks for this post. Maybe I’m wrong but it seems /jail/1stjail/data/local and /jail/1stjail/data/local/etc should be created before symlinking data/local to usr/local or else package installation will fail.

    I don’t know if it’s still true but with past releases of FreeBSD, naming jails with numbers as first character was a bad idea.

    Regards,

  2. I followed the instructions, however I didn’t check until a failure but /jail/template was never created as a mount point. I have
    /zroot
    /zroot/jail
    /zroot/jail/skeleton
    /zroot/jail/template

    I created the ZFS pool zpool create zroot /file1 /file2

    And of course the build failes on the template because the mount point
    /jail/template does not exist.

    You refer to /jail on a number of occasions so I am wondering if I created
    my ZFS mountpoints correctly/

    FreeBSD 9.0-RELEASE #0

    Cheers,
    Mark.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s