by: Matt Simerson
IP: 54.156.47.142
Sunday 24 Sep 17

Booting FreeBSD via PXE (Preboot Execution Environment)

Version: v1.1 (also available: 1.0)
Publish Date: Dec 5, 2000.
Updated: Jan 6, 2001.

Audience: Unix System Administrators

Objective: Document the steps necessary to boot and/or install a FreeBSD system using a DHCP, NFS, & TFTP server.

Background: FreeBSD can boot and install off a variety of mediums. The common and most useful are floppies, cd-roms, and the network (ppp, NFS, & ftp). All the methods work essentially the same, you bootstrap the helpless machine off floppies (yes, even the CD boot uses floppy emulation) and the mini-FreeBSD system then has enough smarts to initialize CD-ROM drives, Ethernet, and PPP interfaces. This all works fairly good for installing FreeBSD on one machine but what happens when you want to install it on 20, or 50, or 1,000 machines? Right now the process is pretty much a manual one on every machine. This is expensive, error prone, and slow.

Excuse: I had quite a few reasons to embark upon this project.

1. It's a Royal PITA [TM] to build a custom FreeBSD install CD. I literally spent days fuddying with cramming all the stuff I wanted onto a 2.88M disk image and burning a lot of coasters getting it perfected. The process of building bootable media isn't documented well and worse yet, the only real documentation is found by searching mailing lists. This is less than convenient as I found myself compiling lots of post-it notes.

2. Even when I successfully built a bootable CD I was severely limited because I was stuck in the confines of 2.88M. You must wrestle with picobsd and the unless you're a good programmer (I'm not) you're pretty much stuck living with the choices that were made by others for their obtuse reasons. No matter how hard I tried, I just couldn't have as much fun as I'd like to when booting off a CD.

3. Building a restore CD (to boot a failed server) that boots, rebuilds, and restores an entire system is not easy. I needed a simpler method of getting a machine bootstrapped than spending hours building a custom boot CD that would let a clue deficient NOC monkey restore my server.

4. The only time the CDROM drive gets used is when I'm installing the OS. It's quite a waste to buy servers with Cd-Roms when you're only going to use them once. You can do a lot more with 1U and 2U hardware when you've got another drive bay to play with.

5. In my spare time I'm building a cluster of FreeBSD machines. It's primary purpose thus far is the conversion of electricity to heat in my garage. I figure at some point I'll find a use for it but until then, just building it has provided me with ample motivation to learn a lot of cool new stuff about FreeBSD. Anyway, I buy Pentium 133 systems (complete with NIC cards) for $35 but without Cd-Roms. It's very slow to get FreeBSD installed so I needed a better way.

At BSDCon 2000 I sat in on a panel taught by Doug White on automated system installations. It gave me the impetus to charge forth and conquer the beast known as PXE.

I'm too lazy to look up some of the facts (again) so anywhere I use <>, please feel free to send me the relevant information and URL's where I can verify and I'll update my documentation.

Some time ago <date please> Intel developed a technology known as PXE and began blessing their wonderful EtherExpress Pro Server Adapters with this spiffy new feature. <Some 3Com NIC cards> also include PXE technology. PXE is designed to allow a NIC card to fetch a configuration from a DHCP server and boot up a computer via it's network interface. John Baldwin and Paul Saab at FreeBSD saw the usefulness of this feature and wrote a little boot loader appropriately named pxeboot. Pxeboot is included in FreeBSD 4.1 and higher.

So, how does it work you ask? Very well, I must say. Once you satisfy it's many dependencies, things work great. Getting everything satisfied wasn't as simple as I'd hoped. Anyway, follow the steps below to reach enlightenment.

Requirement 1: FreeBSD distribution. It would be nice for FreeBSD to just magically appear on your hard drive but alas, we've got to get it from somewhere. I suppose you could fetch it from the FTP server but I simply copied the CD contents to a NFS exported file system on my server.

# mount /cdrom
# mkdir -p /usr/local/export/freebsd4.2
# rsync -avz /cdrom/ /usr/local/export/freebsd4.2
# ln -s /usr/local/export/freebsd4.2 /usr/local/export/freebsd

Requirement 2: Ethernet adapter with PXE boot roms. The Intel Management adapters all include PXE but even our newest batches required flash updating to get FreeBSD booted properly. This is what my oldest Intel's looked like at first:

Intel UNDI, PXE-2.0 (build 067)
Copyright (C) 1997-1998 Intel Corporation

This version of PXE bios does not work. It will load the pxeboot loader but fails soon thereafter. A visit to Intel's web site had me downloading a file named 8255x.zip. Within that archive was three very useful things. The first was a directory name 8255x containing the boot ROM's for the Intel 8255x adapters. Logical huh? :-) The other useful tool was the fboot.exe program. I created a DOS boot floppy and copied these programs onto the boot floppy. I then installed five Intel adapters into the PCI slots of my server and proceeded to update them all.

After installing the new firmware the PXE bios looks like this:

Intel (R) Boot Agent Version 4.0.12
PXE 2.0 Build 082 (Wfm 2.0), RPC v2.7.3
Press Ctrl+S to enter the Setup Menu

Requirement 3: DHCP Server. I already had ISC-DHCP 3.0b installed so I merely had to add a couple lines to my DHCP configuration. Here's what a working configuration looks like:

option broadcast-address 192.168.254.255;
option domain-name-servers 192.168.254.3;
option domain-name "simerson.net";
option routers 192.168.254.1;
option subnet-mask 255.255.255.0;
server-name "DHCPserver";
server-identifier 192.168.254.3;
next-server 192.168.254.3;
default-lease-time -1;

subnet 192.168.254.0 netmask 255.255.255.0 {
range 192.168.254.32 192.168.254.99;
option root-path "/usr/local/export/pxe";
filename "pxeboot";
}
host cm.simerson.net {
hardware ethernet 00:e0:18:98:f0:cc;
fixed-address 192.168.254.126;
}
host c1.simerson.net {
hardware ethernet 00:60:97:0e:bb:a7;
fixed-address 192.168.254.131;
}

Requirement 4: DNS server. I'm not sure it's its necessary but I'm sure it's at least a good idea. Create an entry in your DNS records for dhcpserver.yourdomain.com and records for all the addresses in your DHCP pool. It's saves us a lot of time waiting for DNS timeouts when we've just got DNS set up correctly.

Requirement 5: TFTP server. The DHCP configuration tells the booting client that it's supposed to grab the filename "pxeboot" from the TFTP server (next-server) at 192.168.254.3. Here's how mine is set up:

# grep tftp /etc/inetd.conf
tftp dgram udp wait nobody /usr/libexec/tftpd tftpd -l /tftpboot

# ll /tftpboot
-rw-r--r-- 1 root wheel 165888 Nov 30 11:46 pxeboot

This is pretty easy to configure. On most systems, simply comment out the tftp line in your /etc/inetd.conf and restart inetd (killall -HUP inetd).

NOTE: TFTP has virtually no built in security. You should only enable a TFTP server on an internal (trusted) network or use a firewall to restrict access to it. At a minimum, use TCP wrappers.

Requirement 6: PXEBOOT. Copy the pxeboot file from your /usr/src/sys tree to the /tftpboot directory and you're all set:

# cp /sys/boot/i386/pxeldr/pxeboot /tftpboot

Once your machine has loaded the NIC cards PXE bios, it will (assuming it's the chosen boot device) make the tftp request for the file "pxeboot" from the tftp server. The tftp server, being properly configured will hand it the file "pxeboot" which is comparable to the FreeBSD loader program.

NOTE: The pxeboot program can be compiled to fetch the loader via TFTP or NFS. NFS is the default but you can add this "LOADER_TFTP_SUPPORT=YES" to your /etc/make.conf and recompile pxeboot (#cd /usr/src/sys/boot; make clean; make depend; make; cp i386/pxeldr/pxeboot /tftpboot).

Requirement 7: Boot loader. Once PXE boot is loaded it will fetch the files it needs from the /boot directory that's defined within the root-path directive your DHCP server handed it. Since we've defined a root path of /usr/local/export/pxe, it'll be looking within the /boot directory there for the second and third stage boot loaders. Here's what we've got set up there:

matt# ll /usr/local/export/pxe/boot

-r-xr-xr-x  1 root  wheel     512 Dec  4 15:47 boot1
-r-xr-xr-x  1 root  wheel    7680 Dec  4 15:47 boot2
-r-xr-xr-x  1 root  wheel  163840 Dec  4 15:48 loader
-rw-r--r--  1 root  wheel     504 Jan 27 21:42 loader.rc
-rw-r--r--  1 root  wheel     105 Jan 11 11:30 loader.rc-freebsd-ide
-rw-r--r--  1 root  wheel     105 Jan 27 21:41 loader.rc-freebsd-mailserver
-rw-r--r--  1 root  wheel      93 Jan 11 11:30 loader.rc-freebsd-mylex
-rw-r--r--  1 root  wheel     101 Jan 11 11:30 loader.rc-freebsd-scsi

You can get these files in a couple ways. You can either snag them off the mfsroot.flp (follow the instructions on Alfred Perlstein's page) or just copy them from your source tree (like I did):

# cd /sys/boot
# mkdir -p /usr/local/export/pxe/boot
# cp i386/loader/loader /usr/local/export/pxe/boot
# cp i386/boot2/boot1 /usr/local/export/pxe/boot
# cp i386/boot2/boot2 /usr/local/export/pxe/boot

You'll need to create the contents of the loader.rc file to look something like this:

# more /usr/local/export/pxe/boot/loader.rc
echo Loading Kernel...
load /kernel
load linux.ko
set choice=freebsd-ide
echo
echo You have 5 seconds to select one of the following:
echo
echo freebsd-mylex
echo freebsd-scsi
echo freebsd-ide
echo freebsd-mailserver
echo
read -t 5 -p "Type in your selection EXACTLY: " choice
echo
include /boot/loader.rc-$choice
echo booting...
echo \007\007
echo initializing h0h0magic...
set vfs.root.mountfrom=ufs:/dev/md0c"
#set console="comconsole" #(very useful if you don't have a kvm on the box)
boot

I also wanted to have the ability to select which mfsroot I wanted to boot off so I figured out how to script the loader process a little bit. You'll notice that the loader.rc calls another file (ex. /boot/loader.rc-freebsd-ide). Here's all that file looks like:

# more loader.rc-freebsd-ide
echo Loading FreeBSD 4.2 installer mfsroot for IDE/UDMA drives...
load -t mfs_root /mfsroot-freebsd-ide

.

Requirement 8: NFS Server. Since we're going to use pxeboot's default retrieval method of NFS, we'd better export the /usr/local/export/pxe directory. Once most systems it's as easy as putting a line in /etc/exports that looks like this: "/usr -alldirs -maproot=root -ro". That exports the ent