by: Matt Simerson
Sunday 10 Dec 17

Booting FreeBSD via PXE (Preboot Execution Environment)

Version: (v1.0)
Author: Matt Simerson
Publish Date: Dec 5, 2000.

Audience: System Administrators

Objective: Document the steps necessary to boot and/or install a FreeBSD system using a DHCP, NFS, & TFTP server.

Background: FreeBSD can boot and install off a variety of mediums. The common and most useful are floppies, cd-roms, and the network (ppp, nfs, & ftp). All the methods work essentially the same, you bootstrap the helpless machine off floppies (yes, even the CD boot uses floppy emulation) and the mini-FreeBSD system then has enough smarts to initialize CD-ROM drives, Ethernet, and PPP interfaces. This all works fairly good for installing FreeBSD on one machine but what happens when you want to install it on 20, or 50, or 1,000 machines? Right now the process is pretty much a manual one on every machine. This is expensive, error prone, and slow.

Excuse: I had quite a few reasons to embark upon this project.

1. It's a Royal PITA [TM] to build a custom FreeBSD install CD. I literally spent days fuddying with cramming all the stuff I wanted onto a 2.88M disk image and burning a lot of coasters getting it perfected. The process of building bootable media isn't documented well and worse yet, the only real documentation is found by searching mailing lists. This is less than convenient as I found myself compiling lots of post-it notes.

2. Even when I successfully built a bootable CD I was severely limited because I was stuck in the confines of 2.88M. You must wrestle with picobsd and the unless you're a good programmer (I'm not) you're pretty much stuck living with the choices that were made by others for their obtuse reasons. No matter how hard I tried, I just couldn't have as much fun as I'd like to when booting off a CD.

3. Building a restore CD (to boot a failed server) that boots, rebuilds, and restores an entire system is not easy. I needed a simpler method of getting a machine bootstrapped than spending hours building a custom boot CD that would let a clue deficient NOC monkey restore my server.

4. The only time the CDROM drive gets used is when I'm installing the OS. It's quite a waste to buy servers with Cd-ROM's when you're only going to use them once. You can do a lot more with 1U and 2U hardware when you've got another drive bay to play with.

5. In my spare time I'm building a cluster of FreeBSD machines. It's primary purpose thus far is the conversion of electricity to heat in my garage. I figure at some point I'll find a use for it but until then, just building it has provided me with ample motivation to learn a lot of cool new stuff about FreeBSD. Anyway, I buy Pentium 133 systems (complete with NIC cards) for $35 but without Cd-ROM's. It's very slow to get FreeBSD installed so I needed a better way.

At BSDCon 2000 I sat in on a panel taught by Doug White on automated system installations. It gave me the imputes to charge forth and conquer the beast known as PXE.

I'm too lazy to look up some of the facts (again) so anywhere I use <>, please feel free to send me the relevant information and URL's where I can verify and I'll update my documentation.

Some time ago <date please> Intel developed a technology known as PXE and began blessing their wonderful EtherExpress Pro Server Adapters with this spiffy new feature. <Some 3Com NIC cards> also include PXE technology. PXE is designed to allow a NIC card to fetch a configuration from a DHCP server and boot up a computer via it's network interface. <Some smart folks> at FreeBSD saw the usefulness of this feature and wrote a little boot loader appropriately named pxeboot. Pxeboot is included in FreeBSD 4.1 and higher.

So, how does it work you ask? Very well, I must say. Once you satisfy it's many dependencies, things work great. Getting everything satisfied wasn't as simple as I'd hoped. Anyway, follow the steps below to reach enlightenment.

Requirement 1: FreeBSD distribution. It would be nice for FreeBSD to just magically appear on your hard drive but alas, we've got to get it from somewhere. I suppose you could fetch it from the FTP server but I simply copied the CD contents to a NFS exported file system on my server.

# mount /cdrom
# mkdir -p /usr/local/export/freebsd4.2
# rsync -avz /cdrom/ /usr/local/export/freebsd4.2
# ln -s /usr/local/export/freebsd4.2 /usr/local/export/freebsd

Requirement 2: Ethernet adapter with PXE boot roms. The Intel Management adapters all include PXE but even our newest batches required flash updating to get FreeBSD booted properly. This is what my oldest Intel's looked like at first:

Intel UNDI, PXE-2.0 (build 067)
Copyright (C) 1997-1998 Intel Corporation

This version of PXE bios does not work. It will load the pxeboot loader but fails soon thereafter. A visit to Intel's web site had me downloading a file named Within that archive was three very useful things. The first was a directory name 8255x containing the boot ROM's for the Intel 8255x adapters. Logical huh? :-) The other useful tool was the fboot.exe program. I created a DOS boot floppy and copied these programs onto the boot floppy. I then installed five Intel adapters into the PCI slots of my server and proceeded to update them all.

After installing the new firmware the PXE bios looks like this:

Intel (R) Boot Agent Version 4.0.12
PXE 2.0 Build 082 (Wfm 2.0), RPC v2.7.3
Press Ctrl+S to enter the Setup Menu

Requirement 3: DHCP Server. I already had ISC-DHCP 3.0b installed so I merely had to add a couple lines to my DHCP configuration. Here's what a working configuration looks like:

option broadcast-address;
option domain-name-servers;
option domain-name "";
option routers;
option subnet-mask;
server-name "DHCPserver";
default-lease-time -1;

subnet netmask {
option root-path "/usr/local/export/pxe";
filename "pxeboot";
host {
hardware ethernet 00:e0:18:98:f0:cc;
host {
hardware ethernet 00:60:97:0e:bb:a7;

Requirement 4: DNS server. I'm not sure it's its necessary but I'm sure it's at least a good idea. Create an entry in your DNS records for and records for all the addresses in your DHCP pool. It's saves us a lot of time waiting for DNS timeouts when we've just got DNS set up correctly.

Requirement 5: TFTP server. The DHCP configuration tells the booting client that it's supposed to grab the filename "pxeboot" from the tftp server (next-server) at Here's how mine is set up:

# grep tftp /etc/inetd.conf
tftp dgram udp wait nobody /usr/libexec/tftpd tftpd -l /tftpboot

# ll /tftpboot
-rw-r--r-- 1 root wheel 165888 Nov 30 11:46 pxeboot
-rw-r--r-- 1 root wheel 165888 Nov 30 11:47 pxeboot.tftp

This is pretty easy to configure. On most systems, simply comment out the tftp line in your /etc/inetd.conf and restart inetd (killall -HUP inetd).

NOTE: TFTP has virtually no built in security. You should only enable a TFTP server on an internal (trusted) network or use a firewall to restrict access to it. At a minimum, use TCP wrappers.

Requirement 6: PXEBOOT. Copy the pxeboot file from your /usr/src/sys tree to the /tftpboot directory and you're all set:

# cp /sys/boot/i386/pxeldr/pxeboot /tftpboot

Once your machine has loaded the NIC cards PXE bios, it will (assuming it's the chosen boot device) make the tftp request for the file "pxeboot" from the tftp server. The tftp server, being properly configured will hand it the file "pxeboot" which is comparable to the FreeBSD loader program.

NOTE: The pxeboot program can be compiled to fetch the loader via TFTP or NFS. NFS is the default but you can add this "LOADER_TFTP_SUPPORT=YES" to your /etc/make.conf and recompile pxeboot (#cd /usr/src/sys/boot; make clean; make depend; make; cp i386/pxeldr/pxeboot /tftpboot).

Requirement 7: Boot loader. Once PXE boot is loaded it will fetch the files it needs from the /boot directory that's defined within the root-path directive your DHCP server handed it. Since we've defined a root path of /usr/local/export/pxe, it'll be looking within the /boot directory there for the second and third stage boot loaders. Here's what we've got set up there:

matt# ll /usr/local/export/pxe/boot
total 180
-r--r--r-- 1 root wheel 512 Nov 11 16:57 boot1
-r--r--r-- 1 root wheel 7680 Nov 11 16:57 boot2
-r-xr-xr-x 1 root wheel 163840 Nov 11 16:57 loader
-rw-r--r-- 1 root wheel 194 Dec 1 19:51 loader.rc

You can get these files in a couple ways. You can either snag them off the mfsroot.flp (follow the instructions on Alfred Perlstein's page) or just copy them from your source tree (like I did):

# cd /sys/boot
# mkdir -p /usr/local/export/pxe/boot
# cp i386/loader/loader /usr/local/export/pxe/boot
# cp i386/boot2/boot1 /usr/local/export/pxe/boot
# cp i386/boot2/boot2 /usr/local/export/pxe/boot

You'll need to create the contents of the loader.rc file to look like this:

# more /usr/local/export/pxe/boot/loader.rc
echo Loading Kernel...
load /kernel
echo Loading mfsroot...
load -t mfs_root /mfsroot
echo booting...
echo \007\007
echo initializing h0h0magic...
set vfs.root.mountfrom=ufs:/dev/md0c"
#set console="comconsole" #(very useful if you don't have a kvm on the box)

Requirement 8: NFS Server. Since we're going to use pxeboot's default retrieval method of NFS, we'd better export the /usr/local/export/pxe directory. Once most systems it's as easy as putting a line in /etc/exports that looks like this: "/usr -alldirs -maproot=root -ro". That exports the entire /usr file system with read only permissions. Once you've added that line to the exports file, you need to restart (or start) mountd. To set everything up on a FreeBSD system, just do this:

# echo "/usr -alldirs -maproot=root -ro" >> /etc/exports
# if mountd is running "killlall -HUP mountd`"
# if mountd is not running "nfsd -t -u; mountd"

If you want this machine to be a NFS server every time you reboot, add this to your /etc/rc.conf: "nfs_server_enable="YES".

NOTE: NFS security is adequate at best. Use IPFIREWALL or IPFILTER (I use IPFIREWALL) to restrict access to it.

Requirement 9: Bootable Kernel. At this point your new machine has just run the loader program which, as instructed by the loader.rc file, is going to try loading the file named "kernel" from our root path. Lucky for us, we've got a perfectly good kernel floating around in our / directory so we just copy it from there over to our exported directory.