Booting FreeBSD via PXE (Preboot Execution Environment)
Version: v1.1 (also available: 1.0)
Publish Date: Dec 5, 2000.
Updated: Jan 6, 2001.
Audience: Unix System Administrators
Objective: Document the steps necessary to boot and/or install a FreeBSD system using a DHCP, NFS, & TFTP server.
Background: FreeBSD can boot and install off a variety of mediums. The common and most useful are floppies, cd-roms, and the network (ppp, NFS, & ftp). All the methods work essentially the same, you bootstrap the helpless machine off floppies (yes, even the CD boot uses floppy emulation) and the mini-FreeBSD system then has enough smarts to initialize CD-ROM drives, Ethernet, and PPP interfaces. This all works fairly good for installing FreeBSD on one machine but what happens when you want to install it on 20, or 50, or 1,000 machines? Right now the process is pretty much a manual one on every machine. This is expensive, error prone, and slow.
Excuse: I had quite a few reasons to embark upon this project.
1. It's a Royal PITA [TM] to build a custom FreeBSD install CD. I literally spent days fuddying with cramming all the stuff I wanted onto a 2.88M disk image and burning a lot of coasters getting it perfected. The process of building bootable media isn't documented well and worse yet, the only real documentation is found by searching mailing lists. This is less than convenient as I found myself compiling lots of post-it notes.
2. Even when I successfully built a bootable CD I was severely limited because I was stuck in the confines of 2.88M. You must wrestle with picobsd and the unless you're a good programmer (I'm not) you're pretty much stuck living with the choices that were made by others for their obtuse reasons. No matter how hard I tried, I just couldn't have as much fun as I'd like to when booting off a CD.
3. Building a restore CD (to boot a failed server) that boots, rebuilds, and restores an entire system is not easy. I needed a simpler method of getting a machine bootstrapped than spending hours building a custom boot CD that would let a clue deficient NOC monkey restore my server.
4. The only time the CDROM drive gets used is when I'm installing the OS. It's quite a waste to buy servers with Cd-Roms when you're only going to use them once. You can do a lot more with 1U and 2U hardware when you've got another drive bay to play with.
5. In my spare time I'm building a cluster of FreeBSD machines. It's primary purpose thus far is the conversion of electricity to heat in my garage. I figure at some point I'll find a use for it but until then, just building it has provided me with ample motivation to learn a lot of cool new stuff about FreeBSD. Anyway, I buy Pentium 133 systems (complete with NIC cards) for $35 but without Cd-Roms. It's very slow to get FreeBSD installed so I needed a better way.
At BSDCon 2000 I sat in on a panel taught by Doug White on automated system installations. It gave me the impetus to charge forth and conquer the beast known as PXE.
I'm too lazy to look up some of the facts (again) so anywhere I use <>, please feel free to send me the relevant information and URL's where I can verify and I'll update my documentation.
Some time ago <date please> Intel developed a technology known as PXE and began blessing their wonderful EtherExpress Pro Server Adapters with this spiffy new feature. <Some 3Com NIC cards> also include PXE technology. PXE is designed to allow a NIC card to fetch a configuration from a DHCP server and boot up a computer via it's network interface. John Baldwin and Paul Saab at FreeBSD saw the usefulness of this feature and wrote a little boot loader appropriately named pxeboot. Pxeboot is included in FreeBSD 4.1 and higher.
So, how does it work you ask? Very well, I must say. Once you satisfy it's many dependencies, things work great. Getting everything satisfied wasn't as simple as I'd hoped. Anyway, follow the steps below to reach enlightenment.
Requirement 1: FreeBSD distribution. It would be nice for FreeBSD to just magically appear on your hard drive but alas, we've got to get it from somewhere. I suppose you could fetch it from the FTP server but I simply copied the CD contents to a NFS exported file system on my server.
# mount /cdrom
# mkdir -p /usr/local/export/freebsd4.2
# rsync -avz /cdrom/ /usr/local/export/freebsd4.2
# ln -s /usr/local/export/freebsd4.2 /usr/local/export/freebsd
Requirement 2: Ethernet adapter with PXE boot roms. The Intel Management adapters all include PXE but even our newest batches required flash updating to get FreeBSD booted properly. This is what my oldest Intel's looked like at first:
Intel UNDI, PXE-2.0 (build 067)
Copyright (C) 1997-1998 Intel Corporation
This version of PXE bios does not work. It will load the pxeboot loader but fails soon thereafter. A visit to Intel's web site had me downloading a file named 8255x.zip. Within that archive was three very useful things. The first was a directory name 8255x containing the boot ROM's for the Intel 8255x adapters. Logical huh? :-) The other useful tool was the fboot.exe program. I created a DOS boot floppy and copied these programs onto the boot floppy. I then installed five Intel adapters into the PCI slots of my server and proceeded to update them all.
After installing the new firmware the PXE bios looks like this:
Intel (R) Boot Agent Version 4.0.12
PXE 2.0 Build 082 (Wfm 2.0), RPC v2.7.3
Press Ctrl+S to enter the Setup Menu
Requirement 3: DHCP Server. I already had ISC-DHCP 3.0b installed so I merely had to add a couple lines to my DHCP configuration. Here's what a working configuration looks like:
option broadcast-address 192.168.254.255;
option domain-name-servers 192.168.254.3;
option domain-name "simerson.net";
option routers 192.168.254.1;
option subnet-mask 255.255.255.0;
server-name "DHCPserver";
server-identifier 192.168.254.3;
next-server 192.168.254.3;
default-lease-time -1;
subnet 192.168.254.0 netmask 255.255.255.0 {
range 192.168.254.32 192.168.254.99;
option root-path "/usr/local/export/pxe";
filename "pxeboot";
}
host cm.simerson.net {
hardware ethernet 00:e0:18:98:f0:cc;
fixed-address 192.168.254.126;
}
host c1.simerson.net {
hardware ethernet 00:60:97:0e:bb:a7;
fixed-address 192.168.254.131;
}
Requirement 4: DNS server. I'm not sure it's its necessary but I'm sure it's at least a good idea. Create an entry in your DNS records for dhcpserver.yourdomain.com and records for all the addresses in your DHCP pool. It's saves us a lot of time waiting for DNS timeouts when we've just got DNS set up correctly.
Requirement 5: TFTP server. The DHCP configuration tells the booting client that it's supposed to grab the filename "pxeboot" from the TFTP server (next-server) at 192.168.254.3. Here's how mine is set up:
# grep tftp /etc/inetd.conf
tftp dgram udp wait nobody /usr/libexec/tftpd tftpd -l /tftpboot
# ll /tftpboot
-rw-r--r-- 1 root wheel 165888 Nov 30 11:46 pxeboot
This is pretty easy to configure. On most systems, simply comment out the tftp line in your /etc/inetd.conf and restart inetd (killall -HUP inetd).
NOTE: TFTP has virtually no built in security. You should only enable a TFTP server on an internal (trusted) network or use a firewall to restrict access to it. At a minimum, use TCP wrappers.
Requirement 6: PXEBOOT. Copy the pxeboot file from your /usr/src/sys tree to the /tftpboot directory and you're all set:
# cp /sys/boot/i386/pxeldr/pxeboot /tftpboot
Once your machine has loaded the NIC cards PXE bios, it will (assuming it's the chosen boot device) make the tftp request for the file "pxeboot" from the tftp server. The tftp server, being properly configured will hand it the file "pxeboot" which is comparable to the FreeBSD loader program.
NOTE: The pxeboot program can be compiled to fetch the loader via TFTP or NFS. NFS is the default but you can add this "LOADER_TFTP_SUPPORT=YES" to your /etc/make.conf and recompile pxeboot (#cd /usr/src/sys/boot; make clean; make depend; make; cp i386/pxeldr/pxeboot /tftpboot).
Requirement 7: Boot loader. Once PXE boot is loaded it will fetch the files it needs from the /boot directory that's defined within the root-path directive your DHCP server handed it. Since we've defined a root path of /usr/local/export/pxe, it'll be looking within the /boot directory there for the second and third stage boot loaders. Here's what we've got set up there:
matt# ll /usr/local/export/pxe/boot
-r-xr-xr-x 1 root wheel 512 Dec 4 15:47 boot1 -r-xr-xr-x 1 root wheel 7680 Dec 4 15:47 boot2 -r-xr-xr-x 1 root wheel 163840 Dec 4 15:48 loader -rw-r--r-- 1 root wheel 504 Jan 27 21:42 loader.rc -rw-r--r-- 1 root wheel 105 Jan 11 11:30 loader.rc-freebsd-ide -rw-r--r-- 1 root wheel 105 Jan 27 21:41 loader.rc-freebsd-mailserver -rw-r--r-- 1 root wheel 93 Jan 11 11:30 loader.rc-freebsd-mylex -rw-r--r-- 1 root wheel 101 Jan 11 11:30 loader.rc-freebsd-scsi
You can get these files in a couple ways. You can either snag them off the mfsroot.flp (follow the instructions on Alfred Perlstein's page) or just copy them from your source tree (like I did):
# cd /sys/boot
# mkdir -p /usr/local/export/pxe/boot
# cp i386/loader/loader /usr/local/export/pxe/boot
# cp i386/boot2/boot1 /usr/local/export/pxe/boot
# cp i386/boot2/boot2 /usr/local/export/pxe/boot
You'll need to create the contents of the loader.rc file to look something like this:
# more /usr/local/export/pxe/boot/loader.rc
echo Loading Kernel...
load /kernel
load linux.ko
set choice=freebsd-ide
echo
echo You have 5 seconds to select one of the following:
echo
echo freebsd-mylex
echo freebsd-scsi
echo freebsd-ide
echo freebsd-mailserver
echo
read -t 5 -p "Type in your selection EXACTLY: " choice
echo
include /boot/loader.rc-$choice
echo booting...
echo \007\007
echo initializing h0h0magic...
set vfs.root.mountfrom=ufs:/dev/md0c"
#set console="comconsole" #(very useful if you don't have a kvm on the box)
boot
I also wanted to have the ability to select which mfsroot I wanted to boot off so I figured out how to script the loader process a little bit. You'll notice that the loader.rc calls another file (ex. /boot/loader.rc-freebsd-ide). Here's all that file looks like:
# more loader.rc-freebsd-ide
.
echo Loading FreeBSD 4.2 installer mfsroot for IDE/UDMA drives...
load -t mfs_root /mfsroot-freebsd-ide
Requirement 8: NFS Server. Since we're going to use pxeboot's default retrieval method of NFS, we'd better export the /usr/local/export/pxe directory. Once most systems it's as easy as putting a line in /etc/exports that looks like this: "/usr -alldirs -maproot=root -ro". That exports the entire /usr file system with read only permissions. Once you've added that line to the exports file, you need to restart (or start) mountd. To set everything up on a FreeBSD system, just do this:
# echo "/usr -alldirs -maproot=root -ro" >> /etc/exports
# if mountd is running "killlall -HUP mountd`"
# if mountd is not running "nfsd -t -u; mountd"
If you want this machine to be a NFS server every time you reboot, add this to your /etc/rc.conf: "nfs_server_enable="YES".
NOTE: NFS security is adequate at best. Use IPFIREWALL or IPFILTER (I use IPFIREWALL) to restrict access to it.
Requirement 9: Bootable Kernel. At this point your new machine has just run the loader program which, as instructed by the loader.rc file, is going to try loading the file named "kernel" from our root path. Lucky for us, we've got a perfectly good kernel floating around in our / directory so we just copy it from there over to our exported directory.
# cp /kernel /usr/local/export/pxe/
There are some requirements for this kernel, most specifically the MFS support but if you just use a GENERIC kernel, things will work out just fine for you. The reason you don't want to use the kernel off the boot.flp or kern.flp images is that they pause at the kernel configuration screen. Using a custom kernel also gives you the flexibility to load klm's (kernel loadable modules) for your custom install packages if you need to (I did).
Once the kernel loads we'll proceed to the next line of the loader.rc file and that's fetching the mfsroot file. The mfsroot is just that, a memory based file system that's stored in a file.
Requirement 10: MFSROOT. You have a couple options here. The first is to simply use the mfsroot disk image that comes with the version of FreeBSD you are installing. There are instructions for doing so on Alfred Perlstein's page. If your needs are modest and can be met with the tools provided on the mfsroot floppy then have at it. However, that defeats half the purpose of doing this exercise in the first place.
The real power behind this is that you can create your own mfsroot with no size restrictions. Weehee, let's make a really big 25 megabyte mfsroot:
# cd /usr/local/export/pxe
# dd if=/dev/zero of=mfsroot bs=1k count=25000
# vnconfig -e -s labels vn0 mfsroot
# disklabel -r -w vn0 auto
# newfs /dev/vn0c
# mkdir -p /mnt/mfs
# mount /dev/vn0 /mnt/mfs
There, now we've got 25 megs of space mounted on /mnt to play with. The basic rule of thumb here to keep in mind is that you are building a root file system for FreeBSD. Everything the kernel, sysinstall, and your install programs needs from the root file system is what you need to have in your /mnt directory. To get started we'll copy the contents of the distribution mfsroot.flp to our new mfsroot:
# mkdir /mnt/floppy
# vnconfig vn1 /usr/local/export/freebsd4.2/floppies/mfsroot.flp
# mount /dev/vn1 /mnt/floppy
# cp /mnt/floppy/mfsroot.gz /tmp
# umount /mnt/floppy
# vnconfig -u vn1
# gunzip /tmp/mfsroot.gz
# vnconfig vn1 /tmp/mfsroot
# mount /dev/vn1 /mnt/floppy
# rsync -avzH /mnt/floppy/ /mnt/mfs
# umount /mnt/floppy
# vnconfig -u vn1
Voila, now we've got a bootable mfsroot partition and tons of space to play with. One thing to keep in mind, when you place any of your spiffy utilities on there, make sure you compile them statically. This is usually as easy as changing into the appropriate source directory, editing the Makefile and adding "-static" to the CFLAGS. You've got plenty of room to hack up your own sysinstall or anything else that you want/need on your root partition.
At this point, you can simple umount the mfsroot (umount /mnt/mfs; vnconfig -u vn0) or build it up the way you want/need. We have some pretty ugly kludges we've made which including hacking sysinstall and a few other goodies that use about 20 megs. I'm sure you'll find some creative ways to use all that space. Just drop in your statically compiled binary and run with it. My system looks like this at boot time:
Intel (R) Boot Agent Version 4.0.12
PXE 2.0 Build 082 (Wfm 2.0), RPC v2.7.3
Press Ctrl+S to enter the Setup MenuDHCP MAC ADDR: 00:
CLIENT ID: 192.168.254.133 MASK 255.255.255.0 DHCP IP: 192.168.254.3
GATEWAY IP: 192.168.254.1
PXE Loader 1.00Building the boot loader arguments
Relocating the loader and the BTX
Starting the BTX loaderBTX loader 1.00 BTX Version 1.01
Console: internal video/keyboard
BIOS drive A: is disk0PXE Version 2.1, real mode entry point @9db3:0106
BIOS 639kB/392180kB available memoryFreeBSD/i386 bootstrap loader, Revision 0.8
(root@matt.simerson.net), Thu Nov 30 11:45:41 PST 2000)
pxe_open: server addr: 192.168.254.3
pxe_open: server path: /usr/local/export/pxe
pxe_open: gateway ip: 192.168.254.1
Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [kernel]
Requirement 11: Install.cfg. If you're installing FreeBSD via the Ethernet adapter, (as opposed to just using PXE to netboot) then you'll want to script sysinstall. Why go though all this bother if you have to sit in front of the box and answer questions? Anyway, now our loader.rc has requested the mfsroot file and pulled our 25 meg image across it's 100Megabit interface. The loader.rc also tells the kernel to get it's root file system from memory disk 0 (md0c) that we loaded. The kernel then had it's root partition set up based on the contents of our mfsroot partition. Once it checks out our hardware it looks for /sbin/init which doesn't exist and then falls back to running sysinstall.
Sysinstall checks to see if the file /install.cfg exists and if so, uses it to control it's behavior. The script syntax is documented in "man sysinstall" which you might need to install (cd /usr/src/release/sysinstall; make install). There's a sample install.cfg installed at: /usr/src/release/sysinstall/install.cfg and here's what my install.cfg looks like:
nonInteractive=YES
noWarn=YES
tryDHCP=YES
################################
# My host specific data
hostname=c3
domainname=simerson.net
################################
# Which installation device to use
nfs=192.168.254.3:/usr/local/export/freebsd
netDev=fxp0
tryDHCP=YES
mediaSetNFS
################################
# Select which distributions we want.
dists= bin doc manpages catpages proflibs dict info des compat22 compat3x crypto ports
distSetCustom
################################
# Now set the parameters for the partition editor.
disk=ad0
partition=all
bootManager=standard
diskPartitionEditor
################################
# All sizes are expressed in 512 byte blocks!
#
ad0s1-1=ufs 256000 /
ad0s1-2=swap 256000 none
ad0s1-3=ufs 256000 /var
ad0s1-3=ufs 0 /usr
diskLabelEditor
installCommit
package=rsync-2.4.6
packageAdd
package=ucd-snmp-4.1.2
packageAdd
package=sudo-1.6.3.5
packageAdd
package=gmake-3.79.1
packageAdd
package=libtool-1.3.4_1
packageAdd
package=gdbm-1.8.0
packageAdd
package=cvsup-bin-16.1
packageAdd
package=mypkg-1.0
packageAdd
system /stand/my.custom.installer
shutdown