Tag: freebsd

FreeBSD portsnap vs git clone

FreeBSD is planning to deprecate portsnap in favor of git or svnlite repos. As a heavy git user, part of me thinks, “about time!” There’s a number of discussions about the change on the email lists and the FreeBSD forums. What I wanted is some data on how the change will affect daily use. Should I use a deep or shallow clone? Which git repo should I follow? So I ran a few tests:

portsnapgit clone (freebsd)git clone (github)git clone (freebsd, shallow)
disk used (M)8011,9602,940888
compressratio2.231.221.111.83
checkout time (real)3m 36s41m 49s7m 11s3m 12s
Disk Usage and Time Required to Download

This is a FreeBSD 13.2 host with /usr/ports on a ZFS filesystem with lz4 compression. 

# zfs get compression zroot/usr/ports
zroot/usr/ports  compression  lz4

# rm -rf /usr/ports/* && rm -rf /usr/ports/.* && rm -rf /var/db/portsnap/*
# time portsnap fetch extract
Looking up portsnap.FreeBSD.org mirrors... 5 mirrors found.
Fetching public key from dualstack.aws.portsnap.freebsd.org... done.
Fetching snapshot tag from dualstack.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Sun May  7 17:30:05 PDT 2023:
aa65708d65765ca77e1756616d249f3512ebb192c1ef16         102 MB 8131 kBps    13s
Extracting snapshot... done.
Verifying snapshot integrity... done.
Fetching snapshot tag from dualstack.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Sun May  7 17:30:05 PDT 2023 to Sun May  7 18:18:52 PDT 2023.
Fetching 5 metadata patches... done.
Applying metadata patches... done.
Fetching 0 metadata files... done.
<snip thousands of lines>

# zfs list zroot/usr/ports && zfs get compressratio zroot/usr/ports
NAME              USED  AVAIL     REFER  MOUNTPOINT
zroot/usr/ports   801M  6.96G      813M  /usr/ports
zroot/usr/ports  compressratio  2.23x

# rm -rf /usr/ports/* && rm -rf /usr/ports/.*
# time git clone https://git.freebsd.org/ports.git /usr/ports
Cloning into '/usr/ports'...
remote: Enumerating objects: 5821903, done.
remote: Counting objects: 100% (942/942), done.
remote: Compressing objects: 100% (126/126), done.
remote: Total 5821903 (delta 923), reused 816 (delta 816), pack-reused 5820961
Receiving objects: 100% (5821903/5821903), 1.06 GiB | 488.00 KiB/s, done.
Resolving deltas: 100% (3512457/3512457), done.
Updating files: 100% (157219/157219), done.

real    41m48.997s
user    6m47.535s
sys 1m8.458s

# zfs list zroot/usr/ports && zfs get compressratio zroot/usr/ports
zroot/usr/ports  1.96G  5.80G     1.96G  /usr/ports
zroot/usr/ports  compressratio  1.22x

FreeBSD, Compact Flash, ZFS, and minimum root partition size

The day I booted a FreeBSD system off Compact Flash I was hooked. CF is an extremely robust storage medium with no moving parts. CF cards have emerged completely intact from washing machines, clothes dryers, and impacts that would destroy any spinning disk. After setting up a system to boot from CF, I am confident that henceforth and forevermore, that system will have a functional boot disk.

I’ve stuck CF cards and USB thumb drives into servers in our data centers, our server room at the office, and my server closet. The practice has served me quite well but that is not to say that CF is perfect. Write speed is slow. There is a finite number of write cycles each block can endure. Some CF cards claim DMA support but don’t support it well enough to be useful. Some server boards do not include internal IDE or USB ports. But everywhere else, we use CF.

Because of CF write limits, I always mount the root partition read-only. Files on the / partition are not frequently altered so this rarely causes any inconvenience. We recently built a 6.7 terabyte storage array at work using a HP 320S chassis, a pile of disks, and ZFS. ZFS volumes aren’t bootable in FreeBSD but we had already installed a USB thumb drive as the boot partition.

After working with ZFS, I decided that gmirror was no longer sufficient for my personal file server. It needed ZFS, which meant upgrading to FreeBSD 7. This server has been running off a 256MB CF card for years. The CF card is so old it was actually made in the USA! While upgrading to 7.0 I ran into a snag, the FreeBSD kernel (and modules) now use over 100MB. That means 256MB is no longer enough space for the new kernel and the old one to both fit.