A few years ago I sampled each of the “All My Music In the Cloud” services (iTunes Match, Amazon Cloud, Google Play). For them to stream my music back to all my devices, I first had to first upload all my music (82 GB of data) to each service.
The iTunes Match upload was far smaller because Apple has the worlds largest music library and iTunes Match only uploaded my songs that weren’t already already in their collection. That should have made the upload process quick, except that something about the upload mechanism Apple uses caused severe network congestion and network stalls of 5 full seconds. I blamed it on iTunes and used the built-in IPFW firewall to plumb a 256Kbps pipe so that iTunes Match uploads would stop erring out and I could use my internet connection during the long upload process.
ipfw pipe 1 config bw 256KBytes/s
ipfw add 1 pipe 1 src-port 443
ipfw add 2 pipe 1 dst-port 443
That IPFW solution worked just as well for throttling the other cloud music services.
Fast-forward a couple years to Mac OS 10.10.3 and the new Photos app that stores all my photos in the cloud. There’s a process named photolibraryd and it seems to have that same nasty behavior. The symptoms are identical but I can’t use IPFW because Apple removed it in OS X Yosemite. I understand, as I too stopped using IPFW years ago in favor of PF. But Apple doesn’t provide ALTQ, the PF bandwidth shaper. So the PF firewall has no bandwidth shaping abilities. Or so I thought.
After a bit of hunting, I found the Network Link Conditioner within the Hardware IO Tools for Xcode. Even better, a GUI interface for accomplishing my goal. I downloaded it, set up a 256Kbps upload limit and I could once again let photos upload while I use my internet connection.
By what dark magic has Apple accomplished this task? Inspecting the network interface didn’t turn up anything special so I checked the firewall rules (sudo pfctl -sa) and found dummynet rules! In the PF ruleset! And increasing dummynet packet counters. Hmmmm.
Dummynet is part of IPFW, so apparently rather than implementing ALTQ, Apple decided to modify PF to support dummynet. The man page for pf.conf doesn’t even contain the term ‘dummy’ but I expect that’ll come eventually. In the meantime, the intarwebs can help you find documentation for how to write rules for it.
Hello,
I don’t see any option in the Network Link Conditioner application to slow down only one port or one process.
Sincerely,
Guillaume
There aren’t built-in options for doing such. Just like with IPFW, you have to first create a pipe with dummynet of the desired size. Then you use standard firewall rules to direct the desired traffic (host/port combinations) through the dummynet. In the case of iTunes or photolibraryd, it’s just a matter of creating a rule for all https traffic, or just https traffic destined to Apple/Amazon servers.
Since I’m a complete noob to doing PF, can you give an example of creating the pipe and throttling it using some PF rules?
Bart pointed out on Twitter that Mac OS X has ‘dnctl’, which can be used to manage dummynet pipes and queues.
To see dummynet specific rules with pfctl you have to use the totally obscure and undocumented -d command like “pfctl -sd” to show dummynet rules. On OSX dummynet rules must be placed in a special anchor defined as “dummynet-anchor” in your pf.conf. As of 10.10.3 you have to specify the “in” or “out” parameters for dummynet rules, otherwise you’ll get an error. A good source of examples is “man dnctl”. I’ve just noticed a weird thing: dnctl is in 10.9 source list ad opensource.apple.com, but it is not installed on OSX 10.9. Another good example of how pf/dummynet works is IceFloor on OSX 10.9. On OSX 10.10 it works exactly the same way, you just have to use dnctl instead of ipfw to create pipes. Trick: you can copy the /sbin/ipfw binary from 10.9 to 10.10. Yes, actually Yosemite kernel INCLUDES IPFW đ they have just removed the shell frontend from sbin. I will include bandwidth management for Yosemite (and 10.11) on next releases IceFloor 3 and next major Murus release đ — Hany