TLS management

Let’s Encrypt, TLS certificates, and HAproxy

I’m evolving. As always, the change is being driven by the most pernicious of motivators: pain. I’ve sold, installed, and upgraded SSL/TLS certificates for years. It’s always been mildly painful: I maintain an offline CA where I generate all the keys and CSR (certificate requests). Then I submit the CSRs to whichever Certificate Authority / Reseller has the best current pricing, get back the new signed certificate from the CA, archive it, and finally install the key, crt, and CA chain file at the destination.

It can be painful and annoying enough that clients regularly hire me to install their certificates for them. To reduce the pain, I’ve encouraged long-duration (3+) year certs. I also have custom scripts tailored to my private CA to reduce the keystrokes. Even so, managing a few dozen certificates was onerous. It didn’t help that every application / daemon (apache, nginx, lighttpd, haproxy, dovecot, qmail, postfix, haraka, etc.) has their own special syntax and sometimes format for configuring the TLS certificates.

Two things happened in 2016 that made TLS management not suck:

  1. The Internet Security Research Group released Let’s Encrypt(https://letsencrypt.org). It’s a free and highly automated Certificate Authority that validates domain ownership (via DNS or HTTP) and issues certificates in seconds.
  2. I’ve moved all my web servers behind HAproxy. Now all TLS certs for web servers get deployed to haproxy and the job is done. No messing with lighttpd, apache, or nginx configs. Configure HAproxy get to an A+ at SSLLabs and it covers all the web servers.

Let’s Encrypt provides free signed certificates in just a few seconds, so long as one is willing to invest the time and energy into automating it. I’ve settled on [acme.sh](https://github.com/Neilpang/acme.sh) as my preferred client and once I’ve generated a certificate, it automatically renews and re-deploys it when needed. Just right.

HAproxy now does all the TLS termination, URL routing, scheme upgrades (http -> https), and rewrites. This greatly simplifies the backend web server configs. Need mod_perl, use Apache. Need CGI support, use lighttpd. For everything else I use nginx. Now all of them are simpler to deploy and upgrade.

heat pump water heater

In July I purchased a GE Geospring ($700 at Lowes in Seattle) 50 gallon heat pump water heater. I installed it myself in the basement. It’s wired the same as a typical electric water heater, so I just ran a new circuit of 10 gage wire and hooked it up.

Heat pump water heaters make more noise than traditional water heaters. If I happen to walk by the open door to the basement, I can hear it but I don’t consider it “loud.” It makes a little less noise than a dehumidifier, a lot less noise than an old dishwasher, but a fair bit more noise than my new ultra-quietest-one-available dishwasher. I’d guess in the neighborhood of 65 decibels.

Heat pump water heaters cool the area they’re in. I consider that a feature, as the basement is our “cool dry” storage area. Despite the output of cool air, the basement was about 64° before I put the heat pump water heater in and it’s still usually 64° after. That’s because the concrete floor and walls have lots of thermal mass so it takes a LOT of input to change the temps significantly.

A heat pump also dehumidifies the air. It has a condensate drain where the water obtained is drained off. Over the course of a week, the condensate measured about a quart for our family of four. Not huge, not “replaces a dehumidifier,” but welcome never-the-less.

The install docs recommend installing it in a garage or basement and I agree. You could put it in a large closet or pantry, but you’d want to have insulated doors if it’s adjacent to a “relaxing” area of the house.

Thus far, I’m very fond of my heat pump water heater.

nginx and cronolog

Since the last century, I’ve been in the habit of piping my web server log files through cronolog and off to automatically selected files in the pattern /var/log/http/2015/10/23/access.log. This works quite well for me because way back when, I wrote a little log processing script called Logmonster.

After all these years, Logmonster still runs a while after midnight (via periodic) and:

  • parses the web server logs by date and vhost
  • feeds them through Awstats
  • compresses them

Back when Logmonster was named Apache::Logmonster, it required installing cronolog and making a few small changes to httpd.conf:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %v" logmonster
CustomLog "| /usr/local/sbin/cronolog /var/log/http/%Y/%m/%d/access.log" logmonster
ErrorLog "| /usr/local/sbin/cronolog /var/log/http/%Y/%m/%d/error.log"

Years later, after I got tired of maintaining Apache, lighttpd was all shiny and new and it was similarly easy to configure, making these changes to lighttpd.conf:

accesslog.format = "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %v"
accesslog.filename = "|/usr/local/sbin/cronolog /var/log/http/%Y/%m/%d/access.log"
server.errorlog = "/var/log/http/error.log"

Now, after spending more time than I wanted to determining why lighttpd and haproxy stopped playing nice together (Most HTTP POST commands time out. No good reason why. Remove haproxy, works fine. Replace lighttpd with nginx behind haproxy and it works fine.) so I replaced lighttpd with nginx. That required figuring out how to get cronolog type logging to work in nginx.

Nearly all my cronolog+nginx search returned only instructions for setting up logging to a FIFO, which I thought was a nifty idea. So I created the FIFOs, configured nginx, and upon startup, nginx just hangs. No idea why. It’s also requires setting up the FIFOs before nginx could start up, so I didn’t love that idea. Then I found instructions showing how to configure log rotation within nginx.conf. That’s exactly what I was looking for.

This is my solution for timestamp based logging with nginx:

log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$server_name"';
if ($time_iso8601 ~ "^(?\d{4})-(?\d{2})-(?\d{2})") {}
access_log /var/log/http/$year/$month/$day/access.log main;

Apple data centers on 100% renewable power

Apple is spending an eye popping $850 million to build a ginormous solar farm (280 megawatts) that will power their entire California operations. This new solar farm is not to be confused with the 70MW solar farm they’re building in Arizona, the $55 million “under way” third solar farm (17.5MW) in North Carolina, the two 20MW solar farms they’re building in China, or the existing 20MW solar farm near Reno, NV, or the two existing 20MW solar farms in N. Carolina.

The backstory is that in 2010, Apple wanted to buy renewable energy from Duke to power their Maiden N.C. data center. It wasn’t even legal in N. Carolina. In 2011 Apple bypassed the N.C. coal lobby by purchasing 100 acres of land and in 2012 they finished building (est. $100 million) the first non-utility 20MW solar farm. At the same time, they also built a 5MW fuel cell farm. In 2013 they doubled their fuel cell farm to 10MW and built another 20MW solar farm. Apple has since been producing 100% of the power they need in N.C.

While I believe that Tim Cook is sincere about reducing Apple’s carbon footprint, I also think it’s likely that spending over a billion dollars on solar panels is a very good investment. Apple is famously cash rich and by spending today and owning the solar farms, Apple fixes their energy prices at today’s rates for the next 30 years. Apple has taken a large and variable cost and turned it into a fixed cost that is no longer subject to price inflation or fluctuation. What Apple is also purchasing is energy stability.

Apple is also becoming an energy supplier. For the first 10 years, PG&E will purchase 150MW of production and Apple gets 130MW.  In the last 20 years, Apple gets 100% of production. It’s likely that their operations will have expanded to utilize the power (as has the NC data center) but if not, they’ll have little trouble selling their surplus capacity.

While Apple was first in the, “okay then, we’ll build it ourselves” solar game, the even bigger story is that 2014 was the year solar arrived in Main Street USA. In just 2014, nearly 70% of the worlds solar power generation came online with several companies having more installed solar than Apple: Wal-Mart (105MW), Kohl’s (50MW), and Costco (48MW). IKEA is not far behind with 39MW. Apple isn’t even the largest purchaser of solar as Intel, Kohl’s, Whole Foods, Dell, and Johnson & Johnson all purchase more solar power than Apple. What was so special about solar in 2014?

Swanson’s Law observes that solar modules tend to drop in price by 20% for every doubling of cumulative shipped volume. Apple deployed 60MW between 2012-2014 and during that same time, photo voltaic capacity more than doubled. By being out in front and building not just demand, but also solar capacity, Apple helped 2014 be the year of solar grid parity in 3 NE states, California, Arizona, and Hawaii. It is predicted that grid parity will arrive in “many” US markets in 2015 and Deutsche Bank predicts solar grid parity for all 50 states in 2016. With Apple deploying another 407MW of solar In just 2015-2016, that prediction seems like slam dunk.

Bandwidth shaping on Mac OS X

A few  years ago I sampled each of the “All My Music In the Cloud” services (iTunes Match, Amazon  Cloud, Google Play). For them to stream my music back to all my devices, I first had to first upload all my music (82 GB of data) to each service.

The iTunes Match upload was far smaller because Apple has the worlds largest music library and iTunes Match only uploaded my songs that weren’t already already in their collection. That should have made the upload process quick, except that something about the upload mechanism Apple uses caused severe network congestion and network stalls of 5 full seconds. I blamed it on iTunes and used the built-in IPFW firewall to plumb a 256Kbps pipe so that iTunes Match uploads would stop erring out and I could use my internet connection during the long upload process.

ipfw pipe 1 config bw 256KBytes/s
ipfw add 1 pipe 1 src-port 443
ipfw add 2 pipe 1 dst-port 443

That IPFW solution worked just as well for throttling the other cloud music services.

Fast-forward a couple years to Mac OS 10.10.3 and the new Photos app that stores all my photos in the cloud. There’s a process named photolibraryd and it seems to have that same nasty behavior. The symptoms are identical but I can’t use IPFW because Apple removed it in OS X Yosemite. I understand, as I too stopped using IPFW years ago in favor of PF.  But Apple doesn’t provide ALTQ, the PF bandwidth shaper. So the PF firewall has no bandwidth shaping abilities. Or so I  thought.

After a bit of hunting, I found the Network Link Conditioner within the Hardware IO Tools for Xcode. Even better, a GUI interface for accomplishing my goal. I downloaded it, set up a 256Kbps upload limit and I could once again let photos upload while I use my internet connection.

By what dark magic has Apple accomplished this task?  Inspecting the network interface didn’t turn up anything special so I checked the firewall rules (sudo pfctl -sa) and found dummynet rules! In the PF ruleset! And increasing dummynet packet counters. Hmmmm.

Dummynet is part of IPFW, so apparently rather than implementing ALTQ,  Apple decided to modify PF to support dummynet. The man page for pf.conf doesn’t even contain the term ‘dummy’ but I expect that’ll come eventually. In the meantime, the intarwebs can help you find documentation for how to write rules for it.

An auspicious start to DOS programming

In 1992  I was a young geek of 19 years. My programming experience consisted of the BASIC programs in the manual that came with our Commodore 64 and a few others in our schools Apple II lab. I had also written a few HyperCard and FileMaker apps on the Mac in my bedroom, where I did all the typesetting for my Dad’s print shop. [Thanks so much dad, for buying that first Mac Plus instead of a Compugraphic typesetting machine].

My vocational training in Mechanical Drafting had landed me an entry level position at Kysor/Cadillac as the blueprint clerk. Before long I rearranged the print room to maximize the efficiency of the engineers and myself, leaving me with hours of spare time each day. Often I would roam the engineering department, in search of  engineering projects, much to the delight of the engineers who could often find drudge work to offload.

During one of these lulls, I was chatting with David, a bright young lad who worked in the QA department. David was also quite fond of computers and told me of an escapade in which some students at his school had written a login simulator that captured and stored passwords when users logged into an system infected with their program.

Our engineering files were stored on a Novell Netware server connected by a token ring network. Each DOS computer logged in using a Novell program (login.exe, IIRC). The password capturing program seemed like an interesting challenge so I acquired my first DOS compiler (Qbasic or PowerBasic, I can’t recall which I used for this task) and wrote login.bas. I simulated the login screen perfectly, stored the passwords to a file, and then passed them on to the real login program, logging the user in. It  offered the user no indication that foul play was at hand.

Pleased with my results, I showed Rick, our network admin. I explained that I hadn’t inspected the contents of the file, knew what was in it, and turned my back while he inspected it. It turns out that Rick wasn’t terribly fond of being informed that his network security wasn’t all that secure. A few of his heated words I recall were, “that’s not your job!” He immediately escalated the matter to Keith, our VP of Engineering, intent on having me fired.

On that day, it was quite fortunate for me that I had set a precedent of doing a lot of engineering work that was not my job. Unbeknownst to me, the wheels of my first promotion were already set in motion specifically because of the extra-curricular not my job work I had been doing. That day ended with me getting a stern talking to. Soon thereafter, I was promoted and my new job involved writing software for Kysor.

Motion Sensors

Thanks – very helpful!  I’m curious – what model of motion sensors are you using?  Have you had good luck with them?  What’s the battery life like? Is it practical to put them in pretty much every room of a house?

Every room? That depends on your budget. 🙂

Outside I have Aeon Labs Aeotec Z-Wave MultiSensor: (Motion, Temp, Light level, Humidity) which turns on the lights when motion is detected and it’s dark. I’m not doing anything with the Temp or Humidity yet, but they’re neat to have.

Inside the house, I have Ecolink Z-Wave PIR Motion Detectors. I currently have one in the kitchen, dining room, living room, and front stairwell (split-level). When it’s dark outside (see above) and motion is detected, GE Link bulbs turn on and light that area. I’m going to buy more because my son wants the lights in his bedroom to be smart too.

The motion detectors work fairly well, with limits. They are Passive IR, which is GREAT for battery life, but after tripping they don’t begin watching again for 4 minutes. To prevent going off when someone is sitting idle in the room, I find I can’t “trust” a lack of motion until about 20 minutes after motion stopped. I’ve read that some folks run these detectors in “test” mode, which reduces the 4 minute timeout to 10 seconds, at the cost of shorter battery life. I haven’t tried that yet, but likely will in the dining room. Believe it or not, sometimes my kids actually sit still while doing homework!

I also have a SmartThings SmartSense Motion Sensor that came in the kit with my hub. It usually senses motion and temperature, and after a month’s use six feet from the hub, it’s down to 66% battery. All the Aeon motion detectors are at 100% after a months use. I’m substantially less impressed by this ones reliability (which impacts the WAF) so it got relegated to the garage. I haven’t done any validation of battery life reporting, but there’s a good chance I won’t recharge this guys batteries.

One thing about motion detectors is they don’t detect us until after we come into their field of view. Duh, right? For the motion sensor inside the front door, this means the sensor generally doesn’t “see” us until the door has mostly opened and we’re walking in. During that delay, someone is invariably reaching for the switch at the same time the lights come on. That’s confusing, especially if they flip a 2-way switch and nothing happens. I could put another motion sensor on the other side of the door, but what I like better is…

the Ecolink Z-Wave Door/Window Sensor. I now have one on every exterior door. The second the door starts to open, the lights come on both inside and outside the door (if they weren’t already). The motion sensors are then used as occupancy sensors that turn off the lights after the area hasn’t been occupied for N minutes. A planned automation feature for the door sensors is to automatically yell at my kids if they’re more than N feet from the front door and didn’t close it.

The less “smart” but very useful motion sensors that I’m using are these Mr Beams MB726 Battery Powered Motion Sensing LED Nightlights. They aren’t smart in the Home Automation sense but they are much cheaper. They’re ideal for lighting up dark hallways and stairs where “light it up when I come, and turn if off 30 seconds later” is just perfect. I bought those because I have kids and a couple of our hallways didn’t have power outlets to plug in a motion-activated AC powered nightlight.

One last tangent related to motion sensing, but more on the “smart switch -vs- bulb” topic: With the smart bulbs, one limit is that if someone turns off the switch, the bulbs forget their dim level. A switch never loses power so it remembers. An advantage of smart bulbs is that at homework time, motion turns on all 4 bulbs at 70% brightness. At dinner time, motion turn on 3 bulbs at 50% brightness. After 9PM when the kids are in bed, motion turns on one bulb at 10% brightness. Switches act on all the bulbs or none.

Child Automation and our Yale Deadbolt

When we get home, one of the kids asks for the house keys. The first one to ask gets the keys and gets to unlock the door. They love to unlock the door, so it’s frequently a race. I’ve been thinking it was time to give them their own keys, but its really hard when they can’t yet hold onto the same library card for more than a year. Stashing a key outdoors didn’t fit my sensibilities.

In exploring the options, I found a wide variety of locks. Push buttons. Numeric keypads. Bluetooth. WiFi. Smartphone Apps. Key fobs. So. Many. Options!

Some I was able to weed out straight away. Requiring a smarthphone is a non-starter. For that matter, requiring that we carry anything seems so last century. If someone gets locked out of the house naked, absent their sniggering sibling on the other side of the door, they should be able to get back in.

I found almost exactly what I was looking for in the Yale YRD240-ZW-605. I can easily program a unique key code for each family member. I can add a key-code for the neighbor to feed our pets when we’re on vacation. It has a Z-Wave radio built-in which pairs with my SmartThings hub. I can pull out my phone and lock/unlock the front door from anywhere. Instead of the far-less-secure key backup, this version has a 9-volt battery port which serves as the spare key.

[amazon template=image&asin=B00HS1O5NM]

The Wife Acceptance Factor of this lock is very high. When she walks out the door, she just waves her hand at it and it locks the door. When she gets home, keying in the code is faster and easier than fishing her keys out of her purse. If her phone is already out, she can unlock the door on her way towards it.

The kids adore it, but the entry routine is  a little different than I expected. Before someone unlocked the door and we all piled through. Now one child enters their very own Simerson Secret Door Society code, enters, and then deadbolts the door. Then the next child enters their code and enters. When I pull into the garage, they run out the garage door so they can enter via the front door. Shucks, they still lock themselves outside just to use their secret code and get back in. I’m amazed that the batteries have lasted three weeks. The lock says it’s battery life is still 100%. Amazing.

home automation

I recently gave a presentation on Home Automation to a room full of engineers/programmers. I was asked several times to share the presentation.

The state of home automation has improved greatly since a decade ago when I threw all my X-10 stuff into a box and left it to rot. H.A. still requires an engineer/geek to set up, but it’s far better than a decade ago. It’s not hard to see that in another year or two, setting up home automation will be attainable by non-geeks.