Net Zero mission accomplished

One of my major goals for our home renovations has been achieving Net Zero, meaning our house produces as much energy as it consumes. As one might expect, producing as much energy as we consume yields energy bills of less than $0.  Today I consulted my energy meter and extracted the following data points for calendar year 2017:

  • Energy Produced: 9,288 kWh
  • Energy Consumed: 12,396 kWh
  • Energy Consumed by Auto: 3,216 kWh
  • Energy Consumed by Building: 9,180 kWh

For the purposes of measuring building consumption, I subtracted the auto consumption (measured by our chargepoint home charger) from total consumption. Since our building produced 108 kWh more than it consumed, I can claim the Net Zero goal as accomplished.

Our home and EV together consumed 1,500 kWh less in 2017 than 2016. Looking forward, I anticipate a similar reduction in 2018 because we were using space heaters for part of the house in Jan-Apr of 2017. The heat pump for that area is now installed. Also, the concrete basement and downstairs walls aren’t insulated yet. I have the insulation (XPS & poly-iso) standing by and it will be installed before the next heating season.

I don’t expect an substantial efficiency improvements after 2018. All our energy systems are super efficient, our insulation levels are super, and our energy surpluses (say hello to Energy Plus) will be consumed by our current and future EVs. Any further efficiency improvements would have no economic justification.

Reference points

  • Average home energy consumption: 23,000 kWh
  • Average auto energy consumption: 15,000 kWh
  • A super-insulated (aka: ultra efficient) home typically needs 1/5 the energy for HVAC.
  • On average, EVs consume  1/3 the energy of an ICE vehicle.

Is A Powerwall Worth It?

Primarily for economic reasons, I covered our roof with 10kW of solar panels. As a producer of electricity, it no longer made sense to pay a utility company for natural gas so the furnace was replaced with heat pumps and the water heater was replaced with a HPWH. The fireplace and chimney were anachronisms I was happy to get rid of. I removed all the gas lines and had PSE remove the leaky gas meter as well. From a safety perspective, ridding our home of combustion appliances was a big win.

While on the topic of safety, the Big One is coming. Our home was built in 1955, way before the Nisqually Quake and updated building codes. I’ve done a few seismic retrofits as suggested by the City. The most important thing, literally, is surviving the quake, so making sure our house doesn’t collapse is a good start. The next most important thing is potable drinking water. I warehouse a weeks worth. For lighting we have camping headlamps, Mr. Beam night lights around the house (with rechargeable AA batteries that last about a month) and outside, I’ve got solar LED lights attached to the house. They’re really nice when walking around the house at night and would suffice during a power outage.

What I don’t have is energy storage. For the safety of linemen, when the grid is down, so is my array. I want a Tesla Powerwall so I can operate off grid. For $6k, Tesla provides a 14kW battery and the transfer switch that enables off grid operation. In the summer, our array produces double what we use (including EV charging) so a single Powerwall could easily power our house, plus extension cords to the neighbors for their fridges, indefinitely. In mid-winter, with no visible sun and freezing temps we could drain a Powerwall in just two days. But after a winter disaster, we could turn the heat off, dress warm, and power the house for weeks. Or heat just one room. That’s a compelling use case, but I have yet to conclude that the potential losses from not having a battery (spoiled food, no lights, no cooking, no heat) during an outage outweigh the $6,200 price tag.

EV: how much is 6kWh?

Our first Leaf was the 2013 model year. That was the year Nissan  boosted cold weather driving range by 20+ miles with the switch to heat pumps, for an estimated 85 total miles. While I was able to get 82 miles from one charge (summer hypermiling), in practice the range was quite a bit less. The worst case, mid-winter driving to and from work in the cold and dark, the range was closer to 50 miles. For Jen’s 32 mile round-trip commute, that mean charging the car every night.

On long trips, recharging an EV has the same level of friction as adding gas to an ICE. In the case of commuting, EV refueling is considerably more convenient:

EV

ICE

drive home drive to station
wait in line for pump
park near EV plug pull up to pump
negotiate payment
plug in plug in
wait for tank to fill
unplug (later) unplug
drive away

When the 2016 model year Leaf was released, the battery capacity increased from 24kWh to 30kWh, boosting the EPA rated range from 84 to 107 miles. That extra 23 miles was just enough to get us out to the ski resorts in the winter. It also made quite a difference in how often our Silver Leaf needed to recharge:

snowflake server

I’m in the middle of rebuilding my server. For years I provisioned one-off virtual machines for clients that needed custom solutions. Dedicated IPs for TLS (for shopping carts), custom coded extensions that turned a photo app into a shopping cart, email servers, etc. I’ve been maintaining those VMs for years while the cost of technical debt has been growing.

The base OS in the VMs is years old. As software gets upgraded, the state of the VMs slowly drift and the result is a snowflake server. Upgrades frequently break something. I monitor most services and usually get them fixed before anyone notices. Still. Even on conservative OSes like Debian and FreeBSD, stuff regularly breaks and manual intervention is required. And those manual fixes here and there contribute to the drift.

So I’m rearchitecting everything for composability and simplicity. HAproxy handles all the HTTP redirection and HTTPS termination. The certificate management is now completely automated with Let’s Encrypt and acme.sh. HAproxy routes the requests to the backend web servers. No longer do apache, lighttpd, and nginx handle SSL/TLS or URL manipulation. The web server configs are simpler and require fewer customizations.

Solar 11-month update

On a typical “it rained every single day in April” month, we still managed to skate across the finish line at nearly at Net Zero:

April 2017 household energy
April 2017 household + auto energy budget

Last year I removed all the natural gas appliances and converted everything to electric heat pumps. I sized the 10kW array aiming for Net Zero during the calendar year. That would mean producing enough surplus during the summer to carry us through the winter. It looks like we’re going to miss this year:

2016-2017 household energy budget

Even though we’ll be banking surpluses in May, it won’t close that 7MWh deficit. Our household usage includes over 3MWh of car charging and this last winter was Seattle’s coldest in 32 years. The heat pumps were working overtime to keep the house warm.

Costco Citi Card

While balancing the books I found it disappointing that AmEx dumped access to their site the instant the Costco transferred our accounts to Visa. Companies with better service tend to provide access for a period after cancellation to download statements and the like.

I was at first a little perplexed as the Citi statements begin in July and the last AmEx statement I was able to download was for May. Then I read the fine print in Citi’s site. We just have to create a request and wait 24-48 hours for the PDF statement to appear. Okay, request sent.

Better still, I was able to download all the account transactions and Citi has export formats for any accounting software. Some banks (cough: USAA) can’t seem to understand that exporting account data in OFX/QIF format for accounting software is a useful feature. Anyway, I picked the “since last export” and got transactions starting in mid-2015, so it appears a goodly portion of our account history transferred. Thank you Citi.

TLS management

Let’s Encrypt, TLS certificates, and HAproxy

I’m evolving. As always, the change is being driven by the most pernicious of motivators: pain. I’ve sold, installed, and upgraded SSL/TLS certificates for years. It’s always been mildly painful: I maintain an offline CA where I generate all the keys and CSR (certificate requests). Then I submit the CSRs to whichever Certificate Authority / Reseller has the best current pricing, get back the new signed certificate from the CA, archive it, and finally install the key, crt, and CA chain file at the destination.

It can be painful and annoying enough that clients regularly hire me to install their certificates for them. To reduce the pain, I’ve encouraged long-duration (3+) year certs. I also have custom scripts tailored to my private CA to reduce the keystrokes. Even so, managing a few dozen certificates was onerous. It didn’t help that every application / daemon (apache, nginx, lighttpd, haproxy, dovecot, qmail, postfix, haraka, etc.) has their own special syntax and sometimes format for configuring the TLS certificates.

Two things happened in 2016 that made TLS management not suck:

  1. The Internet Security Research Group released Let’s Encrypt(https://letsencrypt.org). It’s a free and highly automated Certificate Authority that validates domain ownership (via DNS or HTTP) and issues certificates in seconds.
  2. I’ve moved all my web servers behind HAproxy. Now all TLS certs for web servers get deployed to haproxy and the job is done. No messing with lighttpd, apache, or nginx configs. Configure HAproxy get to an A+ at SSLLabs and it covers all the web servers.

Let’s Encrypt provides free signed certificates in just a few seconds, so long as one is willing to invest the time and energy into automating it. I’ve settled on [acme.sh](https://github.com/Neilpang/acme.sh) as my preferred client and once I’ve generated a certificate, it automatically renews and re-deploys it when needed. Just right.

HAproxy now does all the TLS termination, URL routing, scheme upgrades (http -> https), and rewrites. This greatly simplifies the backend web server configs. Need mod_perl, use Apache. Need CGI support, use lighttpd. For everything else I use nginx. Now all of them are simpler to deploy and upgrade.

heat pump water heater

In July I purchased a GE Geospring ($700 at Lowes in Seattle) 50 gallon heat pump water heater. I installed it myself in the basement. It’s wired the same as a typical electric water heater, so I just ran a new circuit of 10 gage wire and hooked it up.

Heat pump water heaters make more noise than traditional water heaters. If I happen to walk by the open door to the basement, I can hear it but I don’t consider it “loud.” It makes a little less noise than a dehumidifier, a lot less noise than an old dishwasher, but a fair bit more noise than my new ultra-quietest-one-available dishwasher. I’d guess in the neighborhood of 65 decibels.

Heat pump water heaters cool the area they’re in. I consider that a feature, as the basement is our “cool dry” storage area. Despite the output of cool air, the basement was about 64° before I put the heat pump water heater in and it’s still usually 64° after. That’s because the concrete floor and walls have lots of thermal mass so it takes a LOT of input to change the temps significantly.

A heat pump also dehumidifies the air. It has a condensate drain where the water obtained is drained off. Over the course of a week, the condensate measured about a quart for our family of four. Not huge, not “replaces a dehumidifier,” but welcome never-the-less.

The install docs recommend installing it in a garage or basement and I agree. You could put it in a large closet or pantry, but you’d want to have insulated doors if it’s adjacent to a “relaxing” area of the house.

Thus far, I’m very fond of my heat pump water heater.

nginx and cronolog

Since the last century, I’ve been in the habit of piping my web server log files through cronolog and off to automatically selected files in the pattern /var/log/http/2015/10/23/access.log. This works quite well for me because way back when, I wrote a little log processing script called Logmonster.

After all these years, Logmonster still runs a while after midnight (via periodic) and:

  • parses the web server logs by date and vhost
  • feeds them through Awstats
  • compresses them

Back when Logmonster was named Apache::Logmonster, it required installing cronolog and making a few small changes to httpd.conf:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %v" logmonster
CustomLog "| /usr/local/sbin/cronolog /var/log/http/%Y/%m/%d/access.log" logmonster
ErrorLog "| /usr/local/sbin/cronolog /var/log/http/%Y/%m/%d/error.log"

Years later, after I got tired of maintaining Apache, lighttpd was all shiny and new and it was similarly easy to configure, making these changes to lighttpd.conf:

accesslog.format = "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %v"
accesslog.filename = "|/usr/local/sbin/cronolog /var/log/http/%Y/%m/%d/access.log"
server.errorlog = "/var/log/http/error.log"

Now, after spending more time than I wanted to determining why lighttpd and haproxy stopped playing nice together (Most HTTP POST commands time out. No good reason why. Remove haproxy, works fine. Replace lighttpd with nginx behind haproxy and it works fine.) so I replaced lighttpd with nginx. That required figuring out how to get cronolog type logging to work in nginx.

Nearly all my cronolog+nginx search returned only instructions for setting up logging to a FIFO, which I thought was a nifty idea. So I created the FIFOs, configured nginx, and upon startup, nginx just hangs. No idea why. It’s also requires setting up the FIFOs before nginx could start up, so I didn’t love that idea. Then I found instructions showing how to configure log rotation within nginx.conf. That’s exactly what I was looking for.

This is my solution for timestamp based logging with nginx:

log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$server_name"';
if ($time_iso8601 ~ "^(?\d{4})-(?\d{2})-(?\d{2})") {}
access_log /var/log/http/$year/$month/$day/access.log main;