April solar update

The coldest Puget Sound winter in decades is receding and with it the heat pumps heavy period of energy use. April showers are upon us, the sun is rising higher each passing week and solar output is crawling out of the winter basement. In the past week, the solar panels produced 75% of our household energy budget. It looks like we’ll be into “solar surplus” territory by the end of April.

Costco Citi Card

While balancing the books I found it disappointing that AmEx dumped access to their site the instant the Costco transferred our accounts to Visa. Companies with better service tend to provide access for a period after cancellation to download statements and the like.

I was at first a little perplexed as the Citi statements begin in July and the last AmEx statement I was able to download was for May. Then I read the fine print in Citi’s site. We just have to create a request and wait 24-48 hours for the PDF statement to appear. Okay, request sent.

Better still, I was able to download all the account transactions and Citi has export formats for any accounting software. Some banks (cough: USAA) can’t seem to understand that exporting account data in OFX/QIF format for accounting software is a useful feature. Anyway, I picked the “since last export” and got transactions starting in mid-2015, so it appears a goodly portion of our account history transferred. Thank you Citi.

Who’s Afraid of the TPP?

Who’s Afraid of the Trans-Pacific Partnership?

Very roughly speaking, DeLong’s argument is this: everyone agrees that Germany is the poster child for an advanced economy with a great manufacturing policy. And yet, their manufacturing employment has steadily declined for the past half century too, just like ours. So if this has happened to Germany, there’s not much of a case for suggesting that the US has done anything especially wrong over the past 50 years. We’ve simply evolved from a (relatively) poor manufacturing nation into a (relatively) rich services and technology nation. This has nothing much to do with trade policy, either. It’s just what rich countries do. What’s more, it’s a decidedly good thing overall, even if it does affect a smallish number of people badly.

This is not terribly different than agricultural employment. At the turn of the 20th century about half of US workers were employed in agriculture. A hundred years later as we skated past Y2K it is about 2%.

TLS management

Let’s Encrypt, TLS certificates, and HAproxy

I’m evolving. As always, the change is being driven by the most pernicious of motivators: pain. I’ve sold, installed, and upgraded SSL/TLS certificates for years. It’s always been mildly painful: I maintain an offline CA where I generate all the keys and CSR (certificate requests). Then I submit the CSRs to whichever Certificate Authority / Reseller has the best current pricing, get back the new signed certificate from the CA, archive it, and finally install the key, crt, and CA chain file at the destination.

It can be painful and annoying enough that clients regularly hire me to install their certificates for them. To reduce the pain, I’ve encouraged long-duration (3+) year certs. I also have custom scripts tailored to my private CA to reduce the keystrokes. Even so, managing a few dozen certificates was onerous. It didn’t help that every application / daemon (apache, nginx, lighttpd, haproxy, dovecot, qmail, postfix, haraka, etc.) has their own special syntax and sometimes format for configuring the TLS certificates.

Two things happened in 2016 that made TLS management not suck:

  1. The Internet Security Research Group released Let’s Encrypt(https://letsencrypt.org). It’s a free and highly automated Certificate Authority that validates domain ownership (via DNS or HTTP) and issues certificates in seconds.
  2. I’ve moved all my web servers behind HAproxy. Now all TLS certs for web servers get deployed to haproxy and the job is done. No messing with lighttpd, apache, or nginx configs. Configure HAproxy get to an A+ at SSLLabs and it covers all the web servers.

Let’s Encrypt provides free signed certificates in just a few seconds, so long as one is willing to invest the time and energy into automating it. I’ve settled on [acme.sh](https://github.com/Neilpang/acme.sh) as my preferred client and once I’ve generated a certificate, it automatically renews and re-deploys it when needed. Just right.

HAproxy now does all the TLS termination, URL routing, scheme upgrades (http -> https), and rewrites. This greatly simplifies the backend web server configs. Need mod_perl, use Apache. Need CGI support, use lighttpd. For everything else I use nginx. Now all of them are simpler to deploy and upgrade.

US manufacturing

The the USA is still one of the largest manufacturers in the world. Our manufacturing sector is producing as much today as it ever has:


source: tradingeconomics.com

While it’s true that some (a small fraction) US manufacturing jobs have moved overseas (especially textiles), the vast majority of manufacturing job losses are due to automation. It is machines that have taken those jobs, not foreigners or immigrants.

On balance, NAFTA was a very big win for the USA and our trading partners Canada and Mexico. The primary reason NAFTA hasn’t helped Mexico far more is due to our ill conceived and almost entirely ineffective war on drugs.

heat pump water heater

In July I purchased a GE Geospring ($700 at Lowes in Seattle) 50 gallon heat pump water heater. I installed it myself in the basement. It’s wired the same as a typical electric water heater, so I just ran a new circuit of 10 gage wire and hooked it up.

Heat pump water heaters make more noise than traditional water heaters. If I happen to walk by the open door to the basement, I can hear it but I don’t consider it “loud.” It makes a little less noise than a dehumidifier, a lot less noise than an old dishwasher, but a fair bit more noise than my new ultra-quietest-one-available dishwasher. I’d guess in the neighborhood of 65 decibels.

Heat pump water heaters cool the area they’re in. I consider that a feature, as the basement is our “cool dry” storage area. Despite the output of cool air, the basement was about 64° before I put the heat pump water heater in and it’s still usually 64° after. That’s because the concrete floor and walls have lots of thermal mass so it takes a LOT of input to change the temps significantly.

A heat pump also dehumidifies the air. It has a condensate drain where the water obtained is drained off. Over the course of a week, the condensate measured about a quart for our family of four. Not huge, not “replaces a dehumidifier,” but welcome never-the-less.

The install docs recommend installing it in a garage or basement and I agree. You could put it in a large closet or pantry, but you’d want to have insulated doors if it’s adjacent to a “relaxing” area of the house.

Thus far, I’m very fond of my heat pump water heater.

nginx and cronolog

Since the last century, I’ve been in the habit of piping my web server log files through cronolog and off to automatically selected files in the pattern /var/log/http/2015/10/23/access.log. This works quite well for me because way back when, I wrote a little log processing script called Logmonster… This is my solution for timestamp based logging with nginx:

Since the last century, I’ve been in the habit of piping my web server log files through cronolog and off to automatically selected files in the pattern /var/log/http/2015/10/23/access.log. This works quite well for me because way back when, I wrote a little log processing script called Logmonster.

After all these years, Logmonster still runs a while after midnight (via periodic) and:

  • parses the web server logs by date and vhost
  • feeds them through Awstats
  • compresses them

Back when Logmonster was named Apache::Logmonster, it required installing cronolog and making a few small changes to httpd.conf:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %v" logmonster
CustomLog "| /usr/local/sbin/cronolog /var/log/http/%Y/%m/%d/access.log" logmonster
ErrorLog "| /usr/local/sbin/cronolog /var/log/http/%Y/%m/%d/error.log"

Years later, after I got tired of maintaining Apache, lighttpd was all shiny and new and it was similarly easy to configure, making these changes to lighttpd.conf:

accesslog.format = "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %v"
accesslog.filename = "|/usr/local/sbin/cronolog /var/log/http/%Y/%m/%d/access.log"
server.errorlog = "/var/log/http/error.log"

Now, after spending more time than I wanted to determining why lighttpd and haproxy stopped playing nice together (Most HTTP POST commands time out. No good reason why. Remove haproxy, works fine. Replace lighttpd with nginx behind haproxy and it works fine.) so I replaced lighttpd with nginx. That required figuring out how to get cronolog type logging to work in nginx.

Nearly all my cronolog+nginx search returned only instructions for setting up logging to a FIFO, which I thought was a nifty idea. So I created the FIFOs, configured nginx, and upon startup, nginx just hangs. No idea why. It’s also requires setting up the FIFOs before nginx could start up, so I didn’t love that idea. Then I found instructions showing how to configure log rotation within nginx.conf. That’s exactly what I was looking for.

This is my solution for timestamp based logging with nginx:

log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$server_name"';
if ($time_iso8601 ~ "^(?\d{4})-(?\d{2})-(?\d{2})") {}
access_log /var/log/http/$year/$month/$day/access.log main;

Is a change of political climate in the air?

On May 21st,  leaders representing 6.5 million companies in 130 countries called on policy makers to shift towards low-carbon economies including carbon pricing and an end to fossil-fuel subsidies.

Yesterday, June 1st,  Six oil and gas “Majors” called on the UN Convention on Climate Change to introduce carbon pricing and markets.

If this keeps up, Fox News will admit climate change is real, Rick Perry will admit that government can create jobs, and lions will lay down with lambs.