snowflake server

I’m in the middle of rebuilding my server. For years I provisioned one-off virtual machines for clients that needed custom solutions. Dedicated IPs for TLS (for shopping carts), custom coded extensions that turned a photo app into a shopping cart, email servers, etc. I’ve been maintaining those VMs for years while the cost of technical debt has been growing.

The base OS in the VMs is years old. As software gets upgraded, the state of the VMs slowly drift and the result is a snowflake server. Upgrades frequently break something. I monitor most services and usually get them fixed before anyone notices. Still. Even on conservative OSes like Debian and FreeBSD, stuff regularly breaks and manual intervention is required. And those manual fixes here and there contribute to the drift.

So I’m rearchitecting everything for composability and simplicity. HAproxy handles all the HTTP redirection and HTTPS termination. The certificate management is now completely automated with Let’s Encrypt and acme.sh. HAproxy routes the requests to the backend web servers. No longer do apache, lighttpd, and nginx handle SSL/TLS or URL manipulation. The web server configs are simpler and require fewer customizations.

Financial Literacy

A Freakonomics Radio podcast I just listened to, Everything You Always Wanted to Know About Money (But Were Afraid to Ask), stated that studies have shown that in the USA and in nearly every other country studied, the percentage of people who are financially literate is under 30%. That’s the bad news.

The good news was Harold Pollack’s “all the financial advice you’ll ever need fits on an index card” conversation. I took notes:

  1. save 20% of your income
  2. pay off your credit card bill in full every month
  3. max out 401k & other tax advantaged accounts
    1. it reduces your tax burden today
    2. matching employer contributions are free money
  4. never buy or sell individual stocks
  5. buy inexpensive and well diversified index and ETF funds
  6. make your financial advisor adhere to the fiduciary standard
  7. buy a home when you’re financially ready
    1. homes are something we use and consume
    2. when are we ready?
      1. have 20% in hand
      2. fixed rate 15/30 year mortgage
      3. still have reserves for home maintenance
  8. insurance: be protected against losses > your reserves
    1. get the largest deductible
  9. do what you can to support the social safety net
    1. bad stuff happens
    2. lots of people need help

Solar ROI update

I now have a full year of electric production and consumption measured. I also have the SCL rate updates for 2017 and 2018 so I have updated my solar ROI estimates. The significant change is that the Net Metering benefit has substantially increased due to:

  1. SCL electric rates are higher in Shoreline than Seattle.
  2. The 2017 and 2018 rate increases are 5.6% (estimated at 4%)
  3. An added RSA surcharge of 1.5%
  4. The coldest winter in 32 years
  5. More electricity use than I predicted.
    • I was still insulating deep into the heating season.
    • I guesstimated the kWh it would require to heat a 1955 house with heat pumps.
    • I installed a fast (level 2) charger for our Leaf. We were able to use it more, offsetting gasoline with electricity.
  6. The increased usage is all at the higher 0.14¢ price tier.

Reasons 1-4 weren’t known during my initial estimates. Reasons 5 and 6 were planned but their scale was unknown. I knew I’d be removing all natural gas appliances (furnace, water heater, fireplace) but I hadn’t yet decided whether to install tankless electric or a heat pump water heater. I hadn’t chosen the heat pumps for house heat yet so I didn’t know their HSPF. I also didn’t know how much more we’d be able to use the Leaf.

The net result is that I now estimate a 100% return on the solar array in the 6th year instead of the 8th year.

Notes

  • I did not include the cost of the heat pumps or the heat pump water heater. Those were efficiency upgrades that I’d have done anyway. If I were keeping natural gas, I’d have replaced the old 80% furnace with a 97% modulating furnace and the “well past its expected lifespan” gas water heater with a gas tankless. In both cases the costs are comparable and just like replacing the fridge, the efficiency increases have their own ROI schedule.

Solar 11-month update

On a typical “it rained every single day in April” month, we still managed to skate across the finish line at nearly at Net Zero:

April 2017 household energy
April 2017 household + auto energy budget

Last year I removed all the natural gas appliances and converted everything to electric heat pumps. I sized the 10kW array aiming for Net Zero during the calendar year. That would mean producing enough surplus during the summer to carry us through the winter. It looks like we’re going to miss this year:

2016-2017 household energy budget

Even though we’ll be banking surpluses in May, it won’t be enough to close that 7MWh deficit. Our household usage includes over 3MWh of car charging and this last winter was Seattle’s coldest in 32 years. The heat pumps were working overtime to keep the house warm.

April solar update

The coldest Puget Sound winter in decades is receding and with it the heat pumps heavy period of energy use. April showers are upon us, the sun is rising higher each passing week and solar output is crawling out of the winter basement. In the past week, the solar panels produced 75% of our household energy budget. It looks like we’ll be into “solar surplus” territory by the end of April.

Costco Citi Card

While balancing the books I found it disappointing that AmEx dumped access to their site the instant the Costco transferred our accounts to Visa. Companies with better service tend to provide access for a period after cancellation to download statements and the like.

I was at first a little perplexed as the Citi statements begin in July and the last AmEx statement I was able to download was for May. Then I read the fine print in Citi’s site. We just have to create a request and wait 24-48 hours for the PDF statement to appear. Okay, request sent.

Better still, I was able to download all the account transactions and Citi has export formats for any accounting software. Some banks (cough: USAA) can’t seem to understand that exporting account data in OFX/QIF format for accounting software is a useful feature. Anyway, I picked the “since last export” and got transactions starting in mid-2015, so it appears a goodly portion of our account history transferred. Thank you Citi.

Who’s Afraid of the TPP?

Who’s Afraid of the Trans-Pacific Partnership?

Very roughly speaking, DeLong’s argument is this: everyone agrees that Germany is the poster child for an advanced economy with a great manufacturing policy. And yet, their manufacturing employment has steadily declined for the past half century too, just like ours. So if this has happened to Germany, there’s not much of a case for suggesting that the US has done anything especially wrong over the past 50 years. We’ve simply evolved from a (relatively) poor manufacturing nation into a (relatively) rich services and technology nation. This has nothing much to do with trade policy, either. It’s just what rich countries do. What’s more, it’s a decidedly good thing overall, even if it does affect a smallish number of people badly.

This is not terribly different than agricultural employment. At the turn of the 20th century about half of US workers were employed in agriculture. A hundred years later as we skated past Y2K it is about 2%.

TLS management

Let’s Encrypt, TLS certificates, and HAproxy

I’m evolving. As always, the change is being driven by the most pernicious of motivators: pain. I’ve sold, installed, and upgraded SSL/TLS certificates for years. It’s always been mildly painful: I maintain an offline CA where I generate all the keys and CSR (certificate requests). Then I submit the CSRs to whichever Certificate Authority / Reseller has the best current pricing, get back the new signed certificate from the CA, archive it, and finally install the key, crt, and CA chain file at the destination.

It can be painful and annoying enough that clients regularly hire me to install their certificates for them. To reduce the pain, I’ve encouraged long-duration (3+) year certs. I also have custom scripts tailored to my private CA to reduce the keystrokes. Even so, managing a few dozen certificates was onerous. It didn’t help that every application / daemon (apache, nginx, lighttpd, haproxy, dovecot, qmail, postfix, haraka, etc.) has their own special syntax and sometimes format for configuring the TLS certificates.

Two things happened in 2016 that made TLS management not suck:

  1. The Internet Security Research Group released Let’s Encrypt(https://letsencrypt.org). It’s a free and highly automated Certificate Authority that validates domain ownership (via DNS or HTTP) and issues certificates in seconds.
  2. I’ve moved all my web servers behind HAproxy. Now all TLS certs for web servers get deployed to haproxy and the job is done. No messing with lighttpd, apache, or nginx configs. Configure HAproxy get to an A+ at SSLLabs and it covers all the web servers.

Let’s Encrypt provides free signed certificates in just a few seconds, so long as one is willing to invest the time and energy into automating it. I’ve settled on [acme.sh](https://github.com/Neilpang/acme.sh) as my preferred client and once I’ve generated a certificate, it automatically renews and re-deploys it when needed. Just right.

HAproxy now does all the TLS termination, URL routing, scheme upgrades (http -> https), and rewrites. This greatly simplifies the backend web server configs. Need mod_perl, use Apache. Need CGI support, use lighttpd. For everything else I use nginx. Now all of them are simpler to deploy and upgrade.