US manufacturing

The the USA is still one of the largest manufacturers in the world. Our manufacturing sector is producing as much today as it ever has:


While it’s true that some (a small fraction) US manufacturing jobs have moved overseas (especially textiles), the vast majority of manufacturing job losses are due to automation. It is machines that have taken those jobs, not foreigners or immigrants.

On balance, NAFTA was a very big win for the USA and our trading partners Canada and Mexico. The primary reason NAFTA hasn’t helped Mexico far more is due to our ill conceived and almost entirely ineffective war on drugs.

heat pump water heater

In July I purchased a GE Geospring ($700 at Lowes in Seattle) 50 gallon heat pump water heater. I installed it myself in the basement. It’s wired the same as a typical electric water heater, so I just ran a new circuit of 10 gage wire and hooked it up.

Heat pump water heaters make more noise than traditional water heaters. If I happen to walk by the open door to the basement, I can hear it but I don’t consider it “loud.” It makes a little less noise than a dehumidifier, a lot less noise than an old dishwasher, but a fair bit more noise than my new ultra-quietest-one-available dishwasher. I’d guess in the neighborhood of 65 decibels.

Heat pump water heaters cool the area they’re in. I consider that a feature, as the basement is our “cool dry” storage area. Despite the output of cool air, the basement was about 64° before I put the heat pump water heater in and it’s still usually 64° after. That’s because the concrete floor and walls have lots of thermal mass so it takes a LOT of input to change the temps significantly.

A heat pump also dehumidifies the air. It has a condensate drain where the water obtained is drained off. Over the course of a week, the condensate measured about a quart for our family of four. Not huge, not “replaces a dehumidifier,” but welcome never-the-less.

The install docs recommend installing it in a garage or basement and I agree. You could put it in a large closet or pantry, but you’d want to have insulated doors if it’s adjacent to a “relaxing” area of the house.

Thus far, I’m very fond of my heat pump water heater.

nginx and cronolog

Since the last century, I’ve been in the habit of piping my web server log files through cronolog and off to automatically selected files in the pattern /var/log/http/2015/10/23/access.log. This works quite well for me because way back when, I wrote a little log processing script called Logmonster… This is my solution for timestamp based logging with nginx:

Since the last century, I’ve been in the habit of piping my web server log files through cronolog and off to automatically selected files in the pattern /var/log/http/2015/10/23/access.log. This works quite well for me because way back when, I wrote a little log processing script called Logmonster.

After all these years, Logmonster still runs a while after midnight (via periodic) and:

  • parses the web server logs by date and vhost
  • feeds them through Awstats
  • compresses them

Back when Logmonster was named Apache::Logmonster, it required installing cronolog and making a few small changes to httpd.conf:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %v" logmonster
CustomLog "| /usr/local/sbin/cronolog /var/log/http/%Y/%m/%d/access.log" logmonster
ErrorLog "| /usr/local/sbin/cronolog /var/log/http/%Y/%m/%d/error.log"

Years later, after I got tired of maintaining Apache, lighttpd was all shiny and new and it was similarly easy to configure, making these changes to lighttpd.conf:

accesslog.format = "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %v"
accesslog.filename = "|/usr/local/sbin/cronolog /var/log/http/%Y/%m/%d/access.log"
server.errorlog = "/var/log/http/error.log"

Now, after spending more time than I wanted to determining why lighttpd and haproxy stopped playing nice together (Most HTTP POST commands time out. No good reason why. Remove haproxy, works fine. Replace lighttpd nginx behind haproxy, works fine.) so I replaced lighttpd with nginx. That required figuring out how to get cronolog type logging to work in nginx.

Nearly all my cronolog+nginx search returned only instructions for setting up logging to a FIFO, which I thought was a nifty idea. So I created the FIFOs, configured nginx, and upon startup, nginx just hangs. No idea why. It’s also requires setting up the FIFOs before nginx could startup, so I didn’t love that idea. Then I found instructions showing how to configure log rotation within nginx.conf. That’s exactly what I was looking for.

This is my solution for timestamp based logging with nginx:

log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$server_name"';
if ($time_iso8601 ~ "^(?\d{4})-(?\d{2})-(?\d{2})") {}
access_log /var/log/http/$year/$month/$day/access.log main;

Is a change of political climate in the air?

On May 21st,  leaders representing 6.5 million companies in 130 countries called on policy makers to shift towards low-carbon economies including carbon pricing and an end to fossil-fuel subsidies.

Yesterday, June 1st,  Six oil and gas “Majors” called on the UN Convention on Climate Change to introduce carbon pricing and markets.

If this keeps up, Fox News will admit climate change is real, Rick Perry will admit that government can create jobs, and lions will lay down with lambs.

Apple data centers on 100% renewable power

Apple is spending an eye popping $850 million to build a ginormous solar farm (280 megawatts) that will power their entire California operations. This new solar farm is not to be confused with the 70MW solar farm they’re building in Arizona, the $55 million “under way” third solar farm (17.5MW) in North Carolina, the two 20MW solar farms they’re building in China, or the existing 20MW solar farm near Reno, NV, or the two existing 20MW solar farms in N. Carolina.

The backstory is that in 2010, Apple wanted to buy renewable energy from Duke to power their Maiden N.C. data center. It wasn’t even legal in N. Carolina. In 2011 Apple bypassed the N.C. coal lobby by purchasing 100 acres of land and in 2012 they finished building (est. $100 million) the first non-utility 20MW solar farm. At the same time, they also built a 5MW fuel cell farm. In 2013 they doubled their fuel cell farm to 10MW and built another 20MW solar farm. Apple has since been producing 100% of the power they need in N.C.

While I believe that Tim Cook is sincere about reducing Apple’s carbon footprint, I also think it’s likely that spending over a billion dollars on solar panels is a very good investment. Apple is famously cash rich and by spending today and owning the solar farms, Apple fixes their energy prices at today’s rates for the next 30 years. Apple has taken a large and variable cost and turned it into a fixed cost that is no longer subject to price inflation or fluctuation. What Apple is also purchasing is energy stability.

Apple is also becoming an energy supplier. For the first 10 years, PG&E will purchase 150MW of production and Apple gets 130MW.  In the last 20 years, Apple gets 100% of production. It’s likely that their operations will have expanded to utilize the power (as has the NC data center) but if not, they’ll have little trouble selling their surplus capacity.

While Apple was first in the, “okay then, we’ll build it ourselves” solar game, the even bigger story is that 2014 was the year solar arrived in Main Street USA. In just 2014, nearly 70% of the worlds solar power generation came online with several companies having more installed solar than Apple: Wal-Mart (105MW), Kohl’s (50MW), and Costco (48MW). IKEA is not far behind with 39MW. Apple isn’t even the largest purchaser of solar as Intel, Kohl’s, Whole Foods, Dell, and Johnson & Johnson all purchase more solar power than Apple. What was so special about solar in 2014?

Swanson’s Law observes that solar modules tend to drop in price by 20% for every doubling of cumulative shipped volume. Apple deployed 60MW between 2012-2014 and during that same time, photo voltaic capacity more than doubled. By being out in front and building not just demand, but also solar capacity, Apple helped 2014 be the year of solar grid parity in 3 NE states, California, Arizona, and Hawaii. It is predicted that grid parity will arrive in “many” US markets in 2015 and Deutsche Bank predicts solar grid parity for all 50 states in 2016. With Apple deploying another 407MW of solar In just 2015-2016, that prediction seems like slam dunk.