Category: Uncategorized

Handy domestic app

A year or two ago, I found GroceryIQ, a nifty shopping list application for the iPhone. It has an enormous built in catalog of grocery items, as well as being able to add custom items and custom stores. So, I can walk into REI, and it’ll show just the items I’m looking for.

The only feature I’ve wished for that it doesn’t have was syncing the list on Jen’s iPhone with the list on mine. A few months back, I poked around and lo and behold, the feature had been added.

Words cannot scarcely describe how wonderful this feature is. Any time Jen adds an item to her list, I get a push messsage that notifies me. And vice versa. When I open the app, the item is in my list.

Because the database is so large, items normally have the brand, size, and quantity as well. So when I buy something, I know I’m getting exactly the right product and quantity.

My walking desk

Last year I bought a standing desk. Along with biking to work a couple days each week, it was part of my plan to shed 20 excess pounds. I like my standing desk. I like it a lot. But what I discovered is that I can’t stand all day long without my knees aching. I adjusted by sitting while at work, and standing while at home.

I read that standing all day is a great way to experience joint pain. The solution is always the same: don’t just stand there, walk. I researched treadmills and decided a Bowflex Series 7 treadmill was the one. And I wanted to pay the lowest possible price. So I set up a RSS feed to watch craigslist. After several months, I didn’t see a single one sell for less than $700. So I started bidding $700 on every one that showed up on craigslist. Finally, 9 months later, I have my treadmill.

I had to move the desk up a few inches to slide the treadmill in under it. I’m tempted to move it up another couple inches. As you can see, my arms are actually sloping downwards from my elbows. I’d rather they were nearly level, but I also want Jen to be able to use my workstation.

I’ve now worked at my walking desk for two days. I walked barefoot for the first 4 miles and found the treadmill belt to be abrasive on my feet. Now I’m alternating between barefeet and socks, until my feet have built up some calluses.

Update: I did move it up 3 more inches. Now it’s just right.

Great Wolf Lodge

The Great Wolf Lodge is a lots of fun for kids. I can see why we’ve heard rave reviews. However, there’s a few things to know before your trip that are not covered in the brochure or web site.
1. the beds are great for jumping on, but not so great for sleeping.
2. the pillows are extra firm, perfect for delivering that knockout blow during a pillowfight. Bring your own for sleeping on. We saw other families hauling them in and now we know why.
3. Beware of the zombies. They look much like normal humans, but because of sleep deprivation and too much chlorine exposure, they merely resemble humans, their faces devoid of emotion, being towed around by kids.
4. If you have really little kids, plan to spend lots of time in the tadpole pond. The water in the big pools is cooler and the 4 and under kids get cold quickly.
5. The wifi coverage is pervasive and slow. I eventually turned off wifi because AT&T 3G was faster.
6. The TV has only paid programming. Bring along options. Their internet is way too slow for streaming NetFlix.

The lodge is designed to be a one stop resort, where you eat, sleep and play without ever leaving the building. It accomplishes this, but after a couple days, we had experienced everything that was age appropriate. The kids would have been happy to stay another day or two but we were looking forward to sleeping in our own bed.

ZFS is production ready

Background

In July of 2008, I was tasked with building a system to back up thousands of Linux based servers. Previous systems using Amanda and Bacula had failed, principally because they required a full time backup administrator to maintain. My job was to build a backup system that required very little maintenance, scaled well, and made restoring data straight forward and easy.

I initially deployed BackupPC which features data deduplication and would likely have reduced our storage needs by more than 60%. I deployed on two SuperMicro systems, each equipped with dual quad-core CPUs, 16GB RAM, and 24 one terabyte disks. I built out one system with OpenSolaris and the other with FreeBSD. After testing, we deployed both with FreeBSD.

BackupPC ended up being inadequate so I wrote my own backup system on top of rsnapshot. My backup system generates rsnapshot config files and then drives multiple concurrent rsnapshot processes on each of the backup servers, pumping data to the backup disks as fast as they’ll take it. I hacked up rsnapshot for better error handling and reporting. I log exactly how much data each remote system has, as well as how much is transferred during each backup.

About ZFS

The main reason we deployed on ZFS was file system compression. After testing several settings, I settled on compression=gzip. I noticed no difference in system performance between compression settings. The backup system has been in production since, with very little attention since deployment.

When initially deployed, each backup server required manual tweaks so that they would only crash once a day. The multiple concurrent rsync processes created a workload that stressed the ZFS memory pools. Working with the lead FreeBSD ZFS developer helped the situation and my systems only crashed once a week. When ZFS v13 was merged into FreeBSD 8-current, memory management improved and my systems only crashed once a month.

Even during the months of using ZFS with frequent crashes, I never lost any data. And there’s no need to fsck the disks after crashes. My confidence in ZFS grew enough that when I upsized the disks in my home file server, I switched from gmirror (tried and true) to ZFS mirrors. I back up my public server to my home file server and saw the same occasional rsync induced crashes. About the time FreeBSD released 8.0 beta releases, I updated and the crashes ceased. So I updated these backup servers and they too have been stable ever since.

I have added another server to the pool and currently store 58 terabytes of data and over a billion files. My compression ratio averages 2.25, more than doubling the effective capacity of the disks we purchased. After FreeBSD 8 was released, I upgraded all the backup servers and could scarcely be more pleased.

And then I learned that deduplication is coming to ZFS. I can’t wait to test it.

ssh bruteforce attacks become sophisticated

SSH scans and bruteforce attacks that have have been common since my first SSH enabled server was attacked in 1996. Back then, attacks were so rare that monitoring logins and manually adding attackers IPs to /etc/hosts.allow (TCP Wrapper) was sufficient to keep systems secure.

In the mid 2000’s, the rise of botnets resulted in distributed bruteforce attacks, in which dozens of IPs (bots) would attempt to bruteforce my SSH daemons. I wrote a shell script that collected the IPs and added them to the TCP Wrappers deny list. A while later, denyhosts, was released and I started using it instead of my shell script.

Since installing denyhosts, I only monitor logins and scan the nightly security reports. In the past few years, the frequency of attacks has slowly risen but occasionally there are significant changes in attack frequency and duration. The last significant escalation I can recall was in the months leading up to McColo being shut down. Immediately after their shutdown, I noticed a dramatic reduction in bruteforce SSH attacks.

It was during that time of increased activity that I wrote Sentry. Like my original shell script and denyhosts, it adds attacking IPs to the TCP Wrappers deny list. Sentry also adds their IPs to my PF firewall. Sentry worked much the same as denyhosts, except that when someone attacks one of my IPs, they got blacklisted on all of them. The number of attacks that made it into my security logs dropped accordingly.

Months later, after McColo was shut down, the distributed attacks all but ceased. Since then, attacks have remained sporadic, perhaps 10 a week. In the last couple weeks, the number of attacks spiked. I’m seeing dozens of new IPs getting blacklisted each day. Unlike previous attacks, the usernames the attackers are using are not being duplicated, which means the command & control network behind this latest round of attacks is more intelligent than most.