chained commands are SMP friendly

I just wrote a command chain (something unix admins do often) that looks like this:

  1. gzcat threeGB-file.gz \
  2. | grep ‘match this’ \
  3. | grep -Ev ‘but not this’ \
  4. | cut -f1 -d’ ‘ \
  5. | uniq \
  6. | /usr/local/bin/idn -u \
  7. | tr ‘[:upper:]’ ‘[:lower:]’ \
  8. > infile.fifo &

That command pipeline decompresses a file, filters out the metadata, weeds out dupes,  converts punycode IDN names to  UTF-8, converts caps to lower case, and makes the data available as a file.

While sucking the contents of that file into a MySQL table, I noticed that each of the commands in the pipeline was running on a separate CPU core. While it’s not the same as Grand Central Dispatch, it’s fun to see just how well the Unix Philosophy had weathered the decades.

Leave a Reply

Your email address will not be published. Required fields are marked *