Why use reverse-proxy caching?

For most public-facing web applications, the significant majority of their traffic is anonymous, non-authenticated users. Even with a variety of internal data-cache mechanisms and other good optimizations, a large amount of code execution goes into executing a PHP application to generate a page even if the content of this page will be the same for many users. Code and query optimization are very important to improving the experience for all users of a web application, but even the most basic “Hello World” script will top out at about 3k requests/second due to the overhead of Apache and PHP — many real applications top out at less than 200 requests/second. Varnish, a light-weight proxy-server that can run on the same host as the webserver, can cache pages in memory and can serve them at rates of more than 10k requests/second with thousands of concurrent connections.

While the point of web-applications is to have content be dynamic and easily changeable, for most applications and most of the anonymous users, receiving content that is slightly stale (cached for 5 minutes or something similar) isn’t a big deal. Sure, visitors to your blog might not see the latest post for a few minutes, but they will get their response in 4 milliseconds rather than 2 seconds.

Should your site get posted on Slashdot, a caching reverse-proxy server will give anonymous visitor #2 and up the same page from cache (until expiration), while authenticated users continue to have their requests passed through to the Apache/PHP back-end. Everyone wins.

Caveats

Before we get into how to set this up, you should be aware of a few caveats (in addition to increased complexity) that come with this scheme.

1. Stale Content

Ideally, pages would always be served from the cache for as long as they don’t change, then the application would expire pages when they are changed on the back-end. Varnish has an API that supports this behavior and Drupal Varnish module is being developed to do this dynamic cache-clearing for Drupal sites, but overall, dynamic cache clearing is much more difficult to set up than time-based cache expiration.

When using time-based cache expiration, the challenge is to balance the needs for content freshness (shorter cache lifetimes) against the efficiency of cache hits (longer cache lifetimes will result in more clients using the cached versions). For content that doesn’t need to be up-to-the-minute fresh, a cache lifetime of around 5 minutes might be a good starting point. If the content only changes daily at certain time, a fixed expiration time (shortly after the data sync) might be appropriate.

2. Cookie Use

If your application only uses a cookies set by PHP’s session_start() function, then lazy_sessions.php should work transparently without modification of either that include file or your application (other than including the file). If your application sets other cookies then these will cause the reverse-proxy not to cache them unless you specifically exclude them in the reverse-proxy server’s configuration.

3. Data Caching in the $_SESSION

If you use the $_SESSION array as a data cache on anonymous requests, then these anonymous requests will be given a session cookie and their requests won’t be served from the reverse-proxy’s cache. Rather than using the $_SESSION array for non-user-specific data, cache such data with APC or memcached. This also has the advantage of such non-user-specific data not having to be rebuilt for every new client.

4. flush() and output buffering

The default PHP session handling mechanism adds the session cookie to the response headers right when session_start() is called and writes the data off to the file-system after the script exits and the data has been sent. This default behavior ensures that users will always get a session cookie and saves the session data as the final processing step after all class destructors have been called.

Since we don’t want to always set a session cookie, we need to remove the Set-Cookie header before headers are sent to the client. Output buffering with ob_start() will ensure that we have a chance to decide to clear the Set-Cookie header at script shutdown.

In some cases (such as incrementally sending large binary files) we want to send the content body (and therefor also the headers) before the script exits using the flush() function. To ensure that the session cookie is properly removed session_write_close() must be called before flush() or any other code that causes headers to be sent.

Implementation

Implementing reverse-proxy caching has three steps: PHP changes to enable lazy sessions, PHP changes to set cache-controlling headers, and finally the reverse-proxy server setup. For this example I’ll use the Varnish reverse-proxy server, but others could be used instead.

1. PHP: Lazy Sessions

The first thing that needs to happen to make anonymous requests cache-able in an application that uses sessions is to ensure that sessions are only started when there is session data to be stored. By default, PHP’s session handling mechanisms add a session cookie to the response header and store a session data file on the server on page-load that calls session_start(). While this behavior makes it easy to write applications that use sessions, it effectively means that there is no way to differentiate between responses that are for a particular user and those that could be for many users.

Including the lazy_sessions.php file before session_start() is called will override the default session-handling mechanism with one that checks to see if there is any data in the $_SESSION array before sending the user a Set-Cookie header and storing a session file:

<?php

// Include files or other pre-session_start code

require_once('lazy_sessions/lazy_sessions.php');
start_session();

// The rest of the application code.
?>

If your application needs to flush content and thereby send headers before script shutdown (such as incrementally sending file data), call session_write_close() if session_start() has been called for that script:

<?php

// Include files or other pre-session_start code

require_once('lazy_sessions/lazy_sessions.php');
start_session();

// other application code.

// If session_write_close() is not called before flushing, then the Set-Cookie
// header will be sent before our custom session handler has a chance to determine
// if a session is even needed.
session_write_close();

print "Hello";
flush();
print " World.";
flush();

?>

2. PHP: Cache-Control headers

Now that we have our cookies straightened out, we need to ensure that our PHP scripts respond with HTTP headers that indicate that downstream clients such as our reverse-proxy and the user’s browser are allowed to cache anonymous pages. There are a number of different Cache-Controlling Headers that may affect whether a particular cache may store a given response. By default, PHP sets all of these headers to indicate that no caches may store any pages, ensuring that they are dynamic.

<?php

// If the session data is empty, then we could assume that there is no per-user data
// and that the response can be cached.
if (!count($_SESSION)) {

// Alternatively, we could check an application-specific value (such as a user-id)
// to determine if the response is for a particular user.
// if (!isset($_SESSION['user_id'])) {

// Cache for 5 minutes
$maxAge = 300;

header('Expires: '.gmdate('D, d M Y H:i:s', time() + $maxAge).' GMT', true);
header('Cache-Control: public, max-age='.$maxAge, true);
header('Pragma: ', true);
}

header('Vary: Cookie,Accept-Encoding', true);

The two most important headers with regard to caching with varnish are the following:

The Cache-Control header.

The Cache-Control: public, max-age=300 header indicates to any clients (such as the Varnish caching proxy) that this response can be cached in public caches valid for many downstream clients. The max-age portion of the header indicates that the cache may store this response for 300 seconds.

As I understand it (possibly wrong) Varnish only looks at the max-age portion of the Cache-Control header when determining how long to store a response. Apparently it ignores the Expires header for its cache-expiration purposes, though this header is passed on to downstream clients.

The Vary header

The Vary: Cookie,Accept-Encoding header tells Varnish (and in-browser caches) that they should not respond with the cached version of a response if the request includes a cookie or a different cookie from the request that previously had its response cached. Similarly, if one client says that it accepts gzip encoding via an Accept-Encoding: gzip request header, then the cached response may be compressed with gzip and should not be sent in response to requests from clients that do not state that they accept gzip encoding.

While Varnish’s behavior is to never cache or respond from cache when cookies are present, without the Vary: Cookie response header, browsers or other downstream caches may respond with a cached response valid for only anonymous users even though a cookie is now present.

See my notes on Cache-Controlling Headers for more details about other headers and how they affect the Varnish cache and in-browser caches.

3. Varnish (Reverse-Proxy) Configuration

The /etc/varnish/default.vcl config file controls how Varnish responds to requests and responses, in particular whether or not it should cache or not. Below is the contents of my default.vcl file.

backend default {
.host = "127.0.0.1";
.port = "80";
}

sub vcl_recv {
// Remove has_js and Google Analytics __* cookies.
set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");
// Remove a ";" prefix, if present.
set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
// Remove empty cookies.
if (req.http.Cookie ~ "^\s*$") {
unset req.http.Cookie;
}

// Cache all requests by default, overriding the
// standard Varnish behavior.
// if (req.request == "GET" || req.request == "HEAD") {
//   return (lookup);
// }
}

sub vcl_hash {
if (req.http.Cookie) {
set req.hash += req.http.Cookie;
}
}

sub vcl_fetch {
if (!beresp.cacheable) {
	return (pass);
}

// If using PHP < 5.3 there is no way to fully delete headers, so empty
// Set-Cookie headers may be in the response. Ignore these empty headers.
if (beresp.http.Set-Cookie ~ "^\s*$") {
	unset beresp.http.Set-Cookie;
}

if (beresp.http.Set-Cookie) {
	return (pass);
}
return (deliver);
}

While LDAP authentication works great, there is one problem: If a person has never logged into our Bugzilla, we can’t add them to the CC list of an issue. This is important for us because issues usually don’t get submitted directly to the bug tracker, but rather come in via calls, emails, tweets, and face-to-face meetings. We are then left to submit issues to Bugzilla ourselves to keep track of our to-do items. Ideally we’d add the original reporter to the bug’s CC list so that they will automatically be notified as we make progress on the issue, but their Bugzilla account must exist before we can add them to the bug.

Searching about the internet I wasn’t able to find anything about how to import LDAP users (or any kind of users) into Bugzilla, though I was able to find some basic instructions on how to create a single user via Bugzilla’s Perl API. To improve on the lack of user-import support I’ve created an Perl script that creates users from lines in a tab-delimited text file (create_users.pl) as well as a companion PHP script that will export an appropriately-formatted list of users from an Active Directory (LDAP) server (export_users.php).

BugzillaImport.zip — Unzip in your Bugzilla directory, run via the command line. See below for examples.

File Listings:

create_users.pl

This script can safely be run repeatedly. Only new users not already in Bugzilla will be added, users matching existing email addresses will be skipped.

#!/usr/bin/env perl
##########################################################
# This is a basic script to import users into Bugzilla.
#
# Users can be imported from tab-delimited text files or
# tab-delimited lines piped to STDIN. Lines should have 3
# columns: login	email	name
#
#
# Author:
#	Adam Franco (afranco@middlebury.edu)
# Date:
#	2010-03-08
# URL:
#	http://www.adamfranco.com/archives/374
# License:
#   The contents of this file are subject to the Mozilla Public
#   License Version 1.1 (the "License"); you may not use this file
#   except in compliance with the License. You may obtain a copy of
#   the License at http://www.mozilla.org/MPL/
#
#   Software distributed under the License is distributed on an "AS
#   IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or
#   implied. See the License for the specific language governing
#   rights and limitations under the License.
##########################################################

use FindBin qw($Bin);
BEGIN {
    push @INC,$Bin;
    push @INC,$Bin."/lib";
    push @INC,$Bin."/lib/x86_64-linux-thread-multi";
}
use Bugzilla;
use Bugzilla::User;
use Error qw(:try);

sub usage {
    print "
Usage:
    $0 ListOfUsers1.txt [ListOfUsers2.txt [...]]
    $0 < ListOfUsers.txt

The ListOfUsers can be passed as either a file argument or passed to STDIN.

The ListOfUsers must be tab-delimited with the following columns:
login   email   name

";
    exit 1;
}

foreach (@ARGV) {
    if ($_ =~ /^-h|--help$/) {
        usage();
    }
}

my $lines = 0;
my $users = 0;
my $usersAdded = 0;
while (<>) {
    chomp; # Remove the trailing new-line.
    my($login, $email, $name) = split(/\t/, $_);

    if ($login && $email && $name && $login =~ /[a-z0-9]+/ &&  $email =~ /[a-z0-9]+.*@.*[a-z0-9]+/ && $name =~ /[a-z]+/) {
        if (is_available_username($email)) {
            try {
                my $user = Bugzilla::User->create({
                    login_name    => $email,
                    realname      => $name,
                    cryptpassword => '*',
                    disable_mail  => 0,
                    extern_id     => $login
                });
                print "Account for " . $user->login . " was created.\n";
                $usersAdded++;
            } catch Error with {
                my $ex = shift;
                my $error = "Error: $ex";
                $error =~ s/\n|\r/ /g;
                print $error."\n";
            };
        }

        $users++;
    }
    $lines++;
    close (ARGV) if (eof);
}

if (!$lines) {
    print "No input lines given.\n\n";
    usage();
}

print "\n$lines lines evaluated, $users user records checked, $usersAdded users added.\n";

exit 0;

export_users.php

#!/usr/bin/env php
<?php
##########################################################
# This is a basic script to export users from an
# MS Active Directory via LDAP in the format required
# by create_users.pl.
#
# Authors:
#	Adam Franco (afranco@middlebury.edu)
#	Ian McBride (imcbride@middlebury.edu)
# Date:
#	2010-03-08
# URL:
#	http://www.adamfranco.com/archives/374
# License:
#   The contents of this file are subject to the Mozilla Public
#   License Version 1.1 (the "License"); you may not use this file
#   except in compliance with the License. You may obtain a copy of
#   the License at http://www.mozilla.org/MPL/
#
#   Software distributed under the License is distributed on an "AS
#   IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or
#   implied. See the License for the specific language governing
#   rights and limitations under the License.
##########################################################

$ldaphost = "ldap.example.com";
$ldapport = 389;
$ldapuser = "username";
$ldappass = "password";
$baseDN = "DC=example,DC=com";

$connection = ldap_connect($ldaphost, $ldapport);

if (!$connection) die();

if (ldap_set_option($connection, LDAP_OPT_PROTOCOL_VERSION,3) === FALSE) die();

if (ldap_set_option($connection, LDAP_OPT_REFERRALS,0) === FALSE) die();

$bind = ldap_bind($connection, $ldapuser, $ldappass);

if (!$bind) die();

$filter = "(&(objectClass=User)(!(objectClass=Computer)))";

$search = ldap_search($connection, $baseDN, $filter, array("samaccountname", "mail", "givenname", "sn"));

$entries = ldap_get_entries($connection, $search);

print "samaccountname\temail\tname\n";

foreach($entries as $entry) {
  if(isset($entry['samaccountname'])) {
    print iconv('UTF-8', 'UTF-8//IGNORE', $entry['samaccountname'][0]);
  }
  print "\t";

  if(isset($entry['mail'])) {
    print iconv('UTF-8', 'UTF-8//IGNORE', $entry['mail'][0]);
  }
  print "\t";

  $name = '';
  if(isset($entry['givenname'])) {
    $name .= iconv('UTF-8', 'UTF-8//IGNORE', $entry['givenname'][0]);
  }  $name .= ' ';
  if(isset($entry['sn'])) {
    $name .= iconv('UTF-8', 'UTF-8//IGNORE', $entry['sn'][0]);
  }
  print trim($name);

  print "\n";
}

Example Usage

After unzipping the scripts in your Bugzilla directory you can use the create_users.pl script right away. To use export_users.php you will need to edit it and add your LDAP server configuration.
[root@hostname /var/www/htdocs/bugzilla/]# ./export_users.php | ./create_users.pl

If you’d rather import users from another source, simply create one or more tab-delimited text files that have the following columns:
login    email    name
[root@hostname /var/www/htdocs/bugzilla/]# ./create_users.pl users.txt otherusers.txt

You can pipe tab-delimited data to the script as well:
[root@hostname /var/www/htdocs/bugzilla/]# head -n 20 users.txt | ./create_users.pl

Update:

• Changed the license statement to the MPL be compatible with the rest of Bugzilla
• Changed the password to ‘*’ based on Max’s suggestion

While the mobile-only lifestyle has generally worked great for us over the years, it does have downsides that have become more apparent as our lifestyles have shifted to a more settled routine. Currently Sarah and I find ourselves generally splitting our time between work and home. At work we each have an office phone supplied, but we had only had our mobile phones at home. An unfortunately common occurrence was for one of us to come home and leave the mobile on silent/vibrate in a coat pocket and become unreachable. After a few incidents of being stranded, stood up, or not getting the message to pick up milk we decided that a home phone was needed — but were shocked to find that a local-only land-line would run us $40 per month (about the same as a cell phone plan in this area).

We were in search of a solution that would allow us to have a phone ringing audibly at home, keep our mobile phones for mobile usage, and come in at less than $120/month (if not lower our bills). Our solution is shown in the diagram below. While it looks a bit complicated, it meets our goals, didn’t require any tricky setup, and comes in at a grand total of $50/month for maintaining two mobile phones and a home phone. It has the added benefits of a single number to reach each of us and Google’s snazzy transcribed-voice-mail service.

New phone system with Google Voice (click to enlarge)

The new home phone: Skype + a handset

The first piece of the puzzle was to purchase a handset (the IPEVO SO-20) that sits at home on our wireless network, signed in to my Skype account. This handset works just like the Skype-application on a desktop computer, but doesn’t require keeping a large computer on to make or receive Skype calls. In addition to making free Skype-to-Skype calls, Skype also offers services for making calls from your Skype client to normal telephone numbers (known as “Skype-Out“) as well as a service which provides you with a telephone number that will ring your Skype client (known as “Skype In“). Skype-Out charges a minimal 2-cents/minute for calls to most of the world and maintaining the Skype-In number costs $3/month with no charge for talk-time.

We’ve been using the Skype phone for a few months now and have been very pleased with it. We notice a 1-1.5 second delay in hearing the caller when we first answer a call. This was a little confusing at first and resulted in a lot of “Hello? Hello? Can you hear me?” back-and-forth with the caller, but the delay is only at connection time and saying “Hello?” and then just pausing for a moment gives the call time to connect fully. Once in a call, the audio quality is generally a bit better than my mobile phone.

Routing calls with Google Voice

With the Skype-phone in place we now had a number that would reliably ring at home and costs us less than $10/month for a few hours of incoming and outgoing calls to anywhere in the world. Now the question is: How do we get people to call us on the Skype-phone rather than our mobile phones? Enter (from stage left) Google Voice.

Google Voice (from here out referred to as “GV”) is at its heart a phone-number forwarding service. The basic idea is that you get a GV phone number and then in your account settings, configure it to forward incoming calls to one or more other phone numbers. When a call comes in, all of your phones ring at the same time (this can be quite shocking if you have them in close proximity) and you pick up whichever one is at hand (and doesn’t incur a usage fee if you want to avoid that). Once you’ve picked up one phone the others stop ringing and you talk away.

I have my GV set up to ring three phones, my mobile number, our Skype-In number, and my work number. Since I spend the majority of my time either at work or home, most of the time I pick up calls at one of those two places. This cuts my mobile phone usage to only a few days per week, opening up other options for cutting costs.

Prepaid mobile + minimal usage = savings

Another driver for this entire phone-system change was that Unicel’s network in Vermont was recently sold to AT&T. After some bad customer-service experiences with Verizon I switched to Unicel in 2007 and was very happy with their service. In particular, they used unlocked GSM phones and didn’t charge for incoming calls or text messages, all for $35/month. With the sale to AT&T I was looking at an increase to $40/month for the minimal plan plus airtime usage for incoming calls.

With the Skype-phone in place and GV forwarding calls to all numbers, our mobile-phone usage wasn’t as high, allowing us to try some other options. Rather than signing up for a new AT&T contract, I instead kept the unlocked phone I used with Unicel and went with a prepaid (“GoPhone“) plan from AT&T. Rather than paying a monthly fee, I pre-pay on my account and then only have my account balance debited when I use the phone. I’m currently using the version of the plan where I pay $1/day on days that I use the phone, plus 10-cents/minute. While this sounds like it would add up, with GV routing calls to my other numbers I’ve averaged $16/month in mobile charges for the past two months. Also, unlike the monthly phone contract this has the potential to get much lower as more friends and family learn of my GV number and stop calling my mobile directly.

All said and done

From a pure cost perspective this telephony setup has been a big success. From two cell phones at $40/month each for a total of $80/month (with additional for a home phone); we’ve now gone to $3/month for the Skype-In number with ~$3/month of Skype-out calls from home, plus about $16/month each in mobile phone charges leaves us with a new total of a bit under $40/month. We had the additional $140 up-front cost for the IPEVO Skype-phone, but amortized over a year that still leaves us at about $50/month, with the potential to drop costs further if our cell-phone usage drops.

The non-monetary benefits are certainly harder to quantify. The biggest benefit I find is the increased control over my phone environment. For example, I could swap out the Skype-phone for something else (or get rid of it entirely) and no callers would know the difference. Once my contacts are all using my GV number, the same is true of my mobile phone.

Other features of GV such as voicemail transcription, caller filtering, scheduling of times when each phone should ring, and free SMS sending are all pretty neat too, but I haven’t yet made heavy use of them.

Now for the downsides:

• Complexity: While I find the increased flexibility valuable and none of the steps are challenging, others may find the whole thing not worth the hassle to set up.
• A new number: While I now have one number that will ring all of my phones, Google currently doesn’t support transferring existing numbers to their service. I’m now trying to wean friends and family off of the mobile number I’ve had for 7 years.
• Apparently some have found that using GV causes delays or other audio degradation. I haven’t noticed this myself.
• One more thing relying on Google. Since all of the phone companies hosted NSA warrentless-wire-tapping computers in their data-centers, I’m not particularly worried about Google having my calling data as well. That said, I’m relying on them to stick around for my email, searching, RSS reading, spreadsheets, and now call-routing.
• Users don’t see my GV number in the caller id. You can make calls with GV so that the person you are calling sees your GV number in the caller-id, but this requires either initiating the call from the GV website (your phone rings first), or dialing your GV number, then from there initiating the call. I find this to be too much hassle so I never bother

One final note: If you are a friend, family, or colleague who I missed in my number-update-email, let me know and I’ll send you my new GV number.

Single Machine Configuration

Pull out the database, use multiple web-servers

The two main components of Drupal (and most similar web applications) are the webserver, which handles PHP execution and file-serving; and the MySQL database, which stores all data with the exception of uploaded files. By putting the database on a separate machine we can can have multiple machines acting as front-end web-servers, both of them reading and writing to the same database. In this way, it doesn’t matter which web-server handles a given request as they will both get the same information out of the database. With two or more web-servers, our platform gains some redundancy since one web-server can fail while the second keeps handling requests.

With both web-servers point at the same database server, the database server still remains a single point of failure. Database clustering can alleviate this problem, but will be the subject of a future post.

Multiple web-server challenges

This redundancy does come at a cost in complexity however, since we need to ensure that any uploaded files are available on both web-servers. There seem to be two primary ways of tackling this problem (without resorting to costly and complex distributed file-system tools). The first is use rsync to copy files between the web-servers every few minutes.

Two web servers with rsync

• Files cannot be deleted in the sync as newly-added files will exist on only one web-server. Since the sync is two-way, there is no way for the rsync processes to tell the difference between a new file and a deleted file.
• Requests that come to the “other” web-server will not be able to access new files until the sync happens.
• If additional web-servers are added, the sync process needs to be updated on every existing web-server to include the new web-server

The other alternative is to store uploaded files on a separate file-server, whose upload directory is mounted on each web-server using NFS. This method eliminates the synchronization problems, since all web-servers are essentially writing to the same directory.

Two web servers with NFS

Best of both worlds

In order to better solve this problem, the approach we took is to go the NFS route, but augment it with a backup copy of the files stored on the local file-system of each web-server. Every ten minutes or so a script (sync_files.sh) runs that checks to see if the shared NFS directory is available, and if so syncs the uploaded-files to a backup location on the web-server’s file-system. This backup copy has its permissions set so that the Apache process cannot write to it, preventing synchronization problems if the shared NFS directory goes offline and we need to serve files out of the backup copy.

Two web servers with NFS and local backup copies.

An important consideration in this setup is that the NFS share is mounted in ‘soft’ mode so that file-access errors will time out quickly and allow for a timely switch-over to our backup files.

files.example.edu:/images       /mnt/files     nfs     soft    0 0

If the default ‘hard’ NFS mount is used, the check_link processes will hang indefinitely while trying to communicate with the file-server and never switch to our backup files.

Here is an example layout on the web-server to accomplish this setup:

# The scripts that will be run by cron:
/usr/local/bin/check_link.sh  # Run every minute
/usr/local/bin/sync_files.sh   # Run every 10 minutes

# The mounted NFS share:
/mnt/files/

# The backup copy of files:
/srv/files_read_only/

# The 'files' symbolic link, pointing normally at the NFS share:
/srv/files/ => /mnt/files/
# On NFS failure, this link will be switched to the backup directory:
/srv/files/ => /srv/files_read_only/

# The Drupal code directory:
/srv/drupal/
# The files directory for a site is a link into the switched files link
/srv/drupal/sites/www.example.com/files/ => /srv/files/www.example.com/files/

By mounting the shared NFS directory, keeping a read-only local copy of the files, and monitoring the state of the NFS directory we gain the following benefits:

• No problems with synchronization as all web-servers share the same remote filesystem.
• Synchronization of the local backup copies is not a problem as this is always a one-way sync rather than a two-way sync between different web-servers.
• While the NFS file-server is still a single point of failure, read access to the uploaded files (via the backup copy) will be restored after a maximum of one minute plus the NFS time-out (2 minutes by default for ‘soft’ mounts).
• The web-servers don’t need to know about each other, easing configuration if additional web-servers are added.

This configuration adds an extra machine to the platform mix and a bit of complexity, but it makes normal operation robust (instant file availability to all web-servers) and allows for graceful degradation (file-access becomes read-only) if the file-server goes down.

Many thanks to our system administrator, Mark Pyfrom, for all of his help in developing and testing this platform.

* Update on 2009-09-10: added note about ‘soft’ NFS mounts and an example file-system layout.

As a tribute to This American Life, I present you with an essay in three acts. In Act One we’ll see my older backup system and how it saved me, yet left me wanting more; Act Two discusses a philosophy of backups appropriate for the moderately paranoid; and in Act Three we’ll go step-by-step through the process of implementing that philosophy.

Act One: A Complex Old (But Successful) Backup System

For the past four years or so I’ve been successfully running automated backups of my Mac laptop using rdiff-backup to send incremental backups over the network to a Linux file-server. These backups occurred every day at 12:30 and every night at 2am when there was a fast-enough network connection. This solution was a bit complex to set up — I wrote my own scripts for testing network connection speed so that it wouldn’t start backing up over dial-up or other slow connections — but it did the job of ensuring that I always had a good copy of my data.

In late 2007 Apple released OS X 10.5 (Leopard) with its built-in Time Machine backup system. As I am a bit paranoid about my data, I set up Time Machine to back up to a 150GB external Firewire drive. Since the external drive was smaller than my laptop’s drive I was only able to back up my user directory, not the whole system. The rdiff-backup system kept running in the background as well.

Fast-forward to June 2009 and my current laptop hard drive has died and gone to meet its maker: Apple (via AppleCare). Incidentally I got a great repair with the display (dead pixels), keyboard, aluminum chassis (deformed), and battery (swelling) all replaced in addition to the faulty drive. Almost a new computer! Now time to restore my data.

Since I didn’t have a full-system Time Machine backup, I couldn’t use the one-click-restore option to get my system back so my recovery process was:

1. Do a fresh install of OS X
2. Restore my user account from my Time Machine backup
3. Restore my applications and other system utilities from my rdiff-backup backups

While this process worked out and I wasn’t in danger of loosing any important data, I did end up wasting an entire weekend getting the whole system back up to snuff, copying over pieces I missed, reinstalling plugins, etc. I wished that I had a full-system Time Machine backup that would have allowed the single-click method to restore my machine. Once I got the laptop back up and running I decided it was time to set up a more robust backup solution that would be easier to restore from and provide a similar or greater amount of protection.

Act Two: A Backup Philosophy For The Moderately Paranoid

If you aren’t doing backups of your computer, please do! Even going the most basic route and plugging in a single external drive and choosing it to be used by Time Machine is an immense improvement over not having any backups. In the past 10 years I’ve had 6 hard drives die on me, making them (and batteries) the most likely parts of a modern computer to fail. Backups are critically important to keeping you safe when those failures happen. Time Machine and other similar programs incrementally back up your data every few hours so that only the files that have recently changed need to be transferred to the backup drive. This allows you to use backup drives that are only a bit bigger than your primary drive as well as to view differences in files over time. Incremental backups are pretty neat and much faster and more space-efficient than duplicating the entire drive every once and a while. Faster and more efficient also mean that there is less penalty (in time and cost) for doing backups, so they happen more frequently, making it much more likely that you have a recent backup when your computer suddenly goes on the fritz.

Unfortunately, there are a few situations that a single backup drive won’t help with:

• Lightning strikes while your backup drive is plugged into your computer, frying both the computer and the backup drive
• Your house burns down, gets hit by a tornado, or is robbed and both the laptop and the drive are gone

The best way to prevent these and similar situations is to have backups stored in a place where your laptop is not. While you can do this by sending your data over the internet to a remote file-server, current DSL upload speeds aren’t fast enough to make this a sure-thing. Another option is to keep an external drive where your computer is not, the catch however is that an external drive needs to be where your computer is in order to get data. My solution to this problem: two backup drives, one in my desk at work, one at home. Since my laptop is either with (or without) me at home, with me at work, or with me somewhere else I always have a copy of my data in a location remote to my laptop.

Though I trust my office-mates, leaving a drive in my desk unattended does open up another possibility of data theft by intruders and a corresponding increase in exposure to it being misused for identity theft. I’m pretty good at using encrypted disk-images to keep my tax documents and similar things safe, but I’d rather not have some else rifling through the rest of my digital life. While it requires a few work-arounds to set up initially, Time Machine will back up to an encrypted disk-image on a USB/Firewire drive, giving you a backup that nothing except your laptop can access without entering your password.

To recap, the primary tenets of this philosophy are:

1. Make backups!
2. Make multiple backups and keep them in different physical locations to mitigate against catastrophies
3. Encrypt your backups to keep prying eyes away when you can’t fully prevent physical access to them by others

A note to the fully paranoid: The safety of your data can be significantly increased (at a significant expense and hassle) by also backing up to a third drive, then periodically rotating one of your backup drives into a safe-deposit box in your bank. Periodically mailing a drive to a friend or family member far away is another option.

Act Three: Step By Step Into Mutiple Encrypted Time Machine Backups

Without further ado, here is how to set this up.

1. Purchase or acquire two USB or Firewire external hard-drives, preferably larger than the drive you want to back up. They may not need to be identical, but mine are. I purchased two Cirago 320GB USB drives since they were cheap ($66 each), big enough, and don’t require a wall-wart for power.
2. Time Machine won’t let you choose a disk image directly as a backup destination, but it does create images for backing up to network drives. We have to jump through a few hoops to get an encrypted image created and onto our external drive. (based on these instructions)
   1. Open the Terminal and run the following command to allow Time Machine to back up to non-TimeCapsule(TM) network drives:
      defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1
   2. Connect to another computer over the network in Finder using “Go” –> “Connect to Server”. I happened to just connect to my desktop at work, but it can be any computer. We won’t be actually backing up there, just getting Time Machine to initialize it’s backup image.
   3. Open the Time Machine preferences and choose to back up to the network drive you just connected to. Time Machine will do a lot of “Preparing for Backup”; once it starts transferring data, just stop the backup.
   4. Time Machine will have created a disk image on your network drive in which it was beginning to back up to. This disk image will be called something like “ComputerName_01a8b93325acf.sparsebundle”. Copy that disk image onto your external hard drive and delete it from the network drive.

   You are now done with the network drive.

3. The disk image is not encrypted yet, so we must do that. Open the Terminal and cd to your external drive (where the disk image should be, the disk image itself shouldn’t be mounted), copy the disk image to a temporary name, encrypt it, then delete the temporary version:

   cd /Volumes/BackupDrive/
   cp ComputerName_01a8b93325acf.sparsebundle ComputerName_01a8b93325acf.sparsebundle-temp
   hdiutil convert -format UDSB -o ComputerName_01a8b93325acf.sparsebundle  -encryption AES-256 ComputerName_01a8b93325acf.sparsebundle-temp
   rm ComputerName_01a8b93325acf.sparsebundle-temp
   

4. You should now have your external drive with the encrypted disk image on it. Now you need to make sure that the System keychain has the password for the encrypted image so that it can back up automatically without prompting you for a password every 10 minutes. Mount the disk image, enter your password, and check the “Save Password in Keychange” box. Then, open the Keychain Access utility and search for ‘sparsebundle’. Right-click on the keychain item and paste it into the “System” keychain.
5. Unmount the disk image.
6. Open Time Machine Preferences and select your external drive as the backup destination. Time Machine is smart enough to use the encrypted image if it sees it because of the disk image name and possibly other file metadata. Time Machine should mount the disk image without prompting for a password and back up to it. This will take a long time.

At this point you should now have Time Machine automatically backing up to an encrypted disk image on one external disk. You can test that the disk image is encrypted by trying to mount it on another machine. It should require a password.

Time Machine keeps its last-backup-position information on the backup drive itself and will happily back up incrementally from the point it left off on that drive no matter how many other drives you back up to in between. Unfortunately the default behavior of the Time Machine preferences requires you to open the preferences and select the new backup drive every time you want to switch. To get around this and have both drives work automatically whenever they are plugged in we need to make the second drive a clone of the first one. That way, Time Machine won’t be able to tell them apart and will back up to whichever one is plugged in.

To clone the drive, plug in both drives and use Disk Utility’s ‘Restore’ tab to restore from the drive with the backup image to your empty second drive. Once the drive is cloned, either one plugged in individually should be automatically used by Time Machine for backups.

Caveats:

• Some people have run into an issue when backing up to disk-images on a network share where the disk fills up and rather than just deleting the oldest few backups, all but the latest backup versions are deleted. It is unknown whether or not this will happen with disk images on USB/Firewire drives.
• While I can browse the backup disk image and see all of the incremental backup versions, this history is not browsable through the Time Machine history-browser user-interface.

References and more help:

• Mac OS X Hints – 10.5: Ease Time Machine networked backups
• Mac OS X Hints – Encrypting Time Machine Backup?

After about 5 evenings of documentation and tutorial reading intermixed with a bit of programming I’m about 1/3 of the way through my first real Rails application and am beginning to have a few thoughts on Ruby, Rails, and the process of learning new languages and frameworks:

Languages are Easy…
This might have something to do with how my brain works, but I find learning new programming languages quite easy and extremely interesting. Some of the things I find really neat about the Ruby language:

• Full-on object-oriented — I really like the consistency
• Blocks and Iterators — I fell in love with blocks in SmallTalk, they are a great concept and so much easier to use than named callback functions
• Method names can end in equals such as name=(newName), parentheses are optional, and spaces seem to be allowed in method names that include ‘=’, so attribute-setting methods can look syntactically identical to variable assignment operators. Example: adam.name=('Adam') is equivalent to adam.name = ('Adam') and is equivalent to adam.name = 'Adam'. Sweet syntactical sugar.
• All instance variables are private. Always. The snazzy setter/getter syntax can spoof the idea of public attributes while still retaining actual methods to do the work. (I never got the point of public attributes other than as a way to avoid other languages’ painful getter/setter syntactical requirements.)
• Classes are never closed — can always add a method
• Single-inheritance prevents ambiguity in the class-hierarchy, but there is this concept of ‘Mix-ins’ where the un-closed nature of classes (I think) allows for a class to obtain method implementations from another class without inheriting from it. It sounds interesting, but I haven’t played with it yet

The Ruby language may have some blemishes (and apparently a slow interpreter), but it seems like a pretty nice language in general and pretty devoid of major syntactical or conceptual warts.

…Frameworks are hard
While I feel like I can [mostly] get my head around Ruby in a few days, Rails is another matter. Rails follows a few architectural concepts very strictly: convention over configuration, DRY, MVC, etc. One of the problems (at least with the tutorials and documentation on the Rails website) is that few of the convention have descriptions on how they should be applied and the documentation is scattered and can be hard to follow. A lot of magic happens behind the scenes when things are named a certain way and it can be hard to sort out what’s up when something goes wrong.

One problem I ran into that took me forever to debug was that by naming one of my classes Connection and using it in a has_many relationship I collided with something internal to ActiveRecord which caused an indecipherable error when trying to save my object. Connection is not one of the reserved words listed for Rails.

It may just be that I need to find a good book for Rails, but the official online documentation seems to be rather sparse, disorganized, and/or scattered. I don’t mean to complain too much and I believe that Rails will become much easier to use after I have a bit more time in the saddle, but it is a complex system that is not easy to learn a bit at a time.

I don’t want to dwell on on comparisons, but I must note that I have been very spoiled by PHP’s excellent official documentation at PHP.net and for the Zend Framework. PHP has more warts than, well, something with a lot of warts, (it has many, many warts), and doesn’t begin to compare to Ruby in syntactical elegance, but good documentation is also a valuable thing.

With all this said, I now have only 5 evenings in Ruby on Rails and I plan to give it at least twice that more. I hope by the end of this first project I’ll be comfortable enough with it to use it on other small to medium-sized applications with relative speed and ease.

One thing that often happens to me though, is that I work for about a half hour to an hour trying to get a new piece of code working and in the process make several sets of changes to one file that are only loosely related.

Let’s say that I am fixing a bug in my ‘MediaLibrary’ class and while doing so notice some some spelling mistakes in some comments that I fix. Now my one file has two changes my bug fix, and the spelling fix. Rather than committing both changes together with one comment describing both changes, I can highlight one of the changes in git-gui and select the “Stage Hunk for Commit” option.

With that one hunk staged I can now commit with a message applicable to that change. Other changes can then be staged and committed with their own messages resulting in a very understandable history of changes.

“Stage Hunk for Commit” can also be used to commit important changes while not including debugging lines inserted in your code.

I wanted to export all 600+ of my tweets, so I wrote the following little php script to accomplish this. I have not yet tested it with many concurrent users or added a form to select which user to update. Until I do so, I won’t be providing it as an end-user service. You are free to put it on your own machine and use it though.

TwitterExport.php

<?php
/**
 * This script will allow the export of complete user time-lines from the twitter
 * service. It joins together all pages of status updates into one large XML block
 * that can then be reformatted/processed with other tools.
 *
 * @since 10/13/08
 *
 * @copyright Copyright © 2008, Adam Franco
 * @license http://www.gnu.org/copyleft/gpl.html GNU General Public License (GPL)
 */

$user = 'afranco_work';	// Replace this with your user name.

header('Content-type: text/plain');

$allDoc = new DOMDocument;
$root = $allDoc->appendChild($allDoc->createElement('statuses'));
$root->setAttribute('type', 'array');

$page = 1;
do {
	$numStatus = 0;

	$pageDoc = new DOMDocument;
	$res = @$pageDoc->load('http://twitter.com/statuses/user_timeline/'.$user.'.xml?page='.$page);
	if (!$res) {
		print "\n\n**** Error loading page $page ****";
		exit;
	}
	foreach ($pageDoc->getElementsByTagName('status') as $status) {
		$root->appendChild($allDoc->createTextNode("\n"));
		$root->appendChild($allDoc->importNode($status, true));
		$numStatus++;
	}

	print "\nLoaded page $page with $numStatus status updates.";
	flush();

	$page ++;
	sleep(1);

} while ($numStatus);

print "\nDone loading timeline.";
print "\n\n\n";

$root->appendChild($allDoc->createTextNode("\n"));
print $allDoc->saveXml();

Usage (assuming PHP is installed)

1. Save the code above on your machine as twitter_export.php
2. Edit the code to change the $user variable to be your own Twitter username
3. From the command line run php twitter_export.php
4. Copy/paste the XML output into a file for safe keeping and further processing

The problem it turns out, is that while one of my video cards (that drives my two primary monitors ) is a high-end ATI Radeon X1900XT that supports transparent menu bars, the second (that drives my two outer monitors used mostly for notes and email) is a cheaper NVIDIA GeForce 7300 GT. The presence of the lower-end card in the system was causing the “Translucent Menu Bar” option to be hidden, even though it was supported by my other card.

Steps to fix:

1. Turn off computer
2. Remove low-end video card
3. Start computer
4. Go to System Preferences –> Desktop & Screen Saver and un-check the “Translucent Menu Bar” option
5. Turn off computer
6. Replace low-end video card
7. Start computer

It is a pain in the butt, but it works to get the damn translucency turned off.

The RiverLevels widget provides an easy way to monitor the amount of water flowing in your favorite streams and rivers right from your Dashboard. The RiverLevels widget is of particular interest to whitewater kayakers and canoeists.

Once any United States Geological Survey (USGS) stream-gauge station is selected, it is automatically refreshed to always provide you with the latest graph of the water-level. As of version 1.2 you can choose between two graph styles: discharge in cubic feet per second (CFS) and water-height in feet.

This widget is Free software, licensed under the GNU General Public License (GPL) version 3 or later.

• Download

• More Info
• Apple’s Download page for RiverLevels.wdgt
• Bug Tracker

Requirements:

• OS X – 10.4 “Tiger” or later

Change Log:

1.2.1 (2008-02-10)

• New zip archive includes the ‘library’ directory missing in the 1.2 release.

1.2 (2008-02-06)

• Fixed Leopard (10.5) compatability bug.
• Added the ability to choose Gauge Height (ft) in addition to discharge (CFS).

1.1 (2007-01-08)

• Fixed graphs extending off bottom of widget
• Fixed invisibility of front refresh icon