Archive for the 'Computers and Technology' Category  

Tips, software, instructions, and thoughts on all things tech.

New Curvature site

November 17th, 2016

Filed under: Computers and Technology , Software

Tags: ,

Several years after creating Curvature –my program that analyzes road-geometry and builds maps of twisty roads– it now has a dedicated site of its own:

roadcurvature.com

The new site is written to help non-techies understand how to use the curvature files with step-by-step instructions and a lot less jargon than in my original post about the program.

I also took this as an opportunity to learn how to do vector graphics in Inkscape and design myself a new logo — with significant design help from Alison.

Regex from the dark lagoon

November 13th, 2013

Filed under: Computers and Technology , Work/Professional

Tags: , , ,

As a software developer or system admin have you ever encountered regular expressions that are just a bit too hard to understand? Kind of frustrating, right? As a rule, regular expressions are often relatively easy to write, but pretty hard to read even if you know what they are supposed to do. Then there is this bugger:

/^(?:([0-9]{4})[\-\/:]?(?:((?:0[1-9])|(?:1[0-2]))[\-\/:]?(?:((?:0[1-9])|(?:(?:1|2)[0-9])
|(?:3[0-1]))[\sT]?(?:((?:[0-1][0-9])|(?:2[0-4]))(?::?([0-5][0-9])?(?::?([0-5][0-9](?:\.[0-9]+)?)?
(Z|(?:([+\-])((?:[0-1][0-9])|(?:2[0-4])):?([0-5][0-9])?))?)?)?)?)?)?)?$/x

This is the most complex regex I’ve ever had need to write and I just had to share. Can you guess what it might do? 😉

Continue Reading »

curvature.py — find the most twisty-turny roads around

December 5th, 2012

Filed under: Computers and Technology , Software

Tags: , , , , , , , ,

Update November, 2016: New dedicated Curvature site — roadcurvature.com

Update October, 2013: Google Earth KML files generated by curvature.py are now available covering the entire world.

In the process of taking up motorcycling this summer I also gained an additional hobby: scouring maps and travel guides to find the roads that would be most fun to ride. While I’ve had great times on dirt roads through farmland and wide open highways, there just isn’t anything that compares to the thrill of leaning through the corners on a winding road.

While I’ve had some good successes in locating roads by map (such as Tracy Road), one of the shortcomings of a map is the tight curves you can really lean into tend to be below the resolution for many maps. Atlases and electronic maps like Google Earth allow you to zoom in, but then there is the problem of finding the gems in the sea of data. What I realized I needed was a way to highlight just the most curvy roads so that I would know where to explore next.
Continue Reading »

River Levels Widget v.1.2.2 available

March 23rd, 2011

Filed under: Computers and Technology , Software

Tags: ,

RiverLevels 1.0 Screen Shot

The RiverLevels widget provides an easy way to monitor the amount of water flowing in your favorite streams and rivers right from your Dashboard. The RiverLevels widget is of particular interest to whitewater kayakers and canoeists.

Once any United States Geological Survey (USGS) stream-gauge station is selected, it is automatically refreshed to always provide you with the latest graph of the water-level. As of version 1.2 you can choose between two graph styles: discharge in cubic feet per second (CFS) and water-height in feet.

This widget is Free software, licensed under the GNU General Public License (GPL) version 3 or later.

Requirements:

  • OS X – 10.4 “Tiger” or later

Change Log:
1.2.2 (2011-03-23)

  • Fix for image URL change in USGS site.

1.2.1 (2008-02-10)

  • New zip archive includes the ‘library’ directory missing in the 1.2 release.

1.2 (2008-02-06)

  • Fixed Leopard (10.5) compatability bug.
  • Added the ability to choose Gauge Height (ft) in addition to discharge (CFS).

1.1 (2007-01-08)

  • Fixed graphs extending off bottom of widget
  • Fixed invisibility of front refresh icon

Git Tip: Grouping feature-branch commits when merging.

December 12th, 2010

Filed under: Computers and Technology , Work/Professional

Tags: , ,

Let’s say you are working on a large feature or update that requires a bunch of commits to complete. You finish up with your work and are then ready to merge it onto your master branch.

For example, here is the history of my drupal repository after some work updating the cas module to the latest version (and to support the new version of phpCAS):

As you can see, I have a number of commits, followed by a merge in with the new module code, followed by some more commits.

Now, if I merge my feature branch (master-cas3-simple) into the master via

git merge  master-cas3-simple

then the history will look like this:

While the history is all there, it isn’t obvious that all of the commits beyond “Convert MS Word quote…” are a single unit of work. They all kind of blend together because git performed a “fast-forward” commit. Usually fast-forward commits are helpful since they keep the history from being cluttered with hundreds of unnecessary merge commits, but in this case we are loosing the context of these commits being a unit of work.

To preserve the grouping of these commits together I can instead force the merge operation to create a merge commit (and even append a message) by using the --no-ff option to git merge:

git merge --no-ff -m "Upgraded CAS support to to cas-6.x-3.x-dev and phpCAS 1.2.0 RC2.5" master-cas3-simple

This results in the history below:

As you can see, merging with the --no-ff option creates a merge commit which very obviously delineates work on this feature. If we decided that we wanted to roll back this feature it would be much easier to sort out where the starting point before the feature was.

Thanks to Vincent Driessen for turning me onto the utility of the the --no-ff merge option via his post “A successful Git branching model“.

Mirroring a Subversion repository on Github

December 5th, 2010

Filed under: Computers and Technology , Work/Professional

Tags: , , , ,

For the past few months I have been doing a lot of work on the phpCAS library, mostly to improve the community trunk of phpCAS so that I wouldn’t have to maintain our own custom fork with support for the CAS attribute format we use at Middlebury College. The phpCAS project lead, Joachim Fritschi, has been great to work with and I’ve had a blast helping out with the project.

The tooling has involved a few challenges however, since Jasig (the organization that hosts the CAS and phpCAS projects) uses Subversion for its source-code repositories and we use Git for all of our projects. Now, I could just suck it up and use Subversion when doing phpCAS development, but there are a few reasons I don’t:

  1. We make use of Git submodules to include phpCAS along with the source-code of our applications, necessitating the use of a public Git repository that includes phpCAS.
  2. The git-svn tools allow me to use git on my end to work with a Subversion repository, which is great because…
  3. I find that Git’s fast history browsing and searching make troubleshooting and bug fixing much easier than any other tools I’ve used.

For the past two years I have been using git-svn to work with the phpCAS repository and every so often pushing changes up to a public Git repository on GitHub. Our applications reference this repository as a submodule when they need to make use of phpCAS. Now that I’ve been doing more work on phpCAS (and am more interested in keeping our applications using up-to-date versions), I’ve decided to automate the process of mirroring the Subversion repository on GitHub. Read on for details of how I’ve set this up and the scripts for keeping the mirror in sync.

Continue Reading »

BASH tip: Top web pages

October 14th, 2010

Filed under: Computers and Technology , Work/Professional

Tags: , , ,

Here is a quick command to generate a list of the top pages in the Apache web-server’s access log:

gawk '{ print $7}' /var/log/httpd/access_log | sort | uniq -c | sort -nr | head -n 20

Parts of the command explained:

  1. gawk '{ print $7}' — return only the 7th [white-space delimited] column of text from the access log, which happens to be the path requested.
  2. sort — sort the lines of the output.
  3. uniq -c — condense the output to unique lines, prepending each line with the number of times that line occurs.
  4. sort -nr — sort the resulting lines numerically in reverse order.
  5. head -n 20 — chop off all but the first 20 lines.

The result should look something like this:

  83361 /
  49582 /feed
  39616 /robots.txt
  36265 /favicon.ico
  17048 /?feed=rss2
  10798 /archives/3
  10036 /wp-content/uploads/2007/05/img_7870_header.jpg
   9913 /wp-includes/images/smilies/icon_smile.gif
   9425 /wp-comments-post.php
   8274 /feed/
   7508 /archives/category/work/feed
   7367 /archives/88
   7312 /photos/10_small/IMG_3023.JPG.jpg
   7175 /photos/10_small/IMG_3028.JPG.jpg
   7151 /photos/10_small/IMG_3024.JPG.jpg
   7096 /photos/10_small/IMG_3026.JPG.jpg
   6381 /photosetToKML.php?set=72157594417350372&size=small
   6253 /qtvr/2007-04-05_back_deck_snow%20-%2010000x5000%20-%20SLIN%20-%20Blended%20Layer0002.jpg
   5798 /photosetToKML.php
   4344 /archives/category/photography

Adding reverse-proxy caching to PHP applications

June 14th, 2010

Filed under: Computers and Technology , Work/Professional

Tags: , , , ,

Note: This is a cross-post of documentation I am writing about Lazy Sessions.

Why use reverse-proxy caching?

For most public-facing web applications, the significant majority of their traffic is anonymous, non-authenticated users. Even with a variety of internal data-cache mechanisms and other good optimizations, a large amount of code execution goes into executing a PHP application to generate a page even if the content of this page will be the same for many users. Code and query optimization are very important to improving the experience for all users of a web application, but even the most basic “Hello World” script will top out at about 3k requests/second due to the overhead of Apache and PHP — many real applications top out at less than 200 requests/second. Varnish, a light-weight proxy-server that can run on the same host as the webserver, can cache pages in memory and can serve them at rates of more than 10k requests/second with thousands of concurrent connections.

While the point of web-applications is to have content be dynamic and easily changeable, for most applications and most of the anonymous users, receiving content that is slightly stale (cached for 5 minutes or something similar) isn’t a big deal. Sure, visitors to your blog might not see the latest post for a few minutes, but they will get their response in 4 milliseconds rather than 2 seconds.

Should your site get posted on Slashdot, a caching reverse-proxy server will give anonymous visitor #2 and up the same page from cache (until expiration), while authenticated users continue to have their requests passed through to the Apache/PHP back-end. Everyone wins.

Continue Reading »

Importing users into Bugzilla

March 8th, 2010

Filed under: Computers and Technology , Software , Work/Professional

Tags: , , ,

For the past 6 months our Web Application Development work-group has been Bugzilla as our issue tracker with quite a bit of success. While it has its warts, Bugzilla seems like a pretty decent issue-tracking system and is flexible enough to fit into a variety of different work-flows. One very important feature of Bugzilla is support for LDAP authentication. This enables any Middlebury College user to log in and report a bug using their standard campus credentials.

While LDAP authentication works great, there is one problem: If a person has never logged into our Bugzilla, we can’t add them to the CC list of an issue. This is important for us because issues usually don’t get submitted directly to the bug tracker, but rather come in via calls, emails, tweets, and face-to-face meetings. We are then left to submit issues to Bugzilla ourselves to keep track of our to-do items. Ideally we’d add the original reporter to the bug’s CC list so that they will automatically be notified as we make progress on the issue, but their Bugzilla account must exist before we can add them to the bug.

Searching about the internet I wasn’t able to find anything about how to import LDAP users (or any kind of users) into Bugzilla, though I was able to find some basic instructions on how to create a single user via Bugzilla’s Perl API. To improve on the lack of user-import support I’ve created an Perl script that creates users from lines in a tab-delimited text file (create_users.pl) as well as a companion PHP script that will export an appropriately-formatted list of users from an Active Directory (LDAP) server (export_users.php).

Continue Reading »

The future of phones: Google Voice, Skype, mobile, and more

November 8th, 2009

Filed under: Computers and Technology

Tags: , , ,

As members of the under-30 club, Sarah and I have come into adulthood in the age of mobile phones. I got my first cell phone right after college and Sarah has had hers since she was 14; neither of us has ever had a land-line of our own.

While the mobile-only lifestyle has generally worked great for us over the years, it does have downsides that have become more apparent as our lifestyles have shifted to a more settled routine. Currently Sarah and I find ourselves generally splitting our time between work and home. At work we each have an office phone supplied, but we had only had our mobile phones at home. An unfortunately common occurrence was for one of us to come home and leave the mobile on silent/vibrate in a coat pocket and become unreachable. After a few incidents of being stranded, stood up, or not getting the message to pick up milk we decided that a home phone was needed — but were shocked to find that a local-only land-line would run us $40 per month (about the same as a cell phone plan in this area).

We were in search of a solution that would allow us to have a phone ringing audibly at home, keep our mobile phones for mobile usage, and come in at less than $120/month (if not lower our bills). Our solution is shown in the diagram below. While it looks a bit complicated, it meets our goals, didn’t require any tricky setup, and comes in at a grand total of $50/month for maintaining two mobile phones and a home phone. It has the added benefits of a single number to reach each of us and Google’s snazzy transcribed-voice-mail service.

New phone system with Google Voice  (click to enlarge)

New phone system with Google Voice (click to enlarge)

The new home phone: Skype + a handset

The first piece of the puzzle was to purchase a handset (the IPEVO SO-20) that sits at home on our wireless network, signed in to my Skype account. This handset works just like the Skype-application on a desktop computer, but doesn’t require keeping a large computer on to make or receive Skype calls. In addition to making free Skype-to-Skype calls, Skype also offers services for making calls from your Skype client to normal telephone numbers (known as “Skype-Out“) as well as a service which provides you with a telephone number that will ring your Skype client (known as “Skype In“). Skype-Out charges a minimal 2-cents/minute for calls to most of the world and maintaining the Skype-In number costs $3/month with no charge for talk-time.

We’ve been using the Skype phone for a few months now and have been very pleased with it. We notice a 1-1.5 second delay in hearing the caller when we first answer a call. This was a little confusing at first and resulted in a lot of “Hello? Hello? Can you hear me?” back-and-forth with the caller, but the delay is only at connection time and saying “Hello?” and then just pausing for a moment gives the call time to connect fully. Once in a call, the audio quality is generally a bit better than my mobile phone.

Routing calls with Google Voice

With the Skype-phone in place we now had a number that would reliably ring at home and costs us less than $10/month for a few hours of incoming and outgoing calls to anywhere in the world. Now the question is: How do we get people to call us on the Skype-phone rather than our mobile phones? Enter (from stage left) Google Voice.

Google Voice (from here out referred to as “GV”) is at its heart a phone-number forwarding service. The basic idea is that you get a GV phone number and then in your account settings, configure it to forward incoming calls to one or more other phone numbers. When a call comes in, all of your phones ring at the same time (this can be quite shocking if you have them in close proximity) and you pick up whichever one is at hand (and doesn’t incur a usage fee if you want to avoid that). Once you’ve picked up one phone the others stop ringing and you talk away.

I have my GV set up to ring three phones, my mobile number, our Skype-In number, and my work number. Since I spend the majority of my time either at work or home, most of the time I pick up calls at one of those two places. This cuts my mobile phone usage to only a few days per week, opening up other options for cutting costs.

Prepaid mobile + minimal usage = savings

Another driver for this entire phone-system change was that Unicel’s network in Vermont was recently sold to AT&T. After some bad customer-service experiences with Verizon I switched to Unicel in 2007 and was very happy with their service. In particular, they used unlocked GSM phones and didn’t charge for incoming calls or text messages, all for $35/month. With the sale to AT&T I was looking at an increase to $40/month for the minimal plan plus airtime usage for incoming calls.

With the Skype-phone in place and GV forwarding calls to all numbers, our mobile-phone usage wasn’t as high, allowing us to try some other options. Rather than signing up for a new AT&T contract, I instead kept the unlocked phone I used with Unicel and went with a prepaid (“GoPhone“) plan from AT&T. Rather than paying a monthly fee, I pre-pay on my account and then only have my account balance debited when I use the phone. I’m currently using the version of the plan where I pay $1/day on days that I use the phone, plus 10-cents/minute. While this sounds like it would add up, with GV routing calls to my other numbers I’ve averaged $16/month in mobile charges for the past two months. Also, unlike the monthly phone contract this has the potential to get much lower as more friends and family learn of my GV number and stop calling my mobile directly.

All said and done

From a pure cost perspective this telephony setup has been a big success. From two cell phones at $40/month each for a total of $80/month (with additional for a home phone); we’ve now gone to $3/month for the Skype-In number with ~$3/month of Skype-out calls from home, plus about $16/month each in mobile phone charges leaves us with a new total of a bit under $40/month. We had the additional $140 up-front cost for the IPEVO Skype-phone, but amortized over a year that still leaves us at about $50/month, with the potential to drop costs further if our cell-phone usage drops.

The non-monetary benefits are certainly harder to quantify. The biggest benefit I find is the increased control over my phone environment. For example, I could swap out the Skype-phone for something else (or get rid of it entirely) and no callers would know the difference. Once my contacts are all using my GV number, the same is true of my mobile phone.

Other features of GV such as voicemail transcription, caller filtering, scheduling of times when each phone should ring, and free SMS sending are all pretty neat too, but I haven’t yet made heavy use of them.

Now for the downsides:

  • Complexity: While I find the increased flexibility valuable and none of the steps are challenging, others may find the whole thing not worth the hassle to set up.
  • A new number: While I now have one number that will ring all of my phones, Google currently doesn’t support transferring existing numbers to their service. I’m now trying to wean friends and family off of the mobile number I’ve had for 7 years.
  • Apparently some have found that using GV causes delays or other audio degradation. I haven’t noticed this myself.
  • One more thing relying on Google. Since all of the phone companies hosted NSA warrentless-wire-tapping computers in their data-centers, I’m not particularly worried about Google having my calling data as well. That said, I’m relying on them to stick around for my email, searching, RSS reading, spreadsheets, and now call-routing.
  • Users don’t see my GV number in the caller id. You can make calls with GV so that the person you are calling sees your GV number in the caller-id, but this requires either initiating the call from the GV website (your phone rings first), or dialing your GV number, then from there initiating the call. I find this to be too much hassle so I never bother

One final note: If you are a friend, family, or colleague who I missed in my number-update-email, let me know and I’ll send you my new GV number. 🙂

Next »