While LDAP authentication works great, there is one problem: If a person has never logged into our Bugzilla, we can’t add them to the CC list of an issue. This is important for us because issues usually don’t get submitted directly to the bug tracker, but rather come in via calls, emails, tweets, and face-to-face meetings. We are then left to submit issues to Bugzilla ourselves to keep track of our to-do items. Ideally we’d add the original reporter to the bug’s CC list so that they will automatically be notified as we make progress on the issue, but their Bugzilla account must exist before we can add them to the bug.

Searching about the internet I wasn’t able to find anything about how to import LDAP users (or any kind of users) into Bugzilla, though I was able to find some basic instructions on how to create a single user via Bugzilla’s Perl API. To improve on the lack of user-import support I’ve created an Perl script that creates users from lines in a tab-delimited text file (create_users.pl) as well as a companion PHP script that will export an appropriately-formatted list of users from an Active Directory (LDAP) server (export_users.php).

BugzillaImport.zip — Unzip in your Bugzilla directory, run via the command line. See below for examples.

File Listings:

create_users.pl

This script can safely be run repeatedly. Only new users not already in Bugzilla will be added, users matching existing email addresses will be skipped.

#!/usr/bin/env perl
##########################################################
# This is a basic script to import users into Bugzilla.
#
# Users can be imported from tab-delimited text files or
# tab-delimited lines piped to STDIN. Lines should have 3
# columns: login	email	name
#
#
# Author:
#	Adam Franco (afranco@middlebury.edu)
# Date:
#	2010-03-08
# URL:
#	http://www.adamfranco.com/archives/374
# License:
#   The contents of this file are subject to the Mozilla Public
#   License Version 1.1 (the "License"); you may not use this file
#   except in compliance with the License. You may obtain a copy of
#   the License at http://www.mozilla.org/MPL/
#
#   Software distributed under the License is distributed on an "AS
#   IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or
#   implied. See the License for the specific language governing
#   rights and limitations under the License.
##########################################################

use FindBin qw($Bin);
BEGIN {
    push @INC,$Bin;
    push @INC,$Bin."/lib";
    push @INC,$Bin."/lib/x86_64-linux-thread-multi";
}
use Bugzilla;
use Bugzilla::User;
use Error qw(:try);

sub usage {
    print "
Usage:
    $0 ListOfUsers1.txt [ListOfUsers2.txt [...]]
    $0 < ListOfUsers.txt

The ListOfUsers can be passed as either a file argument or passed to STDIN.

The ListOfUsers must be tab-delimited with the following columns:
login   email   name

";
    exit 1;
}

foreach (@ARGV) {
    if ($_ =~ /^-h|--help$/) {
        usage();
    }
}

my $lines = 0;
my $users = 0;
my $usersAdded = 0;
while (<>) {
    chomp; # Remove the trailing new-line.
    my($login, $email, $name) = split(/\t/, $_);

    if ($login && $email && $name && $login =~ /[a-z0-9]+/ &&  $email =~ /[a-z0-9]+.*@.*[a-z0-9]+/ && $name =~ /[a-z]+/) {
        if (is_available_username($email)) {
            try {
                my $user = Bugzilla::User->create({
                    login_name    => $email,
                    realname      => $name,
                    cryptpassword => '*',
                    disable_mail  => 0,
                    extern_id     => $login
                });
                print "Account for " . $user->login . " was created.\n";
                $usersAdded++;
            } catch Error with {
                my $ex = shift;
                my $error = "Error: $ex";
                $error =~ s/\n|\r/ /g;
                print $error."\n";
            };
        }

        $users++;
    }
    $lines++;
    close (ARGV) if (eof);
}

if (!$lines) {
    print "No input lines given.\n\n";
    usage();
}

print "\n$lines lines evaluated, $users user records checked, $usersAdded users added.\n";

exit 0;

export_users.php

#!/usr/bin/env php
<?php
##########################################################
# This is a basic script to export users from an
# MS Active Directory via LDAP in the format required
# by create_users.pl.
#
# Authors:
#	Adam Franco (afranco@middlebury.edu)
#	Ian McBride (imcbride@middlebury.edu)
# Date:
#	2010-03-08
# URL:
#	http://www.adamfranco.com/archives/374
# License:
#   The contents of this file are subject to the Mozilla Public
#   License Version 1.1 (the "License"); you may not use this file
#   except in compliance with the License. You may obtain a copy of
#   the License at http://www.mozilla.org/MPL/
#
#   Software distributed under the License is distributed on an "AS
#   IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or
#   implied. See the License for the specific language governing
#   rights and limitations under the License.
##########################################################

$ldaphost = "ldap.example.com";
$ldapport = 389;
$ldapuser = "username";
$ldappass = "password";
$baseDN = "DC=example,DC=com";

$connection = ldap_connect($ldaphost, $ldapport);

if (!$connection) die();

if (ldap_set_option($connection, LDAP_OPT_PROTOCOL_VERSION,3) === FALSE) die();

if (ldap_set_option($connection, LDAP_OPT_REFERRALS,0) === FALSE) die();

$bind = ldap_bind($connection, $ldapuser, $ldappass);

if (!$bind) die();

$filter = "(&(objectClass=User)(!(objectClass=Computer)))";

$search = ldap_search($connection, $baseDN, $filter, array("samaccountname", "mail", "givenname", "sn"));

$entries = ldap_get_entries($connection, $search);

print "samaccountname\temail\tname\n";

foreach($entries as $entry) {
  if(isset($entry['samaccountname'])) {
    print iconv('UTF-8', 'UTF-8//IGNORE', $entry['samaccountname'][0]);
  }
  print "\t";

  if(isset($entry['mail'])) {
    print iconv('UTF-8', 'UTF-8//IGNORE', $entry['mail'][0]);
  }
  print "\t";

  $name = '';
  if(isset($entry['givenname'])) {
    $name .= iconv('UTF-8', 'UTF-8//IGNORE', $entry['givenname'][0]);
  }  $name .= ' ';
  if(isset($entry['sn'])) {
    $name .= iconv('UTF-8', 'UTF-8//IGNORE', $entry['sn'][0]);
  }
  print trim($name);

  print "\n";
}

Example Usage

After unzipping the scripts in your Bugzilla directory you can use the create_users.pl script right away. To use export_users.php you will need to edit it and add your LDAP server configuration.
[root@hostname /var/www/htdocs/bugzilla/]# ./export_users.php | ./create_users.pl

If you’d rather import users from another source, simply create one or more tab-delimited text files that have the following columns:
login    email    name
[root@hostname /var/www/htdocs/bugzilla/]# ./create_users.pl users.txt otherusers.txt

You can pipe tab-delimited data to the script as well:
[root@hostname /var/www/htdocs/bugzilla/]# head -n 20 users.txt | ./create_users.pl

Update:

• Changed the license statement to the MPL be compatible with the rest of Bugzilla
• Changed the password to ‘*’ based on Max’s suggestion

Single Machine Configuration

Pull out the database, use multiple web-servers

The two main components of Drupal (and most similar web applications) are the webserver, which handles PHP execution and file-serving; and the MySQL database, which stores all data with the exception of uploaded files. By putting the database on a separate machine we can can have multiple machines acting as front-end web-servers, both of them reading and writing to the same database. In this way, it doesn’t matter which web-server handles a given request as they will both get the same information out of the database. With two or more web-servers, our platform gains some redundancy since one web-server can fail while the second keeps handling requests.

With both web-servers point at the same database server, the database server still remains a single point of failure. Database clustering can alleviate this problem, but will be the subject of a future post.

Multiple web-server challenges

This redundancy does come at a cost in complexity however, since we need to ensure that any uploaded files are available on both web-servers. There seem to be two primary ways of tackling this problem (without resorting to costly and complex distributed file-system tools). The first is use rsync to copy files between the web-servers every few minutes.

Two web servers with rsync

• Files cannot be deleted in the sync as newly-added files will exist on only one web-server. Since the sync is two-way, there is no way for the rsync processes to tell the difference between a new file and a deleted file.
• Requests that come to the “other” web-server will not be able to access new files until the sync happens.
• If additional web-servers are added, the sync process needs to be updated on every existing web-server to include the new web-server

The other alternative is to store uploaded files on a separate file-server, whose upload directory is mounted on each web-server using NFS. This method eliminates the synchronization problems, since all web-servers are essentially writing to the same directory.

Two web servers with NFS

Best of both worlds

In order to better solve this problem, the approach we took is to go the NFS route, but augment it with a backup copy of the files stored on the local file-system of each web-server. Every ten minutes or so a script (sync_files.sh) runs that checks to see if the shared NFS directory is available, and if so syncs the uploaded-files to a backup location on the web-server’s file-system. This backup copy has its permissions set so that the Apache process cannot write to it, preventing synchronization problems if the shared NFS directory goes offline and we need to serve files out of the backup copy.

Two web servers with NFS and local backup copies.

An important consideration in this setup is that the NFS share is mounted in ’soft’ mode so that file-access errors will time out quickly and allow for a timely switch-over to our backup files.

files.example.edu:/images       /mnt/files     nfs     soft    0 0

If the default ‘hard’ NFS mount is used, the check_link processes will hang indefinitely while trying to communicate with the file-server and never switch to our backup files.

Here is an example layout on the web-server to accomplish this setup:

# The scripts that will be run by cron:
/usr/local/bin/check_link.sh  # Run every minute
/usr/local/bin/sync_files.sh   # Run every 10 minutes

# The mounted NFS share:
/mnt/files/

# The backup copy of files:
/srv/files_read_only/

# The 'files' symbolic link, pointing normally at the NFS share:
/srv/files/ => /mnt/files/
# On NFS failure, this link will be switched to the backup directory:
/srv/files/ => /srv/files_read_only/

# The Drupal code directory:
/srv/drupal/
# The files directory for a site is a link into the switched files link
/srv/drupal/sites/www.example.com/files/ => /srv/files/www.example.com/files/

By mounting the shared NFS directory, keeping a read-only local copy of the files, and monitoring the state of the NFS directory we gain the following benefits:

• No problems with synchronization as all web-servers share the same remote filesystem.
• Synchronization of the local backup copies is not a problem as this is always a one-way sync rather than a two-way sync between different web-servers.
• While the NFS file-server is still a single point of failure, read access to the uploaded files (via the backup copy) will be restored after a maximum of one minute plus the NFS time-out (2 minutes by default for ’soft’ mounts).
• The web-servers don’t need to know about each other, easing configuration if additional web-servers are added.

This configuration adds an extra machine to the platform mix and a bit of complexity, but it makes normal operation robust (instant file availability to all web-servers) and allows for graceful degradation (file-access becomes read-only) if the file-server goes down.

Many thanks to our system administrator, Mark Pyfrom, for all of his help in developing and testing this platform.

* Update on 2009-09-10: added note about ’soft’ NFS mounts and an example file-system layout.

A few months ago Ian and I got CAS installed on campus and began updating applications to work with it rather than maintaining their own internal connections to the Active Directory server. Throughout this process we ran into a few challenges (such as returning attributes with the authentication-success response) and a bug in CAS, but we worked through these and got CAS up and running successfully.

We are now at a point where we need to do some customizations to our CAS installation to deal with changes to the group structure in the Active Directory. As well, the bug I reported was apparently fixed in a new CAS version, an improvement I need to test before we update our production installation. Both of these require a bit more poking at CAS than we can do safely in our production environment, so I am now embarking on the process of setting up a Java/Tomcat development environment on my PC. I’m documenting this process here both for my own benefit (when I have to set this up again on my laptop) and in case it helps anyone else.

Read on for my step-by-step instructions for setting up a CAS development environment on OS X.

Since I’ve recently been successful using MacPorts to install Git (a source-code-management too), I decided to use MacPorts to install Apache Tomcat, Maven, and the other required software.

Part One: Get a default CAS installation up and running

Install the Apache Tomcat server using MacPorts

sudo port install tomcat5

A number of packages will not be successfully found by ports, resulting in the following error:

--->  Verifying checksum(s) for servlet24-api
Error: Checksum (md5) mismatch for apache-tomcat-5.5.25-src.tar.gz
Error: Target org.macports.checksum returned: Unable to verify file checksums

The easiest fix I found was do go to the apache website and directly download the file. Googling will find it. Once you’ve downloaded the file, find the corrupted one on your file-system with

find /opt/local/var/macports -name "apache-tomcat-5.5.25-src.tar.gz"

And replace the corrupted version with the directly downloaded one and then re-try installation with MacPorts:

sudo mv /Users/afranco/Downloads/apache-tomcat-5.5.25-src.tar.gz /opt/local/var/macports/distfiles/servlet24-api/

You will have to do this process several times for different packages that fail to download.

You will likely also need to install the mysql-connector-java. Download the zip archive and copy the .jar file inside to /Library/Java/Extensions/

Install Maven

Same thing as with Tomcat, try installing with ports and replace failed downloads.

sudo port install maven2

Also, make sure that /opt/loca/bin/ is to the front of the search path in your ~/.bash_profile. If not, the built-in mvn command may take precedence.

mvn -v
Apache Maven 2.1.0

Download CAS

I cloned the repository using Git so that I can easily maintain my private branches, but you can download it directly or checkout with subversion.

git-svn clone  https://www.ja-sig.org/svn/cas-clients/phpcas/ --trunk=trunk --branches=branches --tags=tags

Build CAS

Building CAS is accomplished by cd’ing to the directory in which you downloaded CAS and runing:

mvn package install

Because some of the tests rely on network particulars, I can’t get the build to work unless I skip tests by instead using:

mvn -Dmaven.test.skip=true package install

Install CAS

Copy the CAS war and files to the Tomcat webapps directory:

sudo cp -R cas-server-webapp/target/cas-server-webapp-3.x.x  /opt/local/share/java/tomcat5/webapps/cas
sudo cp -R cas-server-webapp/target/cas.war  /opt/local/share/java/tomcat5/webapps/cas.war

Start Tomcat/CAS

Start Tomcat:

sudo tomcatctl start

Point your browser at http://localhost:8080/cas/ and you should see the CAS login page.

Part Two: Customizing CAS

Create the customization overlay

Following the instructions for maintaining local customizations on the CAS wiki, I created a cas-server-midd subdirectory in my CAS-source directory and added a pom.xml file based on the example. Running maven properly generates a war file:

cd cas3/cas-server-midd/

mvn -Dmaven.test.skip=true package install

ls target/
	cas
	cas-server-midd-3.3.3-SNAPSHOT
	cas-server-webapp-3.3.3-SNAPSHOT
	cas.war
	maven-archiver
	pom-transformed.xml
	war

sudo rm -R  /opt/local/share/java/tomcat5/webapps/cas

sudo cp -R target/cas  /opt/local/share/java/tomcat5/webapps/cas

sudo cp target/cas.war  /opt/local/share/java/tomcat5/webapps/cas.war

sudo tomcatctl restart

Note that there is a target/cas/ directory in addition to target/cas-server-midd-3.x.x and target/cas-server-webapp-3.x.x directories. In this case, the target/cas/ one is the one we want to copy to the tomcat5/webapps directory. Refreshing your browser should still show the login page an not an error.

Adding Customizations

With the overlay directory structure in place, any files you add to the overlay directory (in my case cas3/cas-server-midd/src/...) will be used instead of the versions in cas3/cas-server-webapp/src/...

Using the overlay has the same result as editing the files in cas3/cas-server-webapp/src/..., but keeps the changes in their own directory structure and allows the overlay to only contain files with modifications. All other files have their defaults used.

Build and Install the customizations

Run the maven build process again and copy the result from our overlay target to the Tomcat directory:

cd cas3/cas-server-midd/
mvn -Dmaven.test.skip=true package install
sudo rm -R  /opt/local/share/java/tomcat5/webapps/cas
sudo cp -R target/cas  /opt/local/share/java/tomcat5/webapps/cas
sudo cp target/cas.war  /opt/local/share/java/tomcat5/webapps/cas.war
sudo tomcatctl restart

As you go, make more changes on the overlay and rebuild CAS. Lather, rinse, repeat.

Simply adding files to the overlay should be able to support any configuration or JSP (themes, etc) changes needed. I’ll make another post once I figure out where to put class-files for adding customized Principal-Resolver classes.

One thing that often happens to me though, is that I work for about a half hour to an hour trying to get a new piece of code working and in the process make several sets of changes to one file that are only loosely related.

Let’s say that I am fixing a bug in my ‘MediaLibrary’ class and while doing so notice some some spelling mistakes in some comments that I fix. Now my one file has two changes my bug fix, and the spelling fix. Rather than committing both changes together with one comment describing both changes, I can highlight one of the changes in git-gui and select the “Stage Hunk for Commit” option.

With that one hunk staged I can now commit with a message applicable to that change. Other changes can then be staged and committed with their own messages resulting in a very understandable history of changes.

“Stage Hunk for Commit” can also be used to commit important changes while not including debugging lines inserted in your code.

Abstract

Segue and Concerto are two curricular applications built upon Harmoni, an Open Service Interface Definition-based (OSID) service-oriented application framework. This demonstration will show how website content created in Segue is stored as OSID Assets in Harmoni’s OSID Repository. Similarly, the demonstration will show how multimedia assets created in Concerto can be stored same repository. Interoperability will be demonstrated as each application is used to view and make real-time modifications to the OSID Assets created using the other application, while at the same time respecting the authorizations given to those assets. Additionally, an OSID Repository to OAI-PMH gateway will be shown providing the LibraryFind meta-search tool with access to the metadata for content created in Segue, Concerto, and a lightweight, read-only OSID Repository.

• Companion Paper: PDF (76 KB)
• Presentation Slides: PDF (7.4 MB)

Software Demonstrated:

• Harmoni Application Framework (Middlebury College)
• Segue version 2 (Middlebury College)
• Concerto (Middlebury College)
• LibraryFind (Oregon State University)

Visitor registration brings with it a few interesting challenges. As in Segue 1, we want (and need) to be able to allow people outside of the Middlebury community to join in on public discussions hosted in Segue. As well, Middlebury users often need to give access to restricted parts of their sites to people off-campus with whom they are collaborating. Our visitor registration system therefore needs to be easy to use by registrants, keep out spammers, as well as enable searches for visitor accounts by community users.

To keep out spammers, the visitor registration form uses reCAPTCHA to try to verify that a human is sitting at the browser. There are other CAPTCHA systems out there, but I like the philosophy and approach of reCAPTCHA. Starting with words that OCR software had trouble reading seems like a good idea. After the registration form is filled out, Segue sends an email to the address entered with a unique registration code. Until the link in the email is clicked on (and hence the address verified) the account is locked.

To enable easy searching of visitor accounts, visitors are asked to enter their name. While there are a few restrictions on names, these are user-chooseble. To provide some measure of differentiation between verified institution accounts and visitor accounts visitor accounts have the user-chosen name followed by their email domain name in parenthesis, e.g.:

Adam Franco (gmail.com)

I weighed including the entire email address as that is the only verified information we have about the visitor accounts, but I’d rather not open that information up for harvesting by spammers. If abuse becomes an issue, the visitor registration system also supports both black-lists and white-lists of email domains.

I’ve now been working on Segue 2 directly or indirectly for 5 years, since June 2003. It has been a long road and it is wonderful to finally be cresting the last rise. That said, as the feature-request tracker indicates, we still have a lot to do over the next 12 weeks.

Theming
This past week I rebuilt the theming system for the 4th (and last before production) time. The challenge with the theming system is that we wanted to enable end-users to choose from a few straight-forward options for things like ‘overall color scheme’, ‘font size’, corner-treatment — not all of which mapped cleanly to CSS properties. As well, to enable more powerful themes, we needed to let theme developers wrap each content type with HTML tags in order to get some effects that are just not possible with plain CSS when the dimensions of the element are not known. Our first three theming implementations involved different PHP classes for each theme with method for setting various options. Each implementation had its own strengths and weaknesses, but they were all hideously complex and required theme developers to know PHP in order to do more than change the CSS. The new theme implementation scraps all of that complexity and defines themes as a set of CSS files and HTML templates, with associated images. An extension to this simple base adds an option listing (defined in XML) that enables placeholders in the CSS and HTML templates to be replaced with values from end-user-choose-able options.

With the new theming system in place in development Alex has set to work building the first three (Rounded Corners, Shadow Box, and Tabs) of the themes that will be distributed with Segue while I’ve been finishing up the user-interfaces for choosing theme options and enabling more advanced users to customize the theme CSS and HTML in their web-browser. So far Alex and I are pretty happy with the new theming system and its simplicity should give it much longer legs than our previous attempts.

While it won’t make it to production, I eventually plan to have a theme-gallery that users can choose to publish their designs to for use by the rest of the community.

Up Next
With theming out of the way the following are some of the next areas I’ll be working on in addition to fixing bugs and working out smaller kinks:

• Templates – starting points for sites
• Enabling embedded videos from trusted sites (i.e. YouTube, Vimeo, etc)
• Visitor Registration
• Copy/Move tools for Classic Mode
• Display of RSS feeds

Still a lot to do, but with each addition Segue 2 gets much closer to being able to take over as the primary course website system.

If you are only interested in my work-related posts, check my work category or subscibe to its feed. Enjoy!