Archive for the 'Software' Category  

Posts describing software I’ve written.

Creating a Drupal plugin system

November 13th, 2018

Filed under: Computers and Technology , Software , Work/Professional

Drupal 8 has a great Plugin API that not only allows one to write plugins that modify Drupal core’s behavior, but also to define your own pluggable system that you and others can plug into as well. The Plugin API documentation has a number of examples that cover many common use cases for plugins and configuration entities, but none quite covered my use case, so hopefully this article can help fill some of the gaps. I’m going to use an actual example that I built to help ground the discussion of abstract concepts in something more concrete — the abstractions like Annotations, EntityWithPluginCollectionInterface, PluginFormInterface, Derivatives, and others are powerful, but it can be tough to pin down how and when to actually use one versus another.

The Challenge

My end goal is a module that creates Drupal nodes of a custom “Event” content-type based on a variety of external calendaring systems. My campus uses the Resource25 calendaring system which will be the source of most events, while others may be sourced from Outlook calendars, Google Calendars, or other data sources. What the source systems are and how they are connected to will never be completely unified and will change over time, so I want to create a flexible, pluggable, system that will allow new plugins to be written that just cover how to fetch from new source systems without reinventing the whole thing every time we need to add a data source.

There are two complications that lift this implementation out of the realm covered in a straight-forward manner in the Drupal documentation:

  1. Each plugin has both standard configuration options (such as “Enabled”) as well as plugin-specific configuration options.
  2. For a given Drupal site, there might be multiple instances of each source plugin that are configured to pull from different calendar feeds using the same protocols and methods.
Diagram of the relationship between plugins and config instances.

Relationship between plugins and Configuration Entity instances

 

This system is very similar to Drupal-core’s Block system. The Block system defines an API for “Block Plugins” where each block-plugin has PHP code that drives it. Multiple instances of each block-plugin can be placed in the page, each with their own settings. For example, the System Menu Block Plugin in the system module can have multiple instances added to the block-layout, each configured to display a different menu with different settings for initial menu-level and depth of menu-levels visible as well as the standard settings for title and display of the title.

Add block dialog example

 In fact, the Plugin API documentation specifically recommends using the Block system as an example. Unfortunately this is a complicated example to follow because the Block system is scattered between Drupal-core library files and the Block module, rather than all contained in one place.

 

What is needed?

I’m going to skip over the basics of the module skeleton as these are well documented already in order to focus on the parts that are needed to achieve the system described above.

Here are the pieces that we need to get functioning and which I will dive further into:

  • Defining the plugins themselves with a Plugin Manager and discovery mechanisms so that Drupal knows what plugins are available.
  • Defining the Configuration Entity for our plugin system so that instances can be configured and exported along with the rest of our site configuration.
  • Defining dynamic links so that we can create new instances for each plugin installed.
  • Allowing plugins to define extra configuration fields/properties that are particular to their needs.

A snapshot of the resulting module is available on GitHub at adamfranco/middlebury_event_sync. I’ll be linking to code in the module rather than copying it here.

 

Defining the Plugins

For this system, we are going to define a single Plugin Manager that will provide discovery and loading of our Event Source plugins and a Plugin Interface that all of our Event Source plugins will have to implement.

The Plugin Manager gets a listing in our module.services.yml as well as a class that defines what the alter-hooks are and what the plugin name-space is. More on Services and Dependency Injection can be found on drupal.org.

Next up is an Annotation class at src/AnnotationEventSource.php which defines the parameters that will be used in each plugin class‘s doc-block to identify that plugin class as one of our plugins. This is the key to the discovery mechanism we’re using — the annotation in the doc-block registers the plugin so that it is available through our Plugin Manager.

We also define Plugin Interface which defines common methods for all of our Event Source plugins. This isn’t strictly required for all plugin systems as the PluginInspectionInterface and/or ConfigurablePluginInterface can be implemented by plugins directly, but I wanted all plugins to implement a few common methods for standard settings and actual operations.

The manager, service, annotation, and plugin interface are the things we need to define to allow our plugins in src/Plugin/EventSource/ to be discovered and available to Drupal. Now that we have defined our plugins, we can use \Drupal::service('plugin.manager.event_source')->getDefinitions() to access all of the Event Source plugins installed in our site, both those provided by our module as well as those provided by other modules.

 

Configuration Entities

The next task is to define a configuration entity, the instances of which will hold the configuration for each service we are connecting to and reference the plugin that will be used to do the connecting. Configuration in Drupal 8 is  a great new system that allows configuration to be modified in a development environment, exported, version-controlled, and then imported in a production environment. Configuration Entities allow units of configuration to be created, listed, edited, and deleted like other entities in Drupal (nodes, users, taxonomy-terms, menu items, etc). For this module, I need to use config entities because we don’t have just a single global setting, but rather zero or more configuration sets for each plugin.

Our config entity is a single class that extends ConfigEntityBase and has a number of annotations that point to the Controllers and Forms used to list and edit the instances of the Entity. Our config entity class also implements an interface I created that defines how to access the plugin associated with the config entity, not something available by default.

If we just wanted to have a single config entity for each plugin, this would all be enough. However we want to be able to have multiple config entity instances associated with each plugin. To achieve this our config entity must also implement the confusingly named EntityWithPluginCollectionInterface which provides access an EventSourcePluginCollection that we define. This “Plugin Collection” handles the mapping between config entity instances and the plugin instances that house our actual data-fetching code. Via this Plugin Collection we can pass a config entity (or its settings) off to the plugin instance so that the plugin instance has access to data in addition to its methods.

After defining the config entity and getting it working, it is a good idea to define a Configuration Schema in yml. The primary purpose of this is to allow translation of your configuration — it doesn’t seem to affect the actual export of the configuration.

Our config entity is pretty light-weight — it stores its id (a.k.a. machine-name), label, which plugin-id it is associated with, which module is providing that plugin, and finally an arbitrary set of plugin-settings defined by the plugin. These plugin settings include some common ones such as enabled, ttl, and time-shift, as well as plugin-specific settings like URIs, username, and password. There are other ways that the plugin settings could be injected into the config entity, but I felt that putting them in a single container called “settings” was flexible and simple and avoided potential conflicts between the keys needed by the config entity and the plugin.

 

Dynamic Links to add instances

A missing piece with what we have so far are the links to create new config entity instances. Because new plugins may be provided by other installed modules, we can’t hard-code our links to add new instances in our action links. Instead we need to implement a Derivative plugin that generates a list of links based on the available plugins and point to this Deriver from our module’s module.links.action.yml.

Screen shot of config UI with "add" links.

Configuration Entity “add” links provided by our Deriver plugin.

 

Extra configuration fields per plugin

To assist with common settings and functionality, I created an abstract base class that all of my plugins can inherit. This base class implements the PluginFormInterface to provide form fields on the config entity’s editing screens. Each plugin can override these methods to provide and handle additional fields while leaving the common fields to the base class. These fields are handled as a sub-form loaded in our config-entity form class.

Event-source edit form

Event Source editing form showing fields provided by the configuration entity itself (label & machine name), common fields from our abstract base class (TTL, time-shift, enabled) and plugin-specific fields (URI, username, password).

 

 

The Result

Once the items above were in place I now have a system that lets me create as many instances of each plugin as needed. The buttons to add a new instance are provided by the Deriver and the listing itself by a controller.

Event Source admin UI.

Admin UI for adding Event sources. Note that only two plugins are currently installed.

 

Hopefully this overview can point you enough in the right direction to implement your own Drupal plugin systems for other purposes.

Resources

Below are links to the documentation and resources that I used in writing this pluggable system and this article:

New Curvature site

November 17th, 2016

Filed under: Computers and Technology , Software

Tags: , ,

Several years after creating Curvature –my program that analyzes road-geometry and builds maps of twisty roads– it now has a dedicated site of its own:

roadcurvature.com

The new site is written to help non-techies understand how to use the curvature files with step-by-step instructions and a lot less jargon than in my original post about the program.

I also took this as an opportunity to learn how to do vector graphics in Inkscape and design myself a new logo — with significant design help from Alison.

curvature.py — find the most twisty-turny roads around

December 5th, 2012

Filed under: Computers and Technology , Software

Tags: , , , , , , , , ,

Update November, 2016: New dedicated Curvature site — roadcurvature.com

Update October, 2013: Google Earth KML files generated by curvature.py are now available covering the entire world.

In the process of taking up motorcycling this summer I also gained an additional hobby: scouring maps and travel guides to find the roads that would be most fun to ride. While I’ve had great times on dirt roads through farmland and wide open highways, there just isn’t anything that compares to the thrill of leaning through the corners on a winding road.

While I’ve had some good successes in locating roads by map (such as Tracy Road), one of the shortcomings of a map is the tight curves you can really lean into tend to be below the resolution for many maps. Atlases and electronic maps like Google Earth allow you to zoom in, but then there is the problem of finding the gems in the sea of data. What I realized I needed was a way to highlight just the most curvy roads so that I would know where to explore next.
Continue Reading »

River Levels Widget v.1.2.2 available

March 23rd, 2011

Filed under: Computers and Technology , Software

Tags: , ,

RiverLevels 1.0 Screen Shot

The RiverLevels widget provides an easy way to monitor the amount of water flowing in your favorite streams and rivers right from your Dashboard. The RiverLevels widget is of particular interest to whitewater kayakers and canoeists.

Once any United States Geological Survey (USGS) stream-gauge station is selected, it is automatically refreshed to always provide you with the latest graph of the water-level. As of version 1.2 you can choose between two graph styles: discharge in cubic feet per second (CFS) and water-height in feet.

This widget is Free software, licensed under the GNU General Public License (GPL) version 3 or later.

Requirements:

  • OS X – 10.4 “Tiger” or later

Change Log:
1.2.2 (2011-03-23)

  • Fix for image URL change in USGS site.

1.2.1 (2008-02-10)

  • New zip archive includes the ‘library’ directory missing in the 1.2 release.

1.2 (2008-02-06)

  • Fixed Leopard (10.5) compatability bug.
  • Added the ability to choose Gauge Height (ft) in addition to discharge (CFS).

1.1 (2007-01-08)

  • Fixed graphs extending off bottom of widget
  • Fixed invisibility of front refresh icon

Importing users into Bugzilla

March 8th, 2010

Filed under: Computers and Technology , Software , Work/Professional

Tags: , , ,

For the past 6 months our Web Application Development work-group has been Bugzilla as our issue tracker with quite a bit of success. While it has its warts, Bugzilla seems like a pretty decent issue-tracking system and is flexible enough to fit into a variety of different work-flows. One very important feature of Bugzilla is support for LDAP authentication. This enables any Middlebury College user to log in and report a bug using their standard campus credentials.

While LDAP authentication works great, there is one problem: If a person has never logged into our Bugzilla, we can’t add them to the CC list of an issue. This is important for us because issues usually don’t get submitted directly to the bug tracker, but rather come in via calls, emails, tweets, and face-to-face meetings. We are then left to submit issues to Bugzilla ourselves to keep track of our to-do items. Ideally we’d add the original reporter to the bug’s CC list so that they will automatically be notified as we make progress on the issue, but their Bugzilla account must exist before we can add them to the bug.

Searching about the internet I wasn’t able to find anything about how to import LDAP users (or any kind of users) into Bugzilla, though I was able to find some basic instructions on how to create a single user via Bugzilla’s Perl API. To improve on the lack of user-import support I’ve created an Perl script that creates users from lines in a tab-delimited text file (create_users.pl) as well as a companion PHP script that will export an appropriately-formatted list of users from an Active Directory (LDAP) server (export_users.php).

Continue Reading »

Twitter Export Script

October 13th, 2008

Filed under: Computers and Technology , Software

Tags:

I have been using Twitter as a log of my daily doings and wished to export my time-line for reformatting into a calender format. Unfortunately TweetDumpr just retrieves the list of Tweets using a single fetch request which is limited by the Twitter API to a maximum of 200 Tweets. (Update: apparently TweetDumpr can get more than 200 Tweets. It just didn’t say so in its description.)

I wanted to export all 600+ of my tweets, so I wrote the following little php script to accomplish this. I have not yet tested it with many concurrent users or added a form to select which user to update. Until I do so, I won’t be providing it as an end-user service. You are free to put it on your own machine and use it though.

TwitterExport.php

<?php
/**
 * This script will allow the export of complete user time-lines from the twitter
 * service. It joins together all pages of status updates into one large XML block
 * that can then be reformatted/processed with other tools.
 *
 * @since 10/13/08
 *
 * @copyright Copyright © 2008, Adam Franco
 * @license http://www.gnu.org/copyleft/gpl.html GNU General Public License (GPL)
 */

$user = 'afranco_work';	// Replace this with your user name.


header('Content-type: text/plain');

$allDoc = new DOMDocument;
$root = $allDoc->appendChild($allDoc->createElement('statuses'));
$root->setAttribute('type', 'array');

$page = 1;
do {
	$numStatus = 0;

	$pageDoc = new DOMDocument;
	$res = @$pageDoc->load('http://twitter.com/statuses/user_timeline/'.$user.'.xml?page='.$page);
	if (!$res) {
		print "\n\n**** Error loading page $page ****";
		exit;
	}
	foreach ($pageDoc->getElementsByTagName('status') as $status) {
		$root->appendChild($allDoc->createTextNode("\n"));
		$root->appendChild($allDoc->importNode($status, true));
		$numStatus++;
	}

	print "\nLoaded page $page with $numStatus status updates.";
	flush();

	$page ++;
	sleep(1);

} while ($numStatus);

print "\nDone loading timeline.";
print "\n\n\n";

$root->appendChild($allDoc->createTextNode("\n"));
print $allDoc->saveXml();

Usage (assuming PHP is installed)

  1. Save the code above on your machine as twitter_export.php
  2. Edit the code to change the $user variable to be your own Twitter username
  3. From the command line run php twitter_export.php
  4. Copy/paste the XML output into a file for safe keeping and further processing

River Levels Widget v1.2.1

February 10th, 2008

Filed under: Computers and Technology , Software

Tags: ,

A new version (1.2.2) is available that fixes issues with image URLs

This version is a re-release of version 1.2 which had a corrupted archive missing some necessary files.

RiverLevels 1.0 Screen Shot

The RiverLevels widget provides an easy way to monitor the amount of water flowing in your favorite streams and rivers right from your Dashboard. The RiverLevels widget is of particular interest to whitewater kayakers and canoeists.

Once any United States Geological Survey (USGS) stream-gauge station is selected, it is automatically refreshed to always provide you with the latest graph of the water-level. As of version 1.2 you can choose between two graph styles: discharge in cubic feet per second (CFS) and water-height in feet.

This widget is Free software, licensed under the GNU General Public License (GPL) version 3 or later.

Requirements:

  • OS X – 10.4 “Tiger” or later

Change Log:

1.2.1 (2008-02-10)

  • New zip archive includes the ‘library’ directory missing in the 1.2 release.

1.2 (2008-02-06)

  • Fixed Leopard (10.5) compatability bug.
  • Added the ability to choose Gauge Height (ft) in addition to discharge (CFS).

1.1 (2007-01-08)

  • Fixed graphs extending off bottom of widget
  • Fixed invisibility of front refresh icon

River Levels Widget v1.2

February 7th, 2008

Filed under: Computers and Technology , Software

Tags: ,

Update: A new version is available that fixes a corrupted archive in version 1.2.
RiverLevels 1.0 Screen Shot

The RiverLevels widget provides an easy way to monitor the amount of water flowing in your favorite streams and rivers right from your Dashboard. The RiverLevels widget is of particular interest to whitewater kayakers and canoeists.

Once any United States Geological Survey (USGS) stream-gauge station is selected, it is automatically refreshed to always provide you with the latest graph of the water-level. As of version 1.2 you can choose between two graph styles: discharge in cubic feet per second (CFS) and water-height in feet.

This widget is Free software, licensed under the GNU General Public License (GPL) version 3 or later.

Requirements:

  • OS X – 10.4 “Tiger” or later

Change Log:

1.2 (2008-02-06)

  • Fixed Leopard (10.5) compatability bug.
  • Added the ability to choose Gauge Height (ft) in addition to discharge (CFS).

1.1 (2007-01-08)

  • Fixed graphs extending off bottom of widget
  • Fixed invisibility of front refresh icon

Update: A new version is available that fixes a corrupted archive in version 1.2.

WordPress Enclosure Adder

September 6th, 2007

Filed under: Computers and Technology , Software

Tags: ,

I’ve recently developed a small PHP script, the WPEnclosureAdder (source | try) that goes through each item in an RSS feed, looks for links to YouTube videos or GoogleVideo videos, and then adds an enclosure tags for the videos. If multiple videos are found embedded in a post, then that post is duplicated in the feed for each additional URL to provide compatibility with the many RSS readers/video-podcast viewers that expect a single enclosure per post.

I wrote this script because I have been recently making heavy use of Miro (formerly known as “The Democracy Player“) to download videos from YouTube in order to watch them off-line. Miro also provides a nice UI for aggregating videos and remembers my spot when I go back to watching later (nice for long documentaries). Miro however, expects links to videos in RSS enclosure tags, something that WordPress (and probably other blogging software) doesn’t do for embeded videos.

Throw Away Your Telescreen is a video blog done by one of my favorite geo-political bloggers, Dave on Fire, and a few others. In it they link out to the most interesting “documentaries, lectures, and interviews that follow a different editorial line” from the corporate press. I highly recommend all of the videos on it that I have seen.

Throw Away Your Telescreen has all the makings of an indie-news channel, perfect for Miro which was developed to encourage participatory media and culture. The only thing missing was to get the videos embedded in Throw Away Your Telescreen’s posts in such a way that Miro can find them. With the WPEnclosureAdder, this has now been done. Use this feed to view Throw Away Your Telescreen in Miro.


More about the WPEnclosureAdder:

  • View the source-code of the latest version. (save-as to download)
  • License: GNU General Public License (GPL) version 3 or later
  • Requirements (for hosting it yourself): PHP version 5.2 or later
  • Git Repository: http://www2.adamfranco.com/WPEnclosureAdder.git

I wrote this script with Throw Away Your Telescreen in mind, but it should work with any other WordPress blog, and probably with RSS feeds generated from other blogging tools. To point it at another blog’s RSS feed, enter the feed url in the form below:

Using my version will use my default search strings for YouTube and GoogleVideo videos. If you would like to change what is being searched for, please download the script, change the configuration, and host it on your own website. I have licensed the WPEnclosureAdder under the GNU General Public License (GPL) version 3 or later, so you are free to copy and modify this script as per the terms of that license.

KML Joiner

August 29th, 2007

Filed under: Computers and Technology , Software

Tags: , , ,

As of a few days ago, I am now able to generate KML versions of Flickr photosets for viewing in Google Earth/Maps. With that taken care of, I also want to easily combine these KML documents of images together with other KML files that show additional information, such as paths traveled, points of interest, etc.

To accomplish this task, I have written a new script, the KML Joiner that will combine any KML documents on the web together into a single (referenced) KML document. (try it out)

More Detail: for those interested in KML
The resulting document is a collection of network links, each of which points to one of the KML URLs specified. Doing this rather than combining their text together into a static KML document prevents style collisions as well as allows changes in the source data to propagate to the combined document.

Refresh intervals can optionally be specified for every source document allowing for a server-friendly combination of static data with rapidly changing data. By default, no refresh interval is specified, making the linked documents load only once when first accessed.

Example:

View the KML Joiner with fields filled in that generates the map below.


View Larger Map

The map above is of the trip mentioned in a previous blog post, but this time the data sources (1. a static KML file with the path and house placemark, 2. a dynamic KML document generated with my Photo set to KML script) joined together with the KML Joiner script instead of manually put together with a text editor.

Usage:
You are welcome to use this script hosted on my site, or you can download it and run it on your own computer/webserver.

This script is available under the GNU General Public License (GPL) version 3 or later. (Source Code)

Please post any suggestions for fixes or changes. Thanks!

Next »