Phil Wherry
psw at wherry dot com
March 1, 2004
This document describes the technical details behind the automated archiving of EarthDial ED-7 images.
Our server is a Linux-based computer system running the Apache Web server in a fairly standard configuration. The camera we're currently using is connected to another computer physically located near the EarthDial itself. Software on this computer (Webcam32, a Windows application) grabs an image from the camera every 10 minutes and uploads it to the Linux machine using the Internet File Transfer Protocol (FTP).
The Webcam32 software offers a number of options for very basic archiving; in particular, it can maintain a short-term archive of images so that it's easily possible to see the camera at successive 10-minute intervals leading up to the current time. We currently have the camera software configured to maintain a two-hour archive of images.
The mechanism that Webcam32 uses to do this archiving is very simple. Every ten minutes, the camera uploads a file called image.jpg to the server. Before doing this, however, it uses a series of file-renaming operations to create a series of files called image1.jpg, image2.jpg, ... containing older and older images. These don't accumulate forever, though; the software deletes images older than a (configurable) limit.
We decided that we'd like to maintain an archive of every dial image for the past week. This was clearly well beyond the capabilities of the Webcam32 software. Even if it weren't, the file naming scheme doesn't lend itself to long-term archiving.
The solution was to write a couple of small scripts to run on the Linux machine to auto-archive the files. What follows is a detailed explanation of how they work. Be warned that they're not at all simple. But, if you're an expert (or know someone who is), the explanation provided here should be sufficient to allow you to replicate our capabilities in your own environment. Be warned in advance that we're not in a position to provide technical support on these scripts. They work for us and you're welcome to use them, but please don't expect to receive extensive technical support from us as you attempt to use them in your environment. That said, we welcome constructive criticism—so please let us know if you spot any errors!
The scripts are available here. They're in standard Unix tar format. In addition to the scripts, you'll find one very small image; this is the red crosshair that's used in one of the animation-generation scripts. I'd recommend installing the scripts in the /usr/local/bin directory; put the crosshair image in /usr/local/lib. This isn't really recommended practice from a security perspective, but if you were to un-tar the file with / as your current directory, everything will wind up in the right place. Otherwise you'll simply need to move the files into the correct places by hand. Remember that you generally have to be running with the privileges of the root user in order to copy things into the /usr/local tree.
The first script creates archive files from the camera-uploaded image files. Here's the code that does this (this is the file /usr/local/bin/archive-earthdial):
#!/bin/bash
xdate=`date -d "\`ls -l --full-time /home/html/webcam/earthdial/image1.jpg | colrm 1 43 | colrm 25\`" +"%Y-%m-%d-%H%M"`
cp -p /home/html/webcam/earthdial/image1.jpg /home/html/webcam/earthdial/archive/${xdate}.jpg
/usr/local/bin/archive-animate 2 160x120 20
/usr/local/bin/archive-animate 2 320x240 20
/usr/local/bin/archive-animate 4 160x120 20
/usr/local/bin/archive-animate 4 320x240 20
/usr/local/bin/archive-animate 6 160x120 20
/usr/local/bin/archive-animate 8 160x120 20
/usr/local/bin/archive-animate 12 160x120 10
/usr/local/bin/archive-animate 24 160x120 5
Now let's examine this file line by line. This is a relatively terse script, but there's a whole lot going on internally.
#!/bin/bash
This line tells the Linux machine which of several command-line interpreters ("shells") is to be used. The "bash" shell is a flexible and powerful shell that's present on pretty well any Linux machine, so it's a good choice for this sort of application.
xdate=`date -d "\`ls -l --full-time /home/html/webcam/earthdial/image1.jpg | colrm 1 43 | colrm 25\`" +"%Y-%m-%d-%H%M"`
This line does quite a bit of work. The objective is to come up with a file name that's more meaningful than something like image1.jpg. An obvious way to do this is to simply take the current date and time from the server. Unfortunately, however, this won't work well if the camera is down for some reason; it would be possible to take an old image file and assign a current date/time to it in this event.
Instead, we use the modification time on the image file itself; this captures the time on the server when the image was uploaded and therefore solves the problem of what to do if the camera software fails and doesn't update the image. In that case, the modification time doesn't change and we therefore don't create incorrect archive files.
Here's how it works:
cp -p /home/html/webcam/earthdial/image1.jpg /home/html/webcam/earthdial/archive/${xdate}.jpg
This command does the work. The cp command is used to copy the image1.jpg file to an archive directory. Note the use of the ${xdate} construct; this refers to the variable created by the previous command. Using our example above, the image1.jpg file would be copied to a new file named 2004-02-22-1200.jpg. The -p command-line switch in the cp command tells the operating system to preserve as many of the file attributes as possible during the copy; in this case, we're especially interested in preserving the date and time on the file so that it continues to represent when the image file was created, not when the copy occurred.
/usr/local/bin/archive-animate 2 160x120 20
/usr/local/bin/archive-animate 2 320x240 20
/usr/local/bin/archive-animate 4 160x120 20
/usr/local/bin/archive-animate 4 320x240 20
/usr/local/bin/archive-animate 6 160x120 20
/usr/local/bin/archive-animate 8 160x120 20
/usr/local/bin/archive-animate 12 160x120 10
/usr/local/bin/archive-animate 24 160x120 5
The remaining commands in the file are used in support of animations, which are described in more detail in the section on basic animations.
The cron utility is used to invoke this script automatically every ten minutes. See the "Running Things Automatically" section of this document for additional details.
A companion script is responsible for removing old archive images. Once again, it's compact but does a lot. This file is /usr/local/bin/archive-purge in our configuration.
#!/bin/bash
DAYS=7
COMPOSITEDAYS=30
find /home/html/webcam/earthdial/archive -name "*.jpg" -type f -mtime +${DAYS} -exec rm '{}' ';'
find /home/html/webcam/earthdial/archive/composite -name "*.gif" -type f -mtime +${COMPOSITEDAYS} -exec rm '{}' ';'
Once again, a line-by-line analysis:
#!/bin/bash
This specifies the command interpreter ("shell") again. Since this is common to all of the scripts under discussion in this document, I won't explain this again.
DAYS=7
COMPOSITEDAYS=30
This sets a variable called DAYS that contains the number 7. This exists to make it easy to change the retention interval. A similar variable, COMPOSITEDAYS, takes care of the retention for the composite animated images.
find /home/html/webcam/earthdial/archive -name "*.jpg" -type f -mtime +${DAYS} -exec rm '{}' ';'
The find command is used to do all of the work. Specifically, the system is told to look for:
The -exec rm '{}' ';' construct tells the system to execute the rm ("remove") command for each qualifying file found, thereby removing any out-of-date files. The actual filename is substituted for the '{}' construct, and the ';' construct tells find that this is the end of the command to be executed.
find /home/html/webcam/earthdial/archive/composite -name "*.gif" -type f -mtime +${COMPOSITEDAYS} -exec rm '{}' ';'
Another find command does the same thing for the animated GIF images (described in much more detail below).
This script is run by the operating system's cron process daily. See the explanation below on how cron works.
Shortly after putting the images online, we decided it would be instructional to be able to render them as animated GIF images; this allows users to easily see the movement of the gnomon shadow throughout the day. The following script takes a series of archive images and constructs an animated GIF image. This script is named /usr/local/bin/archive-animate in our environment.
#!/bin/bash
HOURS=$1
IMAGESIZE=$2
DELAY=$3
touch -d "${HOURS} hours ago" /tmp/timeflag
/usr/local/bin/convert -resize ${IMAGESIZE} -compress lzw -delay ${DELAY} `find /home/html/webcam/earthdial/archive -name "*.jpg" -newer /tmp/timeflag` /tmp/anim.gif
mv /tmp/anim.gif /home/html/webcam/earthdial/last-${HOURS}-hours-${IMAGESIZE}-${DELAY}0ms-frames.gif
Here's the detailed explanation:
HOURS=$1
IMAGESIZE=$2
DELAY=$3
These three lines pick up "command line arguments." This makes more sense when we examine a few lines from the end of the archive-earthdial script. For example, the command
/usr/local/bin/archive-animate 6 160x120 20
invokes this script. The first command-line argument (6) is assigned to the HOURS variable. The second (160x120) is assigned to the IMAGESIZE variable. Finally, the third argument (20) is assigned to the DELAY variable.
touch -d "${HOURS} hours ago" /tmp/timeflag
The touch command updates the modification time on a file; it will also create the file if it doesn't exist. This command creates a file called /tmp/timeflag with a timestamp in the past by the specified number of hours (in the case of our example, 6 hours). This is necessary to enable a bit of trickery in the next line...
/usr/local/bin/convert -resize ${IMAGESIZE} -compress lzw -delay ${DELAY} `find /home/html/webcam/earthdial/archive -name "*.jpg" -newer /tmp/timeflag` /tmp/anim.gif
This complicated line (and it is all one line, though your browser might render it on more than one depending on the size of your screen) does quite a few things.
mv /tmp/anim.gif /home/html/webcam/earthdial/last-${HOURS}-hours-${IMAGESIZE}-${DELAY}0ms-frames.gif
When completed, this file is moved into another location. The name of the file is based on the command line arguments. In our example, the resulting filename would be
/home/html/webcam/earthdial/last-6-hours-160x120-200ms-frames.gif
We use a move operation on a temporary file in order to avoid problems that might result with a user attempting to read the image file via the Web before it's completely created; by doing thing this way, we ensure that the file available to Web users is always complete.
Once we got this working, we thought it might be interesting to create a daily composite animation that included not only images of the EarthDial, but a couple of computer-generated models of the Earth in order to make the mechanics of a sundial more clear.
A tool called xearth is used to generate the Earth images. We decided to feature xearth images in a three-panel format. The first panel contains a representation of the Earth centered around the subsolar point (the point on Earth where the Sun is directly overhead). The second panel contains an image of the EarthDial. The third panel contains another image of the Earth, but this one is centered in such a way that the position of the EarthDial remains stationary throughout the course of the day (think of a picture of the EarthDial site as might be taken by a geostationary satellite).
Because this wasn't complicated enough, we also thought it would be a good idea to overlay a little red crosshair over the subsolar point and the position of the EarthDial.
Two scripts are involved in the production of the animation. One script controls the selection of images and the assembly of the final animation. The second is responsible for the construction of an individual frame of the animation; this script is invoked repeatedly in order to generate the frames for assembly. Let's start out by looking at the master control script in its entirety. This is a script called /usr/local/bin/archive-xearth in our environment.
#!/bin/bash
cp /home/html/webcam/earthdial/archive/`date -d "24 hours ago" +"%Y-%m-%d"`*.jpg /home/html/webcam/earthdial/archive/composite
cd /home/html/webcam/earthdial/archive/composite
xdate=`date -d "\`ls -l --full-time yesterday.gif | colrm 1 43 | colrm 25\`" +"%Y-%m-%d"`
cp -p yesterday.gif composite-${xdate}.gif
find . -type f -maxdepth 1 -name "????-??-??-????.jpg" -exec /usr/local/bin/archive-xearth-file '{}' ';'
/usr/local/bin/convert -compress lzw -delay 5 -loop 0 montage* yesterday.gif
touch --date="`date -d \"24 hours ago\" +\"%Y-%m-%d 23:59\"`" yesterday.gif
rm -f ????-??-??-????.jpg xearth*-????-??-??-????.gif montage-????-??-??-????.gif intermediate-????-??-??-????.gif
Here's how it works:
cp /home/html/webcam/earthdial/archive/`date -d "24 hours ago" +"%Y-%m-%d"`*.jpg /home/html/webcam/earthdial/archive/composite
This line copies all of the images for a particular day (yesterday) into the directory where the composite images will be assembled. The script runs after midnight (about 12:30 AM), so we ask for yesterday's date in YYYY-MM-DD format. This is used to copy files beginning with YYYY-MM-DD and ending in .jpg to the work directory. The wildcard asterisk allows any characters to appear in between the date and the .jpg suffix, so this results in a copy of all images for the day.
cd /home/html/webcam/earthdial/archive/composite
The rest of the work for this script takes place in the work directory named /home/html/webcam/earthdial/archive/composite. We use the cd command to change the working directory so that we don't have to specify this lengthy directory name every time we reference a file.
xdate=`date -d "\`ls -l --full-time yesterday.gif | colrm 1 43 | colrm 25\`" +"%Y-%m-%d"`
This construction is very similar to the one explained in the archive-earthdial script. It looks at the modification time for the file yesterday.gif and converts it to YYYY-MM-DD format. The result is stuffed into a variable called xdate.
cp -p yesterday.gif composite-${xdate}.gif
The yesterday.gif file is then copied to a file named composite-YYYY-MM-DD.gif; this allows us to maintain an archive of animations.
find . -type f -maxdepth 1 -name "????-??-??-????.jpg" -exec /usr/local/bin/archive-xearth-file '{}' ';'
This command finds all of the EarthDial camera images that were copied into place by the first line of the script. The script archive-xearth-file is run once for every such filename. This creates the individual 3-panel frames that will be animated. This (rather complex) script is explained in more detail in the next section.
/usr/local/bin/convert -compress lzw -delay 5 -loop 0 montage* yesterday.gif
This command takes the resulting set of frames (which all begin with montage) and uses the ImageMagick convert utility to animate them. The resulting file is called yesterday.gif (since it contains all of the images from the prior day).
touch --date="`date -d \"24 hours ago\" +\"%Y-%m-%d 23:59\"`" yesterday.gif
This is a bit of trickery. Since the script runs after midnight, the yesterday.gif file will actually have a timestamp from the time it's created. This would cause problems with the archive file renaming that happens earlier in this script. This touch command rolls the date on the file back to 23:59 (11:59 PM) on the prior day.
rm -f ????-??-??-????.jpg xearth*-????-??-??-????.gif montage-????-??-??-????.gif intermediate-????-??-??-????.gif
The final line of the script is a cleanup command that removes all of the intermediate files used to construct the animations. This is important, since the animation process results in the creation of a large number (over 700) temporary files.
Here's the file that generates the individual 3-panel frames. It's called /usr/local/bin/archive-xearth-file in our configuration.
#!/bin/bash
LONGITUDE=-77
LATITUDE=39
GLOBETILT=10 # tilts globe down in panel 3 by this number of degrees
VERTICALOFFSET=-29 # 60 x sin( -(LATITUDE-GLOBETILT) in degrees)
FNAME=$1
DATEPART=${FNAME:2:10}
TIMEPART=${FNAME:13:2}\:${FNAME:15:2}
let CURSOROFFSET=59+${VERTICALOFFSET}
/usr/X11R6/bin/xearth -time `date -d "${DATEPART} ${TIMEPART}" +"%s"` -nostars -pos "fixed ${GLOBETILT} ${LONGITUDE}" -night 20 -term 15 -nomarkers -size "160 120" -gif >xearth-${FNAME:2:15}.gif
/usr/X11R6/bin/xearth -time `date -d "${DATEPART} ${TIMEPART}" +"%s"` -nostars -night 20 -term 15 -nomarkers -size "160 120" -gif >xearth2-${FNAME:2:15}.gif
/usr/local/bin/montage -geometry 160x120 xearth2-${FNAME:2:15}.gif ${FNAME:2} xearth-${FNAME:2:15}.gif intermediate-${FNAME:2:15}.gif
/usr/local/bin/composite -geometry +77+57 -compose Over /usr/local/lib/crosshair-transparent-small.gif intermediate-${FNAME:2:15}.gif montage-${FNAME:2:15}.gif
mv montage-${FNAME:2:15}.gif intermediate-${FNAME:2:15}.gif
/usr/local/bin/composite -geometry +397+${CURSOROFFSET} -compose Over /usr/local/lib/crosshair-transparent-small.gif intermediate-${FNAME:2:15}.gif montage-${FNAME:2:15}.gif
This is quite a complicated file, but once again we'll dissect it line-by-line.
LONGITUDE=-77
LATITUDE=39
These lines specify the latitude and longitude of the EarthDial. Use a negative number to specify west longitude (and/or south latitude).
GLOBETILT=10 # tilts globe down in panel 3 by this number of degrees
This line defines the tilt of the globe in the third panel. A tilt value of zero will result in a globe image centered on the Equator at the EarthDial's longitude. For our dial (at approximately 39 degrees north latitude), this puts our location rather high on the globe. So we tilt the image down slightly for aesthetic reasons.
VERTICALOFFSET=-29 # 60 x sin( -(LATITUDE-GLOBETILT) in degrees)
A crosshair is positioned over the EarthDial's location in the third frame. The horizontal position of this crosshair is easy to figure out: since the globe is centered on the EarthDial's longitude line, it's always in the center of the frame. Its vertical position, however, is more complicated. The simple application of a little trigonometry can be used to compute the correct position, however. To do this:
FNAME=$1
DATEPART=${FNAME:2:10}
TIMEPART=${FNAME:13:2}\:${FNAME:15:2}
We know that each of the filenames we'll be processing is of the form YYYY-MM-DD-HHMM.jpg (because this is the way the names were generated in the archive-earthdial script). We need, however, to separate the date and time components in order to make parts of this script work.
The FNAME variable starts off containing the entire filename. The output of the find command contains the characters ./ at the beginning of every filename (this literally means "in the current directory"). We therefore need to skip 2 (the "./") characters into the string, then assign the next 10 characters to a variable called DATEPART. This means that DATEPART now contains the date in YYYY-MM-DD format.
We do something similar with the time. The filenames contain the time in HHMM format, and we need a colon between the hour and the minutes. So we skip 13 characters into the file name string, take 2 characters (the hours), then add a literal colon using the \: construct. Then we skip 15 characters into the filename string and take the final 2 characters to get the minutes.
After doing these steps, we've now got the date in YYYY-MM-DD format safely stored in a variable called DATEPART, and then a second variable called TIMEPART that contains the time in HH:MM format.
let CURSOROFFSET=59+${VERTICALOFFSET}
This line computes the vertical offset for our crosshair graphic. We start off centered vertically in the image at Y position 59, then add the vertical offset specified earlier in the file (in our case, -29).
/usr/X11R6/bin/xearth -time `date -d "${DATEPART} ${TIMEPART}" +"%s"` -nostars -pos "fixed ${GLOBETILT} ${LONGITUDE}" -night 20 -term 15 -nomarkers -size "160 120" -gif >xearth-${FNAME:2:15}.gif
This gigantic line (and, once again, it's really a single line) does a whole lot. The xearth command can generate an image for any arbitrary point in time. Unfortunately, however, this time is specified in rather inconvenient units: the number of seconds since January 1, 1970. There's a reason for this: all date/time values in a Unix-like operating system like Linux are stored internally in this format. So we use the date command to convert the date and time into seconds (that's what the +"%s" format string does). The result of this (let's call it NNNNNN for the sake of example) will be fed into the xearth command line. We're also doing variable substitution in the command line in a few places:
This would result in an invocation of xearth that looks like this:
xearth -time NNNNNN -nostars -pos "fixed 10 -77" -night 20 -term 15 -nomarkers -size "160 120" -gif >xearth-YYYY-MM-DD-HHMM.gif
This causes xearth to generate an image in GIF format centered on a particular point (the equator offset by the globe tilt value and using your line of longitude). The result is stored in a file that begins with xearth- followed by the date and time.
/usr/X11R6/bin/xearth -time `date -d "${DATEPART} ${TIMEPART}" +"%s"` -nostars -night 20 -term 15 -nomarkers -size "160 120" -gif >xearth2-${FNAME:2:15}.gif
This does exactly the same thing, but this time the -pos argument is left off; this causes xearth to center on the subsolar point (the point on Earth where the Sun is directly overhead at that moment in time). The results are stored in another file that begins with xearth2- followed by the date and time.
/usr/local/bin/montage -geometry 160x120 xearth2-${FNAME:2:15}.gif ${FNAME:2} xearth-${FNAME:2:15}.gif intermediate-${FNAME:2:15}.gif
Next we use the ImageMagic montage command to construct a three-panel image consisting of:
The result is stored in a file that begins with intermediate- followed by the date and time.
/usr/local/bin/composite -geometry +77+57 -compose Over /usr/local/lib/crosshair-transparent-small.gif intermediate-${FNAME:2:15}.gif montage-${FNAME:2:15}.gif
The ImageMagic composite command is now used to overlay the crosshair on the image. The crosshair is a small (5x5 pixel) transparent GIF image. The resulting file is named montage- followed by the date and time.
mv montage-${FNAME:2:15}.gif intermediate-${FNAME:2:15}.gif
The original version of this script just overlaid one crosshair. When we decided to add one for the dial location, we needed to take a second pass through the composite command. As a consequence, we rename the file beginning with montage- to one beginning with intermediate- (indicative of its less-than-final status). There's already a file by this name in existence at this point; it's simply overwritten.
/usr/local/bin/composite -geometry +397+${CURSOROFFSET} -compose Over /usr/local/lib/crosshair-transparent-small.gif intermediate-${FNAME:2:15}.gif montage-${FNAME:2:15}.gif
This last command overlays the crosshairs that denote the dial position. The result is a file beginning with montage- followed by the date and time that contains the complete 3-panel image. The previously-discussed archive-xearth script constructs an animation once the individual images have all been generated.
We make use of a standard tool called cron in order to schedule the invocation of these tools. Here's the crontab file we use to accomplish this:
5,15,25,35,45,55 * * * * /usr/local/bin/archive-earthdial
30 0 * * * /usr/local/bin/archive-xearth
7 0 * * * /usr/local/bin/archive-purge
Three distinct tasks are accomplished:
We're getting a lot of mileage out of Linux and a few small (if complicated) scripts. If you're able to make use of them, or (better yet) improve upon them, we'd love to hear from you!
Phil Wherry
psw at wherry dot com