This page details how the site was built. If you're looking the user guide, click here.
GitHub Repo: https://github.com/DevinSmithWork/prelinger-stacks-explorer-2
Table of Contents
1. How This Thing Was Put Together: Overview
-
PHOTOS:
- Shooting: This time around, I shot in raw format, resulting in better-quality photos at the expense of some extra conversion steps. I also rigged up a camera mount that attached to the library's rolling ladders.
- Editing: You'll notice the individual photos are now 1:1 with the shelves' dimensions. I wrote a Processing program to facilitate this editing in a streamlined way.
- Deep Zoom Image [.dzi] format: I converted the full-stack composites to .dzi for use with OpenSeadragon (see below). If you're looking to replicate this workflow, there's some edge cases to be aware of.
- DATA ENTRY: During the library's refresh, Megan produced an excellent guide for in-person visitors. I used this guide as a starting point for the zones/subzones organization of the site. The Internet Archive's command-line tool was helpful for creating a starting point for entering data about the scanned items.
- UX: OpenSeadragon [OSD] is an open-source JavaScript [JS] library for displaying .dzi images. OSD is robust, full-featured, and has an active community with a fantastic lead developer. The JS code that translates back-and-forth between library-space and OSD-space is fairly complex; See the section "4 Website Structure" below for details.
2. Photography & Image Processing
2.1 General Overview
At a high level, what's needed is a photo of every shelf and a way to stitch them together -- but along the way, there's some procedural decisions that can influence both the shooting efficiency and the overall aesthetic feel of the website. My process involved (1) rigging up a camera mount for shooting, (2) automating raw-to-tiff conversions with Darktable, (3) writing a Processing program for adjusting the photos' perspective/skew, (4) using Processing to create the shelf composites, and (5) converting the composites to .dzi format for OpenSeadragon with a quick python script. Of the three main parts of this documentation (Photos, Data, and UX), this section includes the most potential for automation and general workflow improvements.
2.2 Photography
I used a camera similar setup as the first version, with a fixed aperture, variable exposure time, and fixed white balance. The fixed aperture provides a consistent depth of field, and the fixed white balance provides a consistent color spectrum when the photos are composited (This helps the the user visually locate themselves in a physical environment).
Once I figured out a feasible way to handle the skew/perspective editing (see below), I opted to shoot in 4:3, the camera's native sensor dimensions. The shooting went quicker this time around because the framing didn't need to be precise, and I rigged some stand clamps from Sammy's onto the library's rolling ladders, which helped with shooting higher shelves.
Idea: Better camera mounting strategies
My DIY rig was adequate for the Prelinger Library's size, but you're planning a similar project for anything larger, exploring better camera mount strategies is 100% worth it.
I tried attaching a camera mount directly to the shelving units, which would create consistently equidistant photos; but I couldn't find a reasonable arrangement which prevented either the rigging from appearing in the photos or blurring from instability.
With a larger budget, the idea rig would be some kind of rolling rack with a tall, vertically repositionable camera mount and locking wheels. Stabilization would need to be considered; This would be an excellent project for a few collegiate engineering students.
Another request for v2 was shooting in raw, a lossless digital format. (I shot v1 in HQ .jpg to reduce filesizes and expedite the Processing workflow.) Raw files require an application to combine the camera's sensor data with metadata to produce a viewable image. Adobe software is the industry standard, but Darktable is the free, open-source solution, and it has a command line interface for automating common tasks. (Like most open-source software, it's a little clunky and there's a learning curve.) Here's the batching workflow I used:
- Toss a couple raw files into the lighttable
- Open one up in the darkroom, and make your desired adjustments with the modules.
- Save your editing history stack as a style.
- Switch back to the lighttable, and apply this style to the other images to test how it looks. Adjust as needed.
- After you've got a style that you like, you apply it to batches using the command-line tool:
darktable-cli [input folder] --style [your saved style] --out-ext [.tiff, .jpeg, etc] [output folder]
Note: .xmp files
Darktable stores image metadata in a separate .xmp "sidecar" file. The data compression of raw is surprising: A 20mb .arw file with a 5k .xmp exports to a 240mb .tif!
Note: Darktable CLI/GUI program conflict
The command line program won't operate if the desktop application is also running. IDK why. It's Chinatown OpenSource, Jake.
Note: CPU throttling for long conversion queues
If you're using an older computer (like me), converting 300+ raw files can redline your CPU for a few hours, which is no bueno. Consider using a tool like cpulimit to manually throttle the batch processing.
Note: 32-bit .tif errors in Processing
Processing had trouble working with 32-bit tiffs: It looks like the color depth was truncated? (Pure white displayed as turquoise, etc.) If you run into similar problems, tack this onto the darktable-cli command above to output 16-bit tifs instead:
--core --conf plugins/imageio/format/tiff/bpp=16
Here's another handy bash command, which displays the metadata differences between two files:
diff <(mdls [file 1]) <(mdls [file 2]) -y --suppress-common-lines
2.3 Adjusting skew/perspective and compositing with Processing
Once the raw files were processed and converted to .tif, the next step was skew/perspective adjustments and resizing, which I wrote a Processing program for. The program asks for a folder of images, which are displayed in order; The user clicks on the four corners of the shelf, after which the clicked-corners are mapped to the shelf's IRL height:width ratio and the resized image is exported. The library has some slight variation in shelf heights and widths, so taking measurements and changing the IRL height & width variables was sometimes needed.
Processing's default texture-mapping renderer is kinda lazy: it draws an "X" connecting the four edge points, and stretches the four resulting triangles... which looks lousy. After some experimentation, I found excellent results from creating a large number of stepped points (i.e., lerping) between both the clicked-corner and export-image-corner points, and drawing them as vertices inside a loop. From a rendering standpoint, this creates so many tiny "X"s that the triangles appear contiguous and the resizing looks smooth.
One unexpected benefit of this strategy is that placing edge points in odd places or out-of-order creates some wild artwork!
Idea: Automate shelf-corner finding with computer vision
One way to speed this up would be adding colored stickers to the shelf edges prior to shooting, then using computer vision to automatically detect the four edge points. Again, this seemed unnecessary for the Prelinger Library's size, but could be a valuable time-saver for a larger library.
Creating large composites from rectangular photos is fairly straightforward, requiring only a few nested loops. But for composites of this size, using an offscreen PGraphics object is advisable to avoid possible pixel limits in size() and redraw() delays.
Idea: Skip the compositing and do the layout in OpenSeadragon
In order to keep the back-end as simple as possible, I created our nine stack composites and converted them to nine large .dzi files. However, OSD's viewer class includes robust options for handling multiple tiled images, and together with a more complex back-end setup, creating a single .dzi for each shelf and arranging them into stacks with OSD may be a more appropriate strategy.
The major benefit of this later strategy is that (1) OSD can identify which image (ie: shelf) was clicked on, and (2) arbitrary JSON data can be stored and retrieved for each image.
2.4 Creating DeepZoom images
Before your very-large stack composite can be used with OpenSeadragon, it needs to be converted to one a few different image tiling formats: .dzi, .iiif, .tms, etc -- See here for a full list. I used deepzoom.py to generate .dzi files; Here's the basic "hello world" they provide:
import os
import deepzoom
SOURCE = "example.jpg"
creator = deepzoom.ImageCreator(
tile_size=128,
tile_overlap=2,
tile_format="jpg",
image_quality=0.8,
resize_filter="bicubic",
)
creator.create(SOURCE, "./example.dzi")
Note: Decompression Bomb errors
Due to the size of my images, deepzoom.py raised an error about "decompression bombs" and exited. (This is apparently a crafty hacker attack where you create a small zip file that unpacks into a gigantic amount of data.) The error's stack trace includes the .py module where this check occurs if you need to disable it.
Note: PIL/Pillow error
One of the image-processing packages used by the deepzoom image creator has been superseded by a newer package, and you may need to install the older version. If you get an error about the PIL package, try:
pip install Pillow==9.5.0
3. Data Entry & Prep for website
3.1 Physical library
The Prelinger Library's idiosyncratic organization creates both opportunities and drawbacks for getting data into a website like this. Detached from a standardized catalog, I instead worked from Megan's guide as a starting point, going shelf-by-shelf and noting the zone/subzone, periodicals, and special collections.
Some areas of the library (oversize, ephemera, and Therkleson in particular) include material from a variety of different zones/subzones on a single shelf. In these cases, the first zone/subzone is displayed first in the sidebar "click view," with the additional zones/subzones under "Material also on this shelf." In the sidebar "list view," every shelf containing a zone/subzone will be highlighted; Hence the user sees every place in the library with material on that topic.
3.2 Scanned Items (Internet Archive collection)
Stacks Explorer v1 used a roundabout method to link scanned items to the physical collection: We entered keywords for each library shelf, and if those keywords returned a non-zero result when searching the Prelinger's IA collection, we included that search link in the interface.
This time around, we instead stepped through every item in the IA collection, and assigned to one-or-more of the physical library's zones/subzones -- That is, we "shelved" the scanned items into the library's physical organization. This means (1) every scanned item is present in the interface, and (2) some scanned items (most, really!) appear in multiple locations.
Note: Internet Archive CLI
The internet archive has a handy command line interface. This tiny program is deceptively powerful, check out the help text. For our purposes -- i.e., "shelving" the digital collection -- we didn't need to download the scanned items, we just needed the metadata. I did this in a two-step process (which you could probably fit into a single step if you're better with bash than me):
ia search "collection:prelinger_library" -i > item_identifiers.txt;
ia download --itemlist="item_identifiers.txt" --destdir=cli-meta-downloads --glob=*_meta.xml --no-directories;
The first command (1) sends a search to IA for collections with the name "prelinger_lirbary", (2) gets a list of all the item identifiers in that collection, and (3) outputs them to a text file. The second command (1) runs through that list of identifiers, and (2) downloads their [identifier]_meta.xml files into a single directory named "cli-meta-downloads."
This multiple-location shelving strategy led to some fascinating consequences, my favorite of which is the unsioling of children's material. Typically, material for youngsters is segregated into a physical location regardless of its topic; But with this new strategy, a kids book about cars sits alongside a technical document on highway design.
Note: Faceted classification
The technical term for this approach is a pure faceted system; And each "facet" (ie: zone/subzone) of the scanned item is given equal weighting (e.g., the ordering of a scanned item's zones/subzones is arbitrary).
3.3 Data Prep
Python's streamlined looping made it the easiest tool for getting our various CSVs into a more digestible .json format for the website. The main data file uses a 3D array structure for storing each shelf's data: That is, library_nested[4][11][2] contains the data for bank 4, shelf 11, row 2. The overall relationship between the files is quasi-2NF: Periodicals, Special collections, and IA items are referred to by index keys in the main file, with the details stored in additional files.
Once again, this data structure responds to our minimal back-end setup and is designed for quick load times. Compared to the data entry CSV files, we dropped from 673kb to 404kb, a 40% compression ratio. (It ain't much, but it's honest work.)
Idea: Libraries with catalogs have a more data options
Traditionally-cataloged libraries will have significantly more options for data encoding and retrieval; particularly when coupled with more complex back-ends setups.
For example, the first and last call number on a shelf could be noted, and used in a database query to retrieve all the holdings on that shelf (and even items in offsite storage!). Coupled with the earlier suggestion about using OSD's tiling layout and ability to store arbitrary data, these start and end call numbers may be the only non-image-layout data needed to power the interface. The call numbers could then be used to dynamically generate a list of topics for the shelves, or link directly to scanned or related material.
4. Website Structure
4.1 General Overview
- Not including the <head>, the index.html is only 100 lines long; Nearly all the content is created dynamically with JS. (Side note: This is terrible for accessibility -- we're brainstorming alternate interfaces for screen readers.)
-
Script.js begins with a chonky object named layout, containing all the variables needed for activities like:
- placing the stacks dzi onto the canvas
- figuring out which shelf the user clicked on
- highlighting shelves when a user selects something in the sidebar's "list view."
- Because several stacks share similar sizing characteristics with only slight variations, the layout.standard object contains a few more variables to accommodate these slight variations.
- The OpenSeadragon [OSD] canvas is loaded (unsurprisingly) into <div id=osd-container></div> in the .html.
- All the .json files are loaded and parsed (also unsurprisingly) in the JS function loadLibraryDataWithCallbacks(). After all the .json files have loaded, (1) the sidebars are populated,(2) the event handler for processing clicks on the OSD canvas (which reference the .json data) is added, and (3) the loading window cross-fades to the menu.
- From there on out, the JS code is primarily concerned with (1) translating back-and-forth between the visual space of the OSD canvas and the library data (see "Idea: Skip the compositing..." above for an alternate strategy); And (2) handling the UI functions in the sidebars.
The interchange between library data ↔ OSD canvas is the hairiest part of the code, so I'll describe how it works in these next two sections. (The non-minimized script.js is fairly well-commented if you want more fine-grain details.)
Note: JS coding style
Reading through script.js, you'll notice some style quirks: I'm typically using "for" loops with set end conditions instead of .forEach, for/in, and spread; The => operator is generally avoided; U16 arrays; No JQuery; etc. etc.
There's a two main reasons for this: (1) Because the sidebar lists are created dynamically, small inefficiencies are multiplied a thousandfold; and (2) Old Man Yells At Cloud: IDK I just kinda dislike jquery for some reason?? For the type of work I do, it feels like a totally unnecessary intermediary layer. But I freely admit that in many cases, it's basically overoptimization for the sake of overoptimization.
4.2 OSD canvas click → library data
The JS function getLocationFromViewportPoint() handles this process:
- OSD provides the stack number from the clicked image; Retrieve tileSources[].layout from the clicked image.
- Using tileSources[].layout, loop layout.bank_widths or uniform_bank_width to find the bank number.
- In tileSources[].layout, select the apropriate shelf_heights array, and loop to find the row number.
- During the steps to find the stack, bank, and row, if the click's viewport point is found to be outside the image's boundaries, false is returned.
- The resulting values are used to get the data in stacks_data[stack][bank][row].
4.3 Library data → OSD canvas/viewer
When the user interacts with an element in the sidebar lists -- e.g. carroting open the zone/subzones <details>, or clicking on periodicals or special collections -- the element calls the JS function hl(this):
- If the element is already highlighted, it's classList will include "text-hl1" or "text-hl2." If so, the element's highlights are removed along with its subelements' highlights. Otherwise:
-
this.id contains an alphanumeric string indicating the type of element (e.g. zone, subzone, periodical, etc), and an index in the relevant data source. For example:
- z-2 = zones_data[2]
- s-4-5 = zones_data[4].sub[5]
- p-139 = periodicals_data[139]
- This alphanumeric string is parsed in processElementForHighlighting(), and the resulting loc array (each item is a stack location, stored in [stack][bank][row] format) is passed to drawHighlights()
DrawHighlights() loops through the loc array, calculating the stack location's pixel position on the OSD viewer with calcViewerHighlight(), and creates the highlights (which are OSD Overlay objects).
CalcViewerHighlight() finds the stack location's pixel values in a 2-step process (x & width and y & height), and references the chonky layout object.
Each highlight (aka OSD Overlay) requires an HTML element so it can be appended to the DOM and styled as needed. The letter in the alphanumeric string determines the CSS styling applied to the Overlay, via classList.add().