Maintaining the Archive

How do you organize your photos? The answer to that is critical. Anyone who generates a lot of images, and that is just about everyone these days needs to answer that question.

XKCD Photo Library Management
Photo Library Management from XKCD (Randall Munroe, Creative Commons Attribution-NonCommercial 2.5 License.)
Keeping my photo archive organized is a bit of a chore. But skip on the effort and it simply gets worse, to the point of being unusable. If you can not find the photos you need why take photos at all?

The trick is to develop a process and to use it… Religiously. I can not tell you how to do it, I can just tell you how I do it and offer a few suggestions.

There are two basic approaches, simply come up with a way to organize the images into a directory using nothing more than your operating system. The other approach is to use some form of photo organizing software to aid in the task. I do make a large assumption here, that the images are in digital format, not negatives and slides. For that you will have to look elsewhere for answers.

Just the OS

You can organize your files very simply using whatever OS you happen to have. Be it Windows, MacOS, or some flavor of Linux, it really does not matter. The goal is to organize your photo collection in a simple and effective way that can be maintained over the decades and easily moved to a new computer or operating system as technology changes. The trick is to keep it simple in order to make the system easy to navigate and to use.

Using the operating system it should not be difficult to come up with a way to organize your into directories. An arrangement where the images are kept in directories by date is usually the answer here, requiring little more than creating directories with dates as the folder name and coping the images into these directories.

Let us consider a possible directory structure… The top level will simply be called something like ‘Archive’, a single folder that will contain everything. Opening this folder you will find a long list of folders, each from a photographic event, each named with a date and subject. For every outing or studio session you create another folder, drop the images in, then move this folder into the archive folder. Simple!

The number of folders or files in a folder used to be limited by the operating system, this is generally no longer the case on modern systems. Windows now allows well over four million files or folders in a single folder, most linux versions are unlimited. This is notably different than other file systems photographers use. The simple file structure used on SD cards drastically limits the number of items in a single folder.

Slides
Slide film awaiting digitizing
One key part is to allow the system to naturally organize itself. The OS will organize the folders or directories alphabetically. Choosing a folder name that puts the date first will allow this feature of all OS to make it easy.

If the folder name is something like ‘20170712 Road Trip‘ where the year, month and day are the first part of the name, the folders will be sorted by date without any further effort on your part. The order is important, Year, Month, Day to get the sort order right. Such a name works for both the computer and the human, you can still read the date in the filename at a glance.

Naming the Files

Another item to consider is renaming the files for each image. the camera will give the file a filename like IMG1234.jpg or GOPRO000123.jpg. You can change this in some cameras, but odds are you will be gathering together files from several different cameras, each with a different naming method.

The Photo Archive
A directory of folders in the photo archive showing the folder naming convention
When gathering together large numbers of images from different cameras it is likely that the filenames will not all be unique, you run the risk of repeat names, overwritten and lost image files. Giving everything a new, truly unique name, is something to consider.

The format of the filename is subject to the same ideas as the folder names we discussed above. Keep it short enough that a human can read it, structure the filename so it sorts properly in a computer directory. Make sure the filename structure can accommodate tens of thousands of unique names so you never have a duplicate.

Obviously renaming hundreds or thousands of files by hand is a time consuming task, probably to the point of impossibility. Some sort of software assistance is going to be needed here. Many photo organizing software aids support this feature as will be discussed below. The other possibility is some sort of simple script, a small program that can do this automatically for a directory of images.

The key part is to properly sit back and think about your solution here. Consider the arrangement, how easy it will be to use, and how it can grow to keep possibly tens of thousands of images organized.

A major advantage of the an all OS style archive is the ability to easily backup or copy your image catalog. Properly arranged it will all be in a single directory, making a copy of everything is as simple as copying that directory to another device, or an external hard drive.

A Software Solution

Besides using a simple OS solution to organize your files, there are several software packages specifically designed to solve this problem for you. I am going to discuss Adobe’s Lightroom as it is an excellent example of this type of photo archive organizing tool and the most widely used.

Adobe Lightroom
The library screen of Adobe’s Lightroom
Available for both PC and MacOS platforms Lightroom is a powerful tool that offers both a photo editor and an archive browser all in a nicely integrated package. The photo editing tools are first rate, with often industry leading capabilities, supporting professional level photo processing. The archive tools are good and can maintain an archive with tens of thousands of images.

One of the most important features of Lightroom is that it does not modify the source files, preserving the integrity of the archive. The software simply stores a recipe, a list of processing steps needed to create the final output image that is applied each time you request an export from the archive. This recipe will contain instruction on cropping, sharpening, color balance, etc.

Lightroom maintains a catalog of all the images in your archive. You must import the images as a specific step before they are cataloged. This can be done straight from the SD card or other storage media, or can be applied to an existing directory structure containing the images. You can also define a file renaming process with a custom format to be done upon importation of the images into the catalog.

One imported you can add keywords to images to aid in sorting and searching. There is also a built in Google map that displays the location data if the image is geotagged by the camera. Or you can select images and apply a geotag simply by clicking on the map.

Using a software package to index your images can automate a great deal of the process, while offering some risk. You should carefully learn and understand everything the software package does to your precious image files before getting started. Where will the file be on your hard drives? What is done to the file? Does the software add metadata to the image file? How are the files renamed? What will you do a decade or two down the road when the software or operating system is no longer available? Learn and think before you import that first image.

What to Archive?

One question you do need to ask yourself is “What do I archive?”. Personally I only archive photos I wish to keep as a permanent record. Images meant for transitory use, documentation, and low quality snapshots are not added to the primary archive. I do not archive my time lapse images, that would add tens of thousand of almost identical images, clogging the entire system. I do not archive my astrophotos for much the same reason. I do keep these images using a similar system, I just do not index them and make them part of the searchable library.

How I Do It…

So? How do I deal with tens of thousands of digital images? Yes, my archive is that large, about 2Tb of images right now. I use a combination of all of the techniques above. I do use software to view and sift through the image archive, but that archive is organized simply by the operating system.

Milky Way Dawn over Mauna Loa
The rising summer Milky Way, Mars and dawn over Mauna Loa
The first step is to dump the memory cards from the cameras to a specific directory. I can dump one card or a dozen from several cameras. The only caution, and this does happen, is to not let any file overwrite another in case two cameras produce the same filename. Just watch for any warnings when copying the files.

I can then do a rough sort on the images, looking through them and seeing what I have. I wil often delete any bad photos at this time. Photos of the ground when I accidentally hit the shutter button, dark photos when I had the settings all wrong, the one second video when I hit record instead of the shutter, the obviously unusable stuff. Be honest, you have this cruft too.

I then run a script that renames all of the files into my naming convention. My format is AANN-NN-YYDDD-FF, where the first AANN-NN is a roll number and frame number. Yes, this filename convention was invented when I still shot film and simply called it a catalog number.

Madonna del Sasso
The Madonna del Sasso church above Locarno, Switzerland
Despite its heritage, this filename still works in the digital era, I just let the roll number and frame number increment sequentially. As this is letter-letter-digit-digit-digit-digit, there are 26×26×10×10×10×10= 6.76 million possible combinations, that will keep me going for a while without a repeat filename.

This filename is also easy to read and remember when looking for a specific frame. I am looking for CD23-12, no problem. The number can be computer sorted and human read. The remainder of the filename is a date, YYDDD, year code and a Julian style day of the year.

The last two characters are a format type FF, two letters to indicate the type of source. I have TS for transparency/slide, DC for digital camera, and a few others. I can also modify this last bit to create different versions if I have edited files that I do not want to mix. Edit the image and make the last two characters of the filename D2 or D3 for each version.

Once the script renames the files I copy to a directory with a name like YYYYMMDD Subject. Drop all of the correct images into that directory and I have one more step, move the new folder to the main archive folder where it joins a thousand others created over a lifetime of shooting images.

As I said I use a combination of a software approach and the OS approach. While renaming and storing the file is handled using only features of the operating system in my method, indexing and viewing the archive is handled by software. Once the folder is copied to the archive I then import it to a Lightroom catalog.

Kupapa'u Lava
An active pāhoehoe breakout at Kupapa’u
The important bit is that I have the preferences set in Lightroom to leave the files in place, not to move, copy, or rename the files. This is not the default setup in Lightroom and must be changed before you start using the software.

Why not just use the built in naming and organizing tools offered by Lightroom? While these features appear to be quite good, they just do not support my system which vastly predates excellent tools like Lightroom to do the job. If I were starting over today I might very well be tempted to use a more supported method in place of my own scripting and directory copying method. On the other hand I worry a bit about the day when Lightroom is gone or something better comes along, it may be better that I have kept a system that is mostly software independent.

This process has worked well for me. In reality it takes just two or three minutes to download all of the images on my cards, rename the files, and move the images into the archive. I have the power of a good indexing and viewing system and I know where everything is and can make a catalog backup without any trouble.

Author: Andrew

An electrical engineer, amateur astronomer, and diver, living and working on the island of Hawaiʻi.

Leave a Reply

Your email address will not be published. Required fields are marked *