Ingestion

Data preservation and data backup

The primary goal of image ingestion is to preserve image data and create backups. Image ingestion from memory card(s), tethered cable, or wireless requires similar steps. In the Capture section, we referred to the ideal set-up of using a backup card when shooting if your camera model supports that function. If you do not use two cards for capture, best practices are:

• Regularly download your single cards to a disc or storage device
• Do not format cards until the downloaded images can be reviewed and backed-up to a second drive
• Use a card reader and not the camera for downloading image files unless you are shooting tethered
• Always use download software - do not use the Finder application to copy the image files

Using the Finder to copy files is potentially unreliable, sometimes resulting in skipped or missing files. Never use the finder to move files from the memory card. If you move (not copy) files or folders between the memory card and computer disk and some glitch happens to the destination volume, or your connection with it, all your files vanish—both on the memory card and on the destination hard drive. There are many other good reasons to use dedicated ingestion software, but image preservation is the most important one.

File naming on ingest

Any download utility you use should support your naming convention and it should be able to create the name automatically during the ingestion process. Be careful if you shoot raw + JPEG and rename on ingest. We found that some software gives the raw files and the JPEG files sequential numbers instead of the same number with different extensions. This behavior can vary according to the model of the camera and whether ingest is done from cards or done auto-imported when the camera is tethered.

Those who use multiple cameras or those who do not want to have gaps in file sequence numbers should not rename on ingest. For everyone else, rename on ingestion is a timesaver. If you do rename at this stage, be sure that your renaming system follows best practices.
Read more about file naming in the file naming section

Add metadata, presets, and optionally catalog information

The second goal of the ingest step is to advance the workflow by adding

• Metadata
• Image adjustment presets
• Applications like Image Ingester Pro software can even add cataloging information as the files come into the computer.

For the cost of some set-up time, batch metadata can be added to the image files via metadata templates. This will save lots of time later in the workflow and doesn’t add significantly to the download time. Adding basic metadata at this stage helps to insure that all derivative files will have the same base metadata as the originals.

We recommend that your basic metadata template should consist of:

• File name (in the Document Title Field)
• Name of the creator/author
• Address
• Phone
• E-mail
• Copyright status
• Copyright notice
• Generic rights usage

This basic set of metadata can be built on to provide more specific custom metadata as the image files go through the editing process.

This additional metadata might include:

• Description and/or headline
• Specific rights usage (possibly PLUS code or usage description)
• Client name
• Location
• Keywords
• Star ratings
• Special Instructions

Ingestion tools

Card readers

Card readers come in many configurations to accommodate different computer connections (USB, USB2, Firewire 400/800) and several types of memory cards (Compact Flash, Secure Digital, Memory Sticks, etc.).
If you have a collection of cameras, from point-and-shoot to high-end digital, you may need an all-in-one card reader to accept a wide variety of card sizes. This type of card reader is usually a USB device.
You may also need a higher speed Firewire 400 or 800 compact flash (CF) card reader. If you photograph events or sports and have a need for speed, you may want to have a set of Firewire 800 card readers. These stackable units can download from multiple cards at the same time.
The gold standard for information on flash memory cards and card readers is available on the Rob Galbraith site. As with most other things digital, plan to periodically replace older memory cards with newer ones that have greater storage capacity as well as increased read and write speed. Luckily, they have gone down in price.

Download/ingest software

Many software applications used for image ingestion are multi-function—meaning they can be used as a browser, parametric image editor, cataloging application, or all of the above, as is the case for such all-in-one programs such as Adobe Lightroom and Apple Aperture.
One software stands out as purely dedicated to image ingestion, the aptly named ImageIngester. Another widely used application, PhotoMechanic, adds a browsing function as well.
Camera manufacturers make downloading utilities and browsers, although their metadata support varies. Adobe Lightroom, Apple Aperture, Adobe Bridge and PhotoMechanic 4.6, include image ingestion along with tethered shooting (Aperture directly, and Lightroom, Bridge, and PhotoMechanic indirectly by means of auto import from a watched folder fed by the manufacturer’s capture utility).
Some cataloging software, such as Expression Media have import functions, although few use it for that purpose since it jumps ahead too far ahead in the workflow.
Phase One’s Capture One software is widely used for tethered ingestion although its metadata support is extremely minimal.
Bibble Labs’s Bibble software also offers tethered shooting. However, its metadata support is only slightly better than Capture One, although like Capture One, it offers advanced parametric image editing capabilities.

Tripwire: Earlier, we said that managing image metadata could be confusing. We face that confusion firsthand in the ingestion step. Some of these applications (example: ImageIngester) write only the XMP variety of metadata. Others (example: Aperture) write only the legacy IIM IPTC variety. Adobe Software writes both varieties. PhotoMechanic lets the user decide whether to use XMP or legacy IPTC and where to put it.

Figure 1 Photo Mechanic 4.6.1 IPTC/XMP preferences pane showing options for add embedded IPTC & IPTC4XMP to TIFF-based RAW unchecked.

PhotoMechanic issues:

PhotoMechanic can embed XMP metadata in proprietary raw files (although not any longer as a default setting). Not only is this a potentially dangerous practice, it can cause a metadata collision with Adobe Camera Raw where a proprietary raw file can have embedded XMP metadata from PhotoMechanic co-existing with attached XMP (sidecar file) from Adobe Camera Raw. This will not appear to be a problem, unless the same files are put back into PhotoMechanic and any change is made to the metadata. A change will cause Adobe Camera Raw to read only the newer embedded XMP, and ignore the older XMP (camera settings in the sidecar file). When this happens, all your parametric edits will be lost, and you’ll have to start over with your color and tone edits. Our advice is to make sure that the PhotoMechanic preference to always create a sidecar file for the XMP data.

Handling memory cards

• Have a plan for keeping shot cards separate from un-shot cards.
• Make sure that the un-shot cards are formatted and ready to go.
• Establish a numbering or coding system for your cards. If you find corrupted files, this will make it easier to track down which card may be creating the problem. It also just helps you keep track of your cards.

Backup

The ideal plan is to use software that writes files to two separate drive destinations. Use of a RAID1 external drive is highly recommended. But if your software only supports writing to one destination, second choice is to regularly download to a second drive, although this doubles your download time. Third choice is to regularly copy files from the first download to a second drive. In all cases, memory cards should not be formatted until downloads have been verified.

Verifying ingested files

The easiest method of verification is to put the downloaded files into a raw processor, and let it build new thumbnails from the raw data.

Be aware that browsers set to use the embedded camera previews cannot be used to verify raw files. The preview JPEG can appear fine, but the underlying raw data can be corrupt. When using parametric image editors (or browsers set to build previews from the raw data) be aware that you need to let them build their cache and create thumbnails from the raw data before you can judge whether the underlying raw data is good.

Convert to DNG (optional)

Some software supports converting files to DNG format on ingestion. We feel that this promises to be the most secure way to handle images because of the data validation hash built into the DNG file. This hash file will serve as a means of telling if any corruption has occurred to the file in subsequent handling and storage. However, you will need to perform at least one visual inspection of the files since the hash will only report any data changes that occur after the DNG file is first created.

Lightroom users have added incentive to convert to DNG on ingest since Lightroom gives you control over when you update DNG previews. This presents a potential workflow bottleneck for Bridge users who choose to update DNG previews automatically when making PIE edits. The Bridge user may have a better workflow experience by waiting to convert to DNG until the end of the initial adjustment process, as the images are being prepared for archiving.