NOTE: I found this article on San Diego photographer
Laurens Antoine's blog and thought it would be helpful for others.
A popular subject amongst photographers is how to store and secure all the photos you've taken. Until not too long ago, we stored all our images on a rack server and backed it up in messy ways. Not anymore.
The price of storage has come down allowing almost anyone to have inexpensive and secure ways to manage data. We're going to explain how we do it.
Our data is explaied into three categories:
? Current data: Recent shoots, and those still being worked on.
? Archived working images: Published and retouched images.
? Archives: All the raw files that came out of the camera.
There are several factors that cause us to maintain our current files the way we do. One is speed, the other is security. We always want at least two copies of all data, even as we shoot. Backups aren't live so our primary storage is a RAID. We back the RAID up regularly.
RAID comes in several types. The full article describes them but for this website we've removed the descriptions.
When you first create a RAID array, it can use part, or all the drive space. On 1 set of drives we use just 1 array, but on another set, we use 2 and we'll explain why later. Each array can have multiple partitions and volumes, but we use just one of each per array.
About RAID cards:
Not all RAIDs are alike. Many motherboards come with RAID on board. These are generally what is called "host" RAID. The thinking comes from the computer's CPU. If all you are doing is striping drives, this can work well, but in a RAID that requires more processing (something RAID 5 does a lot of) a dedicated RAID controller with its own processing can really speed things up. We use an Adaptec 51205 which is fast and can handle up to 12 separate physical drives. These cards run about $1000 and are plenty fast for a small shop like ours. You can fit 12 drives into a number of cases that cool well and cost little.
OUR MAIN SYSTEM IS SET UP IN THREE ARRAYS THAT SPAN TWO SETS OF DRIVES
The system array.
We don't backup our system as often as our data. For the operating system and software we use four drives in a RAID 10. It's about twice as fast as using a single hard drive, and secure. Frankly, we usually use hand me down drives from our data arrays for this. Next time we rebuild it, we'll probably go to mirrored SSDs for the system.
The data array and the faster stripe array.
This is where we start having fun. We have 8 drives that share 2 arrays. The first and main array is a RAID 5 that spans all 8 drives and nets us about 9 TB, and a smaller stripe array of around 300 MB (we presently have 1.5 TB drives loaded).
The RAID 5 data array is redundant, but a bit slow. However, it does hold a lot of data.
The faster stripe array also spans all 8 physical drives. It was created last so it uses the outside of the hard drive platters that spin faster. We primarily use it only for things like swap drives and scratch files. At times, when working on a lot of large image files that get opened and closed a lot, we'll put working copies on there as well. An 8 drive stripe runs really fast reading and writing - as fast or faster than SSDs.
This gives us the best of both worlds. Speed where we really need it and redundancy for our data. And, it's cost efficient.
Backing up the RAID.
First, I'm going to mention that this whole method is fairly new to us. We used to use a rack server with countless small drives that were backed up on tape, and later a variety of external devices. Then we migrated to the RAID system described above but backed it up to an external ESATA RAID. Again, not anymore.
What's changed is the cost and speed of external drives. Now with USB 3.0 and 3TB external drives costing so little, we just do incremental backups to two 3TB external drives that we bought at Costco. When the data RAID approaches 6TB, we archive more data. More on that soon.
ARCHIVING WORKING DATA
Archived working data is different than the archives. This is what we call all the images that have been worked on. This data space is where we keep all the PSD files that get published or otherwise distributed. It's still of a size where we can keep it in a folder on the data RAID so that when someone needs an image that's already been created, it's easily found and accessible. We may soon discontinue that practice; you'll understand why when we describe the newer archives.
We started shooting digital in mass around 2005 when the Canon IDs Mark 2 and the 24MP digital backs came out. Shooting film became much scarcer a couple years later around the time Hasselblad came out with the H3D. Until then we still had quite a few creative and art directors that demanded film. Today we have none, so the amount of data has increased, as has the size each of the actual files.
As mentioned, we used to use a rack server with a tape backup. Before then we had images stored on DVDs, literally thousands of them which have since been migrated on to hard drives.
At some point we started using 1394 drives, but now we use external USB 3.0 drives exclusively. In fact we just recently finished migrating all our old 1394 drives over and have a nice junk pile of them waiting to be washed (secured and permanent deletion of everything on the drive).
We're about to add 4 more drives to clear up the RAID, but our present archives sits on 10 separate drives and the best part is that they are all hooked up through hubs, and all indexed in Lightroom. Everything is reasonably accessible, and the drives are in sleep mode except when needed. This is why we may be rethinking our archived working file location.
One important mote: We strongly recommend buying external drives in pairs. They do fail, but they're inexpensive. We keep one set in house, and another offsite.
Reposted with permission: San Diego photographer Laurens Antoine