Data Asset Management (How I Got Burned)
Preface: This is going to be a long, drawn out technical post about the trials and tribulations of digital asset management. It’s not for the faint of heart nor people with a social life. I recommend you go outside instead. You’ve been warned.
To the non-photographer, it’s hard to express the implications of securing your data. After all, if you were to drop your laptop in a puddle and you’re a normal person, you may have lost a few documents, a few emails, maybe some music. But, if you are a photographer, your memories, your archives, and your profits are all existing as 1′s and 0′s on a high speed rotating platter. Frightening.
I’ve always been moderately attentive to backing my data up. I’ve used a service called BackBlaze for years now to upload all my photos to an off-site location automatically from my laptop. Also, I’d periodically do a data dump to an external hard drive and stuff it in the closet when I felt compelled (not very often).
But, like all of us, I got lazy.
Last September, as I scrambled to get the Skvarla’s Wedding gift video completed, I suffered a laptop hard drive crash. A catastrophic, complete inability to load Windows at all. This was actually the second time that I’ve suffered such a problem on the eve of a video project’s due date. But it’s easy to forget how heartbreaking it feels when it happens.
The saving grace of that evening was that whatever type of error there was, if I plugged the laptop hard drive in to my desktop computer via SATA cables, the file system was still available. I drove to Best Buy and bought a replacement internal laptop hard drive. After a tedious process of copying, pasting, rectifying, and deleting, I had all of my most-important photos restored.
Lesson learned… for a while. Fast forward to last Monday.
I’m sitting at home, browsing the internet, when a brief blue screen flashes across my screen and then permanent darkness. My 7 month old hard drive simply couldn’t load any Windows files. After trying all the normal recovery methods, I began to panic. I forgot to install BackBlaze again on my new computer hard drive, so the last 7 months of work only existed in once location – the broken drive. (See picture to the right).
Gone. Vanished. It’s like someone stuck my hard drive in the washing mashing.
I began to consider the expensive alternatives. I got a quote from Kroll On Track that a simple evaluation of the disc would be $65, and the actual recovery would be somewhere in the realm of $695 – $1295… depending on the damage.
My data is almost priceless… but it does have a price and it wasn’t that.
I called some more local people, and finally wound up with an honest recommendation to try Disc Doctors Windows Data Recovery software. A free trial would show me all the files it could recover. After the first scan (easy mode, I suppose), most of my file-names were just a bunch of random characters. But, I tried the scan again, in their “Turbo Mode” and like magic, most my files were right in front of me! (I suspect there was a major bad sector in the NTFS file map or something). Some data was corrupted, and I’m still assessing which of my photos are gone forever.
So, I needed to actually buy the full version of the software to recover the data. The price was $129. Normally, I’d download it through other means, or try to find a freeware alternative, but I refused to take any chances and pulled out of my credit card. I finally learned not to take any risks with my data.
So now… slowly… piece by piece, I’m pulling old files back, getting them sorted in the right folders. It’s a terrible process. But about 90% of my data seems to be recoverable.
I also bought a new toy, a Drobo FS. This is not your grandmother’s external drive. This is a self-healing, data-striping, expandable vault that is almost fool proof. (No single point of failure can ever be trusted, and power surges, toddlers, and thieves do happen.) But what is nice about this device is that you can stuff it full of up to 5 SATA drives (cheap by memory standards) and it will show up on your desktop as one single drive. I have mine configured to be 2-drive fault tolerant. This means that if 1 or even 2 of my drives fails for any reason, data will be replicated on the remaining array to preserve itself and self-heal when I replace the felons with brand new, cheap drives. The odds of two drives failing at the same are very low, and I’m alerted by email automagically as soon as a drive starts to get cranky. (Also, the FS model is a network drive, so I can use it wireless anywhere in the house).
What should you be doing i.e. What should I have done differently:
First of all… never let a single point of failure exist in your archives.
Really, I only made two simple errors. My first error was that I was negligent in ensuring that my BackBlaze off-site backups were functioning normally. My second error was that I assumed that a brand-spanking, new Western Digital drive would last at least a year or two before I should think about backing it up. So I kept putting off making data dumps to external hard drives. So remember, friends: hard drives are often the first thing to fail in a computer, and they fail unpredictably. Data today can be gone tomorrow.
So here are my rules for happy data.
1) Decide what files you want forever, what files you want for a little while, and files you won’t need tomorrow.
I back up in different ways for different things. For example, my photos are just about priceless. If my photos from high school and college disappeared, I’d be heartbroken. So, it’s essential to maintain them as if they were such. My music, while tediously maintained, rated, organized, and categorized, is replaceable – although at a high cost. Documents and email that are important to me are almost exclusively stored online with services like Google Docs, Dropbox, and Gmail. Stuff just dropped on my desktop probably won’t matter if the laptop gets a little emo.
2) Have a backup strategy.
If I chose to backup all of my photos and music offsite with services like BackBlaze or Carbonite, it would take months of downloading to retrieve everything, not to mention the dangerous penalties that Comcast threatens when you use to much bandwidth. BackBlaze, for $200, can send you a hard drive in the mail, but then you might as well do backups yourself. So this is how I roll now:
Three tiers of backup:
Irreplaceable: My photos, my letters of recommendations from jobs, photography for clients. All of these things could be professionally as well as emotionally damaging if lost. I have my backup software configured to store this type of data on my Drobo ASAP. Additionally, every month or so I keep a standard external hard drive synced with the Drobo and I keep it somewhere else. Some people choose to do this and stuff the drive in a bank’s safety deposit box. Some people mirror their first array over the internet with another array. The point is: if the data can’t be replaced, it needs to exist in multiple geographic locations. Fire, theft, earthquakes, children, and all other natural threats can’t harm your data if its replicated somewhere else on earth.
Inconvenient if lost: If there is room on the Drobo after the first category, I stick other files on it. This is usually associated with my music and iTunes library file (which is priceless in it’s own way). Usually, the volume of this category allows me to keep copies on both the Drobo, my laptop, and maybe an external, and is relatively safe. Important note: I got boned with a corrupted iTunes library file once upon a time. Since I very actively manage my library, the best way I found to revive it is to use CopyTrans to pull the entire library from my iPod. All my ratings, playlists, etc. were restored and it was well worth the $20 to me.
Loose ends: For everything else, I rely on offsite backups via BackBlaze. My pictures, music, and movies are too large to make this solution practical with internet backups for the most part. However, BackBlaze runs continuously with any files on your hard drive, so it excels in its ability to bring you right back to the point of hard drive crash. This is especially useful to use when you are a frequent traveler like me, because even though I can’t always get my files on my Drobo or an external hard drive, I can usually get things up on the internet. Day to day documents, presentations, spreadsheets, whatever else, these are all safe (and also have older versions stored – very sexy).
3) Remove single points of failure.
Is Moammar Gaddafi’s face melting?
Notice that in all three of these categories, data is always in more than one place, and often more. If a hard drive in the Drobo dies, I’d have to lose two more before data would be gone forever. And if it did, it would only be lost up until the point of my last sync with the external backup. I try to do this monthly, so even if the Drobo gets nuked by Muammar Gaddafi, there is no complete loss. My music is always backed up on my laptop, my Drobo, and my iPod. Even little documents typically exist on my laptop, my BackBlaze, my dropbox, and so on. Multiple systems have to fail for me to lose data. This is damn important.
4) Be diligent.
I’ve learned all these tips because I’ve suffered before. I’ve lost pictures I can’t ever get back. But, it’s taught me the lessons that will protect my data in the future, so I’m sure it was well worth it. Sure I’ve lost a couple of clients’ portraits, but now I will know how to protect the photos of my honeymoon. Don’t get burned, let my burning influence you. I have everything set up to do automatic backups (I recommend SyncBack but there are lots of freeware software available). The only manual intervention really comes when I backup the Drobo externally, or when I sync my iPod. But even though these are pretty small tasks, if I let it go months before doing it again, I could potentially lose data if the Drobo dies. So my point is to create manageable habits and strictly follow them.
I’ve learned the hard way, but hopefully all of my friends will learn from my mistakes and never lose a file. My mom has a cabinet full of pictures in the dining room, and you need to see the parallel between her wooden box and my 1′s and 0′s. She has her high school pictures in physical form, which has pros and cons compared to data, but their value is the same in the end.