Conor Cunningham’s blog on SQL Server, data-driven applications, and pretty much whatever other random stuff he decides to post.
I'd like to talk about storage.. well, backup. Not SQL Server backup, just "I wish to keep a copy of all of my files and make sure that I don't lose everything if the house burns down". This happens to include my SQL Server databases, though that's not really my problem case. I have spent the last few weeks trying to come up with a solution to this problem for my house, and I frankly don't have a great answer yet.
So my lovely wife is a photographer (http://www.shannoncunninghamphoto.com/), and a darn good one if I may say so. When she does weddings, she takes lots and lots of pictures, in RAW, with a Nikon D3 (12 megapixels? ~10MB/file). She has 1000-1500 images per wedding, and some number up to say 50 or so in a year. So for those of you uninterested in math, that's 10GB+/wedding. Over the course of the year, this is many terabytes. That's a lot of bytes. That's just the work photos :). We have a wonderful 2 year old girl who has been photographed who knows how many times now, but the size is large enough that I am in trouble when it comes to backing up our personal digital photographs as well.
How does one back all of this stuff up? There are a couple of different "solutions":
1. off-site, internet-based photo storage. Well, most of these charge you for storage, and the scale at which we are working just doesn't fit very well into the business model for a photographer (it costs too much to bother storing pictures like this, especially since in many cases these will not be commonly used).
2. commercial tape drives. Many of these are set up to back up enterprise systems with some IT guy who can sit there and swap tapes for the weekly full backup, with the incremental being small enough to fit on a tape. (80GB? 160GB). That won't handle all of our personal files for a full backup, and perhaps not even a busy month of wedding photos.
3. NAS/Raid 5 array/whatever - these are usually 1-5TB - not big enough in the long run - I need to store the wedding pictures for a year or so, and I rarely need to access them. So, this is overkill. It doesn't really solve the desire to get a remote storage solution as well.
4. Burn DVDs. Well, this is what we do now. Burn 2 copies, take one set somewhere else.
The problem with (4) is that I also have to swap out lots of DVDs. Even if we store just the "good" images, it's still > 1 disk change for a wedding, usually more, and it takes a lot of time and dedication to do this.
I will admit to you all that I am far too lazy to spend time on this for my personal files - my wife does her weddings, but it take a long time.
So I've been looking into a DVD swapping robot arm. (yes, it would be awesome.. I want one to hold my drink while I program as well, now that I think about it). I have concluded that either I'm nuts or I just don't think people realize how much value I place on my laziness. I really, really, really want a robot arm that will just take a stack of DVDs and burn things to it, figuring out how to split the files and such. I'll pay money for it. I promise!
One would think that this would be easy to find. I did eventually find a few things on the search engines by entering "DVD duplicator". There are a few companies that make devices that seem to have all of the parts that I need but focus on making multiple copies of disks. I do actually want 2 copies of everything, but I really want a backup solution that will change the disks for me and put them in a pile. Labeling is nice but I don't need anything fancy - this is a backup.
Then I found a product that is called "DupliQ" that appears to do what I want, in that it claims to work with Nero 6 Express. Perhaps that means it can change disks? Of course, Nero 6 is really old, and DupliQ doesn't even claim Windows 2003 Server support, much less 2008 or 64-bit support. So, even though it only supports 25 disks, I can imagine getting up to put 25 of those at a time in ;).
Nero apparently had a component, at one point, called NeroRobo, which various postings by others on 3rd party sites claims was available for more money. It's not mentioned on the Nero website, which tells me little - perhaps it is in their standard product now? I can't find mention of it in their Nero 8 help files.
The Windows Server 2008 site doesn't mention cross-disc backups for the new program in that version of the OS (it does de-support tape drives, which either means it isn't the way to go or perhaps Microsoft isn't trying to win that market or third party partners are doing a great job on their own).
It seems odd to me that most of these robot arms claim to be "DVD Duplicators" instead of "Automatic DVD Burner Changers". It's like marketing a hammer as merely a "house-building device". Or perhaps these guys just aren't useful for backup and I'm the only one to have imagined this problem...
So, as backing up databases to external storage seems like a similar problem, I'll ask - how should I do this? How should someone backup more data than what fits on a single disc/tape/whatever?
If someone has a solution for a robotic arm to hold my drink and worry about my thirst while I program, I'm also interested in ideas for that.
I worked for a company in Dallas that had a Pioneer DVD jukebox, actually. It was about the size of a fridge. Google for "Pioneer DVD jukebox" or just any dvd jukebox and you'll see tons of 'em. You can buy them used fairly inexpensively through resellers because of a few key problems. For one, you have to have custom software controlling the robot. Ours used an off-the-shelf package from Pioneer that made each DVD disk appear as a folder inside a single virtual drive. When you wanted to burn a new DVD, you created a new folder, and after enough stuff was written there, boom, a DVD was written. Had tons of problems with the software, but that's another story. Second, you can't use it in a multi-user environment because it takes too long for the robot to take an existing DVD out, put it in an empty slot, grab the necessary DVD, put it in the drive, and spin up the drive. Latency is horrible for network use, and Windows will time out waiting for files. I've got plenty of horror stories - ping me if you want more info.