Gizmodo 
Enter your username and password.
Tip your editors:
tips@gizmodo.com
Editorial Director:
Brian Lam | Email | Twitter
Editor:
Jason Chen
Email | AIM | Twitter
Features Editor:
Wilson Rothman
Email | Twitter
Senior Contributing Editor:
Jesus Diaz
Email | AIM | Twitter
Mark Wilson, Reviews
Email | AIM | Twitter
Contributing Editors:
Matt Buchanan
Email | AIM | Twitter
Adam Frucci
Email | Twitter
Sean Fallon
Email | Twitter
Jack Loftus
Email | Twitter
John Herrman
Email | Twitter
Dan Nosowitz
Email
Chris Mascari
Email
Danny Allen
Email | Twitter
Columnist:
Brendan I. Koerner
Interns:
Don Nguyen
Email
Chris Jacob
Email
Comment Intern:
Nick Ellenoff | Email
Comment Account Questions:
Comments@gizmodo.com
Original material is licensed under a Creative Commons License permitting non-commercial sharing with attribution.
10/20/09
It's an oddity how many IT folks preach about backups, disaster recovery, and the like yet walk back into their glass house every single day and fail to practice what they preach.
It doesn't just apply to backups either. Password expiration, weak passwords, bad permissions, one time fixes, and the rest seem to be common place. It's absolutely amazing how lazy IT folks can be when it comes to doing their own job.
10/20/09
And then the disaster hits, and you check your backups... oh, ooops. That 9-track tape you had your data on was failing silently. Ooops, the offsite data got overwritten because your software was buggy. Dang! That disaster recovery plan forgot to actually TEST the backup hardware...
The problem with backups and disaster recovery is that if ANYTHING goes wrong, AT ALL, and there's lots that can go wrong, you look completely incompetent for all that look, even if it was a good plan. Because there's no difference between "it SHOULD have worked but the vendor-provided backup solution had a minor bug which erased your data" and "there was no chance in hell your data was surviving, and those of us over in IT are laughing our asses off at you" to the end user. #tmobile
10/20/09
10/20/09
Why? Because backing up Petabytes is HARD. It takes a long, long time. Testing it takes even longer. If there's corruption in a backup, sometimes even an otherwise good testing routine can miss it. Sometimes, hot-swap hardware is tested and verified, but fails because the data is bad. Sometimes the hot-swap hardware is tested and verified, but fails because it wasn't used for so long, dust built up and caused a short.
And if ANYTHING goes wrong, at any step, you look like a fool. If you pronounce a firm 'recovery' deadline, and miss it because of an unforeseen problem, you look like a fool. Not to mention, on top of this, you have an expectation of staying within budget, and doing it right is EXPENSIVE, but not everybody can afford multiple redundant servers separated geographically, with multiple ISPs, redundant power supplies and power sources, extensive testing of the tape backups, et cetera. When the boss turns any one of those, and it results in downtime, YOU still look like the fool. The user is still out their data.
It's easy to say "they preach it, but don't follow it", but in reality, they preach it because they understand how very, very hard it is, and most people don't understand that.
Wow, that went long. Anyway, long story short: I don't work with backups, and dear god am I glad I don't... #tmobile
10/20/09
I don't question the difficulty because I've been there (not on that scale obviously), but that is what you're paid for and if you can't do the job correctly or have the balls to tell the bosses what more you need, you have no business being there. #tmobile
10/15/09
10/15/09
10/16/09
10/15/09
10/14/09
So... assuming that was technically possible, part of the timebomb then looked and functioned like the backup software, but erased the tapes while reporting that it was backing up data, and did this far enough in advance that all of the sets of tapes were wiped?
Could happen... in theory... but it would take a hell of a lot of work and understanding of the systems they were working on - deep enough to send commands to the tape drive from an original program, or maybe run some console/invisible instance of the real backup program with instructions to silently wipe tapes, while the frontend displayed normal status.
...or they sat there themselves some time and wiped every tape manually - ideally from someone else's account or an admin/service account to defeat the audit logs.
I don't buy it though - that would be some unprecedented malice, risk, and orchestration to make it come together.
10/14/09
10/14/09
Ima Set It Straight, This Watergate
I Can't Stand Rockin' When I'm In Here
'Cause Your Crystal Ball Ain't So Crystal Clear
So, While You Sit Back And Wonder Why
I Got This Fucking Thorn In My Side
Oh My God, It's A Mirage
I'm Tellin' Y'all It's Sabotage
Sabotage
Artist: Beastie Boys
10/14/09
This kind of thing happens all the time. Admins get lazy, tapes don’t get checked, backups are partial, corrupt, or blank, and that is that. I’ve worked at places that had over 12 months of garbage backups when I started, amazing…
If it was sabotage the person would have made sure the backups were useless (not hard to do) and then nuke the system, not delete backup tapes he/she had no access too… I have to believe they did off site storage of tapes.
10/14/09
10/14/09
I've seen at least 10 Admins fired over the years for bad backups and once the CIO all the way down to the PC Tech that was swapping tapes (8 or 9 people including the AS400 guys even though they didn’t loose data they didn’t have backups either) were fired which is also when I started there... What a mess that place was, but it was a fun job.
@OldSchoolGadgetLover: It's not as bad now, but back in the day usually what prevented people from having proper backups was cost. If you did things the right way you had at least 2 weeks of daily's and then 3 months of weeklies and a year of monthlies, but a few of the companies I worked for didn't even have enough money to buy tapes for the monthlies...
I went into a school district around here where I told them to buy a 50 pack of DLTs with their new servers. They cheeped out and bought 6... S I X tapes for 6 servers, I couldn't believe it. I must have brought it up 20 times and they kept saying "sorry we don't have the money". They had also bought an external array which had I think it was 8 or 12 drives in R5 which they were using as storage and backup (again against my advice).
Long story short that array failed due to the incompetence of a Compaq tech and they lost their data.
10/14/09
We found out the hard way that the dailies were worthless because we failed to trap for file locks skipped backing up the incremental changes when users were signed into the system and using files. same with weeklies if batch jobs were running against key files.
When we had a data center AC failure, resulting in multi disk array failures, this little problem came to light, and many full grown men cried.
Guess some things change... and some things stay the same.
10/14/09
I've had some close calls, but never a complete loss. Doing Net/Sys Administration (14 years now) is too forking stressful when your doing it solo. I'm actually an ass hair away from going back to school and switching professions or maybe just do PM instead.
10/14/09
So the timebomb not only wiped all production data, all backup data, all system drives, and also caused the tape jukebox to load each individual tape into the tape drives and wipe all of them (a very time consuming task)?
Not to mention this doesn't even jive with the original explanation that Hitachi was upgrading the SAN and it inadvertently killed off the RAID groups.
Sounds like somebody is fishing for sensationalism to me. Go figure, they said this to Apple Insider of all people.
10/14/09
I just can't believe TMobile didn't have requirements that these guys had to meet.
I just can't believe a company like this didn't use Iron Mountain or a Safety Deposit box or the like to store their tapes. Every day those tapes should have been rotated out with ones from at a minimum 2 weeks prior (daily, obviously weeklies you keep longer).
10/14/09
as far as i know, backup tapes are to be kept off site once backup is complete. Or at least off the contact of any server or machine until the time it is actually really needed.
So how exactly could a backup tape get erased? I've been learning servers and doing my own experimental shit about servers and I've never heard of backup tapes getting erased, until now. And, as far as I know, only a few uses backup tapes these days. They moved to a more revolutionary method called "server mirroring" and uses a more robust system of backups through an off-site server system where backups are stored (and that deletion can only be performed if done within the backup facility, and can not be executed from a remote server).
So I wonder how this saboteur actually performed the trick? I simply can't imagine how stupid the server have been setup for this to happen.
10/14/09
Even with off-site servers you always keep off-site tape. Static copy greater than *
10/14/09
It doesn't have to be some conspiracy...easy explained by apathy.
dimes
10/14/09
[en.wikipedia.org]
@van_line: Occam's Razor is "entities must not be multiplied beyond necessity."
[en.wikipedia.org]
10/14/09
However, you are indeed correct that Hanlon's Razor would be the proper eponymous adage.
...now I'm off to douche up someone else's day :)
10/14/09
hanlon is what i think dimes was looking for, occam was what i hadn't heard of yet so just quick quoted it.
10/14/09
I'd never even heard of hanlon's razor until I heard it on the internet.
10/14/09
10/14/09
10/14/09
10/14/09
10/14/09
10/14/09
Even offsite ones [what, didn't have any?]? Even the ones at the backup site [what, didn't have any?]?
That's a neat trick.
10/13/09
10/13/09
10/13/09
10/13/09
10/13/09