Good day all, this the Angry Systems Administrator. I’ve bludgeoned the Angry Webmaster and tied him to a chair so I can write for a change. I just read a story about a man who wiped out his entire company with one single line of code.
As a Systems Administrator/Engineer, when I read this story I just had to shake my head at how stupid this guy was. He apparently was deleting some data on a UNIX system and he carelessly used the “Destroy everything from here on down!” command, rm -rf *. This command, if run from the wrong location, will irretrievably delete everything in and below the directory you start in, and the -rf switch forces this to happen and not ask “Do you really want to wreck your business? Y/N” Here are the details of the death of a company from The Independent:
A man appears to have deleted his entire company with one mistaken piece of code.
By accidentally telling his computer to delete everything in his servers, hosting provider Marco Marsala has seemingly removed all trace of his company and the websites that he looks after for his customers.
Marco went on some forums asking for advice and assistance. What he got was “You’re Screwed Dude!” instead.
Far from advising them how to fix it, most experts informed him that he had just accidentally deleted the data of his company and its clients, and in so doing had probably destroyed his entire company with just one line of code.
As I mentioned above, Marsala wrote a script and used the rm -rf command, and ran it without verifying what was going to happen. Going on the forums for help in such a circumstance is a good way to be utterly humiliated.
That piece of code is so famously destructive that it has become a joke within some computing circles.
It is a joke, but also no laughing matter when you accidentally run it from either the wrong location, or in this case, from a script that he didn’t bother to double check. Normally, when you do something this stupid, you find out just how good your back up strategy is. In Marco Marsala’s case, not very good would be an understatement.
Mr Marsala confirmed that the code had even deleted all of the backups that he had taken in case of catastrophe. Because the drives that were backing up the computers were mounted to it, the computer managed to wipe all of those, too. “All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script).”
Do I even need to go into how many things this guy did wrong? Why not? I’m feeling really nasty right now. First, he didn’t double check his script. Second, he didn’t make sure his offsite backups were isolated and actually secure. I suspect he had a mounted share on his system and his backup tool pushed the archives to it. Finally, he didn’t have his backups on separate media such as tape. Yes, that is expensive, but not as expensive as this fiasco is going to be for him.
“I feel sorry to say that your company is now essentially dead,” wrote a user called Sven. “You might have an extremely slim chance to recover from this if you turn off everything right now and hand your disks over to a reputable data recovery company.
“This will be extremely expensive and still extremely unlikely to really rescue you, and it will take a lot of time.”
That would be an understatement. Not to bring up Hillary Clinton’s mail server, but, as I understand it, that was a Windows based mail system and when she ordered it erased, the hosting service simply formatted the drive. Since this isn’t a true erasure, the FBI was able to restore the data. Considering the time involved, you can bet this cost a lot of money and manpower with their computer forensics people.
Others agreed that perhaps Mr Marsala was on the wrong forum.
“You’re going out of business,” wrote Michael Hampton. “You don’t need technical advice, you need to call your lawyer.”
Actually, I think he should have gone to the Suicide Prevention and Intervention forums instead. Once his clients figure out they are screwed, they’re going to obliterate him financially. The ever helpful forums were quick to pile on and berate Marsala.
Many of the responses to Mr Marsala’s problem weren’t especially helpful – pointing out that he could have taken steps to stop it happening before it did.
“Well, you should have been thinking about how to protect your customers’ data before nuking them,” wrote one person calling himself Massimo. “I won’t even begin enumerating how many errors are simultaneously required in order to be able to completely erase all your servers and all your backups in a single strike.
“This is not bad luck: it’s astonishingly bad design reinforced by complete carelessness.
You would be amazed how often something like this happens. I’ve had it happen to me, although not to this level of failure. I worked for a company that lost a 1.5TB database and spent a wonderful 36 hours restoring it from multiple backup tapes, and then restoring the incremental backups to bring the database back to the most recent version.
When that happened, it wasn’t something I did, it was something the DBA didn’t do. We upgraded the firmware on our storage array, and he had forgotten to secure a critical configuration file. (He had implemented it between the final backup and the upgrade and didn’t tell anyone)
I feel sorry for Marco Marsala, but I’m sure he will do very well in his new career in the food service industry.
Thank you
~The Angry Systems Administrator~
[yasr_visitor_votes size=”medium”]
