Oops! Tech error wipes out Alaska info

[TechBear_Seattle]

Oops! Tech error wipes out Alaska info

By ANNE SUTTON, Associated Press Writer

JUNEAU, Alaska - Perhaps you know that sinking feeling when a single keystroke accidentally destroys hours of work. Now imagine wiping out a disk drive containing an account worth $38 billion. It happened to a computer technician reformatting a disk drive at the Alaska Department of Revenue.

While doing routine maintenance work, the technician accidentally deleted applicant information for an oil-funded account — one of Alaska residents' biggest perks — and mistakenly reformatted the backup drive, as well.

There was still hope, until the department discovered its third line of defense, backup tapes, were unreadable.

"Nobody panicked, but we instantly went into planning for the worst-case scenario," said Permanent Fund Dividend Division Director Amy Skow. The computer foul-up last July would end up costing the department more than $200,000.

The article continues at http://news.yahoo.com/s/ap/20070320/ap_on_re_us/lost_data

I work in the IT department of a brokerage firm. I have had nightmares like this....

[Atman]

1. "Mistakenly reformatted the backup drive as well?" WTF?

Was this some intern on his first job or something? I have a hard enough time believing one could so easily reformat the primary drive in the first place, but to then go and do it a SECOND drive? This guy should find another line of work. He doesn't belong anywhere near computers.

.

[Buzz Clik]

2. No backups. snark.

Somebody get a hold of Donald Trump. A number of these people need to be fired with flair, and not just the tech.

[Kagemusha]

4. No working backups.

A lot of people think they're backed up and get a rude surprise when they actually have to use those backups.

[Buzz Clik]

6. Yeah, lots of excuses readily available.

But no data. And $39 billion in jeopardy.

There are ways of protecting your data, but it requires a certain level of competence.

"Yer fired."

[notadmblnd]

3. I worked in IT too, performing backups and such

two incrementals were done daily and full backups were done weekly, two backups were performed on the weekends just in case some of the tapes were unreadable because the backup devices were so unreliable. Retention for the dailys were two weeks, weeklys were retained at least 6 months and monthlies were retained a minimum of five years. The only way I can see all data being permently lost is if these procedures were not in place and in that case, shame on their tech support.

[Buzz Clik]

7. bingo.

[Lerkfish]

5. am I the only one who finds this suspicious?

...applicant information for an oil-funded account — one of Alaska residents' biggest perks — and mistakenly reformatted the backup drive, as well.

[Toots]

8. You should since it is an out and out LIE

While the Permanent Fund was originally derived from oil revenues it has been growing annually on it's investments. It is the same thing as your bank account. It draws interest and the interest compounds it's value. But there is a spark of truth. There is still a little oil royalty money added every year but on the whole it is investment driven..

[notadmblnd]

11. No, I find it highly implausable the could have permently lost their data

even if, there were no recoverable digital files, The original paperwork should be somewhere.

[Serendipitous]

12. Read the article. It is.

...And the only backup was the paperwork itself — stored in more than 300 cardboard boxes.

"We had to bring that paper back to the scanning room, and send it through again, and quality control it, and then you have to have a way to link that paper to that person's file," Skow said.

Half a dozen seasonal workers came back to assist the regular division staff, and about 70 people working overtime and weekends re-entered all the lost data by the end of August....

[notadmblnd]

16. well then the story is wrong and it wasn't permenently lost.

there has to be hard copy. like I said, shame on their tech support for having lousy procedures. the money should not be lost, the government can recover the costs for the screwup. Theres generally fines and such written into the SLA's (service level agreements)that are signed between the government and contractor.

[Bandit]

17. No they had backup and they lost nothing but some money

The problem was they never tested their back up system and when the time came to utilize it they found they didn't have server capacity to up load it. It cost the state over $200,000. dollars to buy servers powerful enough to take care of the information. The people received their dividend checks on time and no one suffered except for the state's embarrassment

[Zywiec]

9. Backups occur all the time

What people fail to test is an actual restore. They always assume if data was backed up correctly, it will restore correctly. More than a few times it won't restore.

[mcscajun]

10. Bingo.

Most people (even some companies) who do backups, do backups as an act of faith and never test them.

[TechBear_Seattle]

13. We test our backups regularly

We actually have two databases, one that is "in use" and updated every business day, and one that is used for test purposes (and as the lead programmer for my company, I have had plenty of "oops" moments myself when rewriting our main database application.) The test database is restored from backups at least once every other week, both to keep it up-to-date with the active database and as a test of our backup restoration procedures.

The Securities Exchange Commission (SEC) and National Association of Securities Dealers (NASD) get rather pissy if we get audited and can not show we have an actively pursued data recovery process.

[Serendipitous]

15. Not doing it is a mistake you make once.

I'd bet they have changed their backup practices drastically.

[Bandit]

18. And that is exactly what happened here

They found they were unable to restore the back up with the equipment they had on hand. It cost the state over $200,000. to fix the problem and everyone received their check on time. It was embarrassing was all.

[notadmblnd]

14. I'm sorry, not plausible..unless you're a rinkydink shop

I've performed thousands and thousands of restores over my career, It was rare that I had no success. whether it was a nine track reel or a cassette, if one set of tapes didn't work, there was always a later version to recover from. Same thing with mirrored or striped disk sets, they were backed up religiously. We held a lot of government contracts, NOAA and US Navy among them. I can't imagine losing all data permanently by any IT service group that had any level of competence.