Member-only story
Technology | Database | Crisis
The Database Was Gone, and I Had to Save the Day
Weeks into the job and it was sink or swim time
“Fanculo” (Italian Swear Word)
“It didn’t do what I think it did, did it? Shit, the database is completely gone.”
Stefano’s voice trembled despite his attempt to project calm.
His hands shook visibly, and the panic on his face told the story before his words did. Beads of sweat slid down his temple as the realisation settled in: our mission-critical Oracle database had vanished.
A routine task to remove a single node from the database cluster had spiraled into catastrophe.
Somehow, the software wiped the entire cluster, dropping the database in the process.
It was January 13, 2012 — the same day the Costa Concordia sank off the coast of Italy. Disaster seemed to be in the air.
I was fresh out of the gate, a bright-eyed junior DBA only weeks into the job.
Now I stood in the eye of the storm.
The plan was simple: restore from backups. We had a copy on disk and clung to hope that the process would go smoothly.
