What we do know is that a lot of people spent most of the weekend repairing the damage caused by one little rat that crawled his way into a transformer.
Now, the rat couldn't have known that this transformer powered many of ASU's computer systems and that, besides the small matter of frying his brains, he was going to cause a major service disruption. Fortunately, he sought shelter late on a Saturday night during the lowest period of computer use. We have a UPS but the battery ran out before our power problems were resolved. Technicians worked through the night to grab a generator, hook it up and then restore our equipment.
We were back to full power by 6:30 am Sunday, when the software and hardware people took over to restore services. Support services, such as Blackboard, Exchange and IDEAL, had to come up first so that they'd be available when the Web came back up. Most services were available by 11:30 am, though system functions needed to be reviewed throughout a very long day.
I'd like to thank each and every person who helped get ASU's services back up and running. Many thanks to:
Terry Hinton & his staff - Chris Beke, Robert Blackmon, Alexandro Bonos, Shawn Bryan, Mark Fornefeld, Kevin Lockart, Reyes Mori, James Prather & Brian Shaw; Jack Hsus staff - Chris Coffin, Brent Dunlock, Jim Durand, Jane Flores, Sean Garrett, Paul Harper, Bob Kaneshige, Joe McDonald, OJ Redhair, Jim Salverson, Jeff Scheib, & Greg Wilson; Ken Carl & his staff - James Muscara; Robin Manke-Cassidy & her staff - Gregory Alcorn, Zach Chamberlin, Erich van Sanford & Goska Zapolska; Dave McKee & his staff - Franco Lomonte, Tom Bauer & Duane Woerman; Kari Barlow & her staff - Nancy Lee, Nathan Gudmunson, Andy Beier & Minnie Fontes; John Rome & his staff - Sangeeta Agarwal, Ali Benmalek, Jack Davis, Daren Kahus, Dennis Monce & Bob Sookvong; Max Davis-Johnson & his staff - Walt Ellis, Neil Fritz, Jason Green, Ron Hill, Tasleema Lallmamode, Joe Nadeau & Lori Reents; Sarah Hughes & Samuel DiGangi & his staff - Ruvi Wijesuriya.
And what about Algernon? He may not have made it to grad school, but he did help us learn some things. We knew that we needed redundant power systems and Algernon helped make it a priority. Unfortunately, the more elaborate the infrastructure, the more expensive redundancy becomes. As we move toward a 24 by 7 world, the last couple of 9s are the most expensive. It's interesting to note that services supplied by our partners, Gmail from Google and PeopleSoft functions from CedarCrestone, were not affected by this power outage. Our little rat friend would have had to travel a long way to disrupt them, and due to their scales of operation, they may be a little more varmint-proof than we are.
Comments - even rat jokes - are welcome...