Extensive Planning, Testing and Communication Results in Success
On Sunday, January 6, 2013 from 6 a.m. – 2 p.m. Information Technology Services (ITS) performed important network maintenance that impacted a majority of the IT services at the University. The goal of the maintenance was to bring the data center network (UDC-C) up to the best, current configuration practices and install recommended software. In doing this, the team also corrected critical vendor bugs, fixed vulnerabilities, and increased resilience for the facility.
While the tasks involved in this type of maintenance are straightforward, the planning and coordination is not. The process began a year ago and involved:
- Working with the vendor for the entire year to identify and correct bugs
- Determining the current best practices and implications in the University Data Center
- Developing custom software to simulate traffic flows (something the vendor doesn’t have) then spending months of lab time simulating hardware/software/configuration combinations
- Dedicating weeks to configuration generation and reviews by multiple people
- Developing detailed execution plans and reviews
- Communicating - lots of communicating – to keep campus informed about the maintenance
The extensive preparation for this major maintenance paid off. The average server was unreachable from the Internet for only 9 minutes.
In addition, ITS staff fully implemented the disaster recovery protocol for the Austin Exchange Messaging Service (AEMS) and all its supporting peripheral components, such as load balancers, anti-virus/anti-spam appliance, and domain routing. During the actual outage, these components were switched to backup units and both AEMS and UTmail continued to operate without any down time.
To keep campus informed throughout, ITS staff used the alternate off-site email list, Twitter and the externally hosted emergency website to provide periodic updates during the on Sunday morning maintenance period.
Learn more about ITS maintenance events on the ITS Services Status page: http://www.utexas.edu/its/alerts/main.php