The University of Texas at Austin

ITS Services Status

Current Service Interruptions

8/28/2014
6:09 PM
Secure Cookie Error
Under Investigation

18:04
At around 5:30pm we received several calls and emails regarding an error message when trying to access UT direct and related applications. The exact error message is “Secure Cookie Error” and prevents the user from logging in. We’ve had reports from various students, staff, and faculty members about this issue.

8/28/2014
5:02 PM
RESOLVED: Inbound email backlog
Service Restored

17:02
The inbound mail backlog has cleared on both servers. All pending mail has been delivered and all four inbound servers are operating normally.


16:08
Two of our four inbound email security appliances are backlogged again. ITS Systems staff are working to remediate the backlog. No mail is being lost, only delayed.

8/28/2014
12:56 PM
RESOLVED:  Office 365 customers unable to access Exchange services from off-campus
Service Restored

As of 11:00, ISP issues affected some customers access to O365 Outlook.

As of 11:54, Microsoft Engineers rerouted traffic to restore service and continue to validate service health.

As of 12:00, Emails have been confirmed to be transmitting between external providers and O365 Outlook.

O365 Outlook service is believed to have been restored for UT Austin at this time. 

Email delivery to O365 Outlook may have been delayed, but is now processing.


As of 11:54 via MS

Current Status: Microsoft engineers have rerouted traffic to restore service and are validating service health at this time.

Customer Impact: Affected customers may be unable to access the Exchange Online service.

Next Update: Thursday, August 28, 2014 at 6:00 PM UTC


11:47
Microsoft posted a service alert at 11:00 AM CDT:

Engineers have identified a potential Internet Service Provider (ISP) issue affecting some customers ability to access the Exchange Online service. Affected customers may be unable to access the Exchange Online service.

Next update is scheduled for 12:00 PM CDT.


11:42
The help desk is now receiving numerous calls from users off-campus unable to access Exchange Online. Some users appear to be able to access their mailbox through Outlook Web App but not through clients, while others report the opposite.

8/28/2014
10:55 AM
RESOLVED: BACS Badge Updates Stalled!
Service Restored

10:54
Badge updates have resumed processing and backlog is being reduced.


10:06
We are currently experiencing a backup in the processing of badge updates from the IDCenter Badge Feed.  ITS/BACS staff are investigating.

Past Service Interruptions

Upcoming Maintenance

<<< 09/01/2013 - 09/21/2013 >>>
12:00 AM - 1:00 AM
Mainframe maintenance window
Service: Mainframe
Description: The UT Austin mainframe will undergo planned maintenance early Sunday morning beginning shortly after 12:00 AM midnight and lasting approximately 1 hour into Sunday morning. UT Direct, applytexas.org,... (more)
12:00 AM - 6:00 AM
Austin Exchange Message Service (AEMS) Planned Maintenance
Service: Austin Exchange Messaging Service
Description: This maintenance includes any or all of the following maintenance tasks:

* Installation of security updates and hotfixes.

* Moving of user mailboxes to different mailbox databases to better ... (more)
12:00 AM - 6:00 AM
Austin Disk Services Planned Maintenance Window
Service: Austin Disk Services
Description: Customers utilizing the following dependent services may be affected:

* Austin Disk shares (user home directories and departmental shares)
* Stat Apps Server (user profiles, redirected documents ... (more)
12:01 AM - 6:00 AM
SharePoint 2010 Maintenance
Service: SharePoint
Description: During the weekly maintenance window, SharePoint 2010 farm will have SharePoint updates (product updates and security updates) installed.
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
10:30 AM - 11:30 AM
DEFINE Internals/Enterprise Workflow Non-Disruptive Maintenance
Service: DEFINE Internals/Enterprise Workflow
Description: *DEFINE Internals/Enterprise Workflow will be performing non-disruptive maintenance to deploy user-requested enhancements. Users should not experience any interruption in service.
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
7:00 AM - 8:00 AM
Monthly JIRA Application Maintenance
Service: Jira
Description: JIRA will undergo application maintenance every 1st Tuesday monthly.
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
4:00 PM - 5:00 PM
ERP Web Infrastructure Maintenance Window
Service: ERP UI Web Infrastructure
Description: ERP UI, ERP Application Registry, ERP UI Site generator application upgrades occur during this time. Minimal impact on users should be expected.
5:00 PM - 5:30 PM
UT Direct Planned Maintenance
Service: UT Direct
Description: UT Direct will be performing planned maintenance to deploy user-requested enhancements. Users may experience a very limited interruption in service.
6:00 AM - 8:00 AM
UTLogin routine maintenance
Service: UTLogin
Description: ITS Applications will be conducting maintenance on the UTLogin system that will result in a partial system outage.

Date: 9/4/2013
Start time: 6:00 AM
End time: 8:00 AM

Impact: This ... (more)
7:00 AM - 5:00 PM
NOC-A Rack Power Audit
Service: University Data Center
Description: UDC-FAC will be performing a power audit for all racks and cabinets in NOC-A. We will have electrical panels open from time to time but the dead fronts will remain in place so no electrical hazards ... (more)
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
2:00 PM - 5:00 PM
Ongoing non-disruptive Exchange upgrades
Service: Austin Exchange Messaging Service
Description: More of the ongoing software upgrades on AEMS. No customer visibility is expected
5:00 PM - 7:00 PM
DEFINE Internals/Enterprise Workflow Planned Maintenance
Service: DEFINE Internals/Enterprise Workflow
Description: *DEFINE Internals/Enterprise Workflow will be performing planned maintenance to deploy user-requested enhancements and bug fixes. Users should not experience any interruption in service.
5:00 PM - 10:00 PM
Blog Service Weekly Maintenance
Service: University Blog Service
Description: Scheduled maintenance may occur weekly on Wednesdays between 5 p.m. and 10 p.m. To the maximum extent possible, installation of service, application, and security updates will be performed during ... (more)
7:00 AM - 5:00 PM
NOC-A Rack Power Audit
Service: University Data Center
Description: UDC-FAC will be performing a power audit for all racks and cabinets in NOC-A. We will have electrical panels open from time to time but the dead fronts will remain in place so no electrical hazards ... (more)
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
9:00 AM - 12:00 PM
Install two StarLine modules into live buss
Service: University Data Center
Description: Planned - Non-Disruptive: UDC-FAC Will be installing two 20amp StarLine modules into the StarLine A&B busses in NOC-B.
This work is not expected to have any impact on
exisiting devices on the ... (more)
10:00 AM - 4:00 PM
Non-distruptive software upgrade on AEMS
Service: Austin Exchange Messaging Service
Description: The ongoing process to upgrade AEMS to Exchange 2010 SP3 continues.
4:00 AM - 7:00 AM
Rebuild Blackboard's passive standby database
Service: Blackboard
Description: We need to rebuild Blackboard's standby database. This doesn't modify the active production environment at all, but does load down the production database so needs to be done after prime-time service ... (more)
7:00 AM - 5:00 PM
NOC-A Rack Power Audit
Service: University Data Center
Description: UDC-FAC will be performing a power audit for all racks and cabinets in NOC-A. We will have electrical panels open from time to time but the dead fronts will remain in place so no electrical hazards ... (more)
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
10:30 AM - 12:00 PM
Expedited non-distruptive software upgrade on AEMS
Service: Austin Exchange Messaging Service
Description: The rolling upgrade of all AEMS servers to Exchange 2010 SP3 continues. This update covers the DR mailbox servers, which are not serving customers.
2:00 PM - 3:00 PM
utdirect qual change to pype proxy
Service: UT Direct Q/A environment
Description: We will be changing how we proxy requests to the qual pype applications behind utdirect qual

Rolling restart, with no visible interruption in service
6:00 AM - 11:00 AM
Footprints Weekly Maintenance
Service: Footprints Incident Tracking System
Description: Application or OS patching and updates.
7:00 AM - 5:00 PM
NOC-A Rack Power Audit
Service: University Data Center
Description: UDC-FAC will be performing a power audit for all racks and cabinets in NOC-A. We will have electrical panels open from time to time but the dead fronts will remain in place so no electrical hazards ... (more)
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
12:00 AM - 6:00 AM
Blackboard Weekly Maintenance
Service: Blackboard
Description: Weekly maintenance event for Blackboard (courses.utexas.edu).

The service may be degraded or down during this time.
12:00 AM - 6:00 AM
Austin Exchange Message Service (AEMS) Planned Maintenance
Service: Austin Exchange Messaging Service
Description: This maintenance includes any or all of the following maintenance tasks:

* Installation of security updates and hotfixes.

* Moving of user mailboxes to different mailbox databases to better ... (more)
12:00 AM - 6:00 AM
Austin Disk Services Planned Maintenance Window
Service: Austin Disk Services
Description: Customers utilizing the following dependent services may be affected:

* Austin Disk shares (user home directories and departmental shares)
* Stat Apps Server (user profiles, redirected documents ... (more)
12:01 AM - 6:00 AM
SharePoint 2010 Maintenance
Service: SharePoint
Description: During the weekly maintenance window, SharePoint 2010 farm will have SharePoint updates (product updates and security updates) installed.
7:00 AM - 5:00 PM
NOC-A Rack Power Audit
Service: University Data Center
Description: UDC-FAC will be performing a power audit for all racks and cabinets in NOC-A. We will have electrical panels open from time to time but the dead fronts will remain in place so no electrical hazards ... (more)
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
7:00 AM - 7:30 AM
MSSQL Maintenance for SQL01
Service: Microsoft SQL Server
Description: A failover is required for MSSQL service on SQL01
This will result in a disruption in service for connecting clients during some portion of this maintenance window.
7:00 AM - 5:00 PM
NOC-A Rack Power Audit
Service: University Data Center
Description: UDC-FAC will be performing a power audit for all racks and cabinets in NOC-A. We will have electrical panels open from time to time but the dead fronts will remain in place so no electrical hazards ... (more)
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
3:30 PM - 5:00 PM
PyPE OS Security Patching - Test & Qual
Service: Python Production Environment (PyPE)
Description: Apply OS Patch Stream to PyPE Test and Qual environments. Servers are paired and will be updated in sequence, so PyPE-based web applications will continue to run with diminished capacity but no ... (more)
7:00 AM - 5:00 PM
NOC-A Rack Power Audit
Service: University Data Center
Description: UDC-FAC will be performing a power audit for all racks and cabinets in NOC-A. We will have electrical panels open from time to time but the dead fronts will remain in place so no electrical hazards ... (more)
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
6:00 PM - 8:00 PM
SBC Patch implementation
Service: Voice Services (VoIP)
Description: The vendor for the new VoIP system will be applying a patch to the Session Border Controller. No service should be affected.
6:00 AM - 7:00 AM
AEMS mailbox server failover
Service: Austin Exchange Messaging Service
Description: AEMS mailbox server #1 will be failed over to its standby servers in preparation for upgrading it to SP3. No user impact is anticipated.
7:00 AM - 5:00 PM
NOC-A Rack Power Audit
Service: University Data Center
Description: UDC-FAC will be performing a power audit for all racks and cabinets in NOC-A. We will have electrical panels open from time to time but the dead fronts will remain in place so no electrical hazards ... (more)
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
5:00 PM - 10:00 PM
Blog Service Weekly Maintenance
Service: University Blog Service
Description: Scheduled maintenance may occur weekly on Wednesdays between 5 p.m. and 10 p.m. To the maximum extent possible, installation of service, application, and security updates will be performed during ... (more)
6:00 AM - 7:00 AM
AEMS mailbox server failover
Service: Austin Exchange Messaging Service
Description: AEMS mailbox server #2 will be failed over to its standby servers in preparation for upgrading it to SP3. No user impact is anticipated.
7:00 AM - 5:00 PM
NOC-A Rack Power Audit
Service: University Data Center
Description: UDC-FAC will be performing a power audit for all racks and cabinets in NOC-A. We will have electrical panels open from time to time but the dead fronts will remain in place so no electrical hazards ... (more)
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
12:00 PM - 4:00 PM
Blog Service Maintenance
Service: University Blog Service
Description: The Blog Service will undergo maintenance at 12:00 p.m. on Thursday, September 12. No interruption in service is expected.
6:00 AM - 7:00 AM
AEMS mailbox server failover
Service: Austin Exchange Messaging Service
Description: AEMS mailbox server #3 will be failed over to its standby servers in preparation for upgrading it to SP3. No user impact is anticipated.
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
4:30 PM - 5:00 PM
DirSync server maintenance
Service: Office 365
Description: The DirSync VM will be upgraded to more CPU and memory. The SQL instance will be upgraded to Enterprise.

No customers will be impacted by this maintenance.
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
9:00 AM - 12:00 PM
PROD uTexas Identity Manager (TIM) Routine Maintenance
Service: uTexas Identity Manager (TIM)
Description: We will be conducting routine TIM PROD maintenance. There will be two 10 minute outages during the maintenance period.
12:00 AM - 6:00 AM
Austin Exchange Message Service (AEMS) Planned Maintenance
Service: Austin Exchange Messaging Service
Description: This maintenance includes any or all of the following maintenance tasks:

* Installation of security updates and hotfixes.

* Moving of user mailboxes to different mailbox databases to better ... (more)
12:00 AM - 6:00 AM
Austin Disk Services Planned Maintenance Window
Service: Austin Disk Services
Description: Customers utilizing the following dependent services may be affected:

* Austin Disk shares (user home directories and departmental shares)
* Stat Apps Server (user profiles, redirected documents ... (more)
12:01 AM - 6:00 AM
SharePoint 2010 Maintenance
Service: SharePoint
Description: During the weekly maintenance window, SharePoint 2010 farm will have SharePoint updates (product updates and security updates) installed.
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
10:00 PM - 12:00 AM
Monthly Group E-Mail Maintenance
Service: Group E-Mail
Description: Monthly maintenance window for the Group E-mail server.
12:00 AM - 1:00 AM
Blackboard: Revert to packaged web server
Service: Blackboard
Description: We will revert from the newly installed web server front ends to the version of apache packaged with the application.
6:00 AM - 7:00 AM
AEMS mailbox server failover
Service: Austin Exchange Messaging Service
Description: AEMS mailbox server #4 will be failed over to its standby servers in preparation for upgrading it to SP3. No user impact is anticipated.
7:00 AM - 7:05 AM
DPUSER account lockout process
Service: Mainframe
Description: To comply with best practices, the number of invalid logon attempts in DPUSER before an account will be locked will be reduced from 5 to 3.
7:00 AM - 7:30 AM
Planned Maintenance for Central Web Authentication
Service: Central Web Authentication (CWA)
Description: Central Web Authentication will undergo routine scheduled maintenance. Users should experience no interruption of service.
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
9:00 AM - 10:00 AM
Natclipse server maintenance
Description: The natclipse servers will undergo regular operating system (OS) maintenance on the third Tuesday of every third month. No user disruption is expected due to service architecture.
10:00 AM - 10:30 AM
TRAC site maintenance
Service: Technical Resource Account Control (TRAC)
Description: Minimal content changes will be made to TRAC self-service tool to reflect new service costs. No user impact is anticipated.
4:00 PM - 5:00 PM
ERP Web Infrastructure Maintenance Window
Service: ERP UI Web Infrastructure
Description: ERP UI, ERP Application Registry, ERP UI Site generator application upgrades occur during this time. Minimal impact on users should be expected.
6:00 PM - 10:00 PM
Wiki Service Monthly Maintenance
Description: Scheduled maintenance for the University Wiki Service occurs the third Tuesday of the month from 6 p.m. - 10 p.m. To the maximum extent possible, installation of service, application and security ... (more)
6:00 AM - 7:00 AM
AEMS mailbox server failover
Service: Austin Exchange Messaging Service
Description: AEMS mailbox server #5 will be failed over to its standby servers in preparation for upgrading it to SP3. No user impact is anticipated.
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
5:00 PM - 10:00 PM
Blog Service Weekly Maintenance
Service: University Blog Service
Description: Scheduled maintenance may occur weekly on Wednesdays between 5 p.m. and 10 p.m. To the maximum extent possible, installation of service, application, and security updates will be performed during ... (more)
5:00 PM - 7:00 PM
UT Direct Planned Maintenance
Service: UT Direct
Description: UT Direct will be performing planned maintenance to deploy user-requested enhancements and bug fixes. A small percentage of users may experience a very limited interruption in service.
5:30 PM - 7:30 PM
SBC Patch implementation
Service: Voice Services (VoIP)
Description: Vendor-applied patch to Session Border Controller. (This has been delayed to 9/18/13 - mjr)
All Day
Expedited - Storage Maintenance
Service: UT Enterprise Bulk Storage (UTEBS)
Description: One of the backend arrays for UTEBS will be down.

We are mitigating this outage by temporarily transitioning users to an alternate storage array in a seamless fashion.

However, for latency-... (more)
8:45 AM - 9:15 AM
Unplanned/Expedited maintenance - TRAC site content
Service: Technical Resource Account Control (TRAC)
Description: Maintenance will be performed on TRAC to change site content. No downtime is anticipated.
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
10:00 AM - 2:00 PM
ITS Storage - DDN
Description: A storage array will be updated to the newest generally available firmware. This will require downtime for update dependencies to be performed by vendor technician.
1:00 PM - 5:00 PM
Urchin Planned Maintenance Window
Service: Web Analytics: Urchin
Description: Planned monthly maintenance window for Urchin.
10:00 PM - 11:59 PM
Maintenance for MSSQL Cluster SQL01
Service: Microsoft SQL Server
Description: Maintenance on the MSSQL cluster SQL01 is required to address a recently discovered issue with disk performance. SQL Server services and connected applications including Sharepoint 2010 will ... (more)
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)
7:00 AM - 9:00 AM
WebSpace planned database maintenance
Service: WebSpace
Description: The database server for WebSpace will undergo routine software maintenance. WebSpace will be unavailable while this maintenance is being performed. We expect this maintenance window to be exercised ... (more)
9:00 AM - 5:00 PM
VM storage migrations to permit SAN OS upgrades
Service: Virtual Servers
Description: VMs will be migrated to other storage locations to permit evacuation of 2 SANs to a new firmware revision which cannot be done with the SAN online.

No VM downtime is anticipated. There is a ... (more)

We Can Help

Get help from an expert:

* ITS Help and Service Desk

* Call us at 512-475-9400

* Submit a help request online

We also have a walk-in service in the first floor lobby of the Flawn Academic Center (FAC). Stop by and let us help you!