Multiple Outages and Degraded Service Reported Across Campus

***Updates***

02.27.2024

• The cluster has been stable since 02.22.2024 at 6:00 p.m., so we are moving from "major incident" to "normal operations."

02.26.2024

• 8:30 a.m. - Primary systems such as Banner and websites have remained stable over the weekend. Secondary systems will continue to be restored.

02.23.2024

• 3:30 p.m. - Critical datacenter services have remained stable throughout the day. Additional core services have also been restored. Less critical services will remain offline through the weekend until a more thorough analysis of the situation can be completed.
• 9:00 a.m. - Services continue to be restored and the performance of priority systems like Banner and websites continue to be monitored.

02.22.2024

• 2:30 p.m. - Some services have been restored but may be unstable. Repair work is continuing.
• 11:00 a.m. - Repairs to restore service continue but myWestern and other wwu.edu websites are currently down. Banner users are advised to avoid running any complex process due to current system instability, however, it's okay to perform data entry, general queries, and reports.
• 9:00 a.m. - Instability in datacenter have caused system issues to resurface.

02.21.2024

• 8:00 p.m. - Banner is back up. Final patches and various cleanup and restore tasks are being performed
• 8:00 a.m. - Alert reinstated. An email has been sent to WWU students and employees. A major system stability issue in the campus datacenters is causing many campus applications to periodically go offline. Banner was down overnight Tuesday and remains down first thing Wednesday morning. The nature of the outages is intermittent and unpredictable. Any system hosted on campus servers could become unavailable at any time, for a length of time from 20 minutes to overnight.  Cloud-based systems like Canvas and Microsoft 365 are not impacted, except when they rely on an on-campus system to function.  Information Technology Services is working with Microsoft support engineers to investigate and resolve the stability issues, but Microsoft believes it may take another week to identify the root cause of the issue. Esign is giving a 503 Service Unavailable.

02.16.2024

• 4:42 p.m. - Migration of the services has completed, and services should be restored.
• 8:20 a.m. - Issues have cropped up again and services are being migrated to another node to address stability. There may be some brief performance impacts as the moves take place. EIS is working with the vendor to discuss the analysis on the potential root causes of these service impacts.
   

*** Original Message ***
ITS has identified and is currently investigating outages and degraded service across platforms hosted in the campus datacenters. Work is being done to restore service. 

Services restored:

• Argos
• UISAP
• Library EZProxy
• Drupal
• Rightfax
• Universal password reset
• Banner job submission
• Facilities Online Management
• Alertus
• Some library systems, such as Libloan.
• Student printing and EVPrint.
• Banner as a whole was running slowly. 
• Banner clean was addresses potentially offline.
• The data warehouse did not rebuild. A manual run of data warehouse took place.