Alerts Page

WWU Website Migrations

***UPDATES***

3/21/24 8:35am - Updates and migrations are complete.

***ORIGINAL ALERT***

3/20/24 8am - Some WWU websites are showing as Forbidden or blank while WebTech migrates the drupal servers.

Various System Outages

***UPDATES***

3/20/24 8am - Some WWU websites are showing as Forbidden or blank.

3/19/24 5pm - All systems reported down are now online, with the exception of Transact InstantID.

3/19/24 3pm - MillenniumFAST production and Test are restored. Printing functions and the CBE Decision Sciences server are also online.

3/19/24 9:30am - The MillenniumFAST application is not loading. Users will see a 503 Service Unavailable error. MillFAST test is down as well.

Additional known outages:

• Instability with libloan (OneSearch, Interlibrary Loan)
• FDO Access Control
• FDO Access Control Test
• Transact
• Some printing services
• CBE Decision Sciences server
• CRC

3/18/24 12:00pm - Interlibrary Loan and student printing are partially restored. Banner should be working now as well.

The following services are still down:

• CBE Decision Sciences server
• Account Provisioning Server
• CRC
• AMSEC FOM
• Millfast
• Extron Global Viewer
• Transact Instant ID
• FDO Access Control
• FDO Access Control Test
• MS Print1
• ESIGN Test Server
• PCounterWD
• KeyWin Keybox Management

***ORIGINAL POST***

3/18/24 9:30am -- ITS is continuing work to address system outages that first began on Saturday 3/16. Banner is redirecting to our System Outage page while PPRD is worked on. Libloan is working on the backend but still throwing errors. The student printing release page is not loading. We thank you for your patience while servers are being worked on.

DegreeWorks Down

**UPDATES**
8:45am: DegreeWorks service has been restored.

**ORIGINAL POST**

Monday 3/4/24, 8am: DegreeWorks has been down since Saturday, displaying a 503 Service Unavailable error. ITS is investigating.

Cluster Stability Improvements 6:30pm - 10:30pm Tonight

--UPDATES--
3/4/24 10:30 p.m. - Maintenance is complete. 

--ORIGINAL POST--

3/4/24 4:00 p.m. - The EIS team will be performing maintenance actions on the datacenter infrastructure (HCI cluster) starting at 6:30pm tonight.  Most services will not be impacted.  Some services may briefly lose network connectivity during the maintenance, including campus websites.

These enhancements are necessary to ensure the cluster is ready to support Banner upgrades in time to deliver critical Financial Aid system updates.

Multiple Outages and Degraded Service Reported Across Campus

***Updates***

02.27.2024

• The cluster has been stable since 02.22.2024 at 6:00 p.m., so we are moving from "major incident" to "normal operations."

02.26.2024

• 8:30 a.m. - Primary systems such as Banner and websites have remained stable over the weekend. Secondary systems will continue to be restored.

02.23.2024

• 3:30 p.m. - Critical datacenter services have remained stable throughout the day. Additional core services have also been restored. Less critical services will remain offline through the weekend until a more thorough analysis of the situation can be completed.
• 9:00 a.m. - Services continue to be restored and the performance of priority systems like Banner and websites continue to be monitored.

02.22.2024

• 2:30 p.m. - Some services have been restored but may be unstable. Repair work is continuing.
• 11:00 a.m. - Repairs to restore service continue but myWestern and other wwu.edu websites are currently down. Banner users are advised to avoid running any complex process due to current system instability, however, it's okay to perform data entry, general queries, and reports.
• 9:00 a.m. - Instability in datacenter have caused system issues to resurface.

02.21.2024

• 8:00 p.m. - Banner is back up. Final patches and various cleanup and restore tasks are being performed
• 8:00 a.m. - Alert reinstated. An email has been sent to WWU students and employees. A major system stability issue in the campus datacenters is causing many campus applications to periodically go offline. Banner was down overnight Tuesday and remains down first thing Wednesday morning. The nature of the outages is intermittent and unpredictable. Any system hosted on campus servers could become unavailable at any time, for a length of time from 20 minutes to overnight.  Cloud-based systems like Canvas and Microsoft 365 are not impacted, except when they rely on an on-campus system to function.  Information Technology Services is working with Microsoft support engineers to investigate and resolve the stability issues, but Microsoft believes it may take another week to identify the root cause of the issue. Esign is giving a 503 Service Unavailable.

02.16.2024

• 4:42 p.m. - Migration of the services has completed, and services should be restored.
• 8:20 a.m. - Issues have cropped up again and services are being migrated to another node to address stability. There may be some brief performance impacts as the moves take place. EIS is working with the vendor to discuss the analysis on the potential root causes of these service impacts.
   

*** Original Message ***
ITS has identified and is currently investigating outages and degraded service across platforms hosted in the campus datacenters. Work is being done to restore service. 

Services restored:

• Argos
• UISAP
• Library EZProxy
• Drupal
• Rightfax
• Universal password reset
• Banner job submission
• Facilities Online Management
• Alertus
• Some library systems, such as Libloan.
• Student printing and EVPrint.
• Banner as a whole was running slowly. 
• Banner clean was addresses potentially offline.
• The data warehouse did not rebuild. A manual run of data warehouse took place.

Banner & Related Systems Down

**UPDATES**
2/5/24 1:00 pm - Banner production and test services were restored to functionality last evening, 2/5/24, at 5:00 pm.

**ORIGINAL POST**

2.5.24 - 8:45 am - Enterprise Infrastructure Services and Database Administrators are investigating the root cause of a Banner outage this morning. Folks are unable to log in, seeing an "invalid credentials" error. Services that rely on Banner, such as DegreeWorks, are also impacted.

Microsoft Teams Intermittent Service Impacts

**UPDATES**

1.26.2024 4:30 p.m. - Microsoft has improved connectivity to Teams by applying optimizations and mitigations to their services. There may still be some minor impacts.

**ORIGINAL POST*

1.26.2024 9:20 a.m. - Microsoft is having a large-scale service impact with Microsoft Teams which may impact your ability to log into or send messages within Teams. Another symptom is seeing an "Oops" Error message on the main Teams window. Microsoft is currently investigating the scope and source of the impact and is working to resolve the connectivity problems.

If you experience problems with Teams, please be aware that this issue is international in scope and will be fixed by Microsoft without the need for action on your computer or mobile device.

Windows 11 Upgrades Beginning January 22

Windows 11 will begin downloading to university-owned computers the week of Jan. 22 (most computer labs and classroom computers were upgraded last summer). Installation will begin during nightly maintenance windows. Computers that cannot support Windows 11 or those that have other client health issues (e.g., hard drives with inadequate space) will not be included in this upgrade. Please see article in the January 12, 2024, issue of Western Today for more information.

Wireless and VPN Unavailable, Tonight from 9:00-9:30 p.m.

To troubleshoot some wireless connection issues, maintenance will be done tonight, Thursday, January 18, 2024, from 9:00-9:30 p.m. During that time, users will not be able to login (authenticate) to the VPN or wireless networks. Clients that are already connected should not be affected.