CHI S1 subnet RAID Upgrade Aug 29
#Status Update 3
All hardware upgrades have been completed, while firmware upgrades will have to be completed
in a separate maintenance window and after all production systems will be migrated elsewhere
to avoid any further surprise like that. Thank you for your patience while we worked hard
to complete this critical upgrade.
We are experiencing issues after firmware upgrade which prevent the affected systems
from starting properly. Now working on reverting some changes while keeping necessary
hardware upgrades already installed.
We are sorry for the unexpected extended downtime.
>>>>>> Maintenance Window:Start Date: Thursday, August 29, 2019, 10:00 PM (CDT)
End Date: Thursday, August 29, 2019, 11:45 PM (CDT)
Scope of work:
PERC (RAID) battery and firmware upgrade affecting all instances
in the CHI (Chicago) data center S1 subnet.
We have noticed that long running, duplicate system backups add significant
load on all machines in Chicago using these RAID arrays, which may add both I/O
and RAM pressure, and when the instance has any internal performance issues
due to contrib code or configuration, it may result with slower processing
for all sites in your instance during all backup windows, resulting in
noticeable performance degradation.
Expected downtime:
Please note that while the maintenance window is less than 2 hours, the hardware
maintenance itself which will be carried out by our facility OnSite engineers
is estimated at 30 minutes per machine. However, if it will be determined
that extra steps or firmware upgrades or even partial hardware replacements
are required, it may extend the effective downtime beyond the 30 minutes minimum.
We thank you for your continued support while we work to make these improvements.