The system has continued to be stable since the last update. We will be closing this incident while waiting for an RCA from Microsoft.
Posted Mar 20, 2024 - 13:25 EDT
Monitoring
The issue was identified since the last update as being caused by an outage of a Microsoft Azure resource used by our system.
Errors with that resource have since mostly disappeared, meaning that since around ~10am EDT/2pm UTC most units and Access Manager roles should have stayed online.
We are currently monitoring the system, and waiting for more information from Microsoft. If more errors start appearing again, we will post another update.
Posted Mar 20, 2024 - 11:26 EDT
Update
We are continuing to investigate the issue. We would like to confirm the impacts of this incident are: - Units going offline and then back online sporadically - Synchronization attempts (adding/removing/activating/deactivating cardholders, adding new hardware, etc) to the units might fail if the unit goes offline during one such attempt. The attempt will be retried as soon as the unit comes back online. - No data will be lost in the system
Posted Mar 20, 2024 - 10:26 EDT
Investigating
We are currently investigating an issue causing units and Access Manager roles to go offline/online sporadically