Although Microsoft Azure current status (as of Thursday, Sept. 6 @ 3:30pm pacific time) shows two services still impacted from Tuesday outage, we believe all of our users should be fully back online with their GoFormz services. If you are still experiencing any errors or issues please reach out to us at support@goformz.com and we will do everything we can to help.
Posted Sep 06, 2018 - 15:40 PDT
Update
While some GoFormz customers have reported trouble-free services, others will continue to be impacted as Azure services continue to come back online.
-- Here is the latest from the Microsoft Azure engineering team --
Multiple Services - Applying Extended Mitigation
UPDATE AS OF 15:00 UTC: Engineers have restored storage availability for the majority of impacted services, and customers should be continuing to see improvements to service availability.
CUSTOMER IMPACT: Starting at 09:29 UTC on 04 Sep 2018, customers with resources in South Central US may experience difficulties connecting to resources hosted in this region. A complete list of impacted services can be found below.
PRELIMINARY ROOT CAUSE: A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process.
ENGINEERING STATUS: Engineers have restored access to storage resources for the majority of services, and most customers should be seeing signs of recovery. Engineers are continuing to work on any residual storage impact to fully mitigate this issue. We are also working with the impacted service teams to validate their service health in order to update this status page.
NEXT UPDATE: The next update will be provided by 17:00 UTC 05 Sep 2018 or as events warrant.
While some GoFormz customers have reported trouble-free services, others will continue to be impacted as Azure services continue to come back online.
Microsoft Azure engineers continue to focus at this time on recovery to impacted Storage scale units in South Central US. Mitigation efforts continue for storage-dependent services as well.
-- Here is the latest from Microsoft on the Azure service outage --
Multiple Services - Applying mitigation with some recovery
CUSTOMER IMPACT: There are currently three identified impact workstreams. Starting at 09:29 UTC on 04 Sep 2018:
1) Customers with resources in South Central US may experience difficulties connecting to resources hosted in this region. A complete list of impacted services can be found below. 2) Customers using some non-regional services may experience intermittent failures in multiple regions. 3) Customers may encounter errors when provisioning new subscriptions.
PRELIMINARY ROOT CAUSE:
A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process.
ENGINEERING STATUS: Engineers continue to implement the necessary mitigation steps according to the outlined mitigation workflow:
1) Restore power to the South Central US datacenter (COMPLETED) 2) Recover software load balancers for Azure Storage scale units in South Central US (COMPLETED) 3) Recover impacted Azure Storage scale units in South Central US (In Progress) 4) Recover the remaining Storage-dependent services in South Central US (In Progress)
NEXT UPDATE: The next update will be provided by 05:00 UTC 05 Sep 2018 or as events warrant.
Posted Sep 04, 2018 - 20:40 PDT
Update
Microsoft Azure is still reporting issues with multiple services that are impacting Azure Active Directory across all locations, including but not limited to South Central US.
-- LATEST UPDATE FROM MICROSOFT AZURE --
Warning Multiple Services - Applying mitigation with some recovery
CUSTOMER IMPACT: There are currently two identified impact workstreams:
1) Customers with resources in South Central US may experience difficulties connecting to resources hosted in this region. A complete list of impacted services can be found below. 2) Customers using non-regional services, such as Azure Active Directory, may experience intermittent authentication failures in any region.
PRELIMINARY ROOT CAUSE:
1) A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process. 2) As a result, non-regional services, such as Azure Active Directory, encountered an operational threshold for processing requests through the South Central US datacenter. Initial attempts to fail over into other datacenters resulted in temporary traffic congestion for those regions.
ENGINEERING STATUS: Engineers have successfully restored power to the datacenter. Additionally, engineers have recovered a majority of the impacted network devices. While some services are starting to see signs of recovery, mitigation efforts are still ongoing.
Posted Sep 04, 2018 - 14:22 PDT
Monitoring
Microsoft Azure is still reporting issues with multiple services that are impacting Azure Active Directory and Resource Manager across all locations, including but not limited to South Central US. We are monitoring the issue and will update accordingly. https://azure.microsoft.com/en-us/status/
Posted Sep 04, 2018 - 09:26 PDT
Identified
GoFormz sync and other network services impacted by MS Azure outage. Starting at 09:29 UTC on 04 Sep 2018, Microsoft Azure reported technical issues with their South Central US location. Also, some services may also be experiencing intermittent authentication issues due to downstream Azure Active Directory impact, and engineers are separately working on mitigation options for this also.
Posted Sep 04, 2018 - 04:30 PDT
This incident affected: API (v2) and Mobile Sync API.