Microsoft Azure Service Outage
Incident Report for GoFormz
Postmortem

After service recovery, the Microsoft Azure team is providing more detail regarding the service disruption that occurred on May 2, 2019.

Here is that detail directly from Microsoft:

Network Connectivity - DNS Resolution

Summary of impact: Between 19:43 and 22:35 UTC on 02 May 2019, customers may have experienced intermittent connectivity issues with Azure and other Microsoft services (including M365, Dynamics, DevOps, etc). Most services were recovered by 21:30 UTC with the remaining recovered by 22:35 UTC.

Preliminary root cause: Engineers identified the underlying root cause as a nameserver delegation change affecting DNS resolution and resulting in downstream impact to Compute, Storage, App Service, AAD, and SQL Database services. During the migration of a legacy DNS system to Azure DNS, some domains for Microsoft services were incorrectly updated. No customer DNS records were impacted during this incident, and the availability of Azure DNS remained at 100% throughout the incident. The problem impacted only records for Microsoft services.

Mitigation: To mitigate, engineers corrected the nameserver delegation issue. Applications and services that accessed the incorrectly configured domains may have cached the incorrect information, leading to a longer restoration time until their cached information expired.

Next steps: Engineers will continue to investigate to establish the full root cause and prevent future occurrences. A detailed RCA will be provided within approximately 72 hours.

Posted May 03, 2019 - 13:00 PDT

Resolved
As of 5:00 PM today (May 2nd), Microsoft Azure has updated all services to Operational. GoFormz has been stable since approximately 3:53 PM PDT and we are confident all users should be fully returned to their normal GoFormz services. If you are still experiencing any errors or issues please reach out to us at support@goformz.com.
Posted May 02, 2019 - 17:18 PDT
Update
GoFormz services are performing as operational. We are continuing to monitor this issue. Please see below for the latest update from Microsoft Azure.

"Starting at 19:43 UTC on 02 May 2019, customers may experience intermittent connectivity issues with Azure and other Microsoft services (including M365, Dynamics, DevOps, etc).

Engineers have identified the underlying root cause as a name server delegation issue with DNS resolution, affecting network connectivity and downstream impact to Compute, Storage, App Service, AAD, and SQL Database services. Mitigation has been applied, and engineering teams are clearing resolver cache to fully mitigate the issue. Most services are showing recovery.

This message was last updated at 22:10 UTC on 02 May 2019"
Posted May 02, 2019 - 15:53 PDT
Monitoring
GoFormz is beginning to experience an intermittent return of services. We are continuing to monitor this situation as Microsoft Azure's performance improves.
Please reference the Microsoft Azure status page for more information: https://azure.microsoft.com/en-us/status/
Posted May 02, 2019 - 14:45 PDT
Update
Here is the latest from the Microsoft Azure Status Page:

"Customers may experience intermittent connectivity issues with Azure and Microsoft services. Engineers are investigating DNS resolution issues affecting network connectivity. Connectivity issues may affect the availability of Compute, Storage, and Database services, and some customers may be unable to file support requests. More information will be provided as it becomes available.

This message was last updated at 21:10 UTC on 02 May 2019"

Please refer to the Microsoft Azure Status page for more information: https://azure.microsoft.com/en-us/status/
Posted May 02, 2019 - 14:18 PDT
Update
Here is the latest update from Microsoft Azure Support: "Engineers are aware of intermittent connectivity issues with Azure Services. More information will be provided as events warrant."

Please refer to the Microsoft Azure Status page for more information: https://azure.microsoft.com/en-us/status/
Posted May 02, 2019 - 13:50 PDT
Investigating
Starting at approximately 12:57 PM PDT, GoFormz started experiencing an outage in services provided by Microsoft Azure for our mobile and web platforms. GoFormz sync and other network services are impacted by this outage.

Here is the latest update we have from Microsoft Azure Support:

“Engineers are investigation connectivity issues with Azure Services. More information will be provided as it becomes available.” - Twitter (https://twitter.com/AzureSupport/status/1124046510411460610)
Posted May 02, 2019 - 12:57 PDT
This incident affected: API (v2), Web App, Mobile Sync API, and Automated Workflows.