GoFormz service interruption

Incident Report for GoFormz

Postmortem

Please review the postmortem below shared by the Microsoft Azure Team:

https://status.azure.com/en-us/status/history/

Azure Firewall, Azure Data Explorer, Azure Synapse Analytics, Azure Back up, Azure Search - Multi Region (Tracking ID YVTL-RS0)

Starting at 05:26 UTC on 28 Jun 2022, a subset of customers may experience intermittent network failures and packet loss while trying to connect to the Azure Firewall hosted in affected regions. Additionally, those attempting to perform Backup operations in the affected regions may see intermittent failures. We have determined some downstream services impacted have now been mitigated, this includes Azure Data Explorer and Azure Stream Analytics, with the cause identified as an underlying issue in the Azure Software Load Balancer service.

Workstreams:

Complete: Our first workstream was to update and reset the state in our load balancing platform for impacted endpoints.
In Progress: Our second workstream is a hotfix deployment is in progress to mitigate and is estimated to be completed by 04:00 UTC on July 1. This hotfix is specifically for our software load balancer platform. A subset of customers using service may experience a recurrence of impact during the rollout of this hotfix.

Current Status:

Until our second workstream is completed, an automated mitigation workflow is running to prevent customer impact.

We will continue to provide updates via Service Health in the Azure Portal.

Posted Jun 30, 2022 - 11:29 PDT

Resolved

This incident has been resolved. We will be posting a Postmortem shortly.

Posted Jun 30, 2022 - 11:26 PDT

Update

GoFormz are continuing to monitor, however, all GoFormz services are operational. Please see the latest update below from Microsoft Azure:

Azure Status Page: https://status.azure.com/en-us/status

Azure Data Explorer and Firewall- Global

Starting at 05:26 UTC on 28 Jun 2022, a subset of customers have reported intermittent network failures and packet loss while trying to connect to the service hosted in this region when using Azure Firewall. Customers also reported issues with ingesting data to Azure Data Explorer instances as well as timeout errors when trying to query data. Some other downstream services may be impacted by this incident and their status will be updated via service health.

Current Status: We currently have two workstreams to mitigate the issue: our first workstream is updating and resetting storage nodes. The recent deployment caused traffic to be diverted to compute nodes that were marked unhealthy. When the requests would land of these unhealthy nodes they would fail. We are currently working on addressing these nodes and are 73% completed. Our second workstream is that we are building out a hotfix which was deployed around 23:00 UTC. This hotfix is for our software load balancer and was completed at 13:20 UTC and we have been validating its ability to mitigate across the platform. This hotfix will not reintroduce any impact to customers that have been manually mitigated so far. We expect full mitigation to be completed within 24 hours. The next update will be provided in 2 hours, or as events warrant.

This message was last updated at 23:03 UTC on 29 June 2022

Posted Jun 29, 2022 - 17:18 PDT

Monitoring

All GoFormz services have returned to operational. We are continuing to monitor closely.

Posted Jun 29, 2022 - 09:54 PDT

Update

The Microsoft Azure Networking team is currently investigating network issues impacting various services.

For more information, please visit the Microsoft Azure Status page (https://status.azure.com/en-us/status).

We are continuing to track this situation closely and will provide further updates as more information is learned.

Posted Jun 29, 2022 - 09:44 PDT

Investigating

Users are experiencing trouble logging and and performing actions GoFormz Web App. Engineers are investigating the issue and will report updated status as soon as possible.

Posted Jun 29, 2022 - 09:15 PDT

This incident affected: API (v2), Web App, Mobile Sync API, Automated Workflows, and Email Delivery.