GoFormz engineering has confirmed that all Workflow systems are operational and stable.
Microsoft has issued the following statement:
Intermittent Network Degradation - South Central US - Mitigated Summary of impact: Between 02:48 UTC and 04:52 UTC on 04 Dec 2018, a subset of customers in South Central US may have experienced degraded performance, network drops or time outs when accessing Azure resources hosted in this region. Applications/resources that retried connections or requests may have succeeded, as there are many redundant network devices in the impacted data center and retries are likely to have succeeded through different network devices.
Preliminary root cause and mitigation: Two network devices in the South Central US region experienced an issue updating routing information for packet forwarding. The Azure Fabric failed to automatically remove these devices from service and continued to drop a fraction of the network packets passed through them. Once identified and manually removed from service by Azure engineers the network automatically recovered utilizing alternate network devices.
Next steps: Engineers will continue to investigate to establish the full root cause. A full Root Cause Analysis will be provided in approximately 72 hours. To stay informed on any issues, maintenance events, or advisories, create service health alerts (https://www.aka.ms/ash-alerts) and you will be notified via your preferred communication channel(s): email, SMS, webhook, etc.
GoFormz engineering has also confirmed that our test Workflows are performing as expected.
Even though GoFormz customers should now experience normal Workflow behavior, we will continue monitoring until Microsoft provides updated status.
Posted Dec 03, 2018 - 21:24 PST
Identified
GoFormz engineering has identified the source as an outage with Microsoft Azure. They have published the following:
Starting at approximately 03:30 UTC on 04 Dec 2018, customers in South Central US may experience degraded performance, network drops or time outs when accessing Azure resources hosted in this region. Engineers are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or as events warrant. https://azure.microsoft.com/en-us/status/
GoFormz engineering will continue to monitor Azure's progress and resolve any issues as services return to operating levels.
Posted Dec 03, 2018 - 20:42 PST
Investigating
Some workflows are not completing successfully. GoFormz engineering is investigating the issue and will update as soon as possible.