Disruption in mobile sync and mobile login
Incident Report for GoFormz

Analysis: After further investigation by GoFormz engineers, it was determined that this incident was triggered by an unusually large mobile Sync process initiated by a GoFormz customer that caused GoFormz’s Permission Service to become non-responsive. As designed, the Permission Service started an auto-heal process. However during this auto-heal process the Sync Service continued to receive 500 error codes which degraded performance until the Sync Service was then manually restarted.

Mitigation: In order to mitigate potential recurrence of this specific type of incident, GoFormz’s engineering has scaled up our Permission Service to better handle such unusual usage spikes. We are also reconfiguring the Permission and Sync Service auto-heal policies to appropriately handle 500 error codes from different platform services.

Posted Oct 31, 2018 - 10:26 PDT

From approximately 2:20pm PST to 4:30pm PST, GoFormz mobile sync service suffered degraded performance which resulted in partial disruption of mobile sync and login to some customers. GoFormz engineering performed initial troubleshooting and was able to return the service to fully operational within 2 hours. GoFormz engineering will further investigate this disruption to determine necessary mitigation steps.
Posted Oct 20, 2018 - 16:55 PDT
This incident affected: Mobile Sync API.