At approximately 21:00 UTC on 5/2/22 - customers began to see intermittent 502s as well as slow down in response times. These rates increased through the night and were resolved at approximately 13:39 on 5/3/22.
Root cause was determined to be an influx of requests from an outside source. We have scaled the system accordingly and will continue to monitor the status closely.
Going forward we will continue to 1) investigate the root cause of the issue as well and 2) investigate as to why our alerting system failed to notify the support team.
This is our first customer impacting outage in over 3 years and we'd like to sincerely apologize to any of our partners who were impacted.