Summary

Between April 05 at 21:22 and April 6, 2024, at 00:21 UTC, some Atlassian customers were unable to submit support tickets via the Atlassian Support portal. The event was triggered by a failure release of a route configuration for a support form backend service. The changes included a redirection of an API route in a resource file. The incident was detected within one minute by a customer and mitigated by rolling back the configuration change which put Atlassian Support form system into a known good state. The total time to resolution was about three hours and 21 minutes.

IMPACT

The overall impact was between April 05 at 21:22 and April 6, 2024, at 00:21 UTC on the Atlassian support portal. The Incident caused service disruption to some customers where they attempted to submit a support case through the online portal.

ROOT CAUSE

The issue was caused by a change to a support form backend service resource file.  As a result, the support site could not make online support ticket requests, and the users received support form connectivity errors when attempting to submit the form as well as a banner notification alerting to the system issue. 

REMEDIAL ACTIONS PLAN & NEXT STEPS

We know that outages impact your productivity. While we have a number of testing and preventative processes in place, this specific issue wasn’t identified because the change was related to a very specific service descriptor file change that was not covered by our automated continuous deployment suites and tests.

We are prioritizing the following improvement actions to avoid repeating this type of incident:

• Create additional route configuration automated testing and monitoring
• Consider adding deployment / region specific configuration controls to APIs
• Create additional endpoint PDV (post-deployment verification) monitors for system health
• Updates to hot fix deployment pipeline processes to support incident remediation

We apologize to customers who were impacted during this incident; we are taking immediate steps to improve the Atlassian Support portal availability.

Thanks,

Atlassian Customer Support