The incident impact: Marketplace Simulations web services were inaccessible on March 29, 2023 from 13:39 until 14:38 EDT.

Actions taken to mitigate or resolve the incident: The Marketplace Simulations monitoring service notified the on-call sysadmin about the issue with web services. After troubleshooting, it was determined that there were access/network issues causing the trouble.  A call was placed to the service provider where Marketplace Simulations servers are hosted, to investigate further, and resolve the issue.

The incident's root cause: The root cause was identified as a firewall policy that was inadvertently altered during troubleshooting work, resulting in removal of outbound access for Marketplace Simulations servers via DNS, NTP, HTTP/HTTPS, and SMTP.

Follow-up actions taken to prevent the incident from happening again: Through follow-up conversations with the service provider and our account representatives, the engineer and other staff have been instructed that no work that will potentially impact Marketplace Simulations is to take place outside of an approved maintenance window.