The incident impact: Marketplace Simulations web services were down, and games were not able to process.

The actions taken to mitigate or resolve the incident: The Marketplace Simulations monitoring service notified the on-call sysadmin about the issue with web services. The sysadmin was able to verify that the incident was an issue at the web-hosting provider level, and called immediately to open an incident and to get more details on the issue. During this call, the issue being one at the web hosting provider level was verified, however service was already restored before more information could be gathered.

The incident's root cause: This was a networking issue created by our web hosting/data center provider. While working on provisioning a circuit for a customer, they created a problem for some of their customers in the local network. Their backbone engineering team helped and fixed the issue.

Follow-up actions being taken to prevent the incident from happening again: This is unknown at this time. This is a conversation we will have with the provider (flexential.com), about how they will ensure this mistake will not be repeated.