Outage in Amsterdam 1

Incident Report for BitLaunch

Postmortem

We would like to provide a more detailed explanation of the recent networking outage that affected our services and customers.

Our network experienced a significant outage due to a sudden power loss at the Amsterdam datacenter. This caused our upstream providers network infrastructure to go down. Our hardware remained operational during this time.

Once power was restored, this caused a scenario which unfortunately led to a split-brain issue on upstream routing engines. This continued to cause issues with the connectivity of our services. Engineers worked to get this resolved as soon as possible.

We appreciate your patience during this time.

If you continue to experience any issues, please open a support ticket. Thank you.

Posted Mar 14, 2023 - 15:21 UTC

Resolved

Connectivity has been restored in this region. A post mortem will follow.

We apologize for the inconvenience caused.

Posted Mar 14, 2023 - 14:59 UTC

Monitoring

Upstream networking engineers have completed their fix. We will continue to monitor until we mark this issue as resolved.

Posted Mar 14, 2023 - 14:10 UTC

Identified

Datacenter and Upstream engineers are working on this issue as priority. We will continue to post updates as we get them.

Posted Mar 14, 2023 - 13:04 UTC

Investigating

We are investigating a secondary outage which we believe to be on-going power issues.

Posted Mar 14, 2023 - 12:15 UTC

Monitoring

Network connectivity has been restored in the DC. We will be monitoring for further issues.

Posted Mar 14, 2023 - 10:21 UTC

Update

Emergency power maintenance is still on-going. We are seeking an ETA on resolution.

We sincerely apologise for the inconvenience this has caused.

Posted Mar 14, 2023 - 08:58 UTC

Update

Emergency power maintenance is still on-going. We apologise for this inconvenience.

Posted Mar 14, 2023 - 00:59 UTC

Identified

The DC is performing emergency power maintenance. Unfortunately this was not scheduled. We will continue to update as we receive them

Posted Mar 13, 2023 - 23:35 UTC

Investigating

Engineers are currently looking into this issue.

Posted Mar 13, 2023 - 23:06 UTC

This incident affected: BL Networks (AMS1).