Infrastructure and network failure (13–14 April)

On 13-14 April, our services experienced a prolonged outage due to an account and service provisioning issue with our previous infrastructure provider. This disruption included a period where our systems stayed unreachable over the network even after the initial recovery steps. We restored full service by migrating to a new service provider. Normal operation was confirmed by Apr 14 20:45 Europe/Helsinki (EEST, UTC +3). A rough timeline of events and the steps we're taking to reduce the risk and duration of any similar events in the future follow.

Timeline

All times are approximate and shown in Europe/Helsinki (EEST, UTC +3).

13 April

• 18:05 - We start receiving alerts from our monitoring systems indicating service failure. Customer reports start coming in.
• 20:10 - Partial recovery on the previous provider side: servers are back online but customer-facing connectivity is still impaired. Throughout the evening we are unable to restore normal service.

14 April

• 07:20 - 16:30 - Formal escalation with the previous provider continued in parallel with our own recovery planning. We re-checked monitoring and dependencies, prioritised restoration of critical paths, and assessed risk and time-to-recover for staying versus migrating. After that assessment, we committed to migration as the fastest reliable path back to normal service, and used this window to prepare infrastructure, data movement, and validation steps ahead of the migration starting at 16:45.
• 16:45 - Migration to new service provider begins.
• 20:45 - Migration is complete. All critical systems are back online and verified working.

Incident response and ongoing improvements

What worked

• Our monitoring detected the failure and generated alerts as expected.
• Once we committed to migration, we were able to complete the migration and verification with all critical systems confirmed operational by approximately 20:45 on 14 April.

What did not work well enough

• Communication: customers did not have a single, authoritative place for live status during the incident. This notification is published on our new status page; we should have had that in place earlier.
• Provider escalation: we opened a formal support case later than we should have for a customer-impacting connectivity issue, which extended the window where we depended on asynchronous provider response.

Actions we are taking

• Status page: we are establishing a dedicated status page for service health and incident updates so future disruptions are communicated in one place, in near real time.
• Escalation discipline: we will open formal provider tickets immediately for any production-impacting network or platform fault where we cannot restore service ourselves, rather than waiting while internal troubleshooting continues in parallel.