Hosted phone provider is currently experiencing networking issues.

Incident Report for AYN

Postmortem

Summary:
The issue occurred due to a deadlock bug in the OpenZFS implementation on the primary storage server, which caused corruption in the data replication process. It meant a full backup recovery was required for all 200+ VPS, approximately 3.5TB of data, and caused an outage of about 5hrs in total.

Future Steps:
The provider will implement additional monitoring, upgraded the OpenZFS implementation, and work on the diversity of POP’s for core services to keep communication channels open.

Asyouneed Comments:
We believe the steps outlined once completed are correct.
The bug referenced is a scarce one to be triggered, and it seems they were a bit unlucky, and it was made worse by not monitoring the response of the applications they are hosting, focusing purely on the network. The move to monitor both network and the applications should mean faster response to issues, and we are glad to see they will be diversifying the communication channels to keep us in the loop.

Posted Apr 30, 2021 - 11:41 BST

Resolved

This incident has been resolved.

Posted Apr 28, 2021 - 12:32 BST

Monitoring

Systems are operational again - we will continue to monitor the situation.

Posted Apr 28, 2021 - 10:20 BST

Update

We have had no further update at this stage however it seems they have had a serious system wide issue that has knocked out all their systems.

Posted Apr 28, 2021 - 09:31 BST

Update

Update from provider

"We want to inform you that we are aware that an unplanned event is occurring and our engineers are in the process of investigating and resolving the issue."

Posted Apr 28, 2021 - 08:20 BST

Identified

The issue has been identified and a fix is being implemented.

Posted Apr 28, 2021 - 07:10 BST

Investigating

We are currently investigating this issue.

Posted Apr 28, 2021 - 06:49 BST

This incident affected: Asyouneed 3rd Party Services and Asyouneed Services (Hosted Phone Services).