The Data Centre we are in are this evening experiencing intermittent network issues. There have been two occasions this evening, once at around 18:15, and again at around  20:55, where VPS users have seen varying levels of packet loss lasting between 2 – 4 minutes.  Rest assured the Data Centre have a team of network engineers aware of the issues and currently working to resolve them. We’d like to apologise to customers for the inconvenience caused. We will provide a further update in due course.

Update 22:45 – The network has been stable since the last incident at about 20:55. If we see any further incidents we’ll add a further update but, for now, all is looking fine.

Update Mon 11th Aug, 18:41 – The Data Centre have informed us that the packet loss yesterday was the result of a persistent DDoS attack against one of their other customers. The DC have managed to mitigate against the attack, and we have not seen any further packet loss since yesterday as a result.

Tues 24th June 2014 14:25 – Unfortunately we just suffered a brief loss of connectivity on one of our network segments for approx. 10 minutes, which affected some of our VPS customers. This was as a result of a mistake we made while configuring a switch. Network connectivity is now fully restored, and lessons learned. We’d like to apologise to all customers affected.

We are currently aware that some customers’ VPS have stopped functioning. We are investigating urgently and will update here as soon as we have further information.

Update 06:22AM BST – We have identified the problem and are working hard to restore service for customers affected. Further updates will be posted when we have more information.

Update 06:44AM BST – Unfortunately we’ve had a failure of a RAID card on one of our VPS hosts. We have replaced the card with a new one but we are currently checking the drives to ensure there is no data loss. As soon as we care confident the drives are OK we will be able to restore service. We will keep this page updated.

Update 07:18AM BST – We are able to access the data on the drives and all looks fine, but we’d like to move the data to a fresh set of drives to be absolutely sure there is no further risk to customer data. We’re in the process of doing that and then will be able to restore service to affected customers on this particular VPS host. We will copy each VPS’ data in turn and will be able to restore service to each VPS in turn in this way.

Update 08:47AM BST – Unfortunately the process is taking longer than expected, please accept our apologies for the delay. At the moment we cannot give an ETA, but will continue to update on here.

Update 10:39AM BST – Unfortunately it turns out the drives were not as fine as they initially looked. We found significant corruption and despite all our efforts we have been unable to restore the data. We are devastated to inform customers on this particular VPS host that their VPS data is lost with no hope of recovery. The data was stored on a RAID 10 array of 4 x SAS disks connected to an enterprise level hardware raid card. When the card failed it caused irreversible destruction of data on all 4 disks. As we do not yet offer backups for our VPS services (as stated in our FAQs) there are no backups from which we can restore anything.  Affected customers will now find in the VPS Control Panel that they can start their VPS but they will need to create new fresh virtual drives for their VPS. We now ask affected customers to raise a support ticket with us if you need further assistance with anything. If you have backups of your VPS elsewhere, we will assist in any way we can to help you restore data to your VPS, please raise a ticket so that we can discuss. We can only offer a sincere and heartfelt apology for this dreadful incident. We are now reviewing our storage infrastructure and are going to double our efforts to put in place a backup solution so that customers can take regular backups of their VPS. We will announce when this is ready.

12:48PM: We are currently aware that some customers are experiencing a loss of network connectivity to/from their VPS.  We are investigating the issue and will update this post with further information once we know more.

Update 12:52PM: The data centre have just posted a status update advising that network incident has just occurred causing the issue, which they are investigating.

We are able to see that all customer VPS affected are running fine, this is purely an issue with the Data Centre’s network.  Once they have restored connectivity traffic to/from your VPS will resume without any need for action from the customer.

Update 12:57PM: We have noticed that network connectivity has now been restored by the Data Centre. We are waiting for an update from them but it looks like the incident may be over now.

Update 1:04PM: Network connectivity was restored by the Data Centre at about 12:56PM and has been fine since then, so we are assuming for now that the incident is over. Our monitoring systems detected that the incident started at about 12:40PM so total time of the outage was about 15 minutes. As soon as we have heard from the Data Centre what the reason for the outage was we will post details here.

Reason for outage

The following day after the above outage, the Data Centre we are located in sent us an explanation of what happened, which we have posted below.  The incident affected all of the Data Centre’s customers.  Thankfully they dealt with it very swiftly.  In the whole time we have been with them this is the only significant incident we have ever experienced, which is a testament to the excellent facility that it is, and the fantastic staff that work there, who we have met on many occasions and have always been highly impressed by. If any of our customers have any questions about this incident, please do not hesitate to contact us.

"As part of a routine firewall deployment, some new hardware was connected to our network and BGP sessions were configured on our route reflectors for these new devices.

Unfortunately, when these new sessions connected to our route reflectors, they caused an issue with the software we are using to power them – specifically the length of time taken to process and send the full BGP table to the new devices caused the “hold-timer” to expire on a few of the other sessions. This then resulted in these sessions disconnecting and reconnecting, and doing the same thing as the first sessions, causing further sessions to do the same, ultimately resulting in a loop where by the route reflectors were continuously dealing with sessions disconnecting and reconnecting.

This continued until such time as our Network Engineers identified the issue, and were able to log into the route reflectors and manually restart them to clear all the BGP sessions. The first of which was done at 12:50, the second at 12:58. All BGP sessions were re-established with all routes fully exchanged by 13:00 and traffic once again began to flow normally.

It has become apparent from this incident that we have reached a limit with the current Route Reflector software that we are using, and as such we are now in the process of replacing these with both improved hardware, and alternative software which has better handling of long-running tasks that doesn’t result in cpu-starvation of important tasks such as maintaining hold-timers.

Initially we will run these new route-reflectors alongside the old ones to prove their stability, before ultimately retiring the old devices, a further maintenance window will be scheduled for this which you will be notified about in due course."