Monthly Archives: April 2014

Loss of service for some customers

We are currently aware that some customers’ VPS have stopped functioning. We are investigating urgently and will update here as soon as we have further information.

Update 06:22AM BST – We have identified the problem and are working hard to restore service for customers affected. Further updates will be posted when we have more information.

Update 06:44AM BST – Unfortunately we’ve had a failure of a RAID card on one of our VPS hosts. We have replaced the card with a new one but we are currently checking the drives to ensure there is no data loss. As soon as we care confident the drives are OK we will be able to restore service. We will keep this page updated.

Update 07:18AM BST – We are able to access the data on the drives and all looks fine, but we’d like to move the data to a fresh set of drives to be absolutely sure there is no further risk to customer data. We’re in the process of doing that and then will be able to restore service to affected customers on this particular VPS host. We will copy each VPS’ data in turn and will be able to restore service to each VPS in turn in this way.

Update 08:47AM BST – Unfortunately the process is taking longer than expected, please accept our apologies for the delay. At the moment we cannot give an ETA, but will continue to update on here.

Update 10:39AM BST – Unfortunately it turns out the drives were not as fine as they initially looked. We found significant corruption and despite all our efforts we have been unable to restore the data. We are devastated to inform customers on this particular VPS host that their VPS data is lost with no hope of recovery. The data was stored on a RAID 10 array of 4 x SAS disks connected to an enterprise level hardware raid card. When the card failed it caused irreversible destruction of data on all 4 disks. As we do not yet offer backups for our VPS services (as stated in our FAQs) there are no backups from which we can restore anything.  Affected customers will now find in the VPS Control Panel that they can start their VPS but they will need to create new fresh virtual drives for their VPS. We now ask affected customers to raise a support ticket with us if you need further assistance with anything. If you have backups of your VPS elsewhere, we will assist in any way we can to help you restore data to your VPS, please raise a ticket so that we can discuss. We can only offer a sincere and heartfelt apology for this dreadful incident. We are now reviewing our storage infrastructure and are going to double our efforts to put in place a backup solution so that customers can take regular backups of their VPS. We will announce when this is ready.

Steps we’ve taken to mitigate against Openssl “Heartbleed” bug

It’s quite likely you will have heard by now about the Openssl “Heartbleed” bug, which was made public two days ago, and caught the worlds attention yesterday. If you haven’t heard yet, just type Openssl Heartbleed into your favourite search engine. It is probably the vulnerability that is inflicting the worst damage we’ve ever seen across the world right now. If you haven’t checked if your VPS is vulnerable yet, you really must treat this with urgency if you want to minimise any damage. The vulnerability allows anyone to easily retrieve random portions of memory from services running on your VPS that rely on Openssl to encrypt sensitive data. That memory may contain sensitive data such as session cookies, usernames, passwords, or possibly even private keys.

The purpose of this post is to let you know what action we took yesterday, Tuesday 9th April, to secure our own infrastructure. That is, our own servers and web services. This information does NOT apply to customers’ VPS’. We only offer unmanaged services at the moment, so it is customers’ responsibility to ensure their systems are patched regularly and promptly. If you need assistance with dealing with this, or any other serious security issue, please do not hesitate to raise a support ticket and we will be glad to help.

Early in the morning yesterday (BST) we made sure all affected servers on our infrastructure had the released Openssl update applied, and any affected services were restarted to ensure the update had taken effect. This means it was no longer possible for anyone to directly exploit the vulnerability on our servers, as of mid morning Tuesday 9th April. Judging by the media reports we are seeing today of companies’ web sites being actively exploited, we have acted very swiftly on that front.

However, as you will be hearing again and again from many companies over the coming days and weeks, it is impossible for anyone to know whether this vulnerability has been actively exploited prior to the fix being applied. Therefore, we took to precautionary measure of generating a new private key, contacting our SSL certificate vendor, and having them re-issue a new certificate for all our https protected services.  We also took the opportunity to upgrade from a SHA1 SSL certificate to a stronger SHA256 certificate. By yesterday evening, the new cert was deployed to all our web services, including the Client Billing/Account area, and VPS control panel. What this all means is, if by any chance someone was able to steal our private key via this vulnerability prior to us updating, they would not be able to use it to compromise us.

Despite our swift action, we’d like to ask our customers to reset both their billing account password and VPS Control Panel password at the earliest opportunity. Although we have no evidence of any malicious activity having taken place, it would be very wise for customers to do this purely as a precautionary measure.