Author Archives: Eddie

Network packet loss

The Data Centre we are in are this evening experiencing intermittent network issues. There have been two occasions this evening, once at around 18:15, and again at around  20:55, where VPS users have seen varying levels of packet loss lasting between 2 – 4 minutes.  Rest assured the Data Centre have a team of network engineers aware of the issues and currently working to resolve them. We’d like to apologise to customers for the inconvenience caused. We will provide a further update in due course.

Update 22:45 – The network has been stable since the last incident at about 20:55. If we see any further incidents we’ll add a further update but, for now, all is looking fine.

Update Mon 11th Aug, 18:41 – The Data Centre have informed us that the packet loss yesterday was the result of a persistent DDoS attack against one of their other customers. The DC have managed to mitigate against the attack, and we have not seen any further packet loss since yesterday as a result.

Centos 7 is here, ISOs added

The CentOS team have just announced a few hours ago the very first generally available release of the next major version of CentOS Linux, version 7.

The initial release is for for 64 bit x86 compatible machines only. The CentOS project say on their site homepage “There are many fundamental changes in this release, compared to previous releases of CentOS. Notably the inclusion of SystemD, Gnome3, and a default filesystem of XFS.”  For more information see the release announcement and release notes.

Existing Centos 6 users will be interested to know that, for the first time, there is a supported upgrade path from CentOS 6 to CentOS 7. See the release notes for further info.

We have added the main DVD install ISO to our collection, for customers who would like to install the OS into their VPS already.  We’ve also added two Live CD versions for Gnome and KDE desktops, for those who’d like to explore the CentOS 7 environment without installing yet. The Live CDs will not modify the content of your VPS hard disk(s), unless you choose to install CentOS 7 from within the Live environment.

As always, if there is a particular operating system you’d like to see in our collection, whether you are a prospective customer thinking about joining us, or an existing customer, do not hesitate to suggest it, we will almost certainly say Yes!

Brief network outage

Tues 24th June 2014 14:25 – Unfortunately we just suffered a brief loss of connectivity on one of our network segments for approx. 10 minutes, which affected some of our VPS customers. This was as a result of a mistake we made while configuring a switch. Network connectivity is now fully restored, and lessons learned. We’d like to apologise to all customers affected.

Maintenance Sunday 18th May @1AM

We will be performing essential maintenance work on our VPS host servers between 1AM – 2AM BST on Sunday 18th May. During this window  your VPS will be unavailable for up to 20 mins while the host server is rebooted. Before rebooting we will be sending each VPS an ACPI Shutdown signal. If you have configured your VPS to respond to this, it will start it’s shutdown procedure, and shutdown cleanly. If you would like help configuring your VPS to respond to this, please raise a support ticket and we will be happy to help.

We’d like to apologies in advance for any inconvenience caused by this essential work.

Update 2:35AM – Most of the work has now been completed. However, the work is taking longer than anticipated, and we have gone past the end of the maintenance window, which we’d like to apologise for. Therefore, some customers have yet to see their VPS become unavailable, and will do so soon. We will update here as soon as the work is fully complete.

Update 4:05AM – This work is now fully complete. We’d like to apologise for extended the window, the work took a lot longer than anticipated.  Total downtime experienced for each VPS, however, was not longer than was planned.  All VPS are now running as normal, but if you are experiencing any problems with your VPS at this point, please do raise a support ticket so we can look into it. Thanks to everyone for your patience during this essential work.

Improvements to data storage resilience

We’re pleased to announce that we’ve recently completed moving all VPS data to a new storage subsystem that mirrors all data across more than one physical server.

Prior to this, VPS data was stored on the RAID card attached to the physical server the VPS was running on. Although this provided good data safety by mirroring across multiple drives, the RAID card itself was a single point of failure.

The new storage subsystem eliminates this single points of failure.  Even if we were to suffer the complete failure of all disks attached to a physical machine and/or the complete failure of a raid card, including the situation where a raid card might inflict data corruption to all disks attached to it, this will not affect the VPS disk data on that machine. Additionally, providing that the physical server OS continues running, all VPS’ would be able to continue running without interruption in the face of such a disaster, due to the surviving network mirror on another physical machine.

Another storage related improvement we are planning to introduce over the coming months is the ability for customers to take snapshot backups of their VPS storage data, which will be stored elsewhere on our network. This would allow customers to restore their VPS disk data to a previous “known good” point in time e.g. if their VPS OS becomes unbootable, or is compromised by malicious activity/trojan/virus. We will announce on this blog as soon as we have rolled out this feature.

Loss of service for some customers

We are currently aware that some customers’ VPS have stopped functioning. We are investigating urgently and will update here as soon as we have further information.

Update 06:22AM BST – We have identified the problem and are working hard to restore service for customers affected. Further updates will be posted when we have more information.

Update 06:44AM BST – Unfortunately we’ve had a failure of a RAID card on one of our VPS hosts. We have replaced the card with a new one but we are currently checking the drives to ensure there is no data loss. As soon as we care confident the drives are OK we will be able to restore service. We will keep this page updated.

Update 07:18AM BST – We are able to access the data on the drives and all looks fine, but we’d like to move the data to a fresh set of drives to be absolutely sure there is no further risk to customer data. We’re in the process of doing that and then will be able to restore service to affected customers on this particular VPS host. We will copy each VPS’ data in turn and will be able to restore service to each VPS in turn in this way.

Update 08:47AM BST – Unfortunately the process is taking longer than expected, please accept our apologies for the delay. At the moment we cannot give an ETA, but will continue to update on here.

Update 10:39AM BST – Unfortunately it turns out the drives were not as fine as they initially looked. We found significant corruption and despite all our efforts we have been unable to restore the data. We are devastated to inform customers on this particular VPS host that their VPS data is lost with no hope of recovery. The data was stored on a RAID 10 array of 4 x SAS disks connected to an enterprise level hardware raid card. When the card failed it caused irreversible destruction of data on all 4 disks. As we do not yet offer backups for our VPS services (as stated in our FAQs) there are no backups from which we can restore anything.  Affected customers will now find in the VPS Control Panel that they can start their VPS but they will need to create new fresh virtual drives for their VPS. We now ask affected customers to raise a support ticket with us if you need further assistance with anything. If you have backups of your VPS elsewhere, we will assist in any way we can to help you restore data to your VPS, please raise a ticket so that we can discuss. We can only offer a sincere and heartfelt apology for this dreadful incident. We are now reviewing our storage infrastructure and are going to double our efforts to put in place a backup solution so that customers can take regular backups of their VPS. We will announce when this is ready.

Steps we’ve taken to mitigate against Openssl “Heartbleed” bug

It’s quite likely you will have heard by now about the Openssl “Heartbleed” bug, which was made public two days ago, and caught the worlds attention yesterday. If you haven’t heard yet, just type Openssl Heartbleed into your favourite search engine. It is probably the vulnerability that is inflicting the worst damage we’ve ever seen across the world right now. If you haven’t checked if your VPS is vulnerable yet, you really must treat this with urgency if you want to minimise any damage. The vulnerability allows anyone to easily retrieve random portions of memory from services running on your VPS that rely on Openssl to encrypt sensitive data. That memory may contain sensitive data such as session cookies, usernames, passwords, or possibly even private keys.

The purpose of this post is to let you know what action we took yesterday, Tuesday 9th April, to secure our own infrastructure. That is, our own servers and web services. This information does NOT apply to customers’ VPS’. We only offer unmanaged services at the moment, so it is customers’ responsibility to ensure their systems are patched regularly and promptly. If you need assistance with dealing with this, or any other serious security issue, please do not hesitate to raise a support ticket and we will be glad to help.

Early in the morning yesterday (BST) we made sure all affected servers on our infrastructure had the released Openssl update applied, and any affected services were restarted to ensure the update had taken effect. This means it was no longer possible for anyone to directly exploit the vulnerability on our servers, as of mid morning Tuesday 9th April. Judging by the media reports we are seeing today of companies’ web sites being actively exploited, we have acted very swiftly on that front.

However, as you will be hearing again and again from many companies over the coming days and weeks, it is impossible for anyone to know whether this vulnerability has been actively exploited prior to the fix being applied. Therefore, we took to precautionary measure of generating a new private key, contacting our SSL certificate vendor, and having them re-issue a new certificate for all our https protected services.  We also took the opportunity to upgrade from a SHA1 SSL certificate to a stronger SHA256 certificate. By yesterday evening, the new cert was deployed to all our web services, including the Client Billing/Account area, and VPS control panel. What this all means is, if by any chance someone was able to steal our private key via this vulnerability prior to us updating, they would not be able to use it to compromise us.

Despite our swift action, we’d like to ask our customers to reset both their billing account password and VPS Control Panel password at the earliest opportunity. Although we have no evidence of any malicious activity having taken place, it would be very wise for customers to do this purely as a precautionary measure.

Network outage 3rd March

12:48PM: We are currently aware that some customers are experiencing a loss of network connectivity to/from their VPS.  We are investigating the issue and will update this post with further information once we know more.

Update 12:52PM: The data centre have just posted a status update advising that network incident has just occurred causing the issue, which they are investigating.

We are able to see that all customer VPS affected are running fine, this is purely an issue with the Data Centre’s network.  Once they have restored connectivity traffic to/from your VPS will resume without any need for action from the customer.

Update 12:57PM: We have noticed that network connectivity has now been restored by the Data Centre. We are waiting for an update from them but it looks like the incident may be over now.

Update 1:04PM: Network connectivity was restored by the Data Centre at about 12:56PM and has been fine since then, so we are assuming for now that the incident is over. Our monitoring systems detected that the incident started at about 12:40PM so total time of the outage was about 15 minutes. As soon as we have heard from the Data Centre what the reason for the outage was we will post details here.

Reason for outage

The following day after the above outage, the Data Centre we are located in sent us an explanation of what happened, which we have posted below.  The incident affected all of the Data Centre’s customers.  Thankfully they dealt with it very swiftly.  In the whole time we have been with them this is the only significant incident we have ever experienced, which is a testament to the excellent facility that it is, and the fantastic staff that work there, who we have met on many occasions and have always been highly impressed by. If any of our customers have any questions about this incident, please do not hesitate to contact us.

"As part of a routine firewall deployment, some new hardware was connected to our network and BGP sessions were configured on our route reflectors for these new devices.

Unfortunately, when these new sessions connected to our route reflectors, they caused an issue with the software we are using to power them – specifically the length of time taken to process and send the full BGP table to the new devices caused the “hold-timer” to expire on a few of the other sessions. This then resulted in these sessions disconnecting and reconnecting, and doing the same thing as the first sessions, causing further sessions to do the same, ultimately resulting in a loop where by the route reflectors were continuously dealing with sessions disconnecting and reconnecting.

This continued until such time as our Network Engineers identified the issue, and were able to log into the route reflectors and manually restart them to clear all the BGP sessions. The first of which was done at 12:50, the second at 12:58. All BGP sessions were re-established with all routes fully exchanged by 13:00 and traffic once again began to flow normally.

It has become apparent from this incident that we have reached a limit with the current Route Reflector software that we are using, and as such we are now in the process of replacing these with both improved hardware, and alternative software which has better handling of long-running tasks that doesn’t result in cpu-starvation of important tasks such as maintaining hold-timers.

Initially we will run these new route-reflectors alongside the old ones to prove their stability, before ultimately retiring the old devices, a further maintenance window will be scheduled for this which you will be notified about in due course."

Maintenance 26th February

We will be performing essential maintenance work on our VPS host servers between 5AM – 6AM GMT on Wednesday 26th February. During this window  your VPS will be unavailable for up to 20 mins while the host server is rebooted. Before rebooting we will be sending each VPS an ACPI Shutdown signal. If you have configured your VPS to respond to this, it will start it’s shutdown procedure, and shutdown cleanly. If you would like help configuring your VPS to respond to this, please raise a support ticket and we will be happy to help.

We’d like to apologies in advance for any inconvenience caused by this essential work.

Update 26th Feb 05:49AM – This work has been completed successfully and all VPS are back up and running. If you are experiencing any problems at all at this time with your VPS please raise a support ticket. Many thanks to all customers for their patience during this essential work.

We now accept Bitcoin

We’re delighted to announce that we now accept Bitcoin payments! 

We accept bitcoin

Yes, both new and existing customers can now pay for their VPS with Bitcoin, via BIPS, as well as our existing payment methods of PayPal and bank transfer.

In fact, we might as well go ahead and say it …. 

We love bitcoin !!

That’s right, this is not just an additional method of payment we’re adding, we strongly believe in the ideals that Bitcoin represents. We want to live in a world where everyone has the freedom to be able to pay another individual or organisation without having banks or governments ruling over the transaction, and helping themselves to a cut via hidden fees that are far in excess of what is fair or reasonable.

In the same spirit, we’d also like to announce good news for European customers, which is that they now have the option to pay us via bank transfer from their local bank in EUR, PLN, CHF, NOK, SEK, DKK, HUF, GEL, RON, TRY, CZK or BGN, for a flat fee of only £0.50, with no extra charges from their bank or ours. This has been made possible because of Transferwise, a brilliant company who are also liberating ordinary people everywhere from the stranglehold that big banks have over international money transfers.

Of course, customers in the UK can pay us already via bank transfer without charge, and all customers regardless of where they live in the world can pay us via credit or debit card using PayPal if they prefer, whether they have a PayPal account or not. But we are always looking to increase the number of options available to our customers.