Author Archives: Eddie

Securing our systems against Meltdown and Spectre

At the time of writing (30th of January 2018) it is probably safe to say that this month has been the most significant that we’ve ever seen in the history of computer security.  The revelation, on the 3rd January, of the Meltdown and Spectre security vulnerabilities present in virtually all modern computer processors (CPUs) has sent shock waves throughout the computing world.  Enough has been written online and offline about these vulnerabilities, so no need to write anything here about what they are.  Only to say these vulnerabilities were of particular concern to all VPS providers, due to the fact that a malicious VPS could use these flaws to read private or sensitive data in memory belonging to other customers’ VPS’ running on the same physical server.

So how did we at Manchester VPS react to these revelations?  Here is a timeline:

  • 4th Jan: this was the day the world woke up to the news.  We reacted immediately and on the same day emailed all our customers warning that initial emergency maintenance would be taking place within the next 72 hours to apply software security updates to our systems.
  • 5th Jan: the Linux kernel developers release software updates providing protection against Meltdown only.
  • 6th Jan: we applied the Linux kernel updates released on the 5th to all of our physical servers hosting customer VPS instances, followed by full reboots.  This made our VPS platform safe against the severe and immediate threat presented by Meltdown. This ensured that a malicious VPS would be unable to see private data inside another customers’ VPS.
  • 20th Jan: in the preceding 2 week period, the Linux kernel developer community had been feverishly working on solutions to the less immediate, but still severe, threat posed by the collection of security issues known as Spectre. One of these solutions, known as “retpoline“, soon emerged as the Linux community’s preferred software mitigation technique against variant 2 of Spectre (Spectre has two variants known simply as variants 1 and 2). On the 20th of January we at Manchester VPS updated the Linux kernel on all of our physical servers to version 4.14.14, providing full retpoline protection. Additionally, all software we run on these servers was recompiled with GCC’s new retpoline support, providing full retpoline protection throughout the whole software stack on our VPS platform as of this date. (For the nerds amongst you, /sys/devices/system/cpu/vulnerabilities/spectre_v2 on our servers reports the prized “Mitigation: Full generic retpoline” 🙂 )
  • The present (30th Jan):  At this time our VPS platform is better secured than the majority of VPS providers out there, due to the speed of our response and the skill of our team in being able to put in place all of the available mitigations, some of which have been very difficult to successfully implement.  However, right now there is no suitable protection available to anyone in the computing industry against the one remaining Spectre vulnerability known as Variant 1 (the steps we took above protect us against variant 2 of Spectre and also against Meltdown).  Protection for Variant 1 is expected to be made ready by chip manufacturers very soon, though, and you can be sure we will implement it as soon as it is available.

There are additional improvements to the security of our platform which we have taken the opportunity to implement during the month of January. Here are some of the new security hardening measures we have enabled in the Linux kernel we use on our physical host servers this month: SELinux for more fine grained control of privileges, use of the GCC structleak and randstruct plugins and -fstack-protector-strong switch, KASLR, disabling of User Namespaces, and several other OS hardening features. All of these are only possible because our host server OS is heavily customised by us and thus we have full control over the build process of all software including the kernel.  These new measures are in addition to existing security features we’ve always implemented, such as ensuring each customer VPS qemu process runs as an unprivileged system user.

The next time you are considering a VPS provider, ask them what measures they take to secure the environment that their customer’s VPS run in.  Ask them what they have done about Spectre and Meltdown. Most of the measures we take are not enabled “by default” in the software that we run.  Increasing security takes extra effort and work, and requires increased computing resources, and we know that the majority of VPS providers out there do not make the extra effort.

We’d like to thank our customers for their patience on the 2 days this month when the maintenance work for the above took place.

Loss of network at our Data Centre

10:50AM Tues 27th Sept – the Data Centre where our equipment is located are currently experienceing a total loss of network across a big part of their facility. If you VPS is currently unreachable, please stand by, we are monitoring updates form the DC, who are working to restore service, and will post an update here as soon as we have an update from them.

Update 11:06AM – the issue has now been resolved and you should be able to access your VPS normally. Please accept our apologies for any inconvenience caused.

Loss of network at our Data Centre

12:30PM Tues 5th April – the Data Centre where our equipment is located are currently experienceing a total loss of network across a big part of their facility. If you VPS is currently unreachable, please stand by, we are monitoring updates form the DC, who are working to restore service, and will post an update here as soon as we have an update from them.

Update 12:35PM – we’ve just seen that network service has returned to all of our VPS. You should find your VPS is reachable again now, if it is not, please raise a support ticket so we can look into it.  But please note the DC has yet to confirm that the outage is over, we will post another update once this is confirmed.

Update 12:44PM – unfortunately it appears their problems are not over, as network connectivity has just been lost again at the DC.  Customers will be seeing their VPS’ are unreachable again at this time.

Update 12:53PM – network service has again returned to all of our VPS, you should find your VPS is reachable again now. But please note the DC has yet to confirm that the outage is over, we will post another update once this is confirmed.

Update 13:08PM – We have not seen any further interruptions to service since the last update, and the DC have confirmed that service should now be normal. You should find your VPS is reachable, if it is not, please raise a support ticket so we can look into it.  We’d like to apologise to our customers for the inconvenience caused by this outage. It is likely the DC will communicate to us a reason for this outage once they have completed their investigations. If and when they do, we will communicate that here.

14:20PM – The DC have provided the following update: “We feel we have resolved the immediate issues with the network here we are still considering this investigation as ongoing.  At this time we know an issue with a piece of power infrastructure appears to have led to a routing issue on our network. In turn, this caused network service to be lost across our two data centres.  A root cause analysis will be made available within 7 days.”

We at Manchester VPS will pass on here any further information we get.  Having been with the DC for a long time now we are very confident in their infrastructure, expertise, and excellent service they always provide to us. It is extremely rare for them to suffer a major outage, so we would like to reassure our customers that you are in good hands.

Centos & Ubuntu new releases and ISO images updated

We’re pleased to announce that we’ve updated our Centos and Ubuntu OS ISO collection today, following new releases from both these extremely popular Linux projects.

The Centos 6.6 collection has been updated to the newly released Centos 6.7. More information can be found in the release announcement from 3 days ago.

The Centos 7.0-1406 collection has been updated to the latest rolling ISO media release for May 2015 (1505).  To quote from the Centos project:

“The rolling builds are a point in time snapshot of a given CentOS version including all updates on mirror.centos.org.  This includes all all security, bugfix, enhancement and general updates for CentOS Linux, in this case they include updates up to and including May 28th, 2015.”

We have also expanded our Centos 7 collection to include the Minimal and Net install ISOs. More info about this latest release can be found in the Centos 7 Release 1505 announcement.

The Ubuntu 14.04.2 LTS (Long-Term Support) collection has been updated to the newly released version 14.04.3. More info can be found in the release announcement from 4 days ago.

If you already have an older version of any of these OS installed, you can easily update them, and we urge you to do so as soon as possible.

For Centos, just run the command yum update

For Ubuntu, see Ubuntu’s guide to Trusty updates.

Additionally, as announced recently here, Ubuntu 14.10 (Utopic Unicorn) reached End of Life on July 23, 2015. As such we have deleted it from our ISO collection. If you are still running this version of Ubuntu you should upgrade to 15.04 (Vivid Vervet) as soon as possible so you can continue to release security updates. You can find guidance about doing that here.

If you need help upgrading your OS, we can often perform updates for you at very reasonable hourly rates, just open a support ticket to discuss further.

As always, if there is a particular OS you’d like us to update if you spot there is a newer version available, or if there’s an OS you’d like us to add to the collection that we don’t already have, just let us know, we’ll almost certainly say yes. The full list of OS in the collection can be found here.

What we’ve done to protect against VENOM and Logjam

In the last 7 days the internet community has seen two quite significant (but unrelated) security vulnerabilities made public; affectionately named “VENOM”  (CVE-2015-3456) and “Logjam”.

We’ve taken steps to protect our customers from both.

“VENOM”  (CVE-2015-3456) is a vulnerability affecting the vast majority of Cloud and VPS providers such as ourselves. It is a flaw in the virtualisation software used by most providers, called “qemu” which could potentially allow an administrative user on one VPS to access the memory area of other VPS running on the same physical host. Due to the critical nature of this vulnerability, we applied the fix made available for qemu across all of our physical VPS hosts and, very soon after the vulnerability was made public, all customer VPS were either restarted with a new patched qemu instance or live migrated to a new, patched qemu instance, and thus no longer vulnerable.

“Logjam” is a set of weaknesses identified by security researchers in Diffie-Hellman for TLS, a cryptographic algorithm fundamental to many internet protocols including HTTPS. This research was made public today. The weaknesses allows attackers to downgrade vulnerable TLS connections (including HTTPS) to use 512-bit export-grade cryptography. This allows the attacker to read and modify any data passed over the connection.  We immediately assessed our web servers which rely on HTTPS to encrypt the Client Area and VPS Control Panel, and found that they were not vulnerable to the worst of the weaknesses as we already configured them using a strong set of cipher suites when they were originally deployed, and have hardened the configurations further over the last couple of years as new vulnerabilities have surfaced (such as Heartbleed and Poodle).  As a result we had already disabled export cipher suites, which are vulnerable to Logjam attacks, a long time ago.  However, we found our web servers were vulnerable to one of the weaknesses the Logjam researchers found, which is that most web servers use the same prime numbers for Diffie-Hellman key exchange, making them potentially vulnerable to eavesdropping by a nation state. As a result, we followed their advice and generated a strong, unique Diffie Hellman Group on all our web servers and restarted them, closing the vulnerability.  So our customers can be reassured that their communications with us are extremely secure, as we have always taken the security of our HTTPS protected website areas extremely seriously.

More information about VENOM can be found here, and for information about Logjam see here.

Maintenance Sunday 2nd Nov, 3:00 AM

We will be performing essential maintenance work on our VPS host servers between 3AM – 5AM GMT on Sunday 2nd November. During this window  your VPS will be unavailable for up to 20 mins while the host server is rebooted.

Before rebooting we will be sending each VPS an ACPI Shutdown signal. If you have configured your VPS to respond to this, it will start it’s shutdown procedure, and shutdown cleanly. If you would like help configuring your VPS to respond to this, please raise a support ticket and we will be happy to help.

We will proactively checking that your VPS boots up successfully after reboot. You can help us in this by ensuring that your VPS IP address responds to ICMP ping. If it currently doesn’t, and you would like any assistance, please  raise a support ticket and we will be happy to help.

Updates will be posted here on this blog post during the work.

We’d like to apologies in advance for any inconvenience caused by this essential work.

Update Sun 2nd Nov, 04:55AM – Unfortunately we are running behind schedule, some VPS will have their downtime between 5AM-6AM. Apologies to those affected for the inconvenience this may cause.

Update Sun 2nd Nov, 06:35AM – We have decided to finish the work at 06:20AM and restore normal service, although we still have more VPS hosts to do that we were unable to finish. All VPS which we had to stop are back up again, there should not be any customer without service, so if you find you don’t have service at this point please raise a support ticket. We will re-schedule another maintenance window for the remaining VPS hosts.

Update Sun 2nd Nov, 14:55PM – We have rescheduled the remaining work for tonight, 2AM – 3AM GMT and have informed affected customers by email. This remaining work only affects some customers, who we have emailed today.

Update Mon 3rd Nov, 01:57AM -The remaining work is now complete and any VPS affected are now fully operational again. If you find you don’t have service at this point please raise a support ticket. We’d like to thank all customers for their patience while we completed this essential work.

Some ISO image updates

We’ve updated some ISOs today, the following are the new versions that have replaced the old ones:

 

The Centos 5.10 collection has been updated with the recently released Centos 5.11 equivalents.

Clonezilla Live 2.2.4-12 (amd64)

Clonezilla Live 2.2.4-12 (i686-pae)

KNOPPIX DVD V7.4.2 2014-09-28 EN

System Rescue CD 4.3.1

 

If there is a particular OS you’d like us to update if you spot there is a newer version available, or if there’s an OS you’d like us to add to the collection that we don’t already have, just let us know, we’ll almost certainly say yes. The full list of OS in the collection can be found here.

Network outage Sept 28th

Sun Sept 28th 2014 – 16:50: Unfortunately the data centre we are in has just suffered a brief loss of network connectivity lasting about 5-10 mins. It was very brief so may not have been noticed but our monitoring detected it. We’d like to apologise to any customers affected. At the moment we don’t have any information from the DC as to what the cause was, but we are very pleased to see that they were able to resolve whatever it was so quickly. If we do get any information from them we’ll pass it on here on this post.

Lots of OS ISO images updated

We’re pleased to let everyone know we’ve been going through updating our OS ISO image collection the last couple of days to refresh some of them where newer versions are available. These are the ones we’ve updated:

Ubuntu 12.04.5 Precise Pangolin LTS Server (amd64 & i386)
Ubuntu 14.04.1 Trusty Tahr LTS Server (amd64 & i386)
Debian 6.0.10 Net Install CD (amd64 & i386)
Debian 7.6 DVD Set (amd64 & i386)
System Rescue CD 4.3.0
KNOPPIX V7.4

We’ve also added Desktop versions of Ubuntu 12 and 14, having previously only carried the Server versions, which is a great choice for those wanting to run a Desktop VPS, especially if you are unfamiliar with Linux:

Ubuntu 12.04.5 Precise Pangolin LTS Desktop (amd64 & i386)
Ubuntu 14.04.1 Trusty Tahr LTS Desktop (amd64 & i386)

We’ll be doing some more updating in the very near future, if there is a particular OS you’d like us to update sooner rather than later, or if there’s an OS you’d like us to add to the collection that we don’t already have, just let us know, we’ll almost certainly say yes. The full list of OS in the collection can be found here.

Network Outage

Sat 23rd Aug 2014 11:16AM – Some VPS customers will currently be experiencing a lack of network connectivity to their VPS. We are investigating urgently and will update here as soon as we have further news.

Update 11:28AM – We have identified the cause of the issue and have raised a request with on site technicians at the Data Centre to help resolve the problem.

Update 11:34AM – We’d like to apologise to any customers affected by this, we expect to have this resolved very shortly.

Update 11:36AM – This issue has now been resolved and network connectivity has now resumed. Once again, we apologise for the inconvenience caused, a post mortem will follow shortly.