Category Archives: Security

Securing our systems against Meltdown and Spectre

At the time of writing (30th of January 2018) it is probably safe to say that this month has been the most significant that we’ve ever seen in the history of computer security.  The revelation, on the 3rd January, of the Meltdown and Spectre security vulnerabilities present in virtually all modern computer processors (CPUs) has sent shock waves throughout the computing world.  Enough has been written online and offline about these vulnerabilities, so no need to write anything here about what they are.  Only to say these vulnerabilities were of particular concern to all VPS providers, due to the fact that a malicious VPS could use these flaws to read private or sensitive data in memory belonging to other customers’ VPS’ running on the same physical server.

So how did we at Manchester VPS react to these revelations?  Here is a timeline:

  • 4th Jan: this was the day the world woke up to the news.  We reacted immediately and on the same day emailed all our customers warning that initial emergency maintenance would be taking place within the next 72 hours to apply software security updates to our systems.
  • 5th Jan: the Linux kernel developers release software updates providing protection against Meltdown only.
  • 6th Jan: we applied the Linux kernel updates released on the 5th to all of our physical servers hosting customer VPS instances, followed by full reboots.  This made our VPS platform safe against the severe and immediate threat presented by Meltdown. This ensured that a malicious VPS would be unable to see private data inside another customers’ VPS.
  • 20th Jan: in the preceding 2 week period, the Linux kernel developer community had been feverishly working on solutions to the less immediate, but still severe, threat posed by the collection of security issues known as Spectre. One of these solutions, known as “retpoline“, soon emerged as the Linux community’s preferred software mitigation technique against variant 2 of Spectre (Spectre has two variants known simply as variants 1 and 2). On the 20th of January we at Manchester VPS updated the Linux kernel on all of our physical servers to version 4.14.14, providing full retpoline protection. Additionally, all software we run on these servers was recompiled with GCC’s new retpoline support, providing full retpoline protection throughout the whole software stack on our VPS platform as of this date. (For the nerds amongst you, /sys/devices/system/cpu/vulnerabilities/spectre_v2 on our servers reports the prized “Mitigation: Full generic retpoline” 🙂 )
  • The present (30th Jan):  At this time our VPS platform is better secured than the majority of VPS providers out there, due to the speed of our response and the skill of our team in being able to put in place all of the available mitigations, some of which have been very difficult to successfully implement.  However, right now there is no suitable protection available to anyone in the computing industry against the one remaining Spectre vulnerability known as Variant 1 (the steps we took above protect us against variant 2 of Spectre and also against Meltdown).  Protection for Variant 1 is expected to be made ready by chip manufacturers very soon, though, and you can be sure we will implement it as soon as it is available.

There are additional improvements to the security of our platform which we have taken the opportunity to implement during the month of January. Here are some of the new security hardening measures we have enabled in the Linux kernel we use on our physical host servers this month: SELinux for more fine grained control of privileges, use of the GCC structleak and randstruct plugins and -fstack-protector-strong switch, KASLR, disabling of User Namespaces, and several other OS hardening features. All of these are only possible because our host server OS is heavily customised by us and thus we have full control over the build process of all software including the kernel.  These new measures are in addition to existing security features we’ve always implemented, such as ensuring each customer VPS qemu process runs as an unprivileged system user.

The next time you are considering a VPS provider, ask them what measures they take to secure the environment that their customer’s VPS run in.  Ask them what they have done about Spectre and Meltdown. Most of the measures we take are not enabled “by default” in the software that we run.  Increasing security takes extra effort and work, and requires increased computing resources, and we know that the majority of VPS providers out there do not make the extra effort.

We’d like to thank our customers for their patience on the 2 days this month when the maintenance work for the above took place.

What we’ve done to protect against VENOM and Logjam

In the last 7 days the internet community has seen two quite significant (but unrelated) security vulnerabilities made public; affectionately named “VENOM”  (CVE-2015-3456) and “Logjam”.

We’ve taken steps to protect our customers from both.

“VENOM”  (CVE-2015-3456) is a vulnerability affecting the vast majority of Cloud and VPS providers such as ourselves. It is a flaw in the virtualisation software used by most providers, called “qemu” which could potentially allow an administrative user on one VPS to access the memory area of other VPS running on the same physical host. Due to the critical nature of this vulnerability, we applied the fix made available for qemu across all of our physical VPS hosts and, very soon after the vulnerability was made public, all customer VPS were either restarted with a new patched qemu instance or live migrated to a new, patched qemu instance, and thus no longer vulnerable.

“Logjam” is a set of weaknesses identified by security researchers in Diffie-Hellman for TLS, a cryptographic algorithm fundamental to many internet protocols including HTTPS. This research was made public today. The weaknesses allows attackers to downgrade vulnerable TLS connections (including HTTPS) to use 512-bit export-grade cryptography. This allows the attacker to read and modify any data passed over the connection.  We immediately assessed our web servers which rely on HTTPS to encrypt the Client Area and VPS Control Panel, and found that they were not vulnerable to the worst of the weaknesses as we already configured them using a strong set of cipher suites when they were originally deployed, and have hardened the configurations further over the last couple of years as new vulnerabilities have surfaced (such as Heartbleed and Poodle).  As a result we had already disabled export cipher suites, which are vulnerable to Logjam attacks, a long time ago.  However, we found our web servers were vulnerable to one of the weaknesses the Logjam researchers found, which is that most web servers use the same prime numbers for Diffie-Hellman key exchange, making them potentially vulnerable to eavesdropping by a nation state. As a result, we followed their advice and generated a strong, unique Diffie Hellman Group on all our web servers and restarted them, closing the vulnerability.  So our customers can be reassured that their communications with us are extremely secure, as we have always taken the security of our HTTPS protected website areas extremely seriously.

More information about VENOM can be found here, and for information about Logjam see here.