At the time of writing (30th of January 2018) it is probably safe to say that this month has been the most significant that we’ve ever seen in the history of computer security. The revelation, on the 3rd January, of the Meltdown and Spectre security vulnerabilities present in virtually all modern computer processors (CPUs) has sent shock waves throughout the computing world. Enough has been written online and offline about these vulnerabilities, so no need to write anything here about what they are. Only to say these vulnerabilities were of particular concern to all VPS providers, due to the fact that a malicious VPS could use these flaws to read private or sensitive data in memory belonging to other customers’ VPS’ running on the same physical server.
So how did we at Manchester VPS react to these revelations? Here is a timeline:
- 4th Jan: this was the day the world woke up to the news. We reacted immediately and on the same day emailed all our customers warning that initial emergency maintenance would be taking place within the next 72 hours to apply software security updates to our systems.
- 5th Jan: the Linux kernel developers release software updates providing protection against Meltdown only.
- 6th Jan: we applied the Linux kernel updates released on the 5th to all of our physical servers hosting customer VPS instances, followed by full reboots. This made our VPS platform safe against the severe and immediate threat presented by Meltdown. This ensured that a malicious VPS would be unable to see private data inside another customers’ VPS.
- 20th Jan: in the preceding 2 week period, the Linux kernel developer community had been feverishly working on solutions to the less immediate, but still severe, threat posed by the collection of security issues known as Spectre. One of these solutions, known as “retpoline“, soon emerged as the Linux community’s preferred software mitigation technique against variant 2 of Spectre (Spectre has two variants known simply as variants 1 and 2). On the 20th of January we at Manchester VPS updated the Linux kernel on all of our physical servers to version 4.14.14, providing full retpoline protection. Additionally, all software we run on these servers was recompiled with GCC’s new retpoline support, providing full retpoline protection throughout the whole software stack on our VPS platform as of this date. (For the nerds amongst you, /sys/devices/system/cpu/vulnerabilities/spectre_v2 on our servers reports the prized “Mitigation: Full generic retpoline” 🙂 )
- The present (30th Jan): At this time our VPS platform is better secured than the majority of VPS providers out there, due to the speed of our response and the skill of our team in being able to put in place all of the available mitigations, some of which have been very difficult to successfully implement. However, right now there is no suitable protection available to anyone in the computing industry against the one remaining Spectre vulnerability known as Variant 1 (the steps we took above protect us against variant 2 of Spectre and also against Meltdown). Protection for Variant 1 is expected to be made ready by chip manufacturers very soon, though, and you can be sure we will implement it as soon as it is available.
There are additional improvements to the security of our platform which we have taken the opportunity to implement during the month of January. Here are some of the new security hardening measures we have enabled in the Linux kernel we use on our physical host servers this month: SELinux for more fine grained control of privileges, use of the GCC structleak and randstruct plugins and -fstack-protector-strong switch, KASLR, disabling of User Namespaces, and several other OS hardening features. All of these are only possible because our host server OS is heavily customised by us and thus we have full control over the build process of all software including the kernel. These new measures are in addition to existing security features we’ve always implemented, such as ensuring each customer VPS qemu process runs as an unprivileged system user.
The next time you are considering a VPS provider, ask them what measures they take to secure the environment that their customer’s VPS run in. Ask them what they have done about Spectre and Meltdown. Most of the measures we take are not enabled “by default” in the software that we run. Increasing security takes extra effort and work, and requires increased computing resources, and we know that the majority of VPS providers out there do not make the extra effort.
We’d like to thank our customers for their patience on the 2 days this month when the maintenance work for the above took place.