Misc Tech Notes

Lör 22 Oktober 2016

Cloud fails, DDoS and a dirty cow

Posted by Peter Reuterås in article   

Yesterday I started to receive notifications from Docker Hub about failed builds. Automatic builds was triggered because of my dependencies on the Debian Docker image. The new version of Debian hopefully patched CVE-2016-5195 named Dirty COW because of the dirty bit in handling of copy-on-write. This is a bug in the Linux kernel.

First the problem was related to the DDoS against Dyn. Dyn has some information in their incident report. According to The Register this attack was in part done by hacked IoT devices like smart TVs. The result was that Docker Hub couldn't resolve the hostname for github.com and my builds failed.

When I checked this morning the DDoS against Dyn had stopped and I triggered new builds on Docker Hub for all my containers. Unfortunately this resulted in a new error in the builds for containers that have worked for a long time. The error message can be seen in this failed build report. The error messages is:

Build failed: stat /var/lib/docker/overlay/ddcaf86d8470db2c5db746bff82c79d155f2b37c0d85f72c2c82538da18f101a/merged/entrypoint.sh: no such file or directory

I did a quick search with DuckDuckGo and found this issue on Github for feedback on Docker Hub. It was opened 13 days ago as I write this. It points to and issue in Docker opened as early as 2016-08-22 and closed less then a day ago. It has a target of a new 1.12.3 release of Docker. Hopefully this will resolve the problem and my builds will succeed without needing to try any workarounds. My containers on Docker Hub aren't used in a way that make CVE-2016-5195 easy to exploit so at the moment I wait on the patch.

The thing to take away from this incident is that it is important to have alternatives and continuity plans. Otherwise you might fail like Github, Twitter and some Swedish sites that should have been built to handle crisis like MSB and Krisinformation.se. How would you handle this issue if it affected you at work? Could you change your build process and delivery pipeline if your providers fails? What if your cloud provider fails?

Update (2016-10-25): According to an entry for the issue at Github for Docker Hub a fix has been released and I can confirm that all my builds work now.