A Postmortem on a Hacked MetroAir: Part Two

In the last part, we talked a little bit about the events of a recent hack and how we took actions to mitigate it and restore our content. This article is going to go more into the positives of the incident; specifically we’ll delve into what we learned and how we changed our actions.

Backups

LinodeBackup

Since the incident regular backups of our data has resumed. We initialized the backup services provided by our host that will backup data in 4 slots; We get a backup provided automatically every day and the last two weekly backups. We also get a manual backup slot to take a snapshot of the server at any given moment. While there is a cost associated with this; it is only $5 USD a month. The peace of mind the service provides in the event another catastrophic incident is priceless; Knowing that there is something sitting around that some recent copies of not only our data, but all of our configurations, applications, and scripts that are used everyday to provide teamspeak, up to date FAA charts, VATSIM data, US Airport delays, ACARS, and more are safe and secured is worth a lot more than $5.

Online-Secure-Data-Backup

In addition to the host initiated backups, we enabled other backup protocols. Twice a day snapshots of all of our databases are being output. We found some ways to improve how the dumps are taking place to minimize the impact on our normal operations. We are also getting backups of all of our web files on a daily basis. Each day we ship those backups almost 1,500 miles from MetroAir’s datacenter to a server that Lindle is running in Dallas, TX.

generate_graph

I’m happy to report that those backup have been running flawlessly for almost 2 weeks now.  While both servers have seen a dramatic increase in bandwidth being used, they are still well under our hosts caps thanks to a recent upgrade earlier this year. This is just another example of keeping peace of mind just in case.

Software Updates

**The forum software that was running pre-hack was an older version. While in the past we had kept up with security updates, we recently stopped patching the forums when an update that was done stopped playing nicely with our forum theme. In hindsight, that was silly;  like most things in life, appearance is usually not as important as what is underneath the surface. The hack was a great opportunity to upgrade to the newest version and put the issues of updating the look of the forums out of mind since the priority was restoring access. While we do have plans to update the theme associated with the forums, we aren’t rushing to get that project completed, especially since both of our graphics experts, Rob & Alex, are either on vacation or busy working on other important projects that have a higher priority.

Password & General Security Practices

**The importance of general security practices came rumbling into our views. An informal poll of our execs who have unlimited access to our forums which can cause debilitating effects in the wrongs hands showed no one had updated their passwords in more than 2 years; for some of us it was even longer since they were hired on as staff members. We all hear that we should update our passwords often but rarely do unless forced to. Because of the hacking incident, all of the staff members changed their passwords before going back online. We are also looking into policies of requiring executives to reset their passwords for all MetroAir systems on a recurring basis.

We also expired passwords for all pilots to ensure that the hackers weren’t able to impersonate any of our pilots. If you still haven’t restored your access to the forums or are having issues because your forum email account is no longer active, you should contact staff.

We also cleaned up a number of accounts on the forums that were never activated by pilots or which were dormant for a while. Removing unused accounts is a good security practice since for each account removed is one less method of unauthorized entry. That is also a good reminder for everyone that your forum account and pilot account should be using the same email address. If they differ your forum account may be purged by the system. This has always been policy but was not always enforced. That is going to be more important going forward. It is the pilot’s responsibility to maintain their contact information in both systems.

In summary, we’re feeling more confident about what we are doing now. While it was a painful few days of dealing with the attack, we were grateful for the not-so-subtle reminder that we were not impervious to an attack and to step up our security policies and practices.