A Postmortem on a Hacked MetroAir: Part One

By now most of you know that last weekend MetroAir endured a debilitating attack to the website that lasted a little under 48 hours. Much was learned during the assailment and has helped to reinforce some of our data policies. Join us as we walk through the events of the attack and how the staff mitigated it in part one. Part 2 will concentrate on what we learned and in part 3 we’ll offer some things you can do to improve your own online security.

After a busy Saturday enjoying the the 74°F (23°C) temperatures taking a walk on the beach at Back Bay Refuge and finishing the day off with some NC style barbecue with friends, I returned home at around 9pm to find Skype messages from Sara and James alerting me of an issue with the forums; This is when it all started to go downhill.

Upon first inspection it seemed that only our forum database had been attacked. We were left with about 10 forums messages but everything prior to that was wiped clean from the database. At the time, the forums were not even loading. After fixing a configuration file to get the forums working and with what was thought to be a limited scope at the time, I began focusing on looking for backups of the forum database. After about 4 hours of searching through every nook & cranny of my hard disks and on the MetroAir server, I was left with 2 possible backups, neither of which were promising. The last full backup of the database was in July of 2012. In November we had done another export of the forums in an investigative move to try looking at alternative forums. Unhappy with the results and with drooping eyelids, I decided to call it a night and try to find a better backup the next morning. What no one knew at the time was that the website was still under attack.

hacker

After a restful night of sleep and working out frustrations of the hack at the gym early Sunday morning, I started my work on a plan of restoration. We had different options but none were great. Do we restore the really old backup and lost 8 months worth of content or do we see if we can somehow export data from the testing forum back into the old version of the forums that we were running? Going back and forth and researching options it became apparent the attack was not limited to just the forums or the forum database. During the morning the main MetroAir website homepage was hacked leaving a a marker saying we were attacked by a Kurdish group. Restoring the main page seemed to fix things temporarily but I had no idea while I was taking steps to move forward that there was an unscrupulous individual working to undermine my attempts halfway around the world.

While the website was being attacked I had attempted to log in to the forums. I immediately noticed that I no longer had administrative access to the forums. After correcting that directly in the database and logging in I found that there was only 1 administrator account left. All of the other executives were “demoted”. I was able to reset the password on the account in hopes that the hacking would cease at that point. Unfortunately the hacker had already left a trail and secured backdoors to get back in to cause more havoc.

6a00d8341c54d153ef00e551fff0168833-800wiBefore talking about backups I should probably offer a history of how backups worked. For a long time MetroAir had fairly impressive backup routines, for a VA anyway. Around 5 years ago MetroAir would FTP backups to one of Lindle’s servers. At the time MetroAir and Lindle were both using Windows based servers and were in the same datacenter. We moved MetroAir’s server off of a full dedicated server shortly after that and it went to a new virtual private server. It was linux based with a better price point than paying the monthly fees of a full dedicated server. The servers were no longer in the same datacenter, bandwidth caps were a little tighter and Lindle was in the process of considering alternative servers as well. After moving MetroAir we ended up changing the backups so that it would all be stored on the Amazon Cloud. This worked really well. We were backing up not only regular files but snapshots of the databases every 12 hours. After about 8 months though the backups were starting to get out of hand and the monthly fees for storing backups surpassed the monthly cost of  running the server. We ended up switching those cloud backups off. Around the same time our host began offering its own backup service for a simple and flat $5 a month plan. We were all signed up and never needed the service. The backup service handled everything on the server, but for good measure we were still doing snapshots of the databases every 12 hours. As the MetroAir databases grew, taking the snapshots started to undermine performance of the website. You can probably imagine that 5,676,500+ records (1.1GB) of just ACARS positional data could cause some slowdowns when the database is being locked for backup and the website and ACARS is still trying to access it. We ended up cutting the db backups to be less frequent and eventually stopped them altogether when the performance suffered too much. Early this year we switched all of our services to a brand new server. We wanted to update everything to one of the newest versions of Linux. In doing so, everything had to be set back up again. One of those things was the automated backups with our host. Since it was a different server, it was not automatically backed up. Unfortunately, it was never re-enabled meaning we had no backups. Aghhh!

Early Sunday morning I had got a call from Lindle asking if I had known about the hack and offering his assistance. He also reminded me of the automated backup service from Linode saying that he was going to enable that on his own server in light of recent events. It was a good reminder to get things re-enabled and start the backups on the MetroAir server again.

During all of this, in hopes of restoring the forums an installation of the newest version of the forums was in progress. We had began importing some of the older posts into the newly created forums. William was able to help set up some of the back end of the forums by assigning permissions to certain boards and removing some of the older members who had never logged into the forums. Derrick worked on cleaning up some of the forum accounts as well while James helped to poke his head around trying to find things that were broken in the forums and website.

By 8pm on Sunday it looked like we one of the most devastating blows of the attack was in progress. This time the hacker was deleting files. The entire website was nonfunctional and some of the core files needed to provide services had been deleted from the system. The hacker replaced the website with links to adult websites and lewd images. It was the one time that I wished there was a “recycle bin” I could use to get things back quickly and painlessly. Luckily, that reminder from Lindle to re-enable the backup service couldn’t have been at the most perfect time. I was able to restore the entire server from the very first backup that it had done a mere 7 hours earlier. It was backed up to a completely new server and the IP addresses of the server were changed. We were hoping this would keep the attacker away.

Despite our hopes, we were finding despite the IP address change and restoring things back, the attacker had found us once again. The main website page was left with a simple message for us saying “You all suck.” Finally late Sunday night after combing through webserver access logs I was able to find the source of the continued attempts. The hacker was able to hide his backdoor by naming a new file on the system something that looked like it should be there. Would you have seen something wrong with twittter.php? While I would love to think that our pilots were checking out MetroAir’s twitter feed as much as the access logs were indicating, it was a little skeptical. I went to the page and was quickly devastated seeing just how much access the hacker had to the server’s filesystem. I changed the page around leaving a simple message for the hacker. At that point it seemed as if the backdoor was closed and we would be on the road to recovery.

With everything patched up, we thought we were good to go. Early the next morning while continuing with cleanup efforts we had scoured the system to see if there was any other telltale evidence the hacker left behind. While we were searching for him, he was using one of his last backdoors that he had placed on the server. Twittter wasn’t the only one he had left; it was found in some other innocuous spots on the server as well. This hacker was smart. He wanted to make sure he had access after we had repaired each hole. This time we were a step ahead and found each backdoor that was left on the system and worked to remove them all. For one and for all, order was restored and we were finally confident that things had been cleaned up and we were able to keep the hacker out. We also employed some firewall tactics on the hackers IP range; If you’re in Turkey anytime soon don’t expect to be visiting the MetroAir webpages.

So what exactly did the hacker have his dirty hands on? First he used the forum software itself to purge “old” posts. Unfortunately old for the hacker was everything that wasn’t in the last day. Technically he could have backed up the database and seen email addresses, this hacker seemed to be much more concerned with being malicious and leaving a trail then silently trying to access user data. He wiped files in the forums and main websites. He managed to clean out every single resource we have like gate charts, every download available that we offer, and our latest Monthly Updates. Even the images and scripts used to generate our forum signatures for pilots and the static signatures used for staff were missing. Almost no stone was left unturned but I’m confident that better days are ahead as we complete our restoration efforts of ensuring everything is back to normal. The last big challenge we have is “prettying” the forums with our own spin, though we are just happy with the simple knowledge that it’s back up and running and our pilots can post again without issue.

Part 2 where we discuss what we are doing differently will be out later this week and part 3 discussing your own online security will be out shortly after that.

If you have made it this far, thank you. I’d also like to take the opportunity to thank Sara, James, William, Derrick and Lindle for their concern and cleanup efforts taken to restore the MetroAir that we all know and love!