On Dec 23 2015 there was an attack on the Prykarpattyaoblenergo power grid, in the Ivano-Frankivsk region of Western Ukraine. 30 substations serving some 230,000 customers were taken down for about 6 hours in the depths of winter’s cold just before Christmas. Attacks on critical infrastructure, like power grids, water supplies, and nuclear plants, aren’t just the plot of movies. They do happen in real life. Only the consequences aren’t scripted and the clean up isn’t contained to a movie lot.
In her recent piece for Wired, Kim Zetter writes an excellent analysis of how events unfolded which shows how this was no ordinary attack.And that’s the part where we need to sit up and listen. Somebody invested serious money, manpower and planning to make this happen. Which means they are prepared to do it again. Naturally, all speculation turned to Russia as the perpetrator. Putin, leveraging Russia in his endgame, has been brazen in his quest for control of the region. He annexed Crimea and has been relentless in his onslaught against Ukraine. Poland is very nervous about what will come next. And with Putin, nobody really knows what move he will make. But without absolute certainty, the finger of blame won’t be pointed in this case. Attribution, after all, can be a very dangerous game.
But this is more than just Russia vs Ukraine. This event is a harbinger of major change, a shift in how global conflict will be played out. This is being called the first cyberattack on critical infrastructure, all those things we take for granted in our daily lives: power, water, transportation, fuel, online access. Because today everything is plugged in and connected from the marble palaces of Manhattan to the dirt road villages of Nigeria. That is a lot of attack surface. So many targets. What happened here is the playing field didn’t just get levelled. It changed. Meaning we’ve got a whole new roster of potential players.
But wait – there’s more. A major concern is the deteriorating physical condition of many components that are part of ICS or Industrial Control Systems and of SCADA, the Supervisory Control and Data Acquisition system that oversees wide areas of critical infrastructure. Utilities are known to run things to the point of failure, and don’t change unless they fall under regulatory compliance. NERC CIP does a very good job of maintaining high standards over those utilities they govern over in the US. While the systems in the Ukraine were actually in good condition, the damage they suffered was irreparable in cases and is still not repaired. When these things break, fixes aren’t easy, and parts can be an issue. And it’s interesting to point out that the systems in Ukraine were actually in very good condition, at a state ours should be in. It’s been said before, but I’ll say it again. Attacks against the system are not a matter of “if” but “when”. We’re not ready for what’s coming next – or for the aftermath.
This analysis offers a valuable framework for attack in which there are important lessons we all need to learn. Key here is the matter of time, not just timing. The attackers took the time to set things up because they could find the holes within our systems to go looking for the intel they needed. Think on that for a moment. What else could attackers find? How long could they stay hidden for to mine our systems before we discover them? There’s a big lesson in here about data and security I can’t even begin to get into here but will bring up in future.
Attackers used several methods to bring the main station down. First, a spear phishing campaign helped identify targets within IT and Admin. You don’t go for the biggest target. You build a ladder up, making it harder to track you, and making you less obvious while accumulating more valuable credentials and intel for the attack. Phishing has escalated rapidly to the preferred initial attack method of choice because of its success in nabbing victims. Workers at electrical companies in the Ukraine clicked on an email containing a malicious attachment. They were prompted to enable macros in a pop-up box we have all seen, and activated ourselves. The takeaway here is that malicious macros are considered somewhat old-school as a threat, so the attackers pulled something out of their toolbox knowing it wouldn’t be watched for. Always go where they aren’t looking.
This launched a malicious program, BlackEnergy3, previous varieties of which have infected systems in the US and Europe. This was used to open a backdoor to the attackers. We’ve plenty about backdoors lately, and none of it good. This got the attackers into the corporate networks.
The next challenge was to jump the gap from a segragated network onto the SCADA system. We teach people to build segragated networks. You gotta keep’em separated! But even that wasn’t enough of a deterrent. The attackers used their time wisely. They didn’t go over it, or under it. They went through it using credentials and intel they mined while roaming the other networks for months. How? VPNs or Virtual Private Networks that are supposed to shield traffic being sent online from all the other traffic online, sending it through a virtual tunnel. They had the sign-ons from workers who remotely accessed the SCADA system via secure VPNs. This is an achilles heel for every company because workers need or want to work remotely, but even VPNs are no silver security bullet. According to Chris Sistrunk , an ICS/SCADA security specialist with Mandiant, a FireEye company, “Power companies need to reduce the number of people who can use the remote access, and limit the amount of time a person can use the remote access. Companies also need to use two-factor identification, instead of just a user ID and password.” Nobody expected this wrinkle. Again, go where they aren’t looking.
Once inside the SCADA network, the attackers set up the backup power supply or UPS that would leave the operators in the dark as well as the customers. Robert M. Lee, a former cyber warfare operations officer for the US Air Force and is co-founder of Dragos Security, a critical infrastructure security company, described this move as “a giant F* you” to the power companies.
Like many places, the Ukraine power system was made up of different distribution management systems. That in itself could make things harder for attackers but they used the advantage of time and planning. They studied those disparate systems to know how to write the malicious firmware that would be used to overwrite the existing firmware on serial to ethernet converters. I mention these because they were key to this part of the attack. With these converters disabled, the operators had no means to send commands remotely that would close up the breakers once the blackout happened. Per Lee, “Operation-specific malicious firmware updates (in an industrial control setting) has never been done before. From an attack perspective it was just so awesome.” What’s not so awesome is that the same model of converters is also used in the US grid for power distribution.
Just before attacker launched their plan on Dec 23, they deployed one more clever tactic. They disabled the customer call centers using a phone denial of service attack so that customers could not call in and report outages, buying more time for the attackers. This was important because the call centers were how utilities kept track of outages on distribution lines. While not every utility has smart meters, every customer has a phone number and address which are assigned to the sections of line in the model of the system. If five customers call in the utility knows where the outage specifically is. But if the phone system is down then they can’t see where all of the power is out. Lee cites again the high level of sophistication and planning in this operation.”To me what makes sophistication is logistics and planning and operations and … what’s going on during the length of it. And this was highly sophisticated.” You can bet they weren’t looking at phone systems being brought down. Go where they aren’t looking.
Another hallmark of this attack was the amount of damage. Firmware, the special instructions on a device that tell it how to operate, was overwritten on the converters and the only fix was to physically replace the converters. Then, the attackers used KillDisk, malware that wiped the files from the operator stations, and then made the stations unrecoverable. It overwrites a key component, the master boot record, so the system cannot reboot.
What should we be doing to prepare our systems for what is coming? Chris Sistrunk has some solid recommendation based on his experience in the field. “They should work on monitoring their networks, compiling or “ logging” the data, and looking for signs that they have been infiltrated.” Within the security community, we cannot emphasize enough the importance of keeping logs to track system events so that these can be monitored real time, but also as an invaluable record should something go wrong.
Robust log collection and network traffic monitoring are the foundational components of a defensible ICS [industrial control systems] network. Failure to perform these essential security functions prevents timely detection, pre-emptive response, and accurate incident investigation.
It’s important to note here that the Ukrainians had done an excellent job of keeping those logs, possibly better than we do and certainly something we should emulate. He also recommends that “Distribution companies should also review their industrial control system computer architecture regularly, and review and test their plans for responding to a cyber attack.”
While the clear choice for culprit here would appear to be Putin and Russia, no formal allegations were made. But there was no mistaking the targeted and aggressive nature of this attack. The conflict between Russia and Ukraine is just one of several currently playing out in the global theatre. And what this attack heralds is a new strategy in “conflict resolution.” While we may perceive our biggest threat to be the large established powers who bend the rules, let me reiterate the need to look beyond the expected, to those emerging forces who recognize no rules but their own.
In the end, it isn’t so much about who did it as what was done and fixing the problem. Ukraine asked the US to lend its experience and resources to help fix the damage and analyze what happened. Two months down the line, systems are still offline and unrepaired because of the extent of the attack. There is a big lesson here for us. While NERC CIP ensures those utilities it watches over maintain stringent standards that will help safeguard them against these kinds of tactics, there are other private transmission companies and smaller distribution utilities that don’t fall under this umbrella, and who don’t employ effective security measures like 2 factor authentication for sign on. Clearly we’ve got work to do. This attack reveals key areas of exposure to those major systems fundamental to our way of life. Will we heed the warning?
References:
http://www.wired.com/2016/03/inside-cunning-unprecedented-hack-ukraines-power-grid/
http://www.wired.com/2016/01/everything-we-know-about-ukraines-power-plant-hack/
http://www.archerenergysolutions.com/keys-to-ukraine-power-hack-will-unlock-same-doors-in-u-s/
Issues and Intersections 