Machine Learning: Changing the Cyber Security Paradigm
Today, the financial sector is under continuous attack – our cyber defenses are being tested, probed and targeted at an unprecedented rate by a variety of attackers with different motives, skill levels and techniques.
The good news is that all of the fancy hardware and software we have thrown at the problem generates tons of data to help us prevent, detect and defend against attackers. The bad news is that all of the fancy hardware and software we have thrown at the problem generates tons of data.
Information security professionals are drowning in data. Our firewalls, intrusion detection systems, anti-malware defenses, use logins/logouts, web site logs and system messages from thousands of workstations, servers and network devices (to mention a few data sources) add up to gigabytes, or for the largest organizations, terabytes of data each day. In most cases, the warning signs of bad things happening (or soon to happen) on our networks are buried within this massive set of disparate data – but so are millions of records of perfectly harmless activity.
To make matters worse, sophisticated attackers may use multiple techniques and touch multiple systems or devices to mount their cyber-raids. A single attack may involve:
1. Getting a user to click on a phishing link on their workstation, which
2. Directs them to an “exploit kit” page, which
3. Probes their system for vulnerabilities, which
4. Exploits a detected vulnerability to gain access to their account, which
5. Allows them to install intrusion tools on their workstation, which
6. Allows the attacker access to the systems that the user is authorized to access, which
7. Gives them access to sensitive data which they exfiltrate from the network using another compromised system, which
8. Leads to front page stories in the Wall Street Journal and a CIO and/or CSO “deciding to spend more time with their family”.
Today’s malware is polymorphic – it changes itself each time it runs so that signature-based detection will not notice it.
While the human brain is basically a pattern-matching machine, our capabilities in this area just don’t scale to terabytes of data. Even if we throw more analysts at the problem, the sheer amount of data, the number of formats that the data is in and the number of subtle correlations which need to be made conspire to make manual log review a losing proposition.
Computers are great at ingesting a lot of information and searching it very quickly. Previously, security professionals have relied on systems which look for signatures of known attacks in our logs. Back in the day, when attackers (and our networks) were much less sophisticated, this worked, but only to a point. We knew what we were looking for. Thanks to new technologies that empower our businesses and some pretty excellent R&D by the attacker community, today’s attacks are likely to be just different enough from past attacks to slip by signature-based defenses.
For example, when malware first became a problem, we threw simple pattern matching at it. Anti-virus scanners looked for certain patterns of code or data in executable files and prevented programs with these “bad” patterns from running. Today’s malware is polymorphic – it changes itself each time it runs so that signature-based detection will not notice it.
Enter machine learning.
We are at the beginning of a paradigm shift in defending our networks. Rather than playing the losing game of trying to detect attackers based on past activity, a new generation of cyber defense tools look for activity that is either clearly hostile (for example, a PDF file, which tries to make a network connection to a server in Narniawhen opened) or anomalous, meaning that its activity isn’t normal.
By focusing on detecting potentially hostile or out-of-the-ordinary activities, we avoid the pitfalls of signature-based detection. It doesn’t matter which particular vulnerability the attacker used to gain access to a system. It doesn’t matter if the malware they sent to your CEO encrypts itself to avoid detection – it detects the potentially hostile intent of the code that sets off the alarms.
Using machinelearning tools does not remove the need for cyber security professionals; instead, it makes them much more efficient. Today, a typical security analyst comes in to find hundreds or thousands of “events” in their analysis queue, most of which are false positives. The goal of machine learning is to reduce that daily queue to tens of events which are much more likely to help the analyst detect configuration problems with a security dimension, early signs of attacker reconnaissance or intrusion attempts/successes.
Culling the queue of false positive noise makes the analyst more cost effective and valuable to the organization. It also helps encourage retention of talented (and increasingly scarce) personnel. Asking a highly-trained security analyst to spend his or her days looking at long lists of false positive events is not likely to result in an engaged, loyal employee with a long term future in your organization.
Again, it is early days for machine learning in cyber security, but there is a real promise that this new approach may help us defenders keeps up with the bad guys. Keep this topic on your radar.