Millennial IBM scientists revolutionize cyber security operations from idea to product

Every year, CareerCast publishes a widely-reported list of the top 10 most stressful jobs. The 2017 list includes some obvious careers including military service, firefighter, and police officer, who risk their lives every day. The list also includes several less life-threatening jobs including TV news broadcaster and taxi driver.

But what if you had the task of protecting the entire IT infrastructure of a company, which includes protecting billions of dollars in assets, where one security breach could mean the collapse of the global economy? Or you are responsible for millions of electronic patient medical records needed for insurance reimbursements? In terms of stress, that makes driving a taxi in New York City look like a walk in the park.

These are the responsibilities of security analysts and experts in security operations centers (SOCs), who work 24/7 to keep the bad guys out of mission-critical IT environments.

A typical eight-hour shift for a SOC analyst can be broken down into three pieces:

• 1 hour – updates on the latest security news bulletins
• 3 hours – investigating potential security incidents via online sources: the average organization sees over 200,000 pieces of security events per day
• 4 hours – manually correlating data by copying and pasting information from disparate and silo’ed tools

These are rather mundane tasks where the biggest worry is to overlook an important indicator of a compromise in the latest security news bulletins or miss a clue in the multitude of security events and not to raise an alarm resulting in a disastrous breach.  On the other hand, flagging excessive events results in too many false alarms that need to be manually weeded out by analysts, on average.

“Today’s SOC analysts suffer from cognitive overload where they have to keep abreast of hundreds of threat and vulnerability intelligence reports daily and must be able to connect it to the thousands of security events that they see streaming by,” says J.R. Rao, IBM Fellow and Director of Security Research at IBM.

This unfortunate reality caught the attention of an international team of four IBM scientists who span from Nepal, South Korea and Switzerland and are based at the IBM Research lab in Yorktown Heights, NY.

Marc Ph. Stoecklin, who was recently promoted to Principal Research Staff Member, is a computer scientist and the manager of the Cognitive Cybersecurity Intelligence (CCSI) group at IBM Research said, “As part of our agile design processes, we spent several weeks interviewing and sitting side-by-side with analysts at the IBM Managed Security Services (MSS) SOCs around the world, including Costa Rica, Poland, and the U.S., to understand the pain points of their day-to-day jobs and look at how machine learning, cognitive computing and artificial intelligence could help them to focus on the real priorities — and thereby revolutionize the way they work.”

“Our eureka moment came after we met with the SOC team in Atlanta,” said IBM computer scientist Dhilung Kirat, who received his first training in security engineering at a university in Kathmandu, Nepal, and received his PhD from UC Santa Barbara. “The teams are collecting threat feeds, reports, malware and sandbox outputs and then they are asked to make sense of it all. We realized that there was a cognitive model here, which could drastically augment the abilities of the SOC analyst.”

The results of their research: QRadar Advisor with Watson and Watson for Cyber Security. Both tools were released as the first cognitive offering of IBM Security in February.  Using the input from analyst interviews, the researchers worked intensively, in a start-up like environment, to create sketches, models and an operational research prototype. Eventually they teamed up with product developers to build the products that were released commercially.

Like a doctor who has to read thousands of pages of medical records before diagnosing a patient, an SOC analyst has to absorb a massive corpus of security knowledge (e.g., tens of millions threat feed events and 60,000+ security blog posts published each month, 75,000+ documented software vulnerabilities and 10,000+ security research papers published each year) to determine if an event is  indeed a threat to the IT environment or a false alarm, and like a doctor, the amount of data and noise is enormous and its complete analysis is not humanly possible.

Enter Watson.

To assist the SOC analysts, the team has designed a comprehensive representation of cyber security knowledge that was amenable to analysis by machines.  Then, Watson for Cyber Security was trained by ingesting millions of security documents and threat feeds, both from structured and unstructured data sources, such as threat reports or blog posts.  The security knowledge is encoded in a security knowledge graph and is made accessible to SOC analysts by sophisticated reasoning algorithms, which connect related security facts and derive deep insights about the threat landscape; incidentally, they also produce beautiful images that are museum worthy. Today, the knowledge graph is constructed from over 10 billion data elements from IBM X-Force Exchange and threat feeds (with 4 million added every hour) and over 1.25 million security documents (with 15,000 added every day).

“The knowledge graph is a threat intelligence repository that continuously curates, understands, and consolidates threat information learned from multiple data sources, and our cognitive model is designed to comprehensively reason about a cyber security incident by synthesizing critical evidence,” said Jiyong Jang, an IBM scientist, CMU alumni, and expert on software and network security, and machine learning, who focuses on applying AI to security offenses.

Indeed, the team’s research yielded highly sophisticated algorithms to perform cognitive analytics over cyber security incidents that led to QRadar Advisor with Watson, a smart wingman (or extended brain) for SOC analysts. The algorithms they invented are trained to reason over cybersecurity incidents in an organization’s IT environment and combine them with the security knowledge graph to establish insights, which human analysts would be unable to deduce under normal time and information complexity constraints.

Looking back at the past year Youngja Park, a natural language processing (NLP) scientist on the team researching on information extraction from security articles and threat reports, added, “It was incredibly rewarding to see how we could build a completely new solution and to have our CEO announce it on stage in front of hundreds of clients. ”

One of those clients was SIX, the operator of the infrastructure underpinning the Swiss financial sector, which announced on 24 March that it plans to leverage IBM Watson for Cyber Security in a new cognitive SOC. SIX will now be able to offer advanced security services to its existing financial industry customers, utilizing the IBM Security capabilities as important building blocks for the offering.

For Marc, the story has come full circle.

“Being Swiss I can appreciate the significance of the financial industry on the Swiss economy. It’s incredibly humbling to know that a part of my code is keeping it safe and secure.”

Dhilung, who has been with IBM for less than two years, has a message for students who choose to work for a start-up compared to a global enterprise.

“We often get asked by students and applicants about having an impact at a large company and I’m proud to respond with this project as an example. Whether you are a small or large team or just recently hired, if you have an innovative idea, the best companies, like IBM, can help you scale it and bring it to market fast.”