Outlier Detection

Our Outlier Detection system detects unusual suspicious activities of the monitored entities (e.g., users, servers). Such suspicious activities can reveal otherwise unnoticed security attacks going on in the background. A typical example of such attacks is the insider attack, in which a compromised privileged account is used to steal sensitive information from an organization. Such an attack can originate from the account owner, who may be a disgruntled employee of the organization, or by malicious hijacking of the account, for example by password theft. In any case, insider attacks are more immune to traditional security mechanisms like firewalls, and are much better handled using analytic methods, like outlier detection.

The Outlier Detection system, based on advanced statistical and machine learning techniques, automatically builds behavioral models that describe what constitutes "normal" behavior of the modeled entities. These models are based on historical data and are then compared to ongoing observed activities. The Outlier Detection system doesn't rely on attack signatures and is able to detect zero-day attacks. The system creates a holistic view of the typical behavior of the monitored entities by clustering similar entities together and taking multiple aspects of observed activities into account. This in turn improves precision and accuracy of the detection of suspicious activities. In addition, the system can be easily customized for new types of monitored entities and their related aspects.

The Outlier Detection system is successfully being used in industry by IBM customers for the protection of large-scale database systems as part of IBM’s Guardium platform, where database access logs are analyzed for unusual behavioral patterns of users or servers. The Outlier Detection system can be easily adapted to various domains thanks to its built-in customization mechanism for defining the sources of information, the modeled entities, and anomaly detection methods. Recently the Outlier Detection technology was implemented on Bluemix as a cloud-based application. This application is designed to support higher event traffic volumes and thus requires more scalable Big Data solutions. Scalability is achieved using Spark cluster computing technologies. This solution was already applied for outlier detection in the IoT domain.


Allon Adir, IBM Research - Haifa