Announcing Security Monkey — AWS Security Configuration Monitoring and Analysis
We are pleased to announce the open source availability of Security Monkey, our solution for monitoring and analyzing the security of our Amazon Web Services configurations.
At Netflix, responsibility for delivering the streaming service is distributed and the environment is constantly changing. Code is deployed thousands of times a day, and cloud configuration parameters are modified just as frequently. To understand and manage the risk associated with this velocity, the security team needs to understand how things are changing and how these changes impact our security posture.
Netflix delivers its service primarily out of Amazon Web Services’ (AWS) public cloud, and while AWS provides excellent visibility of systems and configurations, it has limited capabilities in terms of change tracking and evaluation. To address these limitations, we created Security Monkey — the member of the Simian Army responsible for tracking and evaluating security-related changes and configurations in our AWS environments.
Overview of Security Monkey
We envisioned and built the first version of Security Monkey in 2011. At that time, we used a few different AWS accounts and delivered the service from a single AWS region. We now use several dozen AWS accounts and leverage multiple AWS regions to deliver the Netflix service. Over its lifetime, Security Monkey has ‘evolved’ (no pun intended) to meet our changing and growing requirements.
There are a number of security-relevant AWS components and configuration items — for example, security groups, S3 bucket policies, and IAM users. Changes or misconfigurations in any of these items could create an unnecessary and dangerous security risk. We needed a way to understand how AWS configuration changes impacted our security posture. It was also critical to have access to an authoritative configuration history service for forensic and investigative purposes so that we could know how things have changed over time. We also needed these capabilities at scale across the many accounts we manage and many AWS services we use.
These needs are at the heart of what Security Monkey is — an AWS security configuration tracker and analyzer that scales for large and globally distributed cloud environments.
Architecture
At a high-level, Security Monkey consists of the following components:
- Watcher — The component that monitors a given AWS account and technology (e.g. S3, IAM, EC2). The Watcher detects and records changes to configurations. So, if a new IAM user is created or if an S3 bucket policy changes, the Watcher will detect this and store the change in Security Monkey’s database.
- Notifier — The component that lets a user or group of users know when a particular item has changed. This component also provides notification based on the triggering of audit rules.
- Auditor — Component that executes a set of business rules against an AWS configuration to determine the level of risk associated with the configuration. For example, a rule may look for a security group with a rule allowing ingress from 0.0.0.0/0 (meaning the security group is open to the Internet). Or, a rule may look for an S3 policy that allows access from an unknown AWS account (meaning you may be unintentionally sharing the data stored in your S3 bucket). Security Monkey has a number of built-in rules included, and users are free to add their own rules.
In terms of technical components, we run Security Monkey in AWS on Ubuntu Linux, and storage is provided by a PostgreSQL RDS database. We currently run Security Monkey on a single m3.large instance — this instance type has been able to easily monitor our dozens of accounts and many hundreds of changes per day.
The application itself is written in Python using the Flask framework (including a number of Flask plugins). At Netflix, we use our standard single-sign on (SSO) provider for authentication, but for the OSS version we’ve implemented Flask-Login and Flask-Security for user management. The frontend for Security Monkey’s data presentation is written in Angular Dart, and JSON data is also available via a REST API.
General Features and Operations
Security Monkey is relatively straightforward from an operational perspective. Installation and AWS account setup is covered in the installation document, and Security Monkey does not rely on other Netflix OSS components to operate. Generally, operational use includes:
Initial Configuration
- Setting up one or more Security Monkey users to use/administer the application itself.
- Setting up one or more AWS accounts for Security Monkey to monitor.
- Configuring user-specific notification preferences (to determine whether or not a given user should be notified for configuration changes and audit reports).
Typical Use Cases
- Checking historical details for a given configuration item (e.g. the different states a security group has had over time).
- Viewing reports to check what audit issues exist (e.g. all S3 policies that reference unknown accounts or all IAM users that have active access keys).
- Justifying audit issues (providing background or context on why a particular issues exists and is acceptable though it may violate an audit rule).
Note on AWS CloudTrail and AWS Trusted Advisor
CloudTrail is AWS’ service that records and logs API calls. Trusted Advisor is AWS’ premium support service that automatically evaluates your cloud deployment against a set of best practices (including security checks).
Security Monkey predates both of these services and meets a bit of each services’ goals while having unique value of its own:
- CloudTrail provides verbose data on API calls, but has no sense of state in terms of how a particular configuration item (e.g. security group) has changed over time. Security Monkey provides exactly this capability.
- Trusted Advisor has some excellent checks, but it is a paid service and provides no means for the user to add custom security checks. For example, Netflix has a custom check to identify whether a given IAM user matches a Netflix employee user account, something that is impossible to do via Trusted Advisor. Trusted Advisor is also a per-account service, whereas Security Monkey scales to support and monitor an arbitrary number of AWS accounts from a single Security Monkey installation.
Open Items and Future Plans
Security Monkey has been in production use at Netflix since 2011 and we will continue to add additional features. The following list documents some of our planned enhancements.
- Integration with CloudTrail for change detail (including originating IP, instance, IAM account).
- Ability to compare different configuration items across regions or accounts.
- CSRF protections for form POSTs.
- Content Security Policy headers (currently awaiting a Dart issue to be addressed).
- Additional AWS technology and configuration tracking.
- Test integration with moto.
- SSL certificate expiry monitoring.
- Simpler installation script and documentation.
- Roles/authorization capabilities for admin vs. user roles.
- More refined AWS permissions for Security Monkey operations (the current policy in the install docs is a broader read-only role).
- Integration with edda, our general purpose AWS change tracker. On a related note, our friends at Prezi have open sourced reddalert, a security change detector that is itself integrated with edda.
Conclusion
Security Monkey has helped the security teams @ Netflix gain better awareness of changes and security risks in our AWS environment. Its approach fits well with the general Simian Army approach of continuously monitoring and detecting potential anomalies and risky configurations, and we look forward to seeing how other AWS users choose to extend and adapt its capabilities. Security Monkey is now available on our GitHub site.
— Patrick Kelley, Kevin Glisson, and Jason Chan (Netflix Cloud Security Team)
See Also:
Originally published at techblog.netflix.com on June 30, 2014.
