Stream-processing with Mantis
Back in January of 2014 we wrote about the need for better visibility into our complex operational environments:
The core of the message in that post was about the need for fine-grained, contextual and scalable insights into the experiences of our customers and behaviors of our services. While our execution has evolved somewhat differently from our original vision, the underlying principles behind that vision are as relevant today as they were then. In this post we’ll share what we’ve learned building Mantis, a stream-processing service platform that’s processing event streams of up to 8 million events per second and running hundreds of stream-processing jobs around the clock. We’ll describe the architecture of the platform and how we’re using it to solve real-world operational problems.
Why Mantis?
There are more than 75 million Netflix members watching 125 million hours of content every day in over 190 countries around the world. To provide an incredible experience for our members, it’s critical for us to understand our systems at both the coarse-grained service level and fine-grained device level. We’re good at detecting, mitigating, and resolving issues at the application service level — and we’ve got some excellent tools for service-level monitoring — but when you get down to the level of individual devices, titles, and users, identifying and diagnosing issues gets more challenging.
We created Mantis to make it easy for teams to get access to realtime events and build applications on top of them. We named it after the Mantis shrimp, a freakish yet awesome creature that is both incredibly powerful and fast. The Mantis shrimp has sixteen photoreceptors in its eyes compared to humans’ three. It has one of the most unique visual systems of any creature on the planet. Like the shrimp, the Mantis stream-processing platform is all about speed, power, and incredible visibility.
So Mantis is a platform for building low-latency, high throughput stream-processing apps but why do we need it? It’s been said that the Netflix microservices architecture is a metrics generator that occasionally streams movies. It’s a joke, of course, but there’s an element of truth to it; our systems do produce billions of events and metrics on a daily basis. Paradoxically, we often experience the problem of having both too much data and too little at the same time. Situations invariably arise in which you have thousands of metrics at your disposal but none are quite what you need to understand what’s really happening. There are some cases where you do have access to relevant metrics, but the granularity isn’t quite good enough for you to understand and diagnose the problem you’re trying to solve. And there are still other scenarios where you have all the metrics you need, but the signal-to-noise ratio is so high that the problem is virtually impossible to diagnose. Mantis enables us to build highly granular, realtime insights applications that give us deep visibility into the interactions between Netflix devices and our AWS services. It helps us better understand the long tail of problems where some users, on some devices, in some countries are having problems using Netflix.
By making it easier to get visibility into interactions at the device level, Mantis helps us “see” details that other metrics systems can’t. It’s the difference between 3 photoreceptors and 16.
A Deeper Dive
With Mantis, we wanted to abstract developers away from the operational overhead associated with managing their own cluster of machines. Mantis was built from ground up to be cloud native. It manages a cluster of EC2 servers that is used to run stream-processing jobs. Apache Mesos is used to abstract the cluster into a shared pool of computing resources. We built, and open-sourced, a custom scheduling library called Fenzo to intelligently allocate these resources among jobs.
Architecture Overview
The Mantis platform comprises a master and an agent cluster. Users submit stream-processing applications as jobs that run as one or more workers on the agent cluster. The master consists of a Resource Manager that uses Fenzo to optimally assign resources to a jobs’ workers. A Job Manager embodies the operational behavior of a job including metadata, SLAs, artifact locations, job topology and life cycle.
The following image illustrates the high-level architecture of the system:
Mantis Jobs
Mantis provides a flexible model for defining a stream-processing job. A mantis job can be defined as single-stage for basic transformation/aggregation use cases or multi-stage for sharding and processing high-volume, high-cardinality event streams.
There are three main parts to a Mantis job.
- The source is responsible for fetching data from an external source
- One or more processing stages which are responsible for processing incoming event streams using high order RxJava functions
- The sink to collect and output the processed data
RxNetty provides non-blocking access to the event stream for a job and is used to move data between its stages.
To give you a better idea of how a job is structured, let’s take a look at a typical ‘aggregate by group’ example.
Imagine that we are trying to process logs sent by devices to calculate error rates per device type. The job is composed of three stages. The first stage is responsible for fetching events from a device log source job and grouping them based on device ID. The grouped events are then routed to workers in stage 2 such that all events for the same group (i.e., device ID) will get routed to the same worker. Stage 2 is where stateful computations like windowing and reducing — e.g., calculating error rate over a 30 second rolling window — are performed. Finally the aggregated results for each device ID are collected by Stage 3 and made available for dashboards or other applications to consume.
Job Chaining
One of the unique features of Mantis is the ability to chain jobs together. Job chaining allows for efficient data and code reuse. The image below shows an example of an anomaly detector application composed of several jobs chained together. The anomaly detector streams data from a job that serves Zuul request/response events (filtered using a simple SQL-like query) along with output from a “Top N” job that aggregates data from several other source jobs.
Scaling in Action
At Netflix the amount of data that needs to be processed varies widely based on the time of the day. Running with peak capacity all the time is expensive and unnecessary. Mantis autoscales both the cluster size and the individual jobs as needed.
The following chart shows how Fenzo autoscales the Mesos worker cluster by adding and removing EC2 instances in response to demand over the course of a week.
And the chart below shows an individual job’s autoscaling in action, with additional workers being added or removed based on demand over a week.
UI for Self-service, API for Integration
Mantis sports a dedicated UI and API for configuring and managing jobs across AWS regions. Having both a UI and API improves the flexibility of the platform. The UI gives users the ability to quickly and manually interact with jobs and platform functionality while the API enables easy programmatic integration with automated workflows.
The jobs view in the UI, shown below, lets users quickly see which jobs are running across AWS regions along with how many resources the jobs are consuming.
Each job instance is launched as part of a job cluster, which you can think of as a class definition or template for a Mantis job. The job cluster view shown in the image below provides access to configuration data along with a view of running jobs launched from the cluster config. From this view, users are able to update cluster configurations and submit new job instances to run.
How Mantis Helps Us
Now that we’ve taken a quick look at the overall architecture for Mantis, let’s turn our attention to how we’re using it to improve our production operations. Mantis jobs currently process events from about 20 different data sources including services like Zuul, API, Personalization, Playback, and Device Logging to name a few.
Of the growing set of applications built on these data sources, one of the most exciting use cases we’ve explored involves alerting on individual video titles across countries and devices.
One of the challenges of running a large-scale, global Internet service is finding anomalies in high-volume, high-cardinality data in realtime. For example, we may need access to fine-grained insights to figure out if there are playback issues with House of Cards, Season 4, Episode 1 on iPads in Brazil. To do this we have to track millions of unique combinations of data (what we call assets) all the time, a use case right in Mantis’ wheelhouse.
Let’s consider this use case in more detail. The rate of events for a title asset (title * devices * country) shows a lot of variation. So a popular title on a popular device can have orders of magnitude more events than lower usage title and device combinations. Additionally for each asset, there is high variability in event rate based on the time of the day. To detect anomalies, we track rolling windows of unique events per asset. The size of the window and alert thresholds vary dynamically based on the rate of events. When the percentage of anomalous events exceeds the threshold, we generate an alert for our playback and content platform engineering teams. This approach has allowed us to quickly identify and correct problems that would previously go unnoticed or, best case, would be caught by manual testing or be reported via customer service.
Below is a screen from an application for viewing playback stats and alerts on video titles. It surfaces data that helps engineers find the root cause for errors.
In addition to alerting at the individual title level, we also can do realtime alerting on our key performance indicator: SPS. The advantage of Mantis alerting for SPS is that it gives us the ability to ratchet down our time to detect (TTD) from around 8 minutes to less than 1 minute. Faster TTD gives us a chance to resolve issues faster (time to recover, or TTR), which helps us win more moments of truth as members use Netflix around the world.
Where are we going?
We’re just scratching the surface of what’s possible with realtime applications, and we’re exploring ways to help more teams harness the power of stream-processing. For example, we’re working on improving our outlier detection system by integrating Mantis data sources, and we’re working on usability improvements to get teams up and running more quickly using self-service tools provided in the UI.
Mantis has opened up insights capabilities that we couldn’t easily achieve with other technologies and we’re excited to see stream-processing evolve as an important and complementary tool in our operational and insights toolset at Netflix.
If the work described here sounds exciting to you, head over to our jobs page; we’re looking for great engineers to join us on our quest to reinvent TV!
— by Ben Schmaus, Chris Carey, Neeraj Joshi, Nick Mahilani, and Sharma Podila
See Also:
Originally published at techblog.netflix.com on March 14, 2016.
