Streamalert: Serverless, Realtime Data Analysis Framework
Category : Blog
StreamAlert is a serverless, real-time data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define. A serverless framework for real-time data analysis and alerting.
Airbnb needed a product that empowered both engineers and administrators to ingest, analyze, and alert on data in real-time from their respective environments.
Features of Streamalert:
- Deployment is automated: simple, safe and repeatable for any AWS account
- Easily scalable from megabytes to terabytes per day
- Infrastructure maintenance is minimal, no devops expertise required
- Infrastructure security is a default, no security expertise required
- Supports data from different environments (ex: IT, PCI, Engineering)
- Supports data from different environment types (ex: Cloud, Datacenter, Office)
- Supports different types of data (ex: JSON, CSV, Key-Value, or Syslog)
- Supports different use-cases like security, infrastructure, compliance and more
As partially outlined above, StreamAlert has some unique benefits:
- Serverless — StreamAlert utilizes AWS Lambda, which means you don’t have to manage, patch or harden any new servers
- Scalable — StreamAlert utilizes AWS Kinesis Streams, which will “scale from megabytes to terabytes per hour and from thousands to millions of PUT records per second”
- Automated — StreamAlert utilizes Terraform, which means infrastructure and supporting services are represented as code and deployed via automation
- Secure — StreamAlert uses secure transport (TLS), performs data analysis in a container/sandbox, segments data per your defined environments, and uses role-based access control (RBAC)
- Open Source — Anyone can use or contribute to StreamAlert
StreamAlert utilizes the following services:
- AWS Kinesis Streams — Datastream; AWS Lambda polls this stream (stream-based model)
- AWS Kinesis Firehose — Loads streaming data into S3 long-term data storage
- AWS Lambda (Python) — Data analysis and alerting
- AWS SNS — Alert queue
- AWS S3 — Optional datasources, long-term data storage, & long-term alert storage
- AWS Cloudwatch — Infrastructure metrics
- AWS KMS — Encryption and decryption of application secrets
- AWS IAM — Role-based Access Control (RBAC)
If you’re not an AWS customer, StreamAlert can support data such as:
- Host Logs (e.g. Syslog, osquery, auditd)
- Network Logs (e.g. Palo Alto Networks, Cisco)
- Web Application Logs (e.g. Apache, nginx)
- SaaS providers (e.g. Box, OneLogin)
It should be noted that StreamAlert is not intended for analytics, metrics or time series use-cases. There are many great open source and commercial offerings in this space, including but not limited to Prometheus, DataDog and NewRelic.
Open source has allowed us as a community, to both share, collaborate, and iterate on common needs and goals. Now with the ability to represent infrastructure as code, this goal can be further realized with reduced costs for both development and deployment.
We hope StreamAlert serves as an example of this, making deployment simple, repeatable and safe so that anyone can use it easily.
Most Popular Training Courses at Indian Cyber Security Solutions:
Cybersecurity services that can protect your company: