Streamalert: Serverless, Realtime Data Analysis Framework

StreamAlert

Streamalert: Serverless, Realtime Data Analysis Framework

Category : Blog

Streamalert

StreamAlert is a serverless, real-time data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define. A serverless framework for real-time data analysis and alerting.

Airbnb needed a product that empowered both engineers and administrators to ingest, analyze, and alert on data in real-time from their respective environments.

StreamAlert

 

Features of Streamalert:

  • Deployment is automated: simple, safe and repeatable for any AWS account
  • Easily scalable from megabytes to terabytes per day
  • Infrastructure maintenance is minimal, no devops expertise required
  • Infrastructure security is a default, no security expertise required
  • Supports data from different environments (ex: IT, PCI, Engineering)
  • Supports data from different environment types (ex: Cloud, Datacenter, Office)
  • Supports different types of data (ex: JSON, CSV, Key-Value, or Syslog)
  • Supports different use-cases like security, infrastructure, compliance and more

Supports

 

Benefits:

As partially outlined above, StreamAlert has some unique benefits:

  • Serverless — StreamAlert utilizes AWS Lambda, which means you don’t have to manage, patch or harden any new servers
  • Scalable — StreamAlert utilizes AWS Kinesis Streams, which will “scale from megabytes to terabytes per hour and from thousands to millions of PUT records per second”
  • Automated — StreamAlert utilizes Terraform, which means infrastructure and supporting services are represented as code and deployed via automation
  • Secure — StreamAlert uses secure transport (TLS), performs data analysis in a container/sandbox, segments data per your defined environments, and uses role-based access control (RBAC)
  • Open Source — Anyone can use or contribute to StreamAlert

Source

 

StreamAlert utilizes the following services:

  • AWS Kinesis Streams — Datastream; AWS Lambda polls this stream (stream-based model)
  • AWS Kinesis Firehose — Loads streaming data into S3 long-term data storage
  • AWS Lambda (Python) — Data analysis and alerting
  • AWS SNS — Alert queue
  • AWS S3 — Optional datasources, long-term data storage, & long-term alert storage
  • AWS Cloudwatch — Infrastructure metrics
  • AWS KMS — Encryption and decryption of application secrets
  • AWS IAM — Role-based Access Control (RBAC)

AWS

 

If you’re not an AWS customer, StreamAlert can support data such as:

  • Host Logs (e.g. Syslog, osquery, auditd)
  • Network Logs (e.g. Palo Alto Networks, Cisco)
  • Web Application Logs (e.g. Apache, nginx)
  • SaaS providers (e.g. Box, OneLogin)

It should be noted that StreamAlert is not intended for analytics, metrics or time series use-cases. There are many great open source and commercial offerings in this space, including but not limited to Prometheus, DataDog and NewRelic.

Data

 

Concluding Thoughts:

Open source has allowed us as a community, to both share, collaborate, and iterate on common needs and goals. Now with the ability to represent infrastructure as code, this goal can be further realized with reduced costs for both development and deployment.

We hope StreamAlert serves as an example of this, making deployment simple, repeatable and safe so that anyone can use it easily.

Deployment

 

Most Popular Training Courses at Indian Cyber Security Solutions:

 

Summer Training for CSE, IT, BCA & MCA Students 

Network Penetration Tester Training

Ethical Hacking  training

Python Programming training

 RHCE  training

CEH V9  training

Diploma in Network Security Training

Secure Coding in Java

Diploma in Web Application Security 

Certified Web Application Penetration Tester 

Certified Android Penetration Tester 

Certified Python Programming 

Advanced Python Training 

Reverse Engineering Training  

Amazon Web Services Training  

VMware Training 

Digital marketing

CCNA training

Android Training

 

Cybersecurity services that can protect your company:

 

Web Security | Web Penetration Testing

Network Penetration Testing – NPT

Android App Penetration Testing

Source Web Development

Source Code Review

Android App Development

Digital Marketing Consultancy

Data Recovery


Leave a Reply

Show Buttons
Hide Buttons