Streaming data solutions on AWS with Amazon Kinesis

Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application. With Amazon Kinesis, you can ingest real-time data such as video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications. Amazon Kinesis enables you to process and analyze data as it arrives and respond instantly instead of having to wait until all your data is collected before the processing can begin.

Your group has to read and understand the various streaming data solutions possible with Amazon Kinesis streaming with the help of attached white paper published by Amazon. Based on your grasping of this white paper, address the following questions

Q1. For the given scenario descriptions in the paper, identify and briefly explain the components/services used for - [2.5]

a) Data ingestion and data flow management

b) Data enrichment

b) Data processing

c) Data storage

d) Processed data delivery

Q2. What are the different ways by which data can be written into and read from AWS Kinesis Streams? [1.75]

Q3. Briefly explain in your understanding of the situation (excluding given scenarios) where you will consider using Kinesis Streams over Firehose. [2]

Q4. What are the ways by which streaming data can be queried using SQL? [0.75]

Q5. Design and draw a streaming data pipeline using AWS components satisfying the following requirements - [3]

- Ingested streaming data to be stored for 7 days

- Raw streaming data to be stored so that it can be fed to other AWS services

- Scala to be used as language for data enrichment application

- Data at rest to be queried using SQL

- has to have capabilities to search within streaming enriched data

- alert to be sent to the consumers when certain criteria is meet