Step 3 - Streaming data

AmazonKinesis

EcommCo’s Data Lake leverages Amazon Kinesis to support streaming submission ingestion, validation and analytics with real-time updated dashboards in the Elasticsearch Service.

a. Demonstrate support of streaming submissions

The diagram below illustrates how orders stream in from EcommCo’s e-commerce platform and get ingested in real-time by Kinesis Firehose and stored in Redshift. Kinesis Analytics provides real-time analytics on the orders data. Results of these analytics are saved to the Elasticsearch Service, which provides real-time dashboards of business-relevant KPIs.

b. Run streaming submissions

{% include 'error_box.html' %}

When you click this button, the following steps will be performed within your AWS account:

  • Creates Kinesis Analytics applications:
    • clean-orders-app
    • aggregate-orders-app
  • Enables streaming Data Provider which streams Orders data to Kinesis Firehose
  • Streamed Orders are
    • cleaned and analyzed with results published to the Elasticsearch Service
    • persisted in S3 submissions Bucket
    • validated and loaded to Redshift

c. Check that Kinesis analytics applications are running

  1. Visit Kinesis Analytics in your AWS Management Console
  2. Observe applications in 'Starting' state
  3. Wait for applications to be in 'Running' state and proceed to the next step below

d. Observe real-time updated dashboards with results of analytics

  1. Note: Data will be available in Elasticsearch after couple of minutes
  2. Visit Orders analysis dashboard in Elasticsearch Kibana
  3. Observe the real-time updated dashboards including:

Revenue by state

This dashboard shows total revenue for orders grouped by customer state computed in a sliding 1 minute window.

Top SKUs

This dashboard ranks the most popular product SKUs.