Data and analytics on a data lake

Comments

• Organize the bucket in layers, for example raw and curated. Raw data is kept and the cleaned layers feed the queries.
• Athena charges for the amount of data scanned. Partitioning by date and using columnar formats such as Parquet reduces cost.
• Glue Data Catalog stores the table definitions that Athena and QuickSight use. That is why the dashed line goes from Glue to Athena.
• If you need real-time processing rather than delivery, consider Kinesis Data Streams or Amazon Managed Service for Apache Flink.

Information flow

1. Applications

   Applications send records or events to Amazon Data Firehose.

2. Amazon Data Firehose

   Firehose batches the records and writes them to S3.

3. Amazon S3

   Athena queries the S3 files with SQL.

4. Amazon Athena

   QuickSight receives Athena results to visualize them in dashboards.

5. AWS Glue

   Glue maintains the table catalog that Athena uses.