{"id":1089,"date":"2024-11-07T11:32:12","date_gmt":"2024-11-07T11:32:12","guid":{"rendered":"https:\/\/192.168.1.3\/wordpress\/?p=1089"},"modified":"2025-02-27T06:19:53","modified_gmt":"2025-02-27T06:19:53","slug":"aws-certified-data-engineer-associate-dea-c01-review-material-kinesis","status":"publish","type":"post","link":"https:\/\/mylinuxsite.com\/wordpress\/?p=1089","title":{"rendered":"AWS Certified Data Engineer Associate (DEA-C01) Review Material \u2013 Kinesis"},"content":{"rendered":"\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Kinesis Data Stream (KDS)<\/strong><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Overview<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>Used to process large&nbsp;streams of data records in real-time.<\/li><li>Retain data up to 365 days.<\/li><li>Data sent in the stream are immutable.<\/li><li>KDS is a public service (cannot be placed inside a VPC)<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>High-Level Architecture<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>KDS<ul><li>A Kinesis data stream is a set of <strong>shards<\/strong>.<\/li><li>Each shard has a <strong>sequence of data records<\/strong>. <\/li><li>Each data record has a <strong>sequence number<\/strong> that is assigned by Kinesis Data Streams.<\/li><\/ul><\/li><li>Producer<ul><li>Put records into Amazon Kinesis Data Streams<\/li><li>Types of Producers:<ul><li>AWS SDK<\/li><li>KPL<ul><li>Automated and configurable retry mechanism<\/li><li>Support C++ or Java<\/li><\/ul><\/li><li>Kinesis Agents<\/li><li>AWS Managed Services w\/ built-in producer function:<ul><li>Cloudwatch<\/li><li>AWS IoT<\/li><li>KDA<\/li><\/ul><\/li><\/ul><\/li><li><em>KPL Batching<\/em><ul><li>Performs a single action on multiple items instead of repeatedly performing the action on each individual item.<\/li><li>It can incur a delay due to its buffering action.<\/li><li>Two Types:<ol><li><em>Aggregation<\/em><ul><li>Storing multiple records within a single Kinesis Data Streams record.<\/li><li>Aggregation allows customers to increase the number of records sent per API call, which effectively increases producer throughput.<\/li><\/ul><\/li><li><em>Collection<\/em><ol><li>Batching multiple Kinesis Data Streams records and sending them in a single HTTP request with a call to the API operation&nbsp;<code>PutRecords<\/code><\/li><\/ol><\/li><\/ol><\/li><li><img loading=\"lazy\" decoding=\"async\" width=\"500\" height=\"200\" class=\"wp-image-1092\" style=\"width: 500px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-06-16-53-18.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-06-16-53-18.png 1210w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-06-16-53-18-300x120.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-06-16-53-18-1024x410.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-06-16-53-18-768x307.png 768w\" sizes=\"auto, (max-width: 500px) 100vw, 500px\" \/><\/li><\/ul><\/li><\/ul><\/li><li>Consumers<ul><li>Get records from Amazon Kinesis Data Streams and process them<\/li><li>Types of Consumers:<ul><li>AWS SDK<\/li><li>KCL<\/li><li>AWS Managed Services w\/ built-in consumer function:<ul><li>Lambda<\/li><li>KDF<\/li><li>KDA<\/li><\/ul><\/li><\/ul><\/li><li><em>KCL<\/em>:<ul><li>It helps you consume and process data from a Kinesis data stream by taking care of many of the complex tasks associated with distributed computing.&nbsp;<\/li><li>It&nbsp;provides a layer of abstraction around KDS API (AWS SDK)<\/li><li>Uses a concept of <strong>Lease<\/strong>&nbsp;\u2013 data that defines the binding between a worker and a shard.<\/li><li>Uses a <strong>Lease table<\/strong>&nbsp;&#8211; a unique Amazon DynamoDB table that is used to keep track of the shards in a KDS data stream that are being leased and processed by the workers of the KCL<\/li><li><em>Each KCL application must use its own DynamoDB table.<\/em><\/li><\/ul><\/li><\/ul><\/li><li>Shards<ul><li>Uniquely identified sequence of data records in a stream.<\/li><li><em>Reads<\/em>:<ul><li>Each shard can support up to 5 transactions per second.<\/li><li>Maximum data read at an <strong><span style=\"color:#a30400\" class=\"has-inline-color\">enhanced<\/span><\/strong> <strong><span style=\"color:#a30400\" class=\"has-inline-color\">fan-out<\/span><\/strong> mode of 2 MB per second.<\/li><li>Maximum <span style=\"color:#a30d00\" class=\"has-inline-color\">total<\/span> data read rate of 2 MB per second.<\/li><\/ul><\/li><li><em>Writes<\/em>:<ul><li>Each shard can support up to 1000 transactions per second.<\/li><li>Maximum <span style=\"color:#a30d00\" class=\"has-inline-color\">total<\/span> data read rate of 1 MB per second.<\/li><\/ul><\/li><li><em>Partition key:<\/em><ul><li>Used to group data by shard within a stream.<\/li><li>It is associated with each data record to determine which shard a given data record belongs to.&nbsp;<\/li><li>An MD5 hash function is used to map partition keys to 128-bit integer values and to map associated data records to shards using the hash key ranges of the shards.&nbsp;<\/li><\/ul><\/li><li><em>Sequence Number<\/em>:<ul><li>Unique per partition-key within its shard<\/li><li>KDS assigns the sequence number after you write to the stream with&nbsp;<code>client.putRecords<\/code>&nbsp;or&nbsp;<code>client.putRecord<\/code>.&nbsp;<\/li><\/ul><\/li><\/ul><\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Capacity Mode<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>Determines how the capacity of a data stream is managed and how you are charged for the usage of your data stream.<\/li><li>Two (2) Capacity Modes:<ol><li><strong>On-demand<\/strong><ul><li>It requires no capacity planning and automatically scales to handle gigabytes of write and read throughput per minute. <\/li><li>Automatically manages the shards in order to provide the necessary throughput.<\/li><li>It is ideal for addressing the needs of highly variable and unpredictable application traffic.<\/li><li>You pay <em>per GB <\/em>of data <em>written<\/em> and <em>read<\/em> from your data streams.&nbsp;<\/li><\/ul><\/li><li><strong>Provisioned<\/strong><ul><li>You specify the number of shards needed for the data stream. <\/li><li>The <em>total capacity<\/em> of a data stream is the <em>sum of the capacities of its shards<\/em>. <\/li><li>You can increase or decrease the number of shards in a data stream as needed.<\/li><li>You pay by <em>shard hour<\/em> and <em>PUT Payload Units<\/em><\/li><\/ul><\/li><\/ol><\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Resharding a Stream<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>It lets you adjust the number of shards in your stream to adapt to changes in the rate of data flow through the stream.<\/li><li>Two (2) types of operations: <ol><li><em>Split<\/em>:<ul><li>Divide a single shard into two shards.<\/li><li>Increases the number of shards in your stream and therefore increases the data capacity of the stream.<\/li><li>Increases the cost of your stream.&nbsp;<\/li><\/ul><\/li><li><em>Merge<\/em>:<ul><li>Combine two shards into a single shard.&nbsp;&nbsp;<\/li><li>Reduces the number of shards in your stream and therefore decreases the data capacity.<\/li><li>Reduces the cost of your stream.<\/li><\/ul><\/li><\/ol><\/li><li>Resharding is always&nbsp;<em>pairwise,<\/em>&nbsp;i.e. cannot split into <em>more than two shards<\/em> in a single operation, cannot merge more than two shards in a single operation.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Kinesis Data Firehose (KDF)<\/strong><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Overview<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>A managed service for delivering real-time streaming data to a number of destinations.<\/li><li>Sources: <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.aws.amazon.com\/firehose\/latest\/dev\/create-name.html\" data-type=\"URL\" data-id=\"https:\/\/docs.aws.amazon.com\/firehose\/latest\/dev\/create-name.html\" target=\"_blank\">https:\/\/docs.aws.amazon.com\/firehose\/latest\/dev\/create-name.html<\/a><ul><li>Kinesis Agent<\/li><\/ul><ul><li>AWS SDK<\/li><li>CW Log<\/li><li>CW Event<\/li><li>AWS IoT<\/li><li>WAF<\/li><li>MSK<\/li><li>KDS<ul><li>More than one Firehose stream can read from the same Kinesis stream.<\/li><li>Firehose\u2019s PutRecord and PutRecordBatch operations will be disabled, so an agent cannot simultaneously write.<\/li><\/ul><\/li><\/ul><\/li><li>Destinations: <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.aws.amazon.com\/firehose\/latest\/dev\/create-name.html\" data-type=\"URL\" data-id=\"https:\/\/docs.aws.amazon.com\/firehose\/latest\/dev\/create-name.html\" target=\"_blank\">https:\/\/docs.aws.amazon.com\/firehose\/latest\/dev\/create-name.html<\/a><ul><li>S3<\/li><li>RedShift<\/li><li>Amazon Open Search <\/li><li>MSK<\/li><li>HTTP Endpoint<\/li><li>Splunk<\/li><li>Snowflake<\/li><li>Custom HTTP Endpoint<\/li><\/ul><\/li><li>Can convert data from JSON to Apache Parquet or Apache ORC before storing the data in Amazon S3<\/li><li>Near realtime<\/li><li>In rare circumstances, such as a request timeout upon a data delivery attempt, a delivery retry by Firehose could introduce <strong>duplicates<\/strong> if the previous request eventually goes through.<\/li><li>Support compression for S3 target<\/li><li>Automatic Scaling<\/li><li>You pay for the <em>volume<\/em> of data you ingest into the service<\/li><li>It <strong>can invoke a Lambda<\/strong> function to transform incoming source data. Several blueprints are available. (<em>only supports invocation time of up to 5 minutes<\/em>)<\/li><li>It uses the IAM Policy to grant access to the KDF.<\/li><li>Buffers incoming streaming data to a certain size and for a certain period of time before delivering it to the specified destinations.<ul><li>Buffering Hints:<ul><li><strong>buffer size <\/strong>&#8211; measured in MBs (1 MB to 128 MB)<\/li><li><strong>buffer interval<\/strong> &#8211; measured in seconds. (60 to 900 seconds)<\/li><li><sub>When Kinesis Data Firehose&#8217;s delivery stream scales, it can cause an effect on the buffering hints of Data Firehose.<\/sub><\/li><\/ul><\/li><\/ul><\/li><li>Data payload size limit is <strong>1MB<\/strong><br><\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Data Delivery<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>S3<\/strong><ul><li>concatenates multiple incoming records based on the buffering configuration<\/li><li>delivers the records to Amazon S3 as an Amazon S3 object<\/li><li><em>any lambda transformation failure will also be written to an S3 bucket<\/em><\/li><li>A single&nbsp;Firehose stream can currently only deliver data to one Amazon S3 bucket<\/li><\/ul><\/li><li><strong>Redshift<\/strong><ul><li>first, it delivers incoming data to your S3 bucket<\/li><li><span style=\"box-sizing: border-box; margin: 0px; padding: 0px;\">Then<\/span>, an Amazon Redshift&nbsp;COPY&nbsp;command will be issued&nbsp;to load the data from your S3 bucket.<\/li><li><em>Need to unblock KDF IP if Redshift is in a VPC<\/em><\/li><\/ul><\/li><li><strong>OpenSearch<\/strong><ul><li>buffers incoming records based on the buffering configuration<\/li><li>Then, it generates an OpenSearch Service or OpenSearch Serverless bulk request to index multiple records to your OpenSearch service cluster.<\/li><li>Firehose stream and destination Amazon OpenSearch Service domain need to be in the same region<\/li><\/ul><\/li><li><strong>Splunk<\/strong><ul><li>concatenates the bytes that were sent<\/li><\/ul><\/li><li><strong>HTTP endpoint<\/strong><ul><li>can use the integrated Amazon Lambda service to create a function to transform the incoming record(s) to the format that matches the format the service provider&#8217;s integration is expecting<\/li><\/ul><\/li><li><strong>Snowflake<\/strong><ul><li>internally buffers data for one second and uses Snowflake streaming API operations to insert data to Snowflake<\/li><li>can only deliver data to one Snowflake table<\/li><\/ul><\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Kinesis Data Analytics (KDA)<\/strong><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Overview<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>Two (2) types of service offered:<ul><li>KDA for SQL Application (EOL 2026)<\/li><li>Managed Service for Apache Flink<\/li><\/ul><\/li><li><strong>Managed Service for Apache Flink<\/strong><ul><li>Process data streams in real-time with SQL or Apache Flink.<\/li><li>Support Java, Python &amp; Scala<\/li><li>Applications primarily use either the <em>DataStream API<\/em> or the<em> Table API<\/em>.<\/li><li><em>Connectors<\/em>:<ul><li>Software components that move data into and out of an Amazon Managed Service for Apache Flink application<\/li><li>Sample connectors:<ul><li>KDS<\/li><li>KDF<\/li><li>Kafka<\/li><li>OpenSearch<\/li><li>DynamoDB<\/li><\/ul><\/li><\/ul><\/li><\/ul><\/li><li>KDA SQL<ul><li>Support standard SQL operators such as SELECT, INSERT, CREATE<\/li><li>It provides a number of functions, such as:<ul><li>Aggregate<\/li><li>Analytic<\/li><li>Date &amp; Time<\/li><li>Statistical Variance and Deviation e.g.<ul><li><em>RANDOM_CUT_FOREST<\/em> <ul><li>Detects <em>anomalies<\/em> in your data stream.&nbsp; A record is an anomaly if it is distant from other records.<\/li><\/ul><\/li><\/ul><\/li><\/ul><\/li><\/ul><\/li><li>You pay mainly by KPU (Kinesis Processing Unit),  per Hour.<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong><span style=\"color:#2ceb0e\" class=\"has-inline-color\">Hands-On<\/span><\/strong><\/h3>\n\n\n\n<h5 class=\"wp-block-heading\"><strong>1<\/strong>.<strong> Cross-account streaming of CW Logs using KDS<\/strong><\/h5>\n\n\n\n<ul class=\"wp-block-list\"><li>For this hands-on, you need two (2) AWS Accounts: (1) a sending account and (2) a recipient account.<\/li><li>Perform the following in the recipient account:<ol><li>Create a Kinesis Data Stream (KDS):<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"159\" class=\"wp-image-1102\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-17-58-55.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-17-58-55.png 1584w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-17-58-55-300x68.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-17-58-55-1024x232.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-17-58-55-768x174.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-17-58-55-1536x348.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><\/li><li>Create a role with the following policies:<ul><li>Create the following trust policy:<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"369\" class=\"wp-image-1103\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-01-59.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-01-59.png 1554w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-01-59-300x158.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-01-59-1024x540.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-01-59-768x405.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-01-59-1536x810.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><\/li><li>Create the following permission policy:<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"374\" class=\"wp-image-1101\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-02-30.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-02-30.png 1579w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-02-30-300x160.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-02-30-1024x547.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-02-30-768x410.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-02-30-1536x820.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><\/li><\/ul><\/li><li>Create a CW Log destination:<ul><li>Execute the following from the command line:<ul><li>$ <code>aws logs put-destination --destination-name \"mylinuxsite-dea-c01\" --target-arn \"&lt;kds arn&gt;\" --role-arn \"&lt;arn of role created in #2&gt;\"<\/code><\/li><li>Take note of the <em>destination arn<\/em> that will be displayed as part of the output of this command.<\/li><\/ul><\/li><li>Create an access policy file containing the following texts.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"253\" class=\"wp-image-1100\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-19-04.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-19-04.png 738w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-18-19-04-300x109.png 300w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><ul><li>The &#8216;Resource&#8217; is the <em>destination arn<\/em>&nbsp;outputted from the previous command.<\/li><\/ul><\/li><\/ul><\/li><li>Write a simple lambda function that the Kinesis Data Stream triggers. The lambda will read the data and print it out in the CW log.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"462\" class=\"wp-image-1105\" style=\"width: 600px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-14-56.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-14-56.png 1100w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-14-56-300x231.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-14-56-1024x788.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-14-56-768x591.png 768w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/li><\/ul><\/li><\/ol><\/li><li>Perform the following in the sender account:<ul><li>Select one of the log groups and add a subscription filter. In this example, the log group is from a lambda function.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"344\" class=\"wp-image-1106\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-17-28.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-17-28.png 1585w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-17-28-300x148.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-17-28-1024x504.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-17-28-768x378.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-17-28-1536x756.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><li>Use the <em>destination arn<\/em> that was created in the previous step.<\/li><\/ul><\/li><li>Generate a log in your chosen log group.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"800\" height=\"226\" class=\"wp-image-1107\" style=\"width: 800px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-22-42.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-22-42.png 1645w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-22-42-300x85.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-22-42-1024x289.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-22-42-768x217.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-22-42-1536x434.png 1536w\" sizes=\"auto, (max-width: 800px) 100vw, 800px\" \/><\/li><\/ul><\/li><\/ul><\/li><li>Check the lambda function log created in the previous step in the recipient account. It should output the same text as what was record in the sender account lambda log.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"366\" class=\"wp-image-1108\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-27-17.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-27-17.png 1662w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-27-17-300x157.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-27-17-1024x535.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-27-17-768x402.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-07-19-27-17-1536x803.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><\/li><\/ul>\n\n\n\n<h5 class=\"wp-block-heading\"><strong>2. Dumping Cross-account CW Logs to S3 Using KDF.<\/strong><\/h5>\n\n\n\n<ul class=\"wp-block-list\"><li>We will use the same setup we used in Hands-on 1, but we will send the logs to KDF this time.<ol><li>Create a Kinesis Data Firehose (KDF) with the source as the KDS in Hands-on 1 and the destination as an S3 bucket.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"800\" height=\"354\" class=\"wp-image-1125\" style=\"width: 800px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-08-15-23-25.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-08-15-23-25.png 1904w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-08-15-23-25-300x133.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-08-15-23-25-1024x453.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-08-15-23-25-768x340.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-08-15-23-25-1536x680.png 1536w\" sizes=\"auto, (max-width: 800px) 100vw, 800px\" \/><\/li><\/ul><\/li><li>Generate a log in the log group used in Hands-on 1.<\/li><li>Check the destination S3 bucket if a file is created. If you did not modify the KDF buffer interval default value, you may need to wait 5 minutes.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"200\" class=\"wp-image-1124\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-08-15-35-20.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-08-15-35-20.png 1605w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-08-15-35-20-300x86.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-08-15-35-20-1024x293.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-08-15-35-20-768x220.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-08-15-35-20-1536x439.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><\/li><\/ol><\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[11],"tags":[],"class_list":["post-1089","post","type-post","status-publish","format-standard","hentry","category-aws-review-notes"],"_links":{"self":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1089","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1089"}],"version-history":[{"count":40,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1089\/revisions"}],"predecessor-version":[{"id":1517,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1089\/revisions\/1517"}],"wp:attachment":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1089"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1089"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1089"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}