{"id":475,"date":"2022-02-14T01:35:08","date_gmt":"2022-02-14T01:35:08","guid":{"rendered":"https:\/\/192.168.1.3\/wordpress\/?p=475"},"modified":"2025-02-10T07:07:39","modified_gmt":"2025-02-10T07:07:39","slug":"aws-solution-architect-associate-saac02-review-material-message-queues-and-streams","status":"publish","type":"post","link":"https:\/\/mylinuxsite.com\/wordpress\/?p=475","title":{"rendered":"AWS Solution Architect Associate (SAA-C02) Review Material \u2013 Message Queues and Streams (Kinesis, SQS, SNS, &#038; Active MQ)"},"content":{"rendered":"\n<!--more continue reading-->\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Kinesis<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\"><li>A fully managed service that allows you to ingest, buffer, and process streaming data in real-time.<\/li><li>Can handle any amount of streaming data and process data from hundreds of thousands of sources with very low latencies<\/li><li>A <strong>Producer\/Consumer<\/strong> model<\/li><li>Kinesis Capabilities:<ol><li><strong>&nbsp;Kinesis Data Streams(KDS)<\/strong><ul><li>A serverless streaming data service that makes it easy to capture, process, and store data stream at any scale.<\/li><li>Producers use SDK, KPL or through an agent<\/li><li>Producers emit <strong>data records<\/strong> that contain<strong> partition keys<\/strong>. <strong>Partition keys<\/strong> ultimately <strong>determine<\/strong> <strong>which shard ingests the data<\/strong> record for a data stream<\/li><li><strong>Shard <\/strong><ul><li>the base throughput unit of an Amazon Kinesis data stream. <\/li><li>&nbsp;can ingest up to 1000 data records per second, or 1MB\/sec.<\/li><li>grouped into <em>Data Streams <\/em>which will retain data for <strong>24 hours by default, or optionally up to 365 days<\/strong>.<\/li><li><em>scaling is done through shards<\/em><\/li><\/ul><\/li><li>Consumers can be an app(SDK, KCL) or other AWS services such as Lambda, Kinesis FireHorse or Analytics<\/li><li>KCL:<ul><li>Uses <strong>Lease<\/strong> for consumer to lock on the shard.<\/li><li>Consumer <strong>cannot<\/strong>  hold the <strong>Lease<\/strong> of a shard at the same time<\/li><\/ul><\/li><li>Consumed records will have a <strong>sequence no<\/strong> which is added by Kinesis Data Streams<\/li><li>2 types of consumption mechanisms:<ol><li>Shared &#8211; 2 MB\/sec\/shard shared across all consumers<\/li><li>Enhanced &#8211; 2MB\/sec\/shard\/consumer<\/li><\/ol><\/li><li>Billed per shard<\/li><li>Records are immutable.<\/li><li><strong>Realtime, replay capability<\/strong><\/li><li><strong>Kinesis Agent<\/strong> is a stand-alone Java software application that offers an easy way to collect and send data to Kinesis Data Streams.<\/li><\/ul><\/li><li><strong>Kinesis Data Firehorse (KDF)<\/strong><ul><li>A fully managed service that makes it easy to capture, transform, and load massive volumes of streaming data into a data store or analytics tool.<\/li><li>Requires a lambda function to transform data<\/li><li>Write to destination in batches i.e. near real-time<\/li><li>Supported destination includes<strong> Amazon S3, Amazon Redshift, Amazon OpenSearch Service<\/strong>, HTTP endpoints, Datadog, New Relic, MongoDB, and Splunk as destinations.<\/li><li>Failed data can be copied to S3 bucket<\/li><\/ul><\/li><li><strong>Kinesis Data Analytics (KDA)<\/strong><ul><li>A fully managed service for analyzing streaming data in real-time.<\/li><li>&nbsp;You get a console-based editor to build SQL queries.<\/li><li><strong>Kinesis Data Analytics Studio<\/strong> supports sub-second queries with built-in visualizations.<\/li><li>Automatic Scaling<\/li><li>Real-time<\/li><li>Integrates with KDS and KDF<\/li><li>Use cases: Real-time dashboard, metrics, time-series. <\/li><\/ul><\/li><li>Kinesis Video Streams<ul><li>makes it easy to securely stream video from connected devices to AWS for analytics, machine learning (ML), playback, and other processing.&nbsp;<\/li><\/ul><\/li><\/ol><\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">SQS<\/h4>\n\n\n\n<ul class=\"wp-block-list\"><li>A fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.\u00a0<\/li><li>A <strong>Producer\/Consumer<\/strong> model.<\/li><li>Unlimited throughput\/queue<\/li><li>Consumers must delete the message.<\/li><li>Has 2 types:<ul><li><strong>Standard<\/strong><\/li><li><strong>FIFO<\/strong><\/li><\/ul><\/li><li>Best effort ordering and message can have duplicates unless using FIFO.<\/li><li>Can <strong>receive up to max 10 messages.<\/strong><\/li><li><strong>FIFO<\/strong> Queue:<ul><li>Ordered<\/li><li>No duplicates<\/li><li>Limited throughput (compared to <strong>Standard<\/strong>)<\/li><li>Queue name must end with <strong>.fifo<\/strong><\/li><li>Has <strong>batching<\/strong> (support 3000 transactions\/sec per API call)<\/li><li>W\/o <strong>batching<\/strong> supports 300 API\/sec<\/li><li><strong>S3 is not allowed to send notifications to this type of SQS<\/strong><\/li><\/ul><\/li><li>Has access policy (like S3)<\/li><li><strong>Message Group ID<\/strong> <ol><li>A <strong>tag<\/strong> that specifies that a message belongs to a specific message group.<\/li><li>The same message group are a<strong>lways processed one by one, in a strict order<\/strong> relative to the message group.<\/li><\/ol><\/li><li><strong>Message Deduplication ID<\/strong><ul><li> A token that is used for deduplication of sent messages. <\/li><li>Messages sent with the same message deduplication ID are accepted successfully but aren&#8217;t delivered during the 5-minute deduplication interval.<\/li><\/ul><\/li><li><strong>Request-Response Model:<\/strong><ul><li>Used when a producer requires responses from consumers<\/li><li> A producer will send a message containing a &#8216;Correlation Id&#8217; and &#8216;Response Queue Name&#8217; is sent to the Request queue.<\/li><li>A consumer will respond by sending a message containing the same &#8216;Correlation Id&#8217; to the queue specified in the request.<\/li><li>Implemented through <strong>SQS Temporary Queue Client.<\/strong><\/li><\/ul><\/li><li>Important Configurations:<ul><li><strong>Visibility Timeout<\/strong> &#8211; a time when it will not be visible to other consumers. <strong> 0 &#8211; 12 hours<\/strong>. Can be set programmatically(<strong>ChangeMessageVisibility<\/strong> API)<\/li><li><strong>Delivery Delay<\/strong> &#8211; time to delay the first delivery of each message added to the queue. <strong>0 &#8211; 15 minutes<\/strong><\/li><li><strong>Receive Message Timeout<\/strong> &#8211; time that polling will wait for messages to become available to receive. <strong>0 &#8211; 20 seconds<\/strong><\/li><li><strong>Message Retention Period-<\/strong> time that Amazon SQS retains a message that does not get deleted. <strong>1 minute to 14 days<\/strong><\/li><li><strong>Maximum message size<\/strong> &#8211; maximum message size for your queue.   <strong>1 &#8211; 256 KB.<\/strong><\/li><li><strong>Dead Letter Queue<\/strong> <strong>(DLQ)<\/strong><ul><li>message can&#8217;t be consumed successfully can be sent to DLQ<\/li><li>DLQ is another SQS but with <strong>Redrive Allow Policy <\/strong>enabled<\/li><li><strong>Maximum Receives <\/strong>determines when a message will be sent to the DLQ. If the\u00a0<strong>ReceiveCount<\/strong> exceeds this value then the message will go to the DLQ<\/li><\/ul><\/li><\/ul><\/li><li><strong>Delay queues:<\/strong><ul><li>let you postpone the delivery of new messages to consumers for a number of seconds,<\/li><li>similar to\u00a0<a href=\"https:\/\/docs.aws.amazon.com\/AWSSimpleQueueService\/latest\/SQSDeveloperGuide\/sqs-visibility-timeout.html\">visibility timeouts<\/a>\u00a0because both features make messages unavailable to consumers for a specific period of time<\/li><li>\u00a0difference between the two is that, for <strong>delay queues<\/strong>, a message is <strong>hidden\u00a0<em>when it is first added to queue<\/em>,<\/strong> whereas for <strong>visibility timeouts<\/strong> a message is hidden\u00a0<em>only after it is consumed from the queue<\/em>.<\/li><\/ul><\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>SNS<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\"><li>A fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication<\/li><li>A <strong>Pub\/Sub <\/strong>model<\/li><li>Subscribers can be <strong>SQS, Lambda, Email, SMS, HTTP\/HTTPS, Mobile endpoints.<\/strong><\/li><li>Has <em>Access Policy<\/em><\/li><li>Has <em>Subscription Filter<\/em><\/li><li>Can <em>define Message Attributes<\/em><\/li><li>Has <em>DLQ<\/em><\/li><li>Has <em>Message Group ID and Deduplication ID<\/em><\/li><li>Has TTL (only for Mobile Endpoints)<\/li><li>Can be coupled with SQS for the Fan Out pattern. But SQS access policy must allow SNS to write to SQS<\/li><li>Has 2 types (similar to SQS):<ul><li><strong>Standard<\/strong><\/li><li><strong>FIFO<\/strong> &#8211; only allows SQS for subscription<\/li><\/ul><\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Active MQ<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\"><li>A &nbsp;managed message broker service for&nbsp;<a rel=\"noreferrer noopener\" href=\"http:\/\/activemq.apache.org\/components\/classic\/\" target=\"_blank\">Apache ActiveMQ<\/a>&nbsp;and&nbsp;<a rel=\"noreferrer noopener\" href=\"https:\/\/www.rabbitmq.com\/\" target=\"_blank\">RabbitMQ<\/a>.<\/li><li>Supports open\/standard protocols such as MQTT, AMQP, STOMP<\/li><li>HA (Active\/Standby) but requires EFS.<\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[11],"tags":[],"class_list":["post-475","post","type-post","status-publish","format-standard","hentry","category-aws-review-notes"],"_links":{"self":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/475","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=475"}],"version-history":[{"count":29,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/475\/revisions"}],"predecessor-version":[{"id":1467,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/475\/revisions\/1467"}],"wp:attachment":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=475"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=475"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=475"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}