{"id":1236,"date":"2024-11-21T09:11:45","date_gmt":"2024-11-21T09:11:45","guid":{"rendered":"https:\/\/192.168.1.3\/wordpress\/?p=1236"},"modified":"2024-12-18T00:28:48","modified_gmt":"2024-12-18T00:28:48","slug":"aws-certified-data-engineer-associate-dea-c01-review-material-opensearch","status":"publish","type":"post","link":"https:\/\/mylinuxsite.com\/wordpress\/?p=1236","title":{"rendered":"AWS Certified Data Engineer Associate (DEA-C01) Review Material \u2013 OpenSearch"},"content":{"rendered":"\n<!--more-->\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Overview<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>A managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud.&nbsp;<\/li><li>An OpenSearch Service domain is synonymous with an OpenSearch cluster.&nbsp;<\/li><li>Automatically detects and replaces failed OpenSearch Service nodes.<\/li><li>Has the option for a Managed or Serverless cluster.<\/li><li>It can scale out or scale up\/down (no downtime).<\/li><li>It can be placed inside a VPC or made public.<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Managing Indexes<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Storages<\/strong>:<ol><li><strong>UltraWarm<\/strong>:<ul><li>A cost-effective way to store large amounts of <em>read-only<\/em> data.<\/li><li>It uses <em>Amazon S3<\/em> and a sophisticated caching solution to improve performance.<\/li><li>Best-suited to immutable data, such as logs.<\/li><\/ul><\/li><li><strong>Standard<\/strong>:<ul><li>Use &#8220;hot&#8221; storage, which takes the form of instance stores or Amazon EBS volumes attached to each node.<\/li><li>Hot storage provides the fastest possible performance for indexing and searching new data.<\/li><\/ul><\/li><li><strong>Cold:<\/strong><ul><li>Backed by Amazon S3.<\/li><li>Suitable for storing infrequently accessed or historical data.<\/li><li>Data suitable for cold storage include infrequently accessed logs, data that must be preserved to meet compliance requirements, and logs that have historical value.<\/li><\/ul><\/li><li><strong>OR1<\/strong>:<ul><li>An instance family for Amazon OpenSearch Service that provides a cost-effective way to store large amounts of data.<\/li><li>It uses Amazon Elastic Block Store (Amazon EBS)&nbsp;<code>gp3<\/code>&nbsp;or&nbsp;<code>io1<\/code>&nbsp;volumes for primary storage, with data copied synchronously to Amazon S3 as it arrives.<\/li><li>Suitable for running indexing heavy operational analytics workloads such as log analytics, observability, or security analytics.<\/li><li>OR1 instances offer an automatic data recovery option, which improves your domain&#8217;s overall reliability.<\/li><\/ul><\/li><\/ol><\/li><li><strong>Index State Management (ISM)<\/strong>:<ul><li>It lets you define custom management policies that automate routine tasks and apply them to indexes and index patterns.<\/li><li>Done through a policy which is attached to an index.<\/li><li>Examples of policies are:<ul><li>Hot to warm to cold storage<\/li><li>Reduce replica count<\/li><li>Take an index snapshot<\/li><\/ul><\/li><\/ul><\/li><li><strong>Index Rollup<\/strong><ul><li>It reduces storage costs by periodically rolling up old data into <em>summarized indexes<\/em>.<\/li><li>With index rollup, you create a new index with selected fields aggregated into coarser time buckets.<\/li><li>Reduces data <em>granularity<\/em> by rolling up old data into condensed indexes<\/li><\/ul><\/li><li><strong>Index Transform:<\/strong><ul><li>You create a different, summarized view of your data centered around certain fields so you can visualize or analyze the data in different ways.<\/li><\/ul><\/li><li><strong>Cross-cluster replication:<\/strong><ul><li>Replicate user indexes, mappings, and metadata from one OpenSearch Service domain to another.<\/li><li>It can be used for disaster recovery or to reduce latency.<\/li><li>The replication follows an active-passive replication model where the&nbsp;<em>local<\/em>&nbsp;or&nbsp;<em>follower<\/em>&nbsp;index pulls data from the&nbsp;<em>remote<\/em>&nbsp;or&nbsp;<em>leader<\/em>&nbsp;index.<\/li><\/ul><\/li><li><strong> Remote reindex:<\/strong><ul><li>It lets you copy indexes from one Amazon OpenSearch Service domain to another.<\/li><li>You can use it to migrate indexes from one domain to another.<\/li><\/ul><\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Security<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>Encryption at rest (except for manual snapshots)<\/li><li>Encryption in flight, i.e. node-to-node encryption<\/li><li>Resource-based policy &#8211; specify which actions a principal can perform on the domain&#8217;s&nbsp;<em>subresources<\/em><\/li><li>Identity-based policy<\/li><li>IP-based policy &#8211; restrict access to a domain to one or more IP addresses or CIDR blocks<\/li><li>Dashboard access control via:<ul><li>Cognito<\/li><li>SAML<\/li><li>Fine-grained access control with HTTP basic authentication<\/li><li>IP-based policy <\/li><li>Access to a domain that is inside a VPC can be either via Reverse Proxy, Direct Connect, VPN, or Cognito<\/li><\/ul><\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong><span style=\"color:#2ceb0e\" class=\"has-inline-color\">Hands-On<\/span><\/strong><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\"><span class=\"has-inline-color has-ast-global-color-0-color\">Stream a CloudWatch Log to Amazon OpenSearch<\/span><\/h4>\n\n\n\n<h6 class=\"wp-block-heading\">In this hands-on, we will stream a Cloudwatch log from a Lambda function to an Amazon OpenSearch domain.<\/h6>\n\n\n\n<ul class=\"wp-block-list\"><li>Create a domain on a managed cluster:<ul><li>Use the instance type <em>t3.small.search<\/em> to be eligible for a Free Tier.<\/li><\/ul><ul><li>Place the cluster inside a VPC.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"350\" class=\"wp-image-1247\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-54-13.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-54-13.png 1628w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-54-13-300x150.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-54-13-1024x512.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-54-13-768x384.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-54-13-1536x768.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><\/li><li>For the <em>Access Policy<\/em>, change to &#8216;Allow All&#8217;.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"346\" class=\"wp-image-1248\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-55-22.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-55-22.png 1627w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-55-22-300x148.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-55-22-1024x507.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-55-22-768x380.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-55-22-1536x760.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><\/li><\/ul><\/li><li>Since the OpenSearch nodes are inside a  VPC, we need to create a <em><strong>reverse proxy <\/strong><\/em>server that will forward our request to the OpenSearch nodes from outside.<ul><li>Launch an EC2 instance with a <strong>public IP<\/strong> in the same VPC and subnet as the OpenSearch nodes <\/li><li>Install <em>Nginx<\/em> in the EC2 instance.<\/li><li>Configure Nginx as a reverse proxy by modifying its configuration file (nginx.conf). Set the value of the <em>proxy_pass<\/em> to the OpenSearch &#8216;Domain endpoint&#8217;<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"321\" class=\"wp-image-1250\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-56-34.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-56-34.png 1894w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-56-34-300x137.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-56-34-1024x469.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-56-34-768x352.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-56-34-1536x704.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><\/li><li>Start the Nginx server.<\/li><\/ul><\/li><li>Test to see if you can connect to the OpenSearch dashboard:<ul><li>Ensure that the Nginx EC2 instance security group allows access to port 80 (HTTP).<\/li><li>From your browser, connect to the URL <em>http:\/\/<strong>&lt;ec2_public_ip&gt;<\/strong>\/_dashboards<\/em>.<\/li><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"338\" class=\"wp-image-1251\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-55-25.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-55-25.png 1920w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-55-25-300x145.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-55-25-1024x494.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-55-25-768x370.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-55-25-1536x741.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><\/li><li>Create a <em>Amazon OpenSearch Subscription Filter<\/em> in a CloudWatch logroup.<ul><li>For this hands-on, I used a Lambda CloudWatch log group.<\/li><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"368\" class=\"wp-image-1254\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-57-29.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-57-29.png 1615w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-57-29-300x158.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-57-29-1024x538.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-57-29-768x403.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-57-29-1536x807.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"368\" class=\"wp-image-1256\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-57-49-1.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-57-49-1.png 1625w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-57-49-1-300x158.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-57-49-1-1024x539.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-57-49-1-768x404.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-15-57-49-1-1536x808.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><\/li><li>Test the streaming:<ul><li>Generate a new log in the log group.<\/li><li>Check if a new index is created in the OpenSearch domain.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"338\" class=\"wp-image-1259\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-07-20.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-07-20.png 1917w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-07-20-300x145.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-07-20-1024x494.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-07-20-768x371.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-07-20-1536x741.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><\/li><li>Query the streamed data from the Dashboard and compare it with the CloudWatch log.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"319\" class=\"wp-image-1258\" style=\"width: 700px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-06-30.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-06-30.png 1912w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-06-30-300x137.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-06-30-1024x467.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-06-30-768x350.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/11\/Screenshot-from-2024-11-21-16-06-30-1536x701.png 1536w\" sizes=\"auto, (max-width: 700px) 100vw, 700px\" \/><\/li><\/ul><\/li><\/ul><\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[11],"tags":[],"class_list":["post-1236","post","type-post","status-publish","format-standard","hentry","category-aws-review-notes"],"_links":{"self":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1236","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1236"}],"version-history":[{"count":11,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1236\/revisions"}],"predecessor-version":[{"id":1392,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/1236\/revisions\/1392"}],"wp:attachment":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1236"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1236"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1236"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}