{"id":825,"date":"2024-09-26T09:18:42","date_gmt":"2024-09-26T09:18:42","guid":{"rendered":"https:\/\/192.168.1.3\/wordpress\/?p=825"},"modified":"2024-10-21T12:20:46","modified_gmt":"2024-10-21T12:20:46","slug":"aws-certified-data-engineer-associate-dea-c01-review-material-migration","status":"publish","type":"post","link":"https:\/\/mylinuxsite.com\/wordpress\/?p=825","title":{"rendered":"AWS Certified Data Engineer Associate (DEA-C01) Review Material \u2013 Data Sync"},"content":{"rendered":"\n<!--more continue reading-->\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Data Sync<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>An online data transfer and discovery service that simplifies data migration<\/li><li>Transfer data to, from, and between AWS storage services.<\/li><li>Source\/Destinations:<ol><li>On-Prem<ul><li>NFS<\/li><li>SMB<\/li><li>HDFS<\/li><li>Object Storage<\/li><\/ul><\/li><li>AWS<ul><li>S3<\/li><li>EFS<\/li><li>Fsx (Windows File Server, Lustre, Ontap, Open ZFS)<\/li><\/ul><\/li><li>Other Cloud Storage<ul><li>GCP<\/li><li>Azure<\/li><li>Oracle<\/li><li>Alibaba<\/li><li>etc<\/li><\/ul><\/li><\/ol><\/li><li>Most situations that require a DataSync agent involve storage that you or another cloud provider manages, such as on-prem or other cloud providers.<\/li><li>Replication is <strong>scheduled<\/strong>. Not automatic.<\/li><li>File permission and metadata are preserved.<\/li><li>If bandwidth is limited, you can use AWS Snowcone to synchronise data. Snowcone comes with a pre-installed Data Sync agent.<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong><span style=\"color:#4ee30a\" class=\"has-inline-color\">Hands-On<\/span><\/strong><\/h3>\n\n\n\n<p>Note: <em>In this hands-on, we will run the agent on our local machine using Oracle Virtual Box.<\/em><\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Download the Agent. Links will be provided when you create an agent. Select the KVM Hypervisor.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"500\" height=\"456\" class=\"wp-image-836\" style=\"width: 500px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-1.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-1.png 888w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-1-300x274.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-1-768x701.png 768w\" sizes=\"auto, (max-width: 500px) 100vw, 500px\" \/><\/li><\/ul><\/li><li>The agent will be in QCOW format. We must convert the file into VDI to run it in Oracle Virtual Box.<ul><li>$ <code>qemu-img convert -O vdi aws-datasync-2.0.1727187542.1-x86_64.xfs.gpt.qcow2 aws-datasync.vdi<\/code><\/li><\/ul><\/li><li>Create a new VM in the Oracle Virtual Box using the VDI. <ul><li><img loading=\"lazy\" decoding=\"async\" width=\"500\" height=\"433\" class=\"wp-image-837\" style=\"width: 500px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-vb.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-vb.png 964w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-vb-300x260.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-vb-768x665.png 768w\" sizes=\"auto, (max-width: 500px) 100vw, 500px\" \/><\/li><\/ul><\/li><li>Start the Agent VM<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"500\" height=\"337\" class=\"wp-image-838\" style=\"width: 500px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-2.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-2.png 720w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-2-300x202.png 300w\" sizes=\"auto, (max-width: 500px) 100vw, 500px\" \/><\/li><\/ul><\/li><li>Perform the Network Connectivity Test (<strong>Option 2<\/strong>)  to ensure your agent can connect to AWS.<\/li><li>Continue with the agent creation (Step 1). Under the option <em>Activation Key<\/em>, choose <em>Manually enter your agent\u2019s activation key<\/em>.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"500\" height=\"213\" class=\"wp-image-840\" style=\"width: 500px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-create-agent-1.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-create-agent-1.png 805w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-create-agent-1-300x128.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-create-agent-1-768x327.png 768w\" sizes=\"auto, (max-width: 500px) 100vw, 500px\" \/><\/li><\/ul><\/li><li>The activation key can be obtained from your Agent VM. To display it, choose option &#8216;0&#8217;.<\/li><li>Complete the Agent creation.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"800\" height=\"144\" class=\"wp-image-841\" style=\"width: 800px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-create-2.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-create-2.png 1881w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-create-2-300x54.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-create-2-1024x185.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-create-2-768x138.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-create-2-1536x277.png 1536w\" sizes=\"auto, (max-width: 800px) 100vw, 800px\" \/><\/li><\/ul><\/li><li>Spin another VM in your Virtual Box with an NFS server running. Ensure your Agent&#8217;s and NFS servers communicate on the same network. You can test this by choosing <strong>Option 3(<\/strong>Test Connectivity to a Self-Managed Storage) in the Agent&#8217;s VM, then <strong>Option 1<\/strong> (NFS Server). In my test environment, the IP address of the NFS server is 10.0.2.6.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"409\" class=\"wp-image-846\" style=\"width: 600px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-3.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-3.png 720w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-agent-3-300x205.png 300w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/li><\/ul><\/li><li>In this hands-on activity, the NFS server shares the directory <strong>\/scratch.<\/strong><ul><li><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"287\" class=\"wp-image-845\" style=\"width: 600px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-nfs-server.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-nfs-server.png 1624w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-nfs-server-300x143.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-nfs-server-1024x489.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-nfs-server-768x367.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-nfs-server-1536x734.png 1536w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/li><\/ul><\/li><li>Create <strong>two(2) locations<\/strong>. One location is the NFS server, which is the data source. The other is the S3 bucket, which is the data destination.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"800\" height=\"145\" class=\"wp-image-848\" style=\"width: 800px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-locations.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-locations.png 1875w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-locations-300x55.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-locations-1024x186.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-locations-768x140.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-locations-1536x279.png 1536w\" sizes=\"auto, (max-width: 800px) 100vw, 800px\" \/><\/li><\/ul><\/li><li>Create a <strong>task<\/strong> using the NFS location as the source, the S3 as the destination and the agent as the agent you created in <strong>Step 1<\/strong>.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"273\" class=\"wp-image-993\" style=\"width: 600px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/10\/aws-data-sync-task-2.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/10\/aws-data-sync-task-2.png 1860w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/10\/aws-data-sync-task-2-300x136.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/10\/aws-data-sync-task-2-1024x466.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/10\/aws-data-sync-task-2-768x349.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/10\/aws-data-sync-task-2-1536x699.png 1536w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/li><\/ul><\/li><li>Create a file under the \/scratch folder of your NFS Server<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"500\" height=\"174\" class=\"wp-image-851\" style=\"width: 500px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-nfs-file.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-nfs-file.png 1342w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-nfs-file-300x104.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-nfs-file-1024x356.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-nfs-file-768x267.png 768w\" sizes=\"auto, (max-width: 500px) 100vw, 500px\" \/><\/li><\/ul><\/li><li>Start your task. After a few minutes, you will see the file you created in the NFS server copied to the S3 bucket.<ul><li><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"142\" class=\"wp-image-852\" style=\"width: 600px;\" src=\"http:\/\/192.168.1.3\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-s3.png\" alt=\"\" srcset=\"https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-s3.png 1543w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-s3-300x71.png 300w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-s3-1024x243.png 1024w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-s3-768x182.png 768w, https:\/\/mylinuxsite.com\/wordpress\/wp-content\/uploads\/2024\/09\/aws-data-sync-s3-1536x364.png 1536w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/li><\/ul><\/li><\/ol>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[11],"tags":[],"class_list":["post-825","post","type-post","status-publish","format-standard","hentry","category-aws-review-notes"],"_links":{"self":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/825","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=825"}],"version-history":[{"count":26,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/825\/revisions"}],"predecessor-version":[{"id":1051,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/825\/revisions\/1051"}],"wp:attachment":[{"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=825"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=825"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mylinuxsite.com\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=825"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}