Hybrid Activation – Manage Non-EC2 machines from AWS System Manager

Leave a Comment / Articles / By

In this article, we will demonstrate how we can manage Non-EC2 on-premise machines from the AWS System Manager using Hybrid Activation.

For this demonstration, our on-premises machine will be a Linux VM running on Oracle VirtualBox. The flavour of Linux that we will use is CentOS 7. But you may use any other Linux flavour. The reason why I am using CentOS 7 is that my Oracle VirtualBox is old and cannot be updated.

In summary, the procedures we need to take are as follows:

  1. Install the SSM Agent on our Linux machine.
  2. Create an IAM Service Role that our machine or the agents in our machine will use.
  3. Create a Hybrid Activation and register our machine with the AWS System Manager.
  4. Install and configure the CloudWatch agent on our Linux machine using the System Manager.
  5. Install AWS CLI on our Linux machine using the System Manager.

1. Install the SSM Agent

There are several ways to install the SSM agent on your on-premise machine. One option, ideal for managing a large number of machines, is to utilise automation tools such as Ansible, Puppet, or Chef. But for this demonstration, we will log in to our machine and enter the following command:

yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm

Refer to this link if you are using a different Linux flavour.

2. Create an IAM Service Role

The IAM service role that we will create will be used by both the SSM agent and the CloudWatch agent, which will be installed later. When we run the registration command, the SSM agent creates a credentials file with a default profile in the root account (or the account where the registration command is run), containing the ACCESS_ID and ACCESS_KEY derived from the IAM service role.

Create a local text file with the following trust policy:

cat <<EOF > MyOnPremServiceRole-Trust.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ssm.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF

Create a role with the above trust policy:

aws iam create-role \
    --role-name MyOnPremServiceRole \
    --assume-role-policy-document file://MyOnPremServiceRole-Trust.json

Attached the policies (1) AmazonSSMManagedInstanceCore and (2) CloudWatchAgentServerPolicy to the role.

aws iam attach-role-policy \
    --role-name MyOnPremServiceRole \
    --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore  

aws iam attach-role-policy \
    --role-name MyOnPremServiceRole \
    --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

3. Create a Hybrid Activation

Create a Hybrid activation using the following command:

aws ssm create-activation \
  --default-instance-name MyOnPremServers \
  --description "Activation for my on-prem servers" \
  --iam-role MyOnPremServiceRole \
  --expiration-date "2025-07-30T00:00:00" \
  --registration-limit 2 \
  --region us-east-1 \
  --tags "Key=ServerOS,Value=CentOS7"

The command should output the ActivationId and the ActivationCode. Use these values to register your machine.

Log in to your server and run the following command:

amazon-ssm-agent -register -code "<ActivationCode>" -id "<ActivationId>" -region "us-east-1"

Verify the Hybrid activation and registration from the console. First, go to the Hybrid Activations panel. You should see an entry similar to the image below:

Then go to the Fleet Manager panel. You should see your instance listed there with a Ping status set to ‘Online’.

4. Install and configure the CloudWatch agent.

To test and demonstrate that the System Manager can now manage our machine, we will install and configure the CloudWatch agent using the System Manager Run Command.

First, we need to install the CloudWatch agent package using the document AWS-ConfigureAWSPackage.

aws ssm send-command \
    --document-name "AWS-ConfigureAWSPackage" \
    --parameters action="Install",installationType="Uninstall and reinstall",name="AmazonCloudWatchAgent",version="latest" \
    --region us-east-1 \ 
    --targets Key=tag:ServerOS,Values=CentOS7

Verify from the Run Command panel that the run is successful.

You can also verify this from the machine itself to confirm that the package is installed.

Next, we need to instruct the CloudWatch agent to use the /root/.aws/credentials as our credential file. We accomplish this by uncommenting the lines in /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml, specifically the [credentials] section and the shared_credential_profile parameter, and assigning a value of ‘default‘.

We will use the AWS-RunShellScript run document for this.

aws ssm send-command \
    --document-name "AWS-RunShellScript" \
    --parameters '{"workingDirectory":["/opt/aws/amazon-cloudwatch-agent/etc"],"executionTimeout":["3600"],"commands":["if ! test -f common-config.toml.bak; then cp common-config.toml common-config.toml.bak; fi","sed -e '"'"'s/\\# \\[credentials\\]/\\[credentials\\]/'"'"' -e '"'"'s/\\#    shared_credential_profile = \\\"{profile_name}\\\"/ shared_credential_profile = \\\"default\\\"/'"'"' common-config.toml.bak > common-config.toml"]}' \
    --timeout-seconds 600 \
    --region us-east-1 \ 
    --targets Key=tag:ServerOS,Values=CentOS7

Verify the result of the run by taking the diff between the original and the modified file.

Next, we need to add a default region to our /root/.aws/credentials. This is necessary because the Run Command document we will execute later to configure and start the CloudWatch agent will read the tokens and region from this file.

aws ssm send-command \
    --document-name "AWS-RunShellScript" \
    --parameters '{"workingDirectory":["/root/.aws"],"executionTimeout":["3600"],"commands":["if ! test -f credentials.bak; then cp credentials credentials.bak; fi","sed '"'"'/\\[default\\]/a\\region = us-east-1'"'"' credentials.bak > credentials"]}' \
    --timeout-seconds 600 \
    --region us-east-1 \ 
    --targets Key=tag:ServerOS,Values=CentOS7

Verify the result of the run by taking the diff between the original and the modified file.

Next, we will create the CloudWatch agent configuration file and store it in the SSM Parameter Store. 

cat <<EOF > OnPremCWAgent-config.json
{
    "agent": {
        "metrics_collection_interval": 60,
        "run_as_user": "root",
        "region": "us-east-1"
    },
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/var/log/secure",
                        "log_group_class": "STANDARD",
                        "log_group_name": "MyOnPrem",
                        "log_stream_name": "{local_hostname}",
                        "retention_in_days": -1
                    }
                ]
            }
        }
    },
	"metrics": {
                "namespace": "MyOnPremNamespace",
		"metrics_collected": {
			"cpu": {
				"measurement": [
					"cpu_usage_idle"
				],
				"metrics_collection_interval": 60,
				"resources": [
					"*"
				],
				"totalcpu": true
			},
			"disk": {
				"measurement": [
					"used_percent"
				],
				"metrics_collection_interval": 60,
				"resources": [
					"*"
				]
			},
			"diskio": {
				"measurement": [
					"write_bytes",
					"read_bytes",
					"writes",
					"reads"
				],
				"metrics_collection_interval": 60,
				"resources": [
					"*"
				]
			},
			"mem": {
				"measurement": [
					"mem_used_percent"
				],
				"metrics_collection_interval": 60
			},
			"net": {
				"measurement": [
					"bytes_sent",
					"bytes_recv",
					"packets_sent",
					"packets_recv"
				],
				"metrics_collection_interval": 60,
				"resources": [
					"*"
				]
			},
			"swap": {
				"measurement": [
					"swap_used_percent"
				],
				"metrics_collection_interval": 60
			}
		}
	}
}
EOF

aws ssm put-parameter \
    --region us-east-1 \
    --name OnPremCWAgent-config \
    --type String \
    --value file://OnPremCWAgent-config.json

Please note that in our configuration file, we will collect the log file /var/log/secure and store it in the log group MyOnPrem. We will also collect metrics, including disk usage, CPU usage, and memory usage and place them under MyOnPremNamespace namespace.

Finally, start the CloudWatch agent with the configuration file we uploaded to the SSM Parameter store. We will use the AmazonCloudWatch-ManageAgent document for this:

aws ssm send-command \
    --document-name "AmazonCloudWatch-ManageAgent" \
    --parameters '{"action":["configure"],"mode":["onPremise"],"optionalConfigurationSource":["ssm"],"optionalConfigurationLocation":["OnPremCWAgent-config"],"optionalRestart":["yes"]}' \
    --timeout-seconds 600 \
    --region us-east-1 \
    --targets Key=tag:ServerOS,Values=CentOS7

Verify the result of the run by checking if the amazon-cloudwatch-agent.service is running on the machine.

Logs under the log group ‘MyOnPrem’ and metrics under the namespace ‘MyOnPremNamespace’ should also be created.

Log Group
Metrics

5. Install AWS CLI on the Linux machine.

For completion, we will install the AWS CLI on our Linux machine using the AWS RunShellScript document.

aws ssm send-command \
    --document-name "AWS-RunShellScript" \
    --parameters '{"workingDirectory":[""],"executionTimeout":["3600"],"commands":["curl \"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip\" -o \"awscliv2.zip\"","unzip awscliv2.zip","./aws/install","",""]}' \
    --timeout-seconds 600 \
    --region us-east-1 \
    --targets Key=tag:ServerOS,Values=CentOS7

Verify the result of the run by checking if the AWS CLI is installed on the machine.

Leave a Comment