Let us assume that your company has a policy that mandates immediate notification and termination of any EC2 instance accessed via SSH. How do you implement this?
In this article, we will explore one solution that makes use of (1) CloudWatch Log Subscription, (2) Lambda and (3) SNS. I believe this option offers more flexibility compared to other options.
How to Detect SSH Access?
There are a few ways to detect SSH access to your Linux machine. One way is to utilize the system log. For this solution, we will use the log file /var/log/secure to detect SSH logins.
For Linux machines that use the systemd service, such as the AMI Linux 2, the log file /var/log/secure may not exist. The reason is that this log file is generated by the rsyslog service, which may not be installed by default. First, we need to install this service. We also need to install the CloudWatch agent, which is not installed by default in the AMI Linux 2, and prepare its configuration file to specify which log group and log stream to send the SSH logs to.
Prepare the EC2 Instance
In summary, we have to make the following preparations for our EC2:
- Install, enable and start the rsyslog service.
sudo yum -y install rsyslog
sudo systemctl enable rsyslog
sudo systemctl start rsyslog- Install the CloudWatch agent.
yum install amazon-cloudwatch-agent- Create the CloudWatch configuration file. You can create the configuration file either manually or by using the configuration wizard. Name the file as config.json.
To start the configuration wizard, enter the following:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizardBelow is a snippet of the configuration file that we use in the sample EC2 resource, which we will build later using a CloudFormation template.
{
"agent": {
"metrics_collection_interval": 60,
"run_as_user": "cwagent"
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/secure",
"log_group_class": "STANDARD",
"log_group_name": "${LogGroupName}",
"log_stream_name": "{instance_id}",
"retention_in_days": -1
}
]
}
}
},
:
:Two of the fields that you need to decide are the (1) run_as_user, which specifies a user to use to run the CloudWatch agent, and (2) log_group_name, which specifies what to use as the log group name in CloudWatch Logs. Since the above configuration is part of a CloudFormation template, and the log_group_name is set as a variable that will be replaced by Fn::Sub with a parameter value.
Please take note that it is essential to keep the value of the log_stream_name to {instance_id}. This value will be used to identify the instance ID where an SSH login took place.
- If you manually created the configuration file, copy it to its proper location and set its owner to root and its access permission to 0644. If you use the configuration wizard, the file should already be in the appropriate location with the right owner and access permission settings.
sudo cp config.json /opt/aws/amazon-cloudwatch-agent/bin/config.json
sudo chmod 0644 /opt/aws/amazon-cloudwatch-agent/bin/config.json
sudo chown root:root /opt/aws/amazon-cloudwatch-agent/bin/config.json- If you set the CloudWatch agent to run as a non-root user (e.g., cwagent), you need to grant access to the /var/log/secure file. You can do this by modifying the file’s ACL.
sudo setfacl -m u:cwagent:rx /var/log/secure- Start the CloudWatch agent.
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.jsonThe Lambda Function
The Lambda function is the service responsible for sending notifications, via SNS, and terminating the EC2 instance. The Lambda function will receive a message from the CloudWatch subscription filter in the following format: { "awslogs": {"data": "BASE64ENCODED_GZIP_COMPRESSED_DATA"} }
The Lambda function has to decompress the data payload, which will yield the raw data formatted as JSON with the following structure:
{
"messageType": "DATA_MESSAGE",
"owner": "123456789012",
"logGroup": "/aws/ec2/ssh-attempts",
"logStream": "i-0ffe4156be759fb9e",
"subscriptionFilters": [
"SshAccessFilter"
],
"logEvents": [
{
"id": "39081747012568859428767625513120880010897834134375366656",
"timestamp": 1752486146299,
"message": "Jul 14 09:42:21 ip-172-31-34-86 sshd[2374]: pam_unix(sshd:session): session opened for user ec2-user(uid=1000) by (uid=0)",
"extractedFields": {
"mm": "Jul",
"dd": "14",
"ip": "ip-172-31-34-86",
"text1": "session",
"text2": "opened for user ec2-user(uid=1000) by (uid=0)",
"time": "09:42:21",
"cmd": "sshd[2374]:",
"pam": "pam_unix(sshd:session):"
}
}
]
}The key elements that will be used by the Lambda function in this raw data are:
- logStream
This field will contain the instance ID that generates the log, and the Lambda function will use this to identify the EC2 instance that needs to be terminated. This is the result of setting the log_stream_name field in the CloudWatch configuration file to the value of {instance_id}. - message
This field will provide information such as the source IP, the date and time of access, and the user who logged in.
The CloudWatch Subscription Filter
The CloudWatch subscription filter will not send the entire log to the Lambda function. Instead, only a particular log record is sent. This record is a record of a successful login and is identified by the string pattern “opened for user*“. The CloudWatch subscription filter will apply the following filter pattern:
[mm, dd, time, ip, cmd, pam="pam_unix(sshd:session):", text1=session, text2="opened for user*" ]The Sample Infrastructure
This CloudFormation template creates the sample infrastructure that demonstrates the solution described above. The template will create the following AWS services:
- An EC2 instance with a public IP using an AMI Linux 2. The template will automatically install the CloudWatch agent and the rsyslog service. It will also copy a CloudWatch agent configuration file. The EC2 instance profile will be configured to allow you to connect to the instance using the System Manager.
- A Lambda function. Note that the Lambda function will not terminate the instance but only stop it. The reason is to allow you to perform multiple tests on the same instance.
- A CloudWatch log group
- A CloudWatch subscription filter.
The template is written using the SAM model and must therefore be deployed accordingly. Hence, to deploy the template, use the following command:
sam build
sam deploy --parameter-overrides ParameterKey=SubnetId,ParameterValue=subnet-12345 ParameterKey=SecurityGroupId,ParameterValue=sg-12345 ParameterKey=EC2Name,ParameterValue=TestEC2 ParameterKey=SNSTopic,ParameterValue=MySNS The template expects the following parameters to be supplied:
- SubnetId – This subnet must be a public subnet.
- SecurityGroupId – The security group must open port 22 for SSH access.
- EC2Name – The name of the EC2 where an SSH will be performed.
- SNSTopic – The SNS topic where the notification will be sent. The SNS topic is not created in the template. You have to create it separately.
The rest of the parameters are optional and will have the following default value if not supplied:
- AmiId – default to al2023-ami-kernel-6.1-x86_64
- LogGroupName – default to /aws/ec2/ssh-attempts
- LogLevel – This is the Lambda log level, which defaults to INFO.
Testing the Sample Infrastructure
To test the infrastructure, connect to the EC2 using the EC2 Instance Connect. You cannot use an SSH client because the template will not generate any key pairs.
After a couple of seconds, your EC2 instance state should change to ‘stopping’.
To check the notification, create an email subscription or an SQS subscription.
