Encrypt all of your S3 Buckets automatically
Here is a quick guide on how to implement a lambda function which would scan all of your S3 Buckets and set the server side encryption automatically. Once completed, an email would be triggered to indicate which S3 Bucket was modified. This configuration is using Terraform 0.12 version to deploy this project.
This documentation is based on this following GitHub repository, any questions, don’t hesitate to raise it.
I also keep my blog updated with new articles with my consulting company, you can check at https://cdcloudlogix.com/blog for more information :)
I. Terraform Providers
Start by configuring your main AWS account as a Terraform provider as follow:
Once completed, add your Terraform backend:
Don’t forget to modify the S3 bucket name as well as the key name based on your situation.
II. Modules
This project is using a unique module:
- s3-scan-encryption
This module would install the following resources:
- IAM roles and policies
- CloudWatch Events
- Lambda
- SNS Notification
III. Lambda function
This project is using a bash script for installing / testing and zipping our lambda function:
Before applying Terraform plan, the zip file would have to be present on the local file system.
Prepare a virtual environment for python before running this on your terminal:
$ virtualenv -p python3 venv
Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix '/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7'
New python executable in /Users/oli/Documents/venv/bin/python3.7
Also creating executable in /Users/oli/Documents/venv/bin/python
Installing setuptools, pip, wheel...
done.$ source venv/bin/activate
IV. SSM Parameters
To avoid exposing email addresses in a GitHub repository, this project is making the use of the following SSM parameters:
- s3-encryption-exception-list: List of S3 Buckets excluded from Scan(StringList type)
- default-source-email: Default name used as a source (String type)
- default-destination-emails: List of email recipients (StringList type)
Regarding the configuration of each of these SSM parameters, have a look below on these screenshots:
V. Deploy
Package your lambda script first (Truncated output):
19:23 $ ./python_packages.sh
Requirement already satisfied: pytest in /Users/oli/Documents/terraform_s3_encryption_email/venv/lib/python3.7/site-packages (from -r requirements_test.txt (line 1)) (5.4.1)[...]--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)~/Documents/terraform_s3_encryption_email/modules
~/Documents/terraform_s3_encryption_email/modules/s3-scan-encryption ~/Documents/terraform_s3_encryption_email/modules
adding: s3_encryption.py (deflated 69%)
~/Documents/terraform_s3_encryption_email/modules
Then execute terraform init command (Truncated output):
11:51 $ terraform init
Initializing modules...Initializing the backend...Initializing provider plugins...The following providers do not have any version constraints in configuration,
so the latest version was installed.To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.* provider.template: version = "~> 2.1"Terraform has been successfully initialized!You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Finally, apply Terraform (Truncated output):
1:51 $ terraform apply
module.s3-encryption-lambda.data.aws_ssm_parameter.display_name: Refreshing state...
module.s3-encryption-lambda.data.aws_caller_identity.current: Refreshing state...
module.s3-encryption-lambda.data.aws_ssm_parameter.s3_encryption_emails: Refreshing state...
module.s3-encryption-lambda.data.template_file.cloudformation_sns_stack: Refreshing state...An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
<= read (data resources)Terraform will perform the following actions:# module.s3-encryption-lambda.data.aws_iam_policy_document.sns_publish will be read during apply
# (config refers to values not yet known)
<= data "aws_iam_policy_document" "sns_publish" {
+ id = (known after apply)
+ json = (known after apply)+ statement {
+ actions = [
+ "SNS:Publish",
]
+ effect = "Allow"
+ resources = [
+ (known after apply),
]
}
}[...]Plan: 15 to add, 0 to change, 0 to destroy.Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.Enter a value:
If you’re happy with this configuration, just confirm by typing yes
After deploying, each recipient would receive the following email from AWS, don’t forget to confirm that you wish to subscribe:
VI. Details of this repository
The following lambda function will:
- List all your existing buckets
- Compare this list with the exempt list of S3 Buckets present in your SSM parameter
- Check if KMS or SSE is applied on each of your bucket
- Apply SSE on your bucket if it wasn’t apply
- Notify by SNS notification
This lambda would be run from Monday to Friday each week:
Terraform doesn’t support SNS Email configuration natively, hence, the reason why we need to use a Cloudformation template for this purpose:
Combine with the following template file:
VII. Testing
You can test your lambda by using the Test button as follow:
This function would start running and setting up the encryption on each of your existing buckets. Once Completed, you should receive the following email of confirmation:
