Encrypt all of your S3 Buckets automatically


Here is a quick guide on how to implement a lambda function which would scan all of your S3 Buckets and set the server side encryption automatically. Once completed, an email would be triggered to indicate which S3 Bucket was modified. This configuration is using Terraform 0.12 version to deploy this project.

This documentation is based on this following GitHub repository, any questions, don’t hesitate to raise it.

I also keep my blog updated with new articles with my consulting company, you can check at https://cdcloudlogix.com/blog for more information :)

I. Terraform Providers

Start by configuring your main AWS account as a Terraform provider as follow:

Once completed, add your Terraform backend:

Don’t forget to modify the S3 bucket name as well as the key name based on your situation.

II. Modules

This project is using a unique module:

  • s3-scan-encryption

This module would install the following resources:

  • IAM roles and policies
  • CloudWatch Events
  • Lambda
  • SNS Notification

III. Lambda function

This project is using a bash script for installing / testing and zipping our lambda function:

Before applying Terraform plan, the zip file would have to be present on the local file system.

Prepare a virtual environment for python before running this on your terminal:

$ virtualenv -p python3 venv
Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix '/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7'
New python executable in /Users/oli/Documents/venv/bin/python3.7
Also creating executable in /Users/oli/Documents/venv/bin/python
Installing setuptools, pip, wheel...
$ source venv/bin/activate

IV. SSM Parameters

To avoid exposing email addresses in a GitHub repository, this project is making the use of the following SSM parameters:

  • s3-encryption-exception-list: List of S3 Buckets excluded from Scan(StringList type)
  • default-source-email: Default name used as a source (String type)
  • default-destination-emails: List of email recipients (StringList type)

Regarding the configuration of each of these SSM parameters, have a look below on these screenshots:

V. Deploy

Package your lambda script first (Truncated output):

19:23 $ ./python_packages.sh
Requirement already satisfied: pytest in /Users/oli/Documents/terraform_s3_encryption_email/venv/lib/python3.7/site-packages (from -r requirements_test.txt (line 1)) (5.4.1)
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
~/Documents/terraform_s3_encryption_email/modules/s3-scan-encryption ~/Documents/terraform_s3_encryption_email/modules
adding: s3_encryption.py (deflated 69%)

Then execute terraform init command (Truncated output):

11:51 $ terraform init
Initializing modules...
Initializing the backend...Initializing provider plugins...The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.
* provider.template: version = "~> 2.1"Terraform has been successfully initialized!You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Finally, apply Terraform (Truncated output):

1:51 $ terraform apply
module.s3-encryption-lambda.data.aws_ssm_parameter.display_name: Refreshing state...
module.s3-encryption-lambda.data.aws_caller_identity.current: Refreshing state...
module.s3-encryption-lambda.data.aws_ssm_parameter.s3_encryption_emails: Refreshing state...
module.s3-encryption-lambda.data.template_file.cloudformation_sns_stack: Refreshing state...
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
<= read (data resources)
Terraform will perform the following actions:# module.s3-encryption-lambda.data.aws_iam_policy_document.sns_publish will be read during apply
# (config refers to values not yet known)
<= data "aws_iam_policy_document" "sns_publish" {
+ id = (known after apply)
+ json = (known after apply)
+ statement {
+ actions = [
+ "SNS:Publish",
+ effect = "Allow"
+ resources = [
+ (known after apply),
[...]Plan: 15 to add, 0 to change, 0 to destroy.Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value:

If you’re happy with this configuration, just confirm by typing yes

After deploying, each recipient would receive the following email from AWS, don’t forget to confirm that you wish to subscribe:

Subscription email

VI. Details of this repository

The following lambda function will:

  • List all your existing buckets
  • Compare this list with the exempt list of S3 Buckets present in your SSM parameter
  • Check if KMS or SSE is applied on each of your bucket
  • Apply SSE on your bucket if it wasn’t apply
  • Notify by SNS notification

This lambda would be run from Monday to Friday each week:

Terraform doesn’t support SNS Email configuration natively, hence, the reason why we need to use a Cloudformation template for this purpose:

Combine with the following template file:

VII. Testing

You can test your lambda by using the Test button as follow:

Test configuration

This function would start running and setting up the encryption on each of your existing buckets. Once Completed, you should receive the following email of confirmation:

Cloud technologies and solutions lover, https://cdcloudlogix.com