Python in the Cloud

[T] Connect your AWS Lambda Function to RDS the PROPER way with AWS Secrets Manager

So you’re finally ready to take your AWS Lambda functions to the next level: connecting them to your database. But how do you do it the secure, proper way in the Serverless Framework?

You’ve setup your database and prepared your connect statement at the top of your handler.py file:

import psycopg2

connection_parameters = {
    'host': 'localhost',
    'database': 'postgres',
    'user': 'user',
    'password': 'NOT-SECURE-PASSWORD'
}
conn = psycopg2.connect(**connection_parameters)
conn.autocommit = True

def handler(event, context):
    try:
        with conn.cursor() as cursor:
            cursor.execute("SELECT * FROM ...")
            rows = cursor.fetchall()
            ...
    except psycopg2.Error as e:
        ...

But wait! Your precious database password is stored in plaintext at the top of your file for everyone to see!

Why is this a problem? Once you commit your files to GIT (or any kind of version control) they are there for everyone else who has access to your repository to see. Not only will all of your fellow developers have access to the credentials, but if your GIT repository was ever hacked or leaked your passwords will be trivial to steal!

What is the proper way to store secrets in the Serverless Framework?

AWS Secrets Manager to the Rescue

Luckily AWS has come to the rescue with the AWS Secrets Manager. The AWS Secrets Manager allows you to securely store your database passwords (or any other secrets such as API keys) inside AWS itself.

When your application needs a secret, it requests it from the AWS Secrets Manager and responds with the secret if you have the correct IAM permissions. This means that:

• Only the call to the AWS Secrets Manager is stored inside your GIT repository and not the password itself.
• Only users and roles who you have explicitly given IAM Secrets Manager permission to can read your password

Setup a new AWS Secrets Manager Secret

To get started with the AWS Secrets Manager, head over to the AWS Console and navigate to the AWS Secrets Manager

Click the orange “Store a new secret” on the right

Choose “Other type of secret”. You’ll notice that this is just a JSON key/value dictionary.

Enter your database username and password

Enter a name and description for this secret. Note The name cannot be changed later

Disable Automatic Rotation

And finally don’t forget to click the Store button on the bottom

Now that we have the Secret created, open it up and note the Secret ARN — we’ll need that below

Using the AWS Secrets Manager From Python

Now that we have the AWS Secrets Manager setup, let’s see how we actually hook it into our Serverless Python project.

In handler.py we want to request the secret from the AWS Secrets Manager, and then parse the SecretString as a JSON object:

import boto3
import json
import psycopg2

# Setup our secrets manager
secrets_manager = boto3.client('secretsmanager')
rds_credentials = json.loads(
    secrets_manager.get_secret_value(
          SecretId='rds-credentials'
    )['SecretString']
)
username = rds_credentials['username']
password = rds_credentials['password']

# Setup our Postgres connection
connection_parameters = {
    'host': 'localhost',
    'database': 'postgres',
    'user': username,
    'password': password
}
conn = psycopg2.connect(**connection_parameters)
conn.autocommit = True

def handler(event, context):
    try:
        with conn.cursor() as cursor:
            cursor.execute("SELECT * FROM ...")
            rows = cursor.fetchall()
            ...
    except psycopg2.Error as e:
        ...

In serverless.yml we want to grant IAM permissions to get this Secret’s value when running inside of AWS Lambda:

provider:
  name: aws
  runtime: python3.7
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "secretsmanager:GetSecretValue"
      Resource: "<SECRET ARN FROM ABOVE>"

Next Steps?

Looking for a complete working Serverless Python example project with both an RDS Database and AWS Secrets Manager programmatically defined in serverless.yml?