Deploying a Serverless Application on AWS with Python and Terraform

Overview

Want to automate the deployment of serverless endpoints, but desire more low-level control than you get with the Serverless Framework? Look no further than Terraform! This tutorial will go over using Terraform to deploy an API Gateway and Lambda function on AWS.

Other tutorials on this subject tend to proxy through all requests to the Lambda, but I wanted to write one that gives you fine-grained control over the endpoints like you'd get with the Serverless Framework.

This tutorial is broken into 5 sections

1. Initial Setup
2. A Basic Lambda Function
3. Adding the API Gateway
4. An Endpoint with User Input and External Dependencies
5. Tearing Things Down
6. Closing Thoughts

IF YOU DECIDE TO END THE TUTORIAL PARTWAY THROUGH, BE SURE TO COMPLETE SECTION 5 BEFORE YOU FINISH

Tools

The tools used in this project are

• Python3.8
• Terraform 0.12.25

A Note on Costs

AWS offers a generous free-tier for Lambda and API Gateway, so nothing you do in this project should cost you any money. However, if what you build starts attracting traffic, you could eventually be billed for your use of AWS resources. When you finish the tutorial, or if you decide to stop mid-way through, be sure to complete Section 5: Tearing Things Down, to ensure that you never get charged. Finally, there is a chance that AWS has removed the free tiers for Lambda and API Gateway. This post was written on May 27th, 2020. Please consult the Lambda and API Gateway pricing pages to confirm that the free-tier is still present

1. Initial Setup

Note: All instructions that start with >>> are meant to be run as is in the terminal.

Download Python3.8 and Terraform 0.12.25. Create an AWS Account, then download and configure the AWS CLI.

Next, we will set up the project directory:

>>> mkdir serverless_example
>>> cd serverless_example
>>> mkdir src
>>> mkdir terraform

Now, just start your preferred code editor in the serverless_example directory, and we can get to work!

All our Python code will go in the src directory and all our Terraform configuration will go in the terraform directory.

2. A Basic Lambda Function

Function Code

In the src directory make another directory called hello, and then create a file called hello.py in the hello directory with the following code.

# src/hello/hello.py
import json


def my_handler(event, context):
    return {
        "statusCode": 200,
        "headers": {"Content-Type": "application/json"},
        "body": json.dumps("Hello!"),
    }

We are going to configure the Lambda to call my_handler when it is invoked. The event and context arguments are what Lambda is going to pass to the function, though we don't have to worry about them for now. Our my_handler function simply returns a 200 http status code with a nice "Hello" message as the body.

Lambda Configuration

Our infrastructure will have several components.

1. A CloudWatch log group for the function to send logs to
2. An IAM Role for the Lambda to run as, that gives it permission to send logs to the log group
3. The Lambda function itself

Create a file called main.tf in the terraform directory.

We'll start with creating the CloudWatch log group.

Add the following code to main.tf:

# terraform/main.tf
resource "aws_cloudwatch_log_group" "hello" {
  name              = "/aws/lambda/hello"
  retention_in_days = 14
}

This block is relatively straight forward. It creates a log group at the /aws/lambda/hello path, and sets the retention period to 14 days.

Note: the value of the name parameter here is actually quite important. When the Lambda runs it will attempt to send logs to the log group at /aws/lambda/<Name of the lambda>, so our lambda will have to have the name hello as well.

Next, we'll create the IAM Role that has permissions to send logs to the log group we just created.

Add the following code to main.tf. It doesn't matter where. In HCL (the language you configure Terraform in) the order of blocks is not relevant.

data "aws_iam_policy_document" "hello_assume_role" {
  statement {
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "hello" {
  name                 = "hello"
  assume_role_policy   = data.aws_iam_policy_document.hello_assume_role.json
}

All IAM roles in AWS are required to have an assume role policy, which dictates what sorts of entities are allowed to use the role. So, the first thing we make is the aws_iam_policy_document, hello_assume_role. This says "If a role uses this policy as its assume_role_policy, only Lambdas are allowed to assume it."

Next, we create the role itself. We call it hello, and set its assume_role_policy to be the policy we just defined.

Now that the role is created, we will add the permissions that allow it to send logs to the CloudWatch log group.

Add the following code to main.tf:

data "aws_iam_policy_document" "hello_cloudwatch_logging" {
  statement {
    effect = "Allow"
    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = [aws_cloudwatch_log_group.hello.arn]
  }
}

resource "aws_iam_policy" "hello_cloudwatch_logging" {
  name        = "hello-logging"
  description = "Allows the Lambda to log to cloudwatch"
  policy      = data.aws_iam_policy_document.hello_cloudwatch_logging.json
}

resource "aws_iam_role_policy_attachment" "hello_cloudwatch_logging" {
  role       = aws_iam_role.hello.name
  policy_arn = aws_iam_policy.hello_cloudwatch_logging.arn
}

In this section we first define the permissions we want our IAM policy to have in the "aws_iam_policy_document" block. We give it permission to create log streams, and then put log events into those streams. We only allow it to exercise these permissions on the log group we created earlier, thereby only allowing the Lambda to do exactly what it needs to do, following the principal of least privilege.

Next, we create an aws_iam_policy with those permissions.

Finally, we attach the aws_iam_policy to the role we created earlier using a aws_iam_role_policy_attachment

The end result is a role that the Lambda can run as, that will give it permission to send logs to the log group we created for it.

Note: the data block can be a little confusing. This block isn't actually creating anything in AWS, but instead lets us define the policy in a nice format, before passing it to the aws_iam_policy resource as JSON.

Now, we can finally create the Lambda function itself.

Add the following code to main.tf:

data "archive_file" "lambda_package" {
  type        = "zip"
  source_dir  = "../src/hello"
  output_path = "../hello_package.zip"
}

resource "aws_lambda_function" "hello" {
  filename          = "../hello_package.zip"
  function_name     = "hello"
  role              = aws_iam_role.hello.arn
  handler           = "hello.my_handler"
  timeout           = 10
  runtime           = "python3.8"
  source_code_hash  = data.archive_file.lambda_package.output_base64sha256  
}

The aws_lambda_function resource expects an archive file to be passed to filename. Luckily, we can use Terraform to create that archive!

So, we first add an archive_file data source, and then we tell Terraform to look at it for the Lambda's code.

We also set the function name to hello. If we later changed this to something else, say my_new_function, we would also want to change the name of the CloudWatch log group from earlier to /aws/lambda/my_new_function, so that the logging would stay correctly configured.

Next, we set the Lambda's role to be the role we created earlier.

Then we tell AWS where to start executing our code. The format is the filename, hello for hello.py, followed by the function to call within that file, my_handler.

We set the timeout to 10 seconds (if you have a particularly long or short running Lambda this can be changed), and tell AWS to use the Python3.8 runtime.

Finally, we add the source_code_hash parameter. We include this so that terraform knows to update the lambda function when the code changes. If it wasn't there, when we changed the python code terraform wouldn't update the lambda, as as terraform's eyes everything would still be the same. Now, whenever the pythonn code changes, so will its hash, and terraform will know to issue an update.

Now our Lambda is fully configured! Head back to the terminal and we can start actually provisioning resources.

Applying The Terraform

First, cd to the terraform directory.

Next, run

>>> terraform --version

You should see it output

Terraform v0.12.25

If you see an error message, you need to install Terraform, and ensure that the directory containing the executable is in your path. If you see a different version of Terraform, you can continue with the tutorial, but things may not work as expected, or at all. I suggest that you download and use Terraform 0.12.25, to ensure that no problems are caused by a version mismatch.

Next, run:

>>> terraform init
>>> terraform apply

You should see a list of the resources Terraform intends to create, followed by a prompt asking you to confirm their creation.

Type yes at the prompt, and Terraform will start provisioning the resources for you.

Testing the Lambda (optional)

(Note: The layout of the AWS console may have changed since the time of this writing, causing these instructions to become inaccurate)

If your Terraform applied error-free, great! We can now play with our new Lambda function.

If not, try and parse the error message that Terraform threw at you, and go back and make sure your syntax matches the examples.

Open up the AWS console in your browser, and navigate to Services > Lambda > hello.

Click Test, give the default test event a name, and click Create.

Then, click Test, and you should see your lovely Lambda function say Hello! (If you want you can even check out the logs in CloudWatch)

Great! We're done with part 2.

3. Adding the API Gateway

Provisioning Resources

First, we'll add the API Gateway itself.

Add the following code to main.tf:

resource "aws_api_gateway_rest_api" "api_example" {
  name        = "api_example"
  description = "An example REST API"
}

This api_gateway resource will serve as the base as we add resources and the methods to access them.

Add the following code to main.tf:

resource "aws_api_gateway_resource" "hello" {
  rest_api_id = aws_api_gateway_rest_api.api_example.id
  parent_id   = aws_api_gateway_rest_api.api_example.root_resource_id
  path_part   = "hello"
}

resource "aws_api_gateway_method" "hello" {
  rest_api_id      = aws_api_gateway_rest_api.api_example.id
  resource_id      = aws_api_gateway_resource.hello.id
  http_method      = "GET"
  authorization    = "NONE"
}

This section adds an aws_api_gateway_resource at the path hello, as a child of the API Gateway's root resource. The means that hello will be accessible at a url like example-api-domain.amazon.aws.com/hello.

We then define a method that acts on that resource. The method allows GET requests at the hello path, and does not require any authorization.

Now that we have defined a resource and allowed a method on it, we need to tell AWS how to route requests for that resource. In our case, we want to send requests to our Lambda function.

Add the following code to main.tf:

resource "aws_api_gateway_integration" "hello" {
  rest_api_id = aws_api_gateway_rest_api.api_example.id
  resource_id = aws_api_gateway_resource.hello.id
  http_method = "GET"

  integration_http_method = "POST"
  type                    = "AWS_PROXY"
  uri                     = aws_lambda_function.hello.invoke_arn
}

This section says "For any GET request for the hello resource, send it to the hello Lambda function."

Any Lambda integration will have the same integration_http_method and type parameters. For Lambda integrations, AWS requires those two values to be POST and AWS_PROXY, respectively.

We're almost done configuring the API Gateway, and in fact only have two Terraform resources left to define.

First, we need to add a deployment resource. AWS allows you to have multiple deployments for each API Gateway (think dev, prod, stage, etc.). So, to start using out gateway, we need to add its own deployment.

Add the following code to main.tf:

resource "aws_api_gateway_deployment" "api_example" {
  depends_on = [
    aws_api_gateway_integration.hello
  ]

  rest_api_id = aws_api_gateway_rest_api.api_example.id
  stage_name = "prod"
}

I've decided to just name our deployment prod, as using multiple deployments is outside the scope of this tutorial.

We add the depends_on block because AWS will throw an error if we try to make a deployment on an API with no methods. So, we make sure the integration for the method is fully created before attempting to add the deployment.

Finally, we need to grant the API Gateway permission to invoke the Lambda function.

Add the following code to main.tf:

resource "aws_lambda_permission" "hello" {
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.hello.arn
  principal     = "apigateway.amazonaws.com"

  source_arn = "${aws_api_gateway_rest_api.api_example.execution_arn}/*/*/*"
}

This is very similar to the permissions granting we saw before, with a little trickiness in the source_arn

The action, function_name, and principal should be straightforward. The action we want to allow is invoking the function, the function we want to invoke is the one we defined earlier, and the type of thing we want to grant this permission to is an API Gateway.

For the source_arn, the trickiness arises from the fact that, when the gateway actually invokes the Lambda, the source_arn will depend on the stage, method, and resource path. In our case, we want any value for each of these to invoke the Lambda, so we set them all to *.

And we're done!

Go back to the terminal and run:

>>> terraform apply

Approve the prompt, and watch your new API Gateway get created!

Now, we can do some quick testing.

Testing The Gateway

Go to the AWS console and navigate to Services > API Gateway > api_example. From there, navigate to stages > prod > hello > GET, and find the Invoke URL on the top right of the screen.

Click that link and you should see your Lambda's response!

And we're done!

4. An Endpoint with User Input and External Dependencies

Now that we've deployed a very basic Lambda function, let's write a slightly more interesting one. Specifically, one that takes user input and passes it to the Lambda function, and that also requires an external dependency.

Lambda Function

Our external dependency is going to be python-us, which gives data on US states and territories.

Create a new directory in the src folder called state_data. Create a new file in the state_data directory called state_data.py with the following code:

# src/state_data/state_data.py
import json

import us


def handler(event, context):
    if "pathParameters" in event and "state" in event["pathParameters"]:
        state = us.states.lookup(event["pathParameters"]["state"])

        if state is None:
          return {
            "statusCode": 400,
            "headers": {"Content-Type": "application/json"},
            "body": json.dumps("Not a US state"),
          }

        return {
            "statusCode": 200,
            "headers": {"Content-Type": "application/json"},
            "body": json.dumps(
              {
                "state": state.name,
                "fipsCode": state.fips,
                "abbr": state.abbr
              }
            ),
        }

    return {
        "statusCode": 400,
        "headers": {"Content-Type": "application/json"},
        "body": json.dumps("Must include a state"),
    }

This function takes the state attribute from the pathParameters attribute of event, and returns a few facts about the state (along with some error handling).

We will see how to get information from the URL into the pathParameters attribute as we add the Terraform.

Add the following code to main.tf:

resource "aws_cloudwatch_log_group" "state_data" {
  name              = "/aws/lambda/state_data"
  retention_in_days = 14
}

data "aws_iam_policy_document" "state_data_assume_role" {
  statement {
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "state_data" {
  name                 = "state_data"
  assume_role_policy   = data.aws_iam_policy_document.state_data_assume_role.json
}

data "aws_iam_policy_document" "state_data_cloudwatch_logging" {
  statement {
    effect = "Allow"
    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = [aws_cloudwatch_log_group.state_data.arn]
  }
}

resource "aws_iam_policy" "state_data_cloudwatch_logging" {
  name        = "state-data-logging"
  description = "Allows the Lambda to log to cloudwatch"
  policy      = data.aws_iam_policy_document.state_data_cloudwatch_logging.json
}

resource "aws_iam_role_policy_attachment" "state_data_cloudwatch_logging" {
  role       = aws_iam_role.state_data.name
  policy_arn = aws_iam_policy.state_data_cloudwatch_logging.arn
}

data "archive_file" "state_data_lambda_package" {
  type        = "zip"
  source_dir  = "../src/state_data"
  output_path = "../state_data_package.zip"
}

resource "aws_lambda_function" "state_data" {
  filename          = "../state_data_package.zip"
  function_name     = "state_data"
  role              = aws_iam_role.state_data.arn
  handler           = "state_data.handler"
  timeout           = 10
  runtime           = "python3.8"
  source_code_hash  = data.archive_file.state_data_lambda_package.output_base64sha256
}

This code is very similar to the configuration we had for the hello function, with only a few names changed. (In fact, if we were really serious about this, we could create our own Terraform Module to avoid repeating so much code, but that is a topic for another time.)

Before applying the Terraform though, we have to do one more thing. Lambdas are required to have all their dependencies packaged with them, so we need to install the python-us module along with state_data.py so it gets added to the package.

cd to the state_data directory, and run the following command:

>>> pip install -t . us

Note: be sure that you use the python3 version of pip. Installing dependencies meant for older versions of Python can cause errors.

The -t . argument indicates that pip should install the package locally.

The new Lambda is done! Apply the Terraform again and you should see it appear in the console.

API Gateway Integration

Now that the Lambda is in place, all we have to do is add a route to it in the API Gateway.

Add the following code to main.tf

resource "aws_api_gateway_resource" "state_data" {
  rest_api_id = aws_api_gateway_rest_api.api_example.id
  parent_id   = aws_api_gateway_rest_api.api_example.root_resource_id
  path_part   = "state_data"
}

resource "aws_api_gateway_resource" "state" {
  rest_api_id = aws_api_gateway_rest_api.api_example.id
  parent_id   = aws_api_gateway_resource.state_data.id
  path_part   = "{state}"
}

resource "aws_api_gateway_method" "state" {
  rest_api_id      = aws_api_gateway_rest_api.api_example.id
  resource_id      = aws_api_gateway_resource.state.id
  http_method      = "GET"
  authorization    = "NONE"
}

resource "aws_api_gateway_integration" "state" {
  rest_api_id = aws_api_gateway_rest_api.api_example.id
  resource_id = aws_api_gateway_resource.state.id
  http_method = "GET"

  integration_http_method = "POST"
  type                    = "AWS_PROXY"
  uri                     = aws_lambda_function.state_data.invoke_arn
}

resource "aws_lambda_permission" "state_data" {
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.state_data.arn
  principal     = "apigateway.amazonaws.com"

  source_arn = "${aws_api_gateway_rest_api.api_example.execution_arn}/*/*/*"
}

This again should look very similar to what we made before, with a small difference. While before we only defined one resource, hello, now we define two, state_data and state.

You'll notice that while the parent_id of state_data is the root resource, the parent of state is state_data. What this means is that our state_data Lambda function can be accessed at a path like /state_data/california, and then california will be the value of state in the pathParameters attribute.

Before we apply the Terraform for the final time, there is one thing we have to do. In the terminal, in the terraform directory, run the following command:

>>> terraform taint aws_api_gateway_deployment.api_example

In AWS, API Gateway deployments are snapshots of the state of the gateway at a certain point in time. Because of this, the deployment won't update with our new endpoint unless we tell AWS to rebuild it. "Tainting" a resource in Terraform requires it to be re-deployed on the next apply, which is exactly what we want!

We're now finished with all of the configuration! Just apply the Terraform and then move on to the next section.

Note: If you get a message like

Error: Error creating API Gateway Integration: NotFoundException: Invalid Method identifier specified    

try applying the Terraform another time. It may apply the second time without errors.

Testing The New Endpoint

Go to the AWS console and navigate to Services > API Gateway > api_example. From there, navigate to stages > prod > state_data/{state} > GET, and copy the Invoke URL on the top right of the screen. Paste that url into your browser, replace {state} with california, and load the page.

Voilà! You have a new serverless endpoint!

5. Tearing Things Down

When you are ready to destroy your resources, cd into the terraform directory and run the following command:

>>> terraform destroy

Approve the prompt, and you're done!

6. Closing Thoughts

Provisioning an API Gateway using Terraform can be a little daunting because of how many separate resources it uses. However, once you see how they all fit together, it can be relatively straightforward, and gives you full control over your infrastructure.

Thanks for reading!