Terraform Setup and First Install

Terraform is a system that allows you to define your infrastructure in a series of configuration files. These configuration files are linked to a provider library which will execute the infrastructure create, update and teardown commands on the platform you want to deploy to. This is called “Infrastructure as Code” (or IaC) which has the benefits (and drawbacks?) that code has

• Repeatability
• Versioning
• etc…

We’re going to define (in configuration) the steps that we just executed (create a lambda and the configuration associated with it, create an API Gateway trigger and associate it to the lambda, etc…) and then use a single command via terraform to install them. Additionally, since terraform understands the state of the system we can also execute a single command to tear down everything we just did (which is useful for things like uninstalling everything to keep costs down)

Installation and Configuration of Terraform

HashiCorp (the developers of Terraform) have the installation guide. I’m using windows, so I’ll be using chocolatey.

C:\WINDOWS\system32>choco install terraform
Chocolatey v0.10.15
Installing the following packages:
terraform
By installing you accept licenses for the packages.
Progress: Downloading terraform 1.0.2... 100%
...
 The install of terraform was successful.
  Software installed to 'C:\ProgramData\chocolatey\lib\terraform\tools'

Chocolatey installed 1/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

In order to be safe with regards to security, I will also be creating a credentials file to log into AWS. Create a file in your home .aws/ directory (for windows that will be usually c:/Users/${your_name}/.aws) called credentials. In that file, put your access and secret keys from AWS in this format:

$ cat credentials
[default]
aws_access_key_id = AKIASXXXXXXXXXMSHFU
aws_secret_access_key = XXXXXXXXXXC8Yi

Create the Terraform Configuration

Now navigate back to your project directory and let’s start a new branch

$ git checkout -b terraform-local
Switched to a new branch 'terraform-local'

Create a directory called /tf off the root of the project and create a file called variables.tf with this content

variable "aws_region" {
  default = "us-east-1"
}
variable "app-name" {
  default = "helloWorld-lambda"
}

Next, create another file called main.tf with this content

provider "aws" {
  region = "${var.aws_region}"
  shared_credentials_file = "/Users/${your_name}/.aws/credentials"
}


#Assume the role needed to create the lambda
resource "aws_iam_role" "iam_for_helloWorld_lambda" {
  name = "iam_for_helloWorld_lambda"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

#create the lambda from the code in the s3 bucket.  Make sure that all of the policies are in place before
#we attempt to create this
resource "aws_lambda_function" "tf-helloWorld" {
  function_name = "helloWorld-lambda"
  filename = "../target/helloworld-lambda-0.0.1-SNAPSHOT-aws.jar"
  role = aws_iam_role.iam_for_helloWorld_lambda.arn
  handler = "org.springframework.cloud.function.adapter.aws.SpringBootApiGatewayRequestHandler::handleRequest"
  memory_size = 512
  timeout = 15

  runtime = "java11"

  depends_on = [
    aws_iam_role_policy_attachment.helloWorld-log-attach,
    aws_cloudwatch_log_group.helloWorld-logs,
  ]

  environment {
    variables = {
      FUNCTION_NAME = "apiFunction"
    }
  }
}

#create a log group for the lambda
resource "aws_cloudwatch_log_group" "helloWorld-logs" {
  name = "/aws/lambda/${var.app-name}"

  retention_in_days = 30
}

#create a policy to be able to write the logs
resource "aws_iam_policy" "helloWorld-logs" {
  name = "helloWorld_lambda_logging"
  path = "/"
  description = "IAM policy for logging from a lambda"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*",
      "Effect": "Allow"
    }
  ]
}
EOF
}

#associate the log policy privileges to the lambda iam
resource "aws_iam_role_policy_attachment" "helloWorld-log-attach" {
  role = aws_iam_role.iam_for_helloWorld_lambda.name
  policy_arn = aws_iam_policy.helloWorld-logs.arn
}

# Now, we need an API to expose those functions publicly
resource "aws_apigatewayv2_api" "helloWorld-api" {
  name = "Hello API"
  protocol_type = "HTTP"
  target = aws_lambda_function.tf-helloWorld.invoke_arn
}

resource "aws_lambda_permission" "helloWorld-permission" {
  action = "lambda:InvokeFunction"
  function_name = aws_lambda_function.tf-helloWorld.arn
  principal = "apigateway.amazonaws.com"

  source_arn = "${aws_apigatewayv2_api.helloWorld-api.execution_arn}/*/*"
}

There’s a lot here, but I’ll walk you through what each clause is doing:

provider : tells terraform that we’re going to be using AWS commands, so download and install the AWS provider libraries that are needed, also configures our credentials

aws_iam_role : grant access to the lambda functions (create, in our case) to our script

aws_lambda_function : this is the lambda create statement. Note that we’re pulling from the local file system. Also, the depends_on which tells terraform to go ahead and execute those clauses first (its declarative, doesn’t execute from top down). This should look familiar because of the manual install we did before.

aws_cloudwatch_log_group : we’re going to do some logging, so create a place to store them

resource "aws_iam_policy" "helloWorld-logs" : define access to write to the logs

resource "aws_iam_role_policy_attachment" "helloWorld-log-attach" : give permission for the lambda to use the policy we just created (so it can write)

aws_apigatewayv2_api : define an API Gateway instance

aws_lambda_permission : give permission to the API Gateway to trigger the lambda

That’s a lot of stuff! AWS console is doing a bunch of this stuff behind the scenes for you, but pulling it out into terraform means that you can evolve it (and we will in this course) in a safe way with version control.

Initialize and Execute Terraform

Now we need to tell Terraform to get ready to execute our code. Terraform needs to pull in the libraries that we’re going to use (AWS provider) and prepare to execute. From the /tf directory execute this command:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v3.51.0...
- Installed hashicorp/aws v3.51.0 (signed by HashiCorp)
...
Terraform has been successfully initialized!

If at this point you execute terraform plan, you’ll see what terraform plans to execute:

$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_apigatewayv2_api.helloWorld-api will be created
  + resource "aws_apigatewayv2_api" "helloWorld-api" {
...
Plan: 7 to add, 0 to change, 0 to destroy.

Lets go ahead and execute it

$ terraform apply
...
aws_cloudwatch_log_group.helloWorld-logs: Creating...
aws_iam_policy.helloWorld-logs: Creating...
aws_iam_role.iam_for_helloWorld_lambda: Creating...
aws_iam_policy.helloWorld-logs: Creation complete after 1s [id=arn:aws:iam::282457041796:policy/helloWorld_lambda_logging]
aws_cloudwatch_log_group.helloWorld-logs: Creation complete after 1s [id=/aws/lambda/helloWorld]
aws_iam_role.iam_for_helloWorld_lambda: Creation complete after 1s [id=iam_for_helloWorld_lambda]
aws_iam_role_policy_attachment.helloWorld-log-attach: Creating...
aws_iam_role_policy_attachment.helloWorld-log-attach: Creation complete after 1s [id=iam_for_helloWorld_lambda-20210724164455618400000001]
aws_lambda_function.tf-helloWorld: Creating...
aws_lambda_function.tf-helloWorld: Still creating... [10s elapsed]
aws_lambda_function.tf-helloWorld: Still creating... [20s elapsed]
aws_lambda_function.tf-helloWorld: Still creating... [30s elapsed]
aws_lambda_function.tf-helloWorld: Still creating... [40s elapsed]
aws_lambda_function.tf-helloWorld: Creation complete after 49s [id=helloWorld-lambda]
aws_apigatewayv2_api.helloWorld-api: Creating...
aws_apigatewayv2_api.helloWorld-api: Creation complete after 1s [id=m812gnu7x9]
aws_lambda_permission.helloWorld-permission: Creating...
aws_lambda_permission.helloWorld-permission: Creation complete after 0s [id=terraform-20210724164545966500000002]

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Now lets checkout the lambda page in the console and see if we’ve got a new lambda

Now lets grab the API from the lambda trigger

If we paste that into our postman, we can see it respond with success:

Terraform State

If you review the directory structure in /tf now you should see some interesting stuff:

the provider registry is similar to maven repository, its a local copy of all of the providers that you needed to execute (terraform init created this)

Also, you’ll see two files terraform.tfstate and .terraform.lock.hcl. These files are created by terraform apply to track the current state of the system you’ve created. Any modifications will be reviewed against the state before being applied and only the changes will be made. Additionally, this lock file prevents other users from wiping out your changes.

we can see this in action by executing terraform plan again:

$ terraform plan
...
No changes. Your infrastructure matches the configuration.

While this is nice, its not going to help us out when we want to deploy in an automated fashion because our pipeline will create its own terraform state which will conflict with our local version. In the next step in the course we’ll see how to make a ‘central’ state and lock file in the cloud so that everyone can be executing against the same plan.

For now, lets blow away our application

$ terraform destroy
...
aws_lambda_permission.helloWorld-permission: Destroying... [id=terraform-20210724164545966500000002]
aws_lambda_permission.helloWorld-permission: Destruction complete after 1s
aws_apigatewayv2_api.helloWorld-api: Destroying... [id=m812gnu7x9]
aws_apigatewayv2_api.helloWorld-api: Destruction complete after 1s
aws_lambda_function.tf-helloWorld: Destroying... [id=helloWorld-lambda]
aws_lambda_function.tf-helloWorld: Destruction complete after 0s
aws_iam_role_policy_attachment.helloWorld-log-attach: Destroying... [id=iam_for_helloWorld_lambda-20210724164455618400000001]
aws_cloudwatch_log_group.helloWorld-logs: Destroying... [id=/aws/lambda/helloWorld]
aws_iam_role_policy_attachment.helloWorld-log-attach: Destruction complete after 0s
aws_cloudwatch_log_group.helloWorld-logs: Destruction complete after 0s
aws_iam_policy.helloWorld-logs: Destroying... [id=arn:aws:iam::282457041796:policy/helloWorld_lambda_logging]
aws_iam_role.iam_for_helloWorld_lambda: Destroying... [id=iam_for_helloWorld_lambda]
aws_iam_policy.helloWorld-logs: Destruction complete after 1s
aws_iam_role.iam_for_helloWorld_lambda: Destruction complete after 1s

Destroy complete! Resources: 7 destroyed.

we also do not want to check in our state files, so update our .gitignore to ignore them (from here)

# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*
*.terraform.lock.*

# Crash log files
crash.log

# Exclude all .tfvars files, which are likely to contain sentitive data, such as
# password, private keys, and other secrets. These should not be part of version
# control as they are data points which are potentially sensitive and subject
# to change depending on the environment.
#
*.tfvars

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
#
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

# Ignore CLI configuration files
.terraformrc
terraform.rc

Commit and Merge

$ git status
$ git add .
$ git commit -m "first terraform files"
$ git push --set-upstream origin terraform-local
$ git checkout main
$ git merge terraform-local
$ git push