How to build a Serverless API with Go and AWS Lambda
Earlier this year AWS announced that their Lambda service would now be providing first-class support for the Go language, which is a great step forward for any gophers (like myself) who fancy experimenting with serverless technology.
So in this post I'm going to talk through how to create a HTTPS API backed by AWS Lambda, building it up step-by-step. I found there to be quite a few gotchas in the process — especially if you're not familiar the AWS permissions system — and some rough edges in the way that Lamdba interfaces with the other AWS services. But once you get your head around these things it works pretty well.
There's a lot of content to cover in this tutorial, so I've broken it down into the following seven steps:
- Setting up the AWS CLI
- Creating and deploying an Lambda function
- Hooking it up to DynamoDB
- Setting up the HTTPS API
- Working with events
- Deploying the API
- Supporting multiple actions
Throughout this post we'll work towards building an API with two actions:
Method | Path | Action |
---|---|---|
GET | /books?isbn=xxx | Display information about a book with a specific ISBN |
POST | /books | Create a new book |
Where a book is a basic JSON record which looks like this:
I'm keeping the API deliberately simple to avoid getting bogged-down in application-specific code, but once you've grasped the basics it's fairly clear how to extend the API to support additional routes and actions.
Setting up the AWS CLI
-
Throughout this tutorial we'll use the AWS CLI (command line interface) to configure our lambda functions and other AWS services. Installation and basic usage instructions can be found here, but if you’re using a Debian-based system like Ubuntu you can install the CLI with
apt
and run it using theaws
command: -
Next we need to set up an AWS IAM user with programmatic access permission for the CLI to use. A guide on how to do this can be found here. For testing purposes you can attach the all-powerful
AdministratorAccess
managed policy to this user, but in practice I would recommend using a more restrictive policy. At the end of setting up the user you'll be given a access key ID and secret access key. Make a note of these — you’ll need them in the next step. -
Configure the CLI to use the credentials of the IAM user you've just created using the
configure
command. You’ll also need to specify the default region and output format you want the CLI to use.(Throughout this tutorial I'll assume you're using the
us-east-1
region — you'll need to change the code snippets accordingly if you're using a different region.)
Creating and deploying an Lambda function
-
Now for the exciting part: making a lambda function. If you're following along, go to your
$GOPATH/src
folder and create abooks
repository containing amain.go
file. -
Next you'll need to install the
github.com/aws-lambda-go/lambda
package. This provides the essential libraries and types we need for creating a lambda function in Go. -
Then open up the
main.go
file and add the following code:In the
main()
function we calllambda.Start()
and pass in theshow
function as the lambda handler. In this case the handler simply initializes and returns a newbook
object.Lamdba handlers can take a variety of different signatures and reflection is used to determine exactly which signature you're using. The full list of supported forms is…
… where the
TIn
andTOut
parameters are objects that can be marshaled (and unmarshalled) by Go'sencoding/json
package. -
The next step is to build an executable from the
books
package usinggo build
. In the code snippet below I'm using the-o
flag to save the executable to/tmp/main
but you can save it to any location (and name it whatever) you wish.Important: as part of this command we're using
env
to temporarily set two environment variables for the duration for the command (GOOS=linux
andGOARCH=amd64
). These instruct the Go compiler to create an executable suitable for use with a linux OS and amd64 architecture — which is what it will be running on when we deploy it to AWS. -
AWS requires us to upload our lambda functions in a zip file, so let's make a
main.zip
zip file containing the executable we just made:Note that the executable must be in the root of the zip file — not in a folder within the zip file. To ensure this I've used the
-j
flag in the snippet above to junk directory names. -
The next step is a bit awkward, but critical to getting our lambda function working properly. We need to set up an IAM role which defines the permission that our lambda function will have when it is running.
For now let's set up a
lambda-books-executor
role and attach theAWSLambdaBasicExecutionRole
managed policy to it. This will give our lambda function the basic permissions it need to run and log to the AWS cloudwatch service.First we have to create a trust policy JSON file. This will essentially instruct AWS to allow lambda services to assume the
lambda-books-executor
role:Then use the
aws iam create-role
command to create the role with this trust policy:Make a note of the returned ARN (Amazon Resource Name) — you'll need this in the next step.
Now the
lambda-books-executor
role has been created we need to specify the permissions that the role has. The easiest way to do this it to use theaws iam attach-role-policy
command, passing in the ARN ofAWSLambdaBasicExecutionRole
permission policy like so:Note: you can find a list of other permission policies that might be useful here.
-
Now we're ready to actually deploy the lambda function to AWS, which we can do using the
aws lambda create-function
command. This takes the following flags and can take a minute or two to run.--function-name
Thethat name your lambda function will be called within AWS --runtime
The runtime environment for the lambda function (in our case "go1.x"
)--role
The ARN of the role you want the lambda function to assume when it is running (from step 6 above) --handler
The name of the executable in the root of the zip file --zip-file
Path to the zip file Go ahead and try deploying it:
-
So there it is. Our lambda function has been deployed and is now ready to use. You can try it out by using the
aws lambda invoke
command (which requires you to specify an output file for the response — I've used/tmp/output.json
in the snippet below).If you're following along hopefully you've got the same response. Notice how the
book
object we initialized in our Go code has been automatically marshaled to JSON?
Hooking it up to DynamoDB
-
In this section we're going to add a persistence layer for our data which can be accessed by our lambda function. For this I'll use Amazon DynamoDB (it integrates nicely with AWS lambda and has a generous free-usage tier). If you're not familiar with DynamoDB, there's a decent run down of the basics here.
The first thing we need to do is create a
Books
table to hold the book records. DynanmoDB is schema-less, but we do need to define the partion key (a bit like a primary key) on the ISBN field. We can do this in one command like so: -
Then lets add a couple of items using the
put-item
command, which we'll use in the next steps. -
The next thing to do is update our Go code so that our lambda handler can connect to and use the DynamoDB layer. For this you'll need to install the
github.com/aws/aws-sdk-go
package which provides libraries for working with DynamoDB (and other AWS services). -
Now for the code. To keep a bit of separation create a new
db.go
file in yourbooks
repository:And add the following code:
And then update the
main.go
to use this new code: -
Save the files, then rebuild and zip up the lambda function so it's ready to deploy:
-
Re-deploying a lambda function is easier than creating it for the first time — we can use the
aws lambda update-function-code
command like so: -
Let's try executing the lambda function now:
Ah. There's a slight problem. We can see from the output message that our lambda function (specifically, the
lambda-books-executor
role) doesn't have the necessary permissions to runGetItem
on a DynamoDB instance. Let's fix that now. -
Create a privilege policy file that gives
GetItem
andPutItem
privileges on DynamoDB like so:And then attach it to the
lambda-books-executor
role using theaws iam put-role-policy
command:As a side note, AWS has some managed policies called
AWSLambdaDynamoDBExecutionRole
andAWSLambdaInvocation-DynamoDB
which sound like they would do the trick. But neither of them actually provideGetItem
orPutItem
privileges. Hence the need to roll our own policy. -
Let's try executing the lambda function again. It should work smoothly this time and return information about the book with ISBN
978-0486298238
.
Setting up the HTTPS API
-
So our lambda function is now working nicely and communicating with DynamoDB. The next thing to do is set up a way to access the lamdba function over HTTPS, which we can do using the AWS API Gateway service.
But before we go any further, it's worth taking a moment to think about the structure of our project. Let's say we have grand plans for our lamdba function to be part of a bigger
bookstore
API which deals with information about books, customers, recommendations and other things.There's three basic options for structuring this using AWS Lambda:
- Microservice style — Each lambda function is responsible for one action only. For example, there are 3 separate lambda functions for showing, creating and deleting a book.
- Service style — Each lambda function is responsible for a group of related actions. For example, one lambda function handles all book-related actions, but customer-related actions are kept in a separate lambda function.
- Monolith style — One lambda function manages all the bookstore actions.
Each of these options is valid, and theres some good discussion of the pros and cons here.
For this tutorial we'll opt for a service style, and have one
books
lambda function handle the different book-related actions. This means that we'll need to implement some form of routing within our lambda function, which I'll cover later in the post. But for now… -
Go ahead and create a
bookstore
API using theaws apigateway create-rest-api
command like so:Note down the
rest-api-id
value that this returns, we'll be using it a lot in the next few steps. -
Next we need to get the id of the root API resource (
"/"
). We can retrieve this using theaws apigateway get-resources
command like so:Again, keep a note of the
root-path-id
value this returns. -
Now we need to create a new resource under the root path — specifically a resource for the URL path
/books
. We can do this by using theaws apigateway create-resource
command with the--path-part
parameter like so:Again, note the
resource-id
this returns, we'll need it in the next step.Note that it's possible to include placeholders within your path by wrapping part of the path in curly braces. For example, a
--path-part
parameter ofbooks/{id}
would match requests to/books/foo
and/books/bar
, and the value ofid
would be made available to your lambda function via an events object (which we'll cover later in the post). You can also make a placeholder greedy by postfixing it with a+
. A common idiom is to use the parameter--path-part {proxy+}
if you want to match all requests regardless of their path. -
But we're not doing either of those things. Let's get back to our
/books
resource and use theaws apigateway put-method
command to register the HTTP method ofANY
. This will mean that our/books
resource will respond to all requests regardless of their HTTP method. -
Now we're all set to integrate the resource with our lambda function, which we can do using the
aws apigateway put-integration
command. This command has a few parameters that need a quick explanation:- The
--type
parameter should beAWS_PROXY
. When this is used the AWS API Gateway will send information about the HTTP request as an 'event' to the lambda function. It will also automatically transform the output from the lambda function to a HTTP response. - The
--integration-http-method
parameter must bePOST
. Don't confuse this with what HTTP methods your API resource responds to. - The
--uri
parameter needs to take the format:
With those things in mind, your command should look a bit like this:
- The
-
Alright, let's give this a whirl. We can send a test request to the resource we just made using the
aws apigateway test-invoke-method
command like so:Ah. So that hasn't quite worked. If you take a look through the outputted log information you should see that the problem appears to be:
Execution failed due to configuration error: Invalid permissions on Lambda function
This is happening because our
bookstore
API gateway doesn't have permissions to execute our lambda function. -
The easiest way to fix that is to use the
aws lambda add-permission
command to give our API permissions to invoke it, like so:Note that the
--statement-id
parameter needs to be a globally unique identifier. This could be a random ID or something more descriptive. Alright, let's try again:
So unfortunately there's still an error, but the message has now changed:
Execution failed due to configuration error: Malformed Lambda proxy response
And if you look closely at the output you'll see the information:
Endpoint response body before transformations: {\"isbn\":\"978-0486298238\",\"title\":\"Meditations\",\"author\":\"Marcus Aurelius\"}
So there's some definite progress here. Our API is talking to our lambda function and is receiving the correct response (a
book
object marshalled to JSON). It's just that the AWS API Gateway considers the response to be in the wrong format.This is because, when you're using the API Gateway's lambda proxy integration, the return value from the lambda function must be in the following JSON format:
So to fix this it's time to head back to our Go code and make some alterations.
Working with events
-
The easiest way to provide the responses that the AWS API Gateway needs is to install the
github.com/aws/aws-lambda-go/events
package:This provides a couple of useful types (
APIGatewayProxyRequest
andAPIGatewayProxyResponse
) which contain information about incoming HTTP requests and allow us to construct responses that the API Gateway understands. -
Let's go back to our
main.go
file and update our lambda handler so that it uses the signature:Essentially, the handler will accept a
APIGatewayProxyRequest
object which contains a bunch of information about the HTTP request, and return aAPIGatewayProxyResponse
object (which is marshalable into a JSON response suitable for the AWS API Gateway).Notice how in all cases the
error
value returned from our lambda handler is nownil
? We have to do this because the API Gateway doesn't accepterror
objects when you're using it in conjunction with a lambda proxy integration (they would result in a 'malformed response' errors again). So we need to manage errors fully within our lambda function and return the appropriate HTTP response. In essence, this means that the return parameter oferror
is superfluous, but we still need to include it to have a valid signature for the lambda function. -
Anyway, save the file and rebuild and redeploy the lambda function:
-
And if you test it again now it should work as expected. Give it a try with different
isbn
values in the query string: -
As a side note, anything sent to
os.Stderr
will be logged to the AWS Cloudwatch service. So if you've set up an error logger like we have in the code above, you can query Cloudwatch for errors like so:
Deploying the API
-
Now that the API Gateway is working properly it's time to make it live. We can do this with the
aws apigateway create-deployment
command like so:In the code above I've given the deployed API using the name
staging
, but you can call it anything that you wish. -
Once deployed your API should be accessible at the URL:
Go ahead and give it a try using curl. It should work as you expect:
Supporting multiple actions
-
Let's add support for a
POST /books
action. We want this to read and validate a new book record (from a JSON HTTP request body) and then add it to the DynamoDB table.Now that the different AWS services are hooked up, extending our lambda function to support additional actions is perhaps the most straightforward part of this tutorial, as it can be managed purely within our Go code.
First update the
db.go
file to include a newputItem
function like so:And then update the
main.go
function so that thelambda.Start()
method calls a newrouter
function, which does a switch on the HTTP request method to determine which action to take. Like so: -
Rebuild and zip up the lambda function, then deploy it as normal:
-
And now when you hit the API using different HTTP methods it should call the appropriate action: