Introduction to SAM Part II: Template and architecture
by Alex Harvey
This is Part II of my 3-part blog series on Amazon’s Serverless Application Model (SAM). Part I is here and I recommend reading that first.
Table of contents
- Overview to Part II
- Important documentation
- Architecture
- The SAM template
- Generated resources
- Summary
- Further reading
Overview to Part II
In this second part of my series, I look at the SAM Translator and its docs, and discuss line by line the example “hello world” template. Along the way I discuss key concepts in SAM such as the AWS::Serverless types, the implicit API, the default Role and Stage, as well as the generated CloudFormation resources.
Important documentation
I begin with an overview of SAM’s documentation, extended from Part I to cover the SAM Translator. I do this because, at the time of writing, the official docs are not always up to date or well organised and understanding key concepts, such as the implicit API, can involve consulting multiple documents.
So, the additional documents to look at for the SAM Translator and the template language are:
- The Markdown docs in the source code repo, especially:
- The SAM Specification 2016-10-31 document, which provides a technical description of the template language, the Resource Types, their Properties, Data Types and so on.
- The docs directory, which includes:
- globals.rst, which documents the SAM template’s Globals section and is a key reference for this section
- internals/generated_resources.rst, which is another key reference that documents the CloudFormation resources that are generated by the SAM resources and also their naming conventions, which is important if you need to refer to them via !Ref or !GetAtt.
- cloudformation_compatibility.rst, which discusses the support for CloudFormation intrinsic functions implemented so far in SAM. (See also the HOWTO.md.)
- faq.rst, is a SAM FAQ that has just three questions at this time, including how to manage multiple environments, and how to enable API Gateway logs.
- policy_templates.rst discusses SAM’s out-of-the-box IAM Policies that are found in the policy_templates.json file.
- And finally, safe_lambda_deployments.rst discusses traffic shifting for safe production deployments. (Also discussed again in the SAM Developer Guide here.)
- The SAM developer guide, especially:
- AWS SAM Template Basics documents the resource types (but I expect the specification document above to be more up to date); contains info on the Metadata section of templates and how it relates to publishing SAM apps in the AWS Serverless Application Repository; and nested applications, which seems to be simply the nested stacks feature of CloudFormation.
- Finally, the examples directory in the source code is especially useful for finding SAM apps and templates you can use a starting point for your own app.
Architecture
As mentioned earlier, SAM’s architecture consists of two main parts: the SAM CLI (discussed in Part I) and the SAM Translator, also known as the SAM Transformer.
The SAM Translator is called by the CloudFormation Service. CloudFormation recognises a top-level template key Transform: AWS::Serverless-2016-10-31 and invokes the SAM Translator if it is found. The Translator then takes the SAM template and expands it into a full CloudFormation template that is executed by CloudFormation to create, update and/or delete AWS resources.
The entry point in the SAM source code is in the Translator class here. It can be seen the translate method expects a template and its parameters as arguments, then it iterates through AWS::Serverless resources, and returns a CloudFormation template.
The SAM template
In Part I, I used the SAM CLI to initialise, build and deploy the example “hello world” Python 2.7 application. In this section, I look at the SAM template that was created.
The “hello world” code example
Here is the template in full1:
---
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sam-app
Sample SAM Template for sam-app
# More info about Globals:
# https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
#
Globals:
Function:
Timeout: 3
Resources:
# More info about Function Resource:
# https://github.com/awslabs/serverless-application-model/blob/master/
# versions/2016-10-31.md#awsserverlessfunction
#
HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: hello_world/
Handler: app.lambda_handler
Runtime: python2.7
# More info about API Event Source:
# https://github.com/awslabs/serverless-application-model/blob/master/
# versions/2016-10-31.md#api
#
Events:
HelloWorld:
Type: Api
Properties:
Path: /hello
Method: get
Outputs:
# ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
# Find out more about other implicit resources you can reference within SAM
# https://github.com/awslabs/serverless-application-model/blob/master/
# docs/internals/generated_resources.rst#api
#
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
HelloWorldFunction:
Description: "Hello World Lambda Function ARN"
Value: !GetAtt HelloWorldFunction.Arn
HelloWorldFunctionIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt HelloWorldFunctionRole.Arn
In the remainder of this section I discuss this line by line with reference to the SAM template features.
The Transform
The first thing to note is that the SAM template is a marked up CloudFormation template. The line AWSTemplateFormatVersion: '2010-09-09'
identifies the CloudFormation template format version. And CloudFormation users will be well familiar with that.
More interesting is the second line Transform: AWS::Serverless-2016-10-31
that identifies the SAM template version. And as already mentioned, it is the presence of this line that causes the CloudFormation service to call the SAM Translator in the backend.
The Globals section
The Globals section is one of the template language extensions read by the SAM Translator.
In the example template, the following snippet sets the Lambda function timeout value to 3 seconds globally:
Globals:
Function:
Timeout: 3
Values specified in the Globals section are merged into the configs of the generated CloudFormation resources. The thinking in the SAM design is that resources in a SAM template will tend to have shared configuration like runtime, memory settings, environment variables, CORS config and so on and this config would otherwise be duplicated.
There is important documentation about the rules for merging here, but in summary: scalars are replaced; maps are merged; and lists are additive.
At the time of writing, the SAM types AWS::Serverless::Function, AWS::Serverless::Api and AWS::Serverless::SimpleTable can all be further configured from the Globals section.
The serverless function
The Lambda function itself is the HelloWorldFunction resource of type AWS::Serverless::Function:
HelloWorldFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: hello_world/
Handler: app.lambda_handler
Runtime: python2.7
Events:
HelloWorld:
Type: Api
Properties:
Path: /hello
Method: get
The AWS::Serverless::Function type is documented at the versions/2016-10-31.md#awsserverlessfunction page mentioned in the template.
Comparing the function types
The AWS::Serverless::Function type is of course central to SAM and it is worth comparing it with the CloudFormation AWS::Lambda::Function type that SAM transforms it into.
Here, I present an sdiff on the signatures of these types from the documentation in order to show their similarities and differences:
Type: "AWS::Lambda::Function" | Type: "AWS::Serverless::Function"
Properties: Properties:
> AutoPublishAlias: String
Code: | CodeUri:
Code | CodeUri
DeadLetterConfig: | DeadLetterQueue:
DeadLetterConfig | DeadLetterQueue
> DeploymentPreference:
> DeploymentPreference
Description: String Description: String
Environment: Environment:
Environment Environment
> Events:
> Events
FunctionName: String FunctionName: String
Handler: String Handler: String
> InlineCode: String
KmsKeyArn: String KmsKeyArn: String
Layers: Layers:
- String - String
MemorySize: Integer MemorySize: Integer
> Policies:
> Policies
ReservedConcurrentExecutions: Integer ReservedConcurrentExecutions: Integer
Role: String Role: String
Runtime: String Runtime: String
Timeout: Integer Timeout: Integer
TracingConfig: | Tracing: String
TracingConfig <
VpcConfig: VpcConfig:
VPCConfig VPCConfig
Tags: | Tags:
Resource Tag Resource Tag
It is important to be aware of this:
Some of the apparent differences between the SAM type and its corresponding CloudFormation type are just naming inconsistencies. Thus, the DeadLetterConfig and TracingConfig properties in the CloudFormation type have been renamed DeadLetterQueue and Tracing in the SAM type. Likewise, the CodeUri and InlineCode properties are renamed features of the Code property of the CloudFormation type.
Meanwhile, The Policies property is a special property that just allows us to modify the Policies of the default IAM Role. (More on the default Role below.)
Now, leaving aside superficial differences, the features provided by the SAM type that didn’t exist in the CloudFormation type are:
Property | Description |
---|---|
Events | A map of Event source objects that defines the events that trigger this function. More on this when I discuss the implicit API below. |
AutoPublishAlias | Name of the Alias. Read the AutoPublishAlias Guide for how it works. |
DeploymentPreference | Settings to enable Safe Lambda Deployments. Read the usage guide for detailed information. |
Of these, I haven’t looked at AutoPublishAlias or DeploymentPreference, and I will omit further discussion of them at this time.
The implicit API and the Events source
A selling point of SAM is that it is requires less CloudFormation code to create some of the frequently required supporting AWS resources like the API Gateway and Lambda execution Role. In support of that are the “implicit API” and the “default” resources, the default IAM Role, the default Stage, the default Deployment, all of which are a bit confusing at first.
In the Events property of the HelloWorldFunction, we have:
Events:
HelloWorld:
Type: Api
Properties:
Path: /hello
Method: get
And now the confusing bit:
By mentioning an event source of Type: Api that isn’t otherwise defined in the template and referred to via the RestApiId property, a resource of type AWS::Serverless::Api and a Swagger definition is created by SAM. These are known in the documentation as “implicit APIs”.
Meanwhile, the new type AWS::Serverless::Api creates a collection of API Gateway resources, allowing us to avoid defining a complex system of API Gateway resources in CloudFormation the old way. And to be clear, this is a big win, because an API Gateway consists of a lot of CloudFormation resources and a lot of code that is both difficult to understand and time-consuming to write.
Note that the implict API can be referenced within the template using the special variable ${ServerlessRestApi}.
The default Role
Another implicit resource create by this template is the default Lambda Execution Role, usually referred to as the “default Role” (although referred to in the “hello world” template as the “implicit IAM Role”). As with the implicit API, this default IAM Role is created silently unless you refer via the Role property of the function resource to either an IAM Role that you define explicitly in the template or to an IAM Role that is otherwise known to exist.
Note that the “hello world” template refers to the default Role in the Outputs section:
HelloWorldFunctionIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt HelloWorldFunctionRole.Arn
Note that its name is the function name + Role and its ARN can be obtained as shown, using !GetAtt. This naming is documented here.
Now I mentioned above that the function has a Policies property that the CloudFormation function doesn’t have. By specifying a list of IAM Policies on the AWS::Serverless::Function type, these are added to the default role.
The default Stage
There is also a default stage named “Prod” created that cannot be configured.
The default stage is also mentioned indirectly in the Outputs section in the API endpoint:
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
The default stage is documented here.
Generated resources
The business of building and deploying the SAM template was discussed in Part I, and in this section I head to the AWS Console to find the CloudFormation stack I created when I deployed the SAM template.
The documentation has an important section on the resources generated by SAM, but it’s sometimes easiest to understand what SAM is doing just by inspecting the actual CloudFormation stack after deployment. Here is the view of the stack resources:
This is familiar territory for CloudFormation users, although it will be immediately apparent that the actual resources in the stack are different from the ones we put in the template. And of course, that’s the point.
More interesting is the new section available for SAM templates, the View Original Template and View Processed Template options, as seen here:
To find out what the SAM Translator did I can use a combination of the cfn_flip library to convert from JSON to YAML, and then the Ruby HashDiff library.
To get the processed template body of the processed template as YAML from the AWS CLI2:
▶ aws cloudformation get-template --stack-name HelloWorld \
--template-stage Processed --query TemplateBody | cfn-flip -y > processed.yml
And then to compare the differences using the Ruby HashDiff library:
▶ ruby -rhashdiff -rawesome_print -ryaml \
-e "ap HashDiff.diff(*ARGV.map{|f| YAML.load_file(f)})" template.yaml processed.yml
This command returns an array of 15 diffs.
For readability I will present the first 9 as they appear in the Ruby Hashdiff output, and then the remainder - all of which add CloudFormation generated resources to the processed template - I’ll simply present as the generated YAML.
Here are the first 9 diffs:
[
[ 0] [
[0] "-",
[1] "Globals", [2] { "Function" => { "Timeout" => 3 } }
],
[ 1] [
[0] "-",
[1] "Transform", [2] "AWS::Serverless-2016-10-31"
],
[ 2] [
[0] "-",
[1] "Resources.HelloWorldFunction.Properties.CodeUri", [2] "hello_world/"
],
[ 3] [
[0] "-",
[1] "Resources.HelloWorldFunction.Properties.Events", [2] {
"HelloWorld" => {
"Type" => "Api",
"Properties" => {
"Path" => "/hello",
"Method" => "get"
}
}
}
],
[ 4] [
[0] "+",
[1] "Resources.HelloWorldFunction.Properties.Code", [2] {
"S3Bucket" => "alexharvey3118",
"S3Key" => "447c06bbc03dcd1b23220d2450918b99"
}
],
[ 5] [
[0] "+",
[1] "Resources.HelloWorldFunction.Properties.Role", [2] "HelloWorldFunctionRole.Arn"
],
[ 6] [
[0] "+",
[1] "Resources.HelloWorldFunction.Properties.Tags", [2] [
[0] {
"Value" => "SAM",
"Key" => "lambda:createdBy"
}
]
],
[ 7] [
[0] "+",
[1] "Resources.HelloWorldFunction.Properties.Timeout", [2] 3
],
[ 8] [
[0] "~",
[1] "Resources.HelloWorldFunction.Type",
[2] "AWS::Serverless::Function",
[3] "AWS::Lambda::Function"
],
To summarise these diffs:
- the Globals and Transform sections were removed, as was the Events properties of the HelloWorld function
- the HelloWorldFunction’s type is changed from AWS::Serverless::Function to AWS::Lambda::Function
- its CodeUri is changed from:
CodeUri: hello_world/
to:
Code:
S3Bucket: alexharvey3118
S3Key: 447c06bbc03dcd1b23220d2450918b99
- a Role property was added and set to HelloWorldFunctionRole.Arn
- the timeout of 3 was added based on the value that was specified in the Globals section
- and a Stack Tag was added:
Tags:
- Value: SAM
Key: lambda:createdBy
The generated HelloWorldFunction resource is thus:
HelloWorldFunction:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: alexharvey3118
S3Key: 447c06bbc03dcd1b23220d2450918b99
Handler: app.lambda_handler
Runtime: python2.7
Tags:
- Value: SAM
Key: lambda:createdBy
Role: !GetAtt 'HelloWorldFunctionRole.Arn'
Timeout: 3
In addition to these changes, the following CloudFormation code is also generated, showing the default Role, the default Lambda permission, the implicit API named ServerlessRestApi, and the default Stage and Deployment:
HelloWorldFunctionRole:
Type: AWS::IAM::Role
Properties:
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
HelloWorldFunctionHelloWorldPermissionProd:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:invokeFunction
Principal: apigateway.amazonaws.com
FunctionName: !Ref 'HelloWorldFunction'
SourceArn: !Sub
- arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/GET/hello
- __Stage__: Prod
__ApiId__: !Ref 'ServerlessRestApi'
HelloWorldFunctionHelloWorldPermissionTest:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:invokeFunction
Principal: apigateway.amazonaws.com
FunctionName: !Ref 'HelloWorldFunction'
SourceArn: !Sub
- arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/GET/hello
- __Stage__: '*'
__ApiId__: !Ref 'ServerlessRestApi'
ServerlessRestApi:
Type: AWS::ApiGateway::RestApi
Properties:
Body:
swagger: '2.0'
info:
version: '1.0'
title: !Ref 'AWS::StackName'
paths:
/hello:
get:
x-amazon-apigateway-integration:
httpMethod: POST
type: aws_proxy
uri: !Sub 'arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloWorldFunction.Arn}/invocations'
responses: {}
ServerlessRestApiProdStage:
Type: AWS::ApiGateway::Stage
Properties:
DeploymentId: !Ref 'ServerlessRestApiDeployment47fc2d5f9d'
RestApiId: !Ref 'ServerlessRestApi'
StageName: Prod
ServerlessRestApiDeployment47fc2d5f9d:
Type: AWS::ApiGateway::Deployment
Properties:
RestApiId: !Ref 'ServerlessRestApi'
Description: 'RestApi deployment id: 47fc2d5f9d21ad56f83937abe2779d0e26d7095e'
StageName: Stage
Of these, note this undocumented behaviour:
- A Lambda Permission HelloWorldFunctionHelloWorldPermissionTest is created. According to this comment in GitHub, this is apparently there so that the Test button will work in the AWS Lambda Console.
- Also note the StageName “Stage” referenced in the ServerlessRestApiDeployment47fc2d5f9d resource, and note this open bug that discusses it.
Of the remainder, all of them are documented in the docs/internals/generated_resources.rst page at GitHub but I found it a lot easier to understand when I used HashDiff on the before and after templates and inspected the generated code.
Understanding the generated resources becomes especially important if you need to refer via !Ref or !GetAtt to the generated resources in your SAM template. I’ll do just that in Part III.
Summary
So, that’s it for Part II. In summary, I have again looked at some important documentation links to have handy when learning SAM, discussed the “hello world” template in some detail with reference to the SAM template language features, shown how to get the CloudFormation template after it was processed by the SAM Translator, and discussed in some detail what the Translator actually changed.
Stay with me for Part III, where I modify the SAM template to add a CORS configuration and a proxy+ endpoint.
Further reading
- Orr Weinstein, 18 Nov 2016, Introducing Simplified Serverless Application Deployment and Management.
- Michael Wittig, 24 Apr 2017, AWS Velocity Series: Serverless app.
1 I have reformatted the YAML for better presentation in this blog. Whitespace and indentation changes only.
2 There is another way of getting the processed template from sam validate --debug:
▶ sam validate -t template.yaml --debug 2>&1 | \
gsed '0,/Translated template is/d;/is a valid SAM Template/Q' > processed.yml
The GNU sed command there is a fancy way of getting all lines in between two patterns. This will be formatted slightly differently, however, and be missing some of the data of the deployed stack, such as the bucket name and S3 key. It is useful for testing.
tags: sam - lambda - cors