2 March 2019

Introduction to SAM Part II: Template and architecture

by Alex Harvey

This is Part II of my 3-part blog series on Amazon’s Serverless Application Model (SAM). Part I is here and I recommend reading that first.

Overview to Part II
Important documentation
Architecture
The SAM template
Generated resources
Summary
Further reading

Overview to Part II

In this second part of my series, I look at the SAM Translator and its docs, and discuss line by line the example “hello world” template. Along the way I discuss key concepts in SAM such as the AWS::Serverless types, the implicit API, the default Role and Stage, as well as the generated CloudFormation resources.

Important documentation

I begin with an overview of SAM’s documentation, extended from Part I to cover the SAM Translator. I do this because, at the time of writing, the official docs are not always up to date or well organised and understanding key concepts, such as the implicit API, can involve consulting multiple documents.

So, the additional documents to look at for the SAM Translator and the template language are:

The Markdown docs in the source code repo, especially:
- The SAM Specification 2016-10-31 document, which provides a technical description of the template language, the Resource Types, their Properties, Data Types and so on.
- The docs directory, which includes:
  - globals.rst, which documents the SAM template’s Globals section and is a key reference for this section
  - internals/generated_resources.rst, which is another key reference that documents the CloudFormation resources that are generated by the SAM resources and also their naming conventions, which is important if you need to refer to them via !Ref or !GetAtt.
  - cloudformation_compatibility.rst, which discusses the support for CloudFormation intrinsic functions implemented so far in SAM. (See also the HOWTO.md.)
  - faq.rst, is a SAM FAQ that has just three questions at this time, including how to manage multiple environments, and how to enable API Gateway logs.
  - policy_templates.rst discusses SAM’s out-of-the-box IAM Policies that are found in the policy_templates.json file.
  - And finally, safe_lambda_deployments.rst discusses traffic shifting for safe production deployments. (Also discussed again in the SAM Developer Guide here.)
The SAM developer guide, especially:
- AWS SAM Template Basics documents the resource types (but I expect the specification document above to be more up to date); contains info on the Metadata section of templates and how it relates to publishing SAM apps in the AWS Serverless Application Repository; and nested applications, which seems to be simply the nested stacks feature of CloudFormation.
Finally, the examples directory in the source code is especially useful for finding SAM apps and templates you can use a starting point for your own app.

Architecture

As mentioned earlier, SAM’s architecture consists of two main parts: the SAM CLI (discussed in Part I) and the SAM Translator, also known as the SAM Transformer.

The SAM Translator is called by the CloudFormation Service. CloudFormation recognises a top-level template key Transform: AWS::Serverless-2016-10-31 and invokes the SAM Translator if it is found. The Translator then takes the SAM template and expands it into a full CloudFormation template that is executed by CloudFormation to create, update and/or delete AWS resources.

The entry point in the SAM source code is in the Translator class here. It can be seen the translate method expects a template and its parameters as arguments, then it iterates through AWS::Serverless resources, and returns a CloudFormation template.

The SAM template

In Part I, I used the SAM CLI to initialise, build and deploy the example “hello world” Python 2.7 application. In this section, I look at the SAM template that was created.

The “hello world” code example

Here is the template in full¹:

---
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app
  Sample SAM Template for sam-app

# More info about Globals:
# https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
#
Globals:
  Function:
    Timeout: 3

Resources:

  # More info about Function Resource:
  # https://github.com/awslabs/serverless-application-model/blob/master/
  #   versions/2016-10-31.md#awsserverlessfunction
  #
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python2.7

      # More info about API Event Source:
      # https://github.com/awslabs/serverless-application-model/blob/master/
      #   versions/2016-10-31.md#api
      #
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get

Outputs:

  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/
  #   docs/internals/generated_resources.rst#api
  #
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"

  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn

  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

In the remainder of this section I discuss this line by line with reference to the SAM template features.

The Transform

The first thing to note is that the SAM template is a marked up CloudFormation template. The line AWSTemplateFormatVersion: '2010-09-09' identifies the CloudFormation template format version. And CloudFormation users will be well familiar with that.

More interesting is the second line Transform: AWS::Serverless-2016-10-31 that identifies the SAM template version. And as already mentioned, it is the presence of this line that causes the CloudFormation service to call the SAM Translator in the backend.

The Globals section

The Globals section is one of the template language extensions read by the SAM Translator.

In the example template, the following snippet sets the Lambda function timeout value to 3 seconds globally:

Globals:
  Function:
    Timeout: 3

Values specified in the Globals section are merged into the configs of the generated CloudFormation resources. The thinking in the SAM design is that resources in a SAM template will tend to have shared configuration like runtime, memory settings, environment variables, CORS config and so on and this config would otherwise be duplicated.

There is important documentation about the rules for merging here, but in summary: scalars are replaced; maps are merged; and lists are additive.

At the time of writing, the SAM types AWS::Serverless::Function, AWS::Serverless::Api and AWS::Serverless::SimpleTable can all be further configured from the Globals section.

The serverless function

The Lambda function itself is the HelloWorldFunction resource of type AWS::Serverless::Function:

HelloWorldFunction:
  Type: AWS::Serverless::Function
  Properties:
    CodeUri: hello_world/
    Handler: app.lambda_handler
    Runtime: python2.7
    Events:
      HelloWorld:
        Type: Api
        Properties:
          Path: /hello
          Method: get

The AWS::Serverless::Function type is documented at the versions/2016-10-31.md#awsserverlessfunction page mentioned in the template.

Comparing the function types

The AWS::Serverless::Function type is of course central to SAM and it is worth comparing it with the CloudFormation AWS::Lambda::Function type that SAM transforms it into.

Here, I present an sdiff on the signatures of these types from the documentation in order to show their similarities and differences:

Type: "AWS::Lambda::Function"           | Type: "AWS::Serverless::Function"
Properties:                               Properties:
                                        >   AutoPublishAlias: String
  Code:                                 |   CodeUri:
    Code                                |     CodeUri
  DeadLetterConfig:                     |   DeadLetterQueue:
    DeadLetterConfig                    |     DeadLetterQueue
                                        >   DeploymentPreference:
                                        >     DeploymentPreference
  Description: String                       Description: String
  Environment:                              Environment:
    Environment                               Environment
                                        >   Events:
                                        >     Events
  FunctionName: String                      FunctionName: String
  Handler: String                           Handler: String
                                        >   InlineCode: String
  KmsKeyArn: String                         KmsKeyArn: String
  Layers:                                   Layers:
    - String                                  - String
  MemorySize: Integer                       MemorySize: Integer
                                        >   Policies:
                                        >     Policies
  ReservedConcurrentExecutions: Integer     ReservedConcurrentExecutions: Integer
  Role: String                              Role: String
  Runtime: String                           Runtime: String
  Timeout: Integer                          Timeout: Integer
  TracingConfig:                        |   Tracing: String
    TracingConfig                       <
  VpcConfig:                                VpcConfig:
    VPCConfig                                 VPCConfig
  Tags:                                 |   Tags:
    Resource Tag                              Resource Tag

It is important to be aware of this:

Some of the apparent differences between the SAM type and its corresponding CloudFormation type are just naming inconsistencies. Thus, the DeadLetterConfig and TracingConfig properties in the CloudFormation type have been renamed DeadLetterQueue and Tracing in the SAM type. Likewise, the CodeUri and InlineCode properties are renamed features of the Code property of the CloudFormation type.

Meanwhile, The Policies property is a special property that just allows us to modify the Policies of the default IAM Role. (More on the default Role below.)

Now, leaving aside superficial differences, the features provided by the SAM type that didn’t exist in the CloudFormation type are:

Property	Description
Events	A map of Event source objects that defines the events that trigger this function. More on this when I discuss the implicit API below.
AutoPublishAlias	Name of the Alias. Read the AutoPublishAlias Guide for how it works.
DeploymentPreference	Settings to enable Safe Lambda Deployments. Read the usage guide for detailed information.

Of these, I haven’t looked at AutoPublishAlias or DeploymentPreference, and I will omit further discussion of them at this time.

The implicit API and the Events source

A selling point of SAM is that it is requires less CloudFormation code to create some of the frequently required supporting AWS resources like the API Gateway and Lambda execution Role. In support of that are the “implicit API” and the “default” resources, the default IAM Role, the default Stage, the default Deployment, all of which are a bit confusing at first.

In the Events property of the HelloWorldFunction, we have:

Events:
  HelloWorld:
    Type: Api
    Properties:
      Path: /hello
      Method: get

And now the confusing bit:

By mentioning an event source of Type: Api that isn’t otherwise defined in the template and referred to via the RestApiId property, a resource of type AWS::Serverless::Api and a Swagger definition is created by SAM. These are known in the documentation as “implicit APIs”.

Meanwhile, the new type AWS::Serverless::Api creates a collection of API Gateway resources, allowing us to avoid defining a complex system of API Gateway resources in CloudFormation the old way. And to be clear, this is a big win, because an API Gateway consists of a lot of CloudFormation resources and a lot of code that is both difficult to understand and time-consuming to write.

Note that the implict API can be referenced within the template using the special variable ${ServerlessRestApi}.

The default Role

Another implicit resource create by this template is the default Lambda Execution Role, usually referred to as the “default Role” (although referred to in the “hello world” template as the “implicit IAM Role”). As with the implicit API, this default IAM Role is created silently unless you refer via the Role property of the function resource to either an IAM Role that you define explicitly in the template or to an IAM Role that is otherwise known to exist.

Note that the “hello world” template refers to the default Role in the Outputs section:

HelloWorldFunctionIamRole:
  Description: "Implicit IAM Role created for Hello World function"
  Value: !GetAtt HelloWorldFunctionRole.Arn

Note that its name is the function name + Role and its ARN can be obtained as shown, using !GetAtt. This naming is documented here.

Now I mentioned above that the function has a Policies property that the CloudFormation function doesn’t have. By specifying a list of IAM Policies on the AWS::Serverless::Function type, these are added to the default role.

The default Stage

There is also a default stage named “Prod” created that cannot be configured.

The default stage is also mentioned indirectly in the Outputs section in the API endpoint:

HelloWorldApi:
  Description: "API Gateway endpoint URL for Prod stage for Hello World function"
  Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"

The default stage is documented here.

Generated resources

The business of building and deploying the SAM template was discussed in Part I, and in this section I head to the AWS Console to find the CloudFormation stack I created when I deployed the SAM template.

The documentation has an important section on the resources generated by SAM, but it’s sometimes easiest to understand what SAM is doing just by inspecting the actual CloudFormation stack after deployment. Here is the view of the stack resources:

SAM resources

This is familiar territory for CloudFormation users, although it will be immediately apparent that the actual resources in the stack are different from the ones we put in the template. And of course, that’s the point.

More interesting is the new section available for SAM templates, the View Original Template and View Processed Template options, as seen here:

SAM processed template

To find out what the SAM Translator did I can use a combination of the cfn_flip library to convert from JSON to YAML, and then the Ruby HashDiff library.

To get the processed template body of the processed template as YAML from the AWS CLI²:

▶ aws cloudformation get-template --stack-name HelloWorld \
    --template-stage Processed --query TemplateBody | cfn-flip -y > processed.yml

And then to compare the differences using the Ruby HashDiff library:

▶ ruby -rhashdiff -rawesome_print -ryaml \
    -e "ap HashDiff.diff(*ARGV.map{|f| YAML.load_file(f)})" template.yaml processed.yml

This command returns an array of 15 diffs.

For readability I will present the first 9 as they appear in the Ruby Hashdiff output, and then the remainder - all of which add CloudFormation generated resources to the processed template - I’ll simply present as the generated YAML.

Here are the first 9 diffs:

[
  [ 0] [
    [0] "-",
    [1] "Globals", [2] { "Function" => { "Timeout" => 3 } }
  ],
  [ 1] [
    [0] "-",
    [1] "Transform", [2] "AWS::Serverless-2016-10-31"
  ],
  [ 2] [
    [0] "-",
    [1] "Resources.HelloWorldFunction.Properties.CodeUri", [2] "hello_world/"
  ],
  [ 3] [
    [0] "-",
    [1] "Resources.HelloWorldFunction.Properties.Events", [2] {
      "HelloWorld" => {
        "Type"       => "Api",
        "Properties" => {
          "Path"   => "/hello",
          "Method" => "get"
        }
      }
    }
  ],
  [ 4] [
    [0] "+",
    [1] "Resources.HelloWorldFunction.Properties.Code", [2] {
      "S3Bucket" => "alexharvey3118",
      "S3Key"    => "447c06bbc03dcd1b23220d2450918b99"
    }
  ],
  [ 5] [
    [0] "+",
    [1] "Resources.HelloWorldFunction.Properties.Role", [2] "HelloWorldFunctionRole.Arn"
  ],
  [ 6] [
    [0] "+",
    [1] "Resources.HelloWorldFunction.Properties.Tags", [2] [
      [0] {
        "Value" => "SAM",
        "Key"   => "lambda:createdBy"
      }
    ]
  ],
  [ 7] [
    [0] "+",
    [1] "Resources.HelloWorldFunction.Properties.Timeout", [2] 3
  ],
  [ 8] [
    [0] "~",
    [1] "Resources.HelloWorldFunction.Type",
    [2] "AWS::Serverless::Function",
    [3] "AWS::Lambda::Function"
  ],

To summarise these diffs:

the Globals and Transform sections were removed, as was the Events properties of the HelloWorld function
the HelloWorldFunction’s type is changed from AWS::Serverless::Function to AWS::Lambda::Function
its CodeUri is changed from:

CodeUri: hello_world/

to:

Code:
  S3Bucket: alexharvey3118
  S3Key: 447c06bbc03dcd1b23220d2450918b99

a Role property was added and set to HelloWorldFunctionRole.Arn
the timeout of 3 was added based on the value that was specified in the Globals section
and a Stack Tag was added:

Tags:
  - Value: SAM
    Key: lambda:createdBy

The generated HelloWorldFunction resource is thus:

HelloWorldFunction:
  Type: AWS::Lambda::Function
  Properties:
    Code:
      S3Bucket: alexharvey3118
      S3Key: 447c06bbc03dcd1b23220d2450918b99
    Handler: app.lambda_handler
    Runtime: python2.7
    Tags:
      - Value: SAM
        Key: lambda:createdBy
    Role: !GetAtt 'HelloWorldFunctionRole.Arn'
    Timeout: 3

In addition to these changes, the following CloudFormation code is also generated, showing the default Role, the default Lambda permission, the implicit API named ServerlessRestApi, and the default Stage and Deployment:

HelloWorldFunctionRole:
  Type: AWS::IAM::Role
  Properties:
    ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Action:
            - sts:AssumeRole
          Effect: Allow
          Principal:
            Service:
              - lambda.amazonaws.com
HelloWorldFunctionHelloWorldPermissionProd:
  Type: AWS::Lambda::Permission
  Properties:
    Action: lambda:invokeFunction
    Principal: apigateway.amazonaws.com
    FunctionName: !Ref 'HelloWorldFunction'
    SourceArn: !Sub
      - arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/GET/hello
      - __Stage__: Prod
        __ApiId__: !Ref 'ServerlessRestApi'
HelloWorldFunctionHelloWorldPermissionTest:
  Type: AWS::Lambda::Permission
  Properties:
    Action: lambda:invokeFunction
    Principal: apigateway.amazonaws.com
    FunctionName: !Ref 'HelloWorldFunction'
    SourceArn: !Sub
      - arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/GET/hello
      - __Stage__: '*'
        __ApiId__: !Ref 'ServerlessRestApi'
ServerlessRestApi:
  Type: AWS::ApiGateway::RestApi
  Properties:
    Body:
      swagger: '2.0'
      info:
        version: '1.0'
        title: !Ref 'AWS::StackName'
      paths:
        /hello:
          get:
            x-amazon-apigateway-integration:
              httpMethod: POST
              type: aws_proxy
              uri: !Sub 'arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloWorldFunction.Arn}/invocations'
            responses: {}
ServerlessRestApiProdStage:
  Type: AWS::ApiGateway::Stage
  Properties:
    DeploymentId: !Ref 'ServerlessRestApiDeployment47fc2d5f9d'
    RestApiId: !Ref 'ServerlessRestApi'
    StageName: Prod
ServerlessRestApiDeployment47fc2d5f9d:
  Type: AWS::ApiGateway::Deployment
  Properties:
    RestApiId: !Ref 'ServerlessRestApi'
    Description: 'RestApi deployment id: 47fc2d5f9d21ad56f83937abe2779d0e26d7095e'
    StageName: Stage

Of these, note this undocumented behaviour:

A Lambda Permission HelloWorldFunctionHelloWorldPermissionTest is created. According to this comment in GitHub, this is apparently there so that the Test button will work in the AWS Lambda Console.
Also note the StageName “Stage” referenced in the ServerlessRestApiDeployment47fc2d5f9d resource, and note this open bug that discusses it.

Of the remainder, all of them are documented in the docs/internals/generated_resources.rst page at GitHub but I found it a lot easier to understand when I used HashDiff on the before and after templates and inspected the generated code.

Understanding the generated resources becomes especially important if you need to refer via !Ref or !GetAtt to the generated resources in your SAM template. I’ll do just that in Part III.

Summary

So, that’s it for Part II. In summary, I have again looked at some important documentation links to have handy when learning SAM, discussed the “hello world” template in some detail with reference to the SAM template language features, shown how to get the CloudFormation template after it was processed by the SAM Translator, and discussed in some detail what the Translator actually changed.

Stay with me for Part III, where I modify the SAM template to add a CORS configuration and a proxy+ endpoint.

alexharv074.github.io

My blog