Auto Create ECR Repos in Guided Deploy

What is the problem?

With the release of Lambda Container Images Support in SAM CLI, customers today have to specify a ECR Repo URI location where images will need to be uploaded by SAM CLI after having been built. This means that customers need to have pre-created resources ready (in this case ECR repos) to go, so that they can supply them during the deploy process. This introduces friction and break in the seamless workflow that sam deploy --guided normally offers, with customers having to figure out how to create a ECR repo and how to find correct ECR URI to specify.

Current flow for deploying a template with image based function:

Create ECR repo: aws ecr create-repository --repository-name myecr_repo
Deploy with SAM CLI guided: sam deploy --guided
Specify image repo: Image Repository for HelloWorldFunction []: 12345.dkr.ecr.us-west-2.amazonaws.com/helloworldfunctionrepo

What will be changed?

When deploying with guided, SAM CLI will prompt the option to auto create ECR repos for image based functions. The auto created ECR repos will reside in a companion stack that gets deployed along with the actual stack.

During each guided deploy, the functions and repos will be synced.

Each function without an image repo specified will have a corresponding repo created in the companion stack. If a function is deleted from the template and has an auto created image repo previously associated with it, the auto created image repo will also be removed.

There will be an escape hatch to use non SAM CLI managed repos by specifying --image-repositories or changing samconfig.toml.

Success criteria for the change

No context switching from SAM CLI to creating resources using other tools.

Out-of-Scope

SAM CLI will not manage lifecycles, creation and deletion, of the auto created resources outside of SAM CLI. For auto created image repos, modifications to the functions throught other means like console will not modify the associated image repo until the next deployment with SAM CLI.
Auto create repo only concerns about guided experience. Repos are assumed to be provided in CI/CD situations. However the option --resolve-image-repos will be added for auto creating repos without going through guided.

User Experience Walkthrough

sam deploy --guided

Creating New Repos

======================
Looking for config file [samconfig.toml] : Not found

Setting default arguments for 'sam deploy'
=========================================
Stack Name [sam-app]: images-app
AWS Region [us-east-1]: us-east-2
#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
Confirm changes before deploy [y/N]: y
#SAM needs permission to be able to create roles to connect to the resources in your template
Allow SAM CLI IAM role creation [Y/n]: y
Save arguments to configuration file [Y/n]: y
SAM configuration file [samconfig.toml]:
SAM configuration environment [default]:
Looking for resources needed for deployment:
 S3 bucket: Found! (aws-sam-cli-managed-default-samclisourcebucket-abcdef)
 Image repositories: Not found.
 #Managed repositories will be deleted when their functions are removed from the template and deployed
 Create managed ECR repositories for all functions? [Y/n]: Y
 Creating the required resources...
 Successfully created resources for deployment!

Successfully saved arguments to config file!
 #Running 'sam deploy' for future deployments will use the parameters saved above.
 #The above parameters can be changed by modifying samconfig.toml 
 #Learn more about samconfig.toml syntax at 
 #https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

Uploading to abcdefg/7dd33d96eafcae1086a1356df982d38e 539284 / 539284.0 (100.00%)	

Deploying with the following values
===================================
Stack name                 : test-stack
Region                     : us-east-2
Confirm changeset          : True
Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-abcdef
Image repositories         : [
 {“helloWorldFunction1”:"12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction1-abcde"},
 {“helloWorldFunction2”:"12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction2-abcde"},
 {“helloWorldFunction3”:"12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction3-abcde”}
]
Capabilities               : ["CAPABILITY_IAM"]
Parameter overrides        : {}
Signing profiles           : {}

Initiating deployment

Deleting Unreferenced Repos

Configuring SAM deploy
======================

        Looking for config file [samconfig.toml] :  Found
        Reading default arguments  :  Success

        Setting default arguments for 'sam deploy'       
        =========================================        
        Stack Name [test-stack]:
        AWS Region [us-east-2]: 
        #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
        Confirm changes before deploy [Y/n]: y
        #SAM needs permission to be able to create roles to connect to the resources in your template
        Allow SAM CLI IAM role creation [Y/n]: y
        HelloWorldFunction4 may not have authorization defined, Is this okay? [y/N]: y
        HelloWorldFunction5 may not have authorization defined, Is this okay? [y/N]: y
        Save arguments to configuration file [Y/n]: y
        SAM configuration file [samconfig.toml]: 
        SAM configuration environment [default]: 

        Looking for resources needed for deployment: Found!

                Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-abcdef
                A different default S3 bucket can be set in samconfig.toml

                Image repositories: Not found.
                #Managed repositories will be deleted when their functions are removed from the template and deployed
                Create managed ECR repositories for all functions? [Y/n]: y
                Checking for unreferenced ECR repositories to clean-up: 2 found
                 12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction1-abcde
                 12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction2-abcde
                 Delete the unreferenced repositories listed above when deploying? [y/N]: y

        helloworldfunction4:python3.8-v1 to be pushed to 12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction4-abcde:helloworldfunction4-7bfff073dfcf-python3.8-v1
        helloworldfunction5:python3.8-v1 to be pushed to 12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction5-abcde:helloworldfunction5-7bfff073dfcf-python3.8-v1


        Saved arguments to config file
        Running 'sam deploy' for future deployments will use the parameters saved above.
        The above parameters can be changed by modifying samconfig.toml
        Learn more about samconfig.toml syntax at
        https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html


        Deploying with following values
        ===============================
        Stack name                   : auto-ecr-guided-test
        Region                       : us-west-2
        Confirm changeset            : True
        Deployment image repository  :
                                       {
                                           "HelloWorldFunction3": "12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction3-abcde",
                                           "HelloWorldFunction4": "12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction4-abcde",
                                           "HelloWorldFunction5": "12345.dkr.ecr.us-east-2.amazonaws.com/helloworldfunction5-abcde"
                                       }
        Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-abcde
        Capabilities                 : ["CAPABILITY_IAM"]
        Parameter overrides          : {}
        Signing Profiles             : {}

--resolve-image-repos

This new option will have the exact behavior as running guided with confirmation for all image repo related prompts.

To be more specific, this option will:

Create image repos for functions are not linked to image repos with the --image-repositories
Delete auto created image repos that do not have functions associated with them anymore

Destructive operations are done for the following reasons:

This will keep a consistent behavior as the guided. In guided, SAM CLI will abort deployment if deletion of auto created image repos is denied.
For UX, this will avoid image repos and functions mapping to an invalid state where orphaned image repos exist. For this case, we will also need to track which repos should be kept and makes the sync less robust.
From security perspective, keeping old image repos will increase the impact radius of an information leakage. A customer might expect a sensitive image repo will be deleted as soon as the function itself is also removed like the guided experience.

Implementation

CLI Changes

Add new prompt for guided deploy.
- Create managed ECR repositories for all functions? [Y/n]: y
Add --resolve-image-repos to non-guided deploy

Breaking Change

Not a breaking change.

Design

Companion Stack Naming Scheme

#Escaped StackName with only common accpetable characters
escaped_stack_name = re.sub(r"[^a-z0-9]", "", stack_name.lower())
#Escaped LambdaName with only common accpetable characters
escaped_lambda_logical_id = re.sub(r"[^a-z0-9]", "", lambda_logical_id.lower())
#MD5 of the original StackName.
stack_md5 = hash.str_checksum(stack_name)
#MD5 of the original LambdaName
function_md5 = hash.str_checksum(lambda_logical_id)

#MD5 is used to avoid two having the same escaped name with different Lambda Functions
#For example: Helloworld and HELLO-WORLD
repo_logical_id =
    lambda_logical_id [:52] + function_md5 [:8] + "Repo"
    #52 + 8 + 4 = 64 max char
    
repo_name = 
    escaped_stack_name + stack_md5[:8] + "/" + escaped_lambda_logical_id + function_md5[:8] + "repo"
    #128        +         8      +  1  +     64      +        8      +      4 = 213 max char
    
companion_stack_name = 
    stack_name[:104] + "-" + stack_md5[:8] + "-" + "CompanionStack"
    #104 + 1 + 8 + 1 + 14 = 128 max char
    
repo_output_logical_id = 
    lambda_logical_id[:52] + function_md5 [:8] + "Out"
    #52 + 8 + 3 = 63 max char

Exmaple:
    Input:
        Customer Stack Name: Hello-World-Stack
        Function 1 Logical ID: TestFunction01
        Function 2 Logical ID: AnotherTestFunction02
    Output:
        Companion Stack Name: Hello-World-Stack-925976eb-CompanionStack
        Function 1 Repo Logical ID: TestFunction0150919004Repo
        Function 1 Repo Name: helloworldstack925976eb/testfunction0150919004repo
        Function 2 Repo Logical ID: AnotherTestFunction025c2cfd8cRepo
        Function 2 Repo Name: helloworldstack925976eb/anothertestfunction025c2cfd8crepo

Companion Stack Structure

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: AWS SAM CLI Managed ECR Repo Stack
Metadata:
  SamCliInfo: 1.18.0
  CompanionStackname:  Hello-World-Stack-925976eb-CompanionStack

Resources:

  TestFunction0150919004Repo:
    Type: AWS::ECR::Repository
    Properties:
      RepositoryName: helloworldstack925976eb/testfunction0150919004repo
      Tags:
        - Key: ManagedStackSource
          Value: AwsSamCli
        - Key: AwsSamCliCompanionStack
          Value: Hello-World-Stack-925976eb-CompanionStack

      RepositoryPolicyText: 
        Version: "2012-10-17"
        Statement:
          -
            Sid: AllowLambdaSLR
            Effect: Allow
            Principal:
              Service:
                - "lambda.amazonaws.com"
            Action:
                - "ecr:GetDownloadUrlForLayer"
                - "ecr:GetRepositoryPolicy"
                - "ecr:BatchGetImage"

  AnotherTestFunction025c2cfd8cRepo:
    Type: AWS::ECR::Repository
    Properties:
      RepositoryName: helloworldstack925976eb/anothertestfunction025c2cfd8crepo
      Tags:
        - Key: ManagedStackSource
          Value: AwsSamCli
        - Key: AwsSamCliCompanionStack
          Value: Hello-World-Stack-925976eb-CompanionStack

      RepositoryPolicyText: 
        Version: "2012-10-17"
        Statement:
          -
            Sid: AllowLambdaSLR
            Effect: Allow
            Principal:
              Service:
                - "lambda.amazonaws.com"
            Action:
                - "ecr:GetDownloadUrlForLayer"
                - "ecr:GetRepositoryPolicy"
                - "ecr:BatchGetImage"

Outputs:

  None:
    Value: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.${AWS::URLSuffix}/${TestFunction0150919004Repo}

  None:
    Value: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.${AWS::URLSuffix}/${AnotherTestFunction025c2cfd8cRepo}

Documentation Changes

New option --resolve-image-repos. This option will auto create/delete repos without the needs of going through guided experience.

Open Issues

https://github.com/aws/aws-sam-cli/issues/2447

Task Breakdown

[x] Send a Pull Request with this design document
[ ] Build Companion Stack Manager
[ ] Change Deploy CLI
[ ] Unit tests
[ ] Functional Tests
[ ] Integration tests
[ ] Update documentation

Auto Create Ecr