How to handle multiple environments in CodePipeline?
I'm using code pipeline to deploy my infrastructure and I would like to be able to deploy it in different environments (dev, staging, prod,...).
I currently have a buildspec.yml file containing some "pip install" instructions and the "aws cloudformation package" command. I also created 2 pipelines, one for production and another for development pointing to 2 different branches on github. The problem I have is that since in both branches the files contains similar resources, I have name collision on the S3 buckets for example.
When using the AWS CLI and cloudformation to create or update a stack you can pass parameters using the --parameters option. I would like to do something similar in the 2 pipelines I've created.
What would be the best solution to tackle this issue?
The final goal is to automate the deployment of our infrastructure. Our infrastructure is composed of Users, KMS keys, Lamdbas (in python), Groups and Buckets.
I've created two pipelines following the tutorial: http://docs.aws.amazon.com/lambda/latest/dg/automating-deployment.html
The first pipeline is linked to the master branch of the repo containing the code and the second one to the staging branch. My goal is to automate the deployment of the master branch in the production environment using the first pipeline and the staging branch in the staging environment using the second one.
My buildspec.yml file look like:
version: 0.1
phases:
install:
commands:
- pip install requests -t .
- pip install simplejson -t .
- pip install Image -t .
- aws cloudformation package --template-file image_processing_sam.yml --s3-bucket package-bucket --output-template-file new_image_processing_sam.yml
artifacts:
type: zip
files:
- new_image_processing_sam.yml
The image_processing_sam.yml file look like:
AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: Create a thumbnail for an image uploaded to S3
Resources:
ThumbnailFunction:
Type: "AWS::Serverless::Function"
Properties:
Role: !GetAtt LambdaExecutionRole.Arn
Handler: create_thumbnail.handler
Runtime: python2.7
Timeout: 30
Description: "A function computing the thumbnail for an image."
LambdaSecretEncryptionKey:
Type: "AWS::KMS::Key"
Properties:
Description: "A key used to encrypt secrets used in the Lambda functions"
Enabled: True
EnableKeyRotation: False
KeyPolicy:
Version: "2012-10-17"
Id: "lambda-secret-encryption-key"
Statement:
-
Sid: "Allow administration of the key"
Effect: "Allow"
Principal:
AWS: "arn:aws:iam::xxxxxxxxxxxxx:role/cloudformation-lambda-execution-role"
Action:
- "kms:Create*"
- "kms:Describe*"
- "kms:Enable*"
- "kms:List*"
- "kms:Put*"
- "kms:Update*"
- "kms:Revoke*"
- "kms:Disable*"
- "kms:Get*"
- "kms:Delete*"
- "kms:ScheduleKeyDeletion"
- "kms:CancelKeyDeletion"
Resource: "*"
-
Sid: "Allow use of the key"
Effect: "Allow"
Principal:
AWS:
- !GetAtt LambdaExecutionRole.Arn
Action:
- "kms:Encrypt"
- "kms:Decrypt"
- "kms:ReEncrypt*"
- "kms:GenerateDataKey*"
- "kms:DescribeKey"
Resource: "*"
LambdaExecutionRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: "LambdaExecutionRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- "lambda.amazonaws.com"
Action:
- "sts:AssumeRole"
Policies:
-
PolicyName: LambdaKMS
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- "kms:Decrypt"
Resource: "*"
-
Effect: "Allow"
Action:
- "lambda:InvokeFunction"
Resource: "*"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
UserGroup:
Type: "AWS::IAM::Group"
LambdaTriggerUser:
Type: "AWS::IAM::User"
Properties:
UserName: "LambdaTriggerUser"
LambdaTriggerUserKeys:
Type: "AWS::IAM::AccessKey"
Properties:
UserName:
Ref: LambdaTriggerUser
Users:
Type: "AWS::IAM::UserToGroupAddition"
Properties:
GroupName:
Ref: UserGroup
Users:
- Ref: LambdaTriggerUser
Policies:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: UserPolicy
PolicyDocument:
Statement:
-
Effect: "Allow"
Action:
- "lambda:InvokeFunction"
Resource:
- !GetAtt DispatcherFunction.Arn
Groups:
- Ref: UserGroup
PackageBucket:
Type: "AWS::S3::Bucket"
Properties:
BucketName: "package-bucket"
VersioningConfiguration:
Status: "Enabled"
Outputs:
LambdaTriggerUserAccessKey:
Value:
Ref: "LambdaTriggerUserKeys"
Description: "AWSAccessKeyId of LambdaTriggerUser"
LambdaTriggerUserSecretKey:
Value: !GetAtt LambdaTriggerUserKeys.SecretAccessKey
Description: "AWSSecretKey of LambdaTriggerUser"
I've added a deploy action in both pipelines to execute the change set computed during the beta action.
The first pipeline works like a charm and does everything I expect it to do. Each time I push code in the master branch, it's deployed.
The issue I'm facing is that when I push code in the staging branch, everything works in the pipeline until the deploy action is reached. The deploy action try to create a new stack but since it's exactly the same buildspec.yml and the image_processing_sam.yml that is processed, I reach name collision like below.
package-bucket already exists in stack arn:aws:cloudformation:eu-west-1:xxxxxxxxxxxx:stack/master/xxxxxx-xxxx-xxx-xxxx
LambdaTriggerUser already exists in stack arn:aws:cloudformation:eu-west-1:xxxxxxxxxxxx:stack/master/xxxxxx-xxxx-xxx-xxxx
LambdaExecutionRole already exists in stack arn:aws:cloudformation:eu-west-1:xxxxxxxxxxxx:stack/master/xxxxxx-xxxx-xxx-xxxx
...
Is there a way to parameterize the buildspec.yml to be able to add a suffix to the name of the resources in the image_processing_sam.yml? Any other idea to achieve this is welcome.
Best regards.
Answer
Template configuration files are applied to the CloudFormation in the CodePipeline via a parameter file like this:
{
"Parameters" : {
"DBName" : "TestWordPressDB",
"DBPassword" : "TestDBRootPassword",
"DBRootPassword" : "TestDBRootPassword",
"DBUser" : "TestDBuser",
"KeyName" : "TestEC2KeyName"
}
}
Place these files in the root of your repo and they can be referenced in at least 2 ways.
In your CodePipeline CloudFormation:
Configuration:
ActionMode: REPLACE_ON_FAILURE
RoleArn: !GetAtt [CFNRole, Arn]
StackName: !Ref TestStackName
TemplateConfiguration: !Sub "TemplateSource::${TestStackConfig}"
TemplatePath: !Sub "TemplateSource::${TemplateFileName}"
Or in the console in the Template configuration field:
It is worth noting the config file format is different from CloudFormation via cli using
-- parameters
--parameters uses this format:
[
{
"ParameterKey": "team",
"ParameterValue": "AD-Student Life Applications"
},
{
"ParameterKey": "env",
"ParameterValue": "dev"
},
{
"ParameterKey": "dataSensitivity",
"ParameterValue": "public"
},
{
"ParameterKey": "app",
"ParameterValue": "events-list-test"
}
]
CodePipeline Cloudformation template configuration files use this format:
{
"Parameters" : {
"DBName" : "TestWordPressDB",
"DBPassword" : "TestDBRootPassword",
"DBRootPassword" : "TestDBRootPassword",
"DBUser" : "TestDBuser",
"KeyName" : "TestEC2KeyName"
}
}