Amazon Simple Storage Service (also known as Amazon S3) is a well-known Amazon Web Services (AWS) service that customers can use to store data securely and reliably. Using Amazon S3, businesses will be able to build a low-cost, yet highly available storage solution. To deserve 11 numbers 9 (99.9999999999%) in terms of SLAs, Amazon S3 offers features available for replication to another region, which allows for easy automatic backups of various regions (Multi-region Replication).
Overview
The article below references an AWS CDK (Cloud Development Kit) project using TypeScript. Using AWS CDK together with AWS CloudFormation StackSets, customers can deploy the following resources:
-
Amazon S3 Bucket on the primary region with custom KMS key.
-
AWS CloudFormation StackSet replicates copies to different regions.
-
AWS IAM Role with access to the primary region and copies.
Interestingly, you can deploy these resources on the desired Regions, WITH ONLY 01 SINGLE COMMAND !!!
Steps to proceed deploying Multi-Region S3 Replication
Step 1: Prerequisite
To use Self-managed StackSets, you first need to create two IAM roles. You can create them manually using the AWS Management Console or use the official CloudFormation templates provided by AWS, click to download:
This project is taken from a Repository containing pre-packaged templates on Github, link here. Just adjust some parameters for CloudFormation Stack to be usable.
// Configured ./aws/index.ts
const option = {
env: {
region: 'ap-southeast-1'
},
replications: [
'ap-northeast-1',
'ap-southeast-2',
'...'
]
}Then, run using the command below to deploy everything.
npx cdk deployStep 2: Edit parameters of Primary Region and Data Source
Changes to data inside amazon S3 buckets in primary regions are replicated to other AWS regions, for example here the main region VTI Cloud is making ap-southeast-1 (Singapore) and other regions ap-northeast-1 (Tokyo) and ap-southeast-2 (Sydney).
In primary region, you need Amazon S3 bucket with custom KMS (Key Management System) key for encryption.
Learn more about AWS KMS here: https://aws.amazon.com/kms/.
import * as s3 from '@aws-cdk/aws-s3'
import * as kms from '@aws-cdk/aws-kms'
const key = new kms.Key(this, 'Key')
const alias = key.addAlias('archive')
const bucket = new s3.Bucket(this, 'Bucket', {
bucketName: `${props.prefix}-archive`,
encryption: s3.BucketEncryption.KMS,
encryptionKey: alias,
blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
bucketKeyEnabled: true,
versioned: true,
removalPolicy: cdk.RemovalPolicy.RETAIN
})To use S3 bucket replication, you need to create an IAM role with permission to access data in Amazon S3 and use kms key:
import * as iam from '@aws-cdk/aws-iam'
const role = new iam.Role(this, 'ReplicationRole', {
assumedBy: new iam.ServicePrincipal('s3.amazonaws.com'),
path: '/service-role/'
});
role.addToPolicy(
new iam.PolicyStatement({
resources: [
bucket.bucketArn
],
actions: [
's3:GetReplicationConfiguration',
's3:ListBucket'
]
})
);
role.addToPolicy(
new iam.PolicyStatement({
resources: [
bucket.arnForObjects('*')
],
actions: [
's3:GetObjectVersion',
's3:GetObjectVersionAcl',
's3:GetObjectVersionForReplication',
's3:GetObjectVersionTagging'
]
})
);
role.addToPolicy(
new iam.PolicyStatement({
resources: [
key.keyArn
],
actions: [
'kms:Decrypt'
]
})
);After completing the above steps, the next step is to create an Amazon S3 bucket with a KMS key that can be used in any region you want to replicate, here VTI Cloud configures the KMS key in the regionap-northeast-1 (Tokyo) and ap-southeast-2 (Sydney).
Step 3: Creat CloudFormation StackSet for Multi-Region S3 Replication
To avoid having to create each CloudFormation Stack in each region you want to replicate amazon S3 bucket data, AWS CloudFormation StackSet is used to automate deployment from the region. Currently, AWS CDK only supports low-level access to CloudFormation StackSet resources:
import * as cdk from '@aws-cdk/core'
new cdk.CfnStackSet(this, "StackSet", {
stackSetName: `${props.prefix}-archive-replication`,
permissionModel: "SELF_MANAGED",
parameters: [
{
parameterKey: 'Prefix',
parameterValue: props.prefix
},
{
parameterKey: 'ReplicationRole',
parameterValue: role.roleArn
}
],
stackInstancesGroup: [
{
regions: props.replications,
deploymentTargets: {
accounts: [AccountId],
},
},
],
templateBody:templateReplicationData, #đây là template chứa Amazon S3 và KMS cho mọi region
});The parameterReplicationRole is required to grant access to KMS keys by region that IAM role uses to perform replication.
Parameters:
Prefix:
Type: String
ReplicationRole:
Type: String
Resources:
Key:
Type: AWS::KMS::Key
Properties:
KeyPolicy:
Version: 2012-10-17
Id: access-account
Statement:
- Sid: Enable IAM User Permissions
Effect: Allow
Principal:
AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
Action: kms:*
Resource: '*'
- Sid: Replication
Effect: Allow
Principal:
AWS: !Ref ReplicationRole
Action:
- kms:Encrypt
- kms:ReEncrypt*
- kms:GenerateDataKey*
- kms:DescribeKey
Resource: '*'
KeyAlias:
Type: AWS::KMS::Alias
Properties:
AliasName: alias/archive/replication
TargetKeyId: !Ref Key
Bucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
Properties:
BucketName: !Sub ${Prefix}-archive-replication-${AWS::Region}
AccessControl: Private
PublicAccessBlockConfiguration:
BlockPublicAcls: Yes
BlockPublicPolicy: Yes
IgnorePublicAcls: Yes
RestrictPublicBuckets: Yes
VersioningConfiguration:
Status: Enabled
BucketEncryption:
ServerSideEncryptionConfiguration:
- BucketKeyEnabled: Yes
ServerSideEncryptionByDefault:
SSEAlgorithm: aws:kms
KMSMasterKeyID: !Sub arn:aws:kms:${AWS::Region}:${AWS::AccountId}:${KeyAlias}AWS resources in the region are replicated using their own name patterns to differentiate, VTI Cloud will example is ap-northeast-1 in the following configuration:
-
KMS Key
arn:aws:kms:ap-northeast-1:11223344:alias/archive/replication -
S3 Bucket
arn:aws:s3:::prefix-archive-replication-ap-northeast-1
Using this name pattern, you can extend the IAM role used for region replication from the primary region:
role.addToPolicy( new iam.PolicyStatement({
resources: props.replications.map(
region => `arn:aws:kms:${region}:$(AccountId):alias/archive/replication`
),
actions: [
'kms:Encrypt'
]
})
);
role.addToPolicy(
new iam.PolicyStatement({
resources: props.replications.map(
region => `arn:aws:s3:::${props.prefix}-archive-replication-${region}/*`
),
actions: [
's3:ReplicateDelete',
's3:ReplicateObject',
's3:ReplicateTags'
]
})
);
role.addToPolicy(
new iam.PolicyStatement({
resources: props.replications.map(
region => `arn:aws:s3:::${props.prefix}-archive-replication-${region}`
),
actions: [
's3:List*',
's3:GetBucketVersioning',
's3:PutBucketVersioning'
]
})
);Step 4: Configure to deploy Multi-Region S3 Replication
Set up S3 Replication Configuration at Amazon S3 bucket in the primary region. In this step, AWS CDK can still use low-level objects to configure replication:
const cfnBucket = bucket.node.defaultChild as s3.CfnBucket;
cfnBucket.replicationConfiguration = {
role: role.roleArn,
rules: props.replications.map(
(region, index) => (
{
id: region,
destination: {
bucket: `arn:aws:s3:::${props.prefix}-archive-replication-${region}`,
encryptionConfiguration: {
replicaKmsKeyId: `arn:aws:kms:${region}:${AccountID}:alias/archive/replication`
}
},
priority: index,
deleteMarkerReplication: {
status: 'Enabled'
},
filter: {
prefix: ''
},
sourceSelectionCriteria: {
sseKmsEncryptedObjects: {
status: 'Enabled'
}
},
status: 'Enabled'
}
)
)
}Step 5: Deploy
As said above, after running using the command below, everything will be implemented with this command:
npx cdk deployConclusion
With only the AWS Cloud Development Kit, you can create the AWS CloudFormation template.
From this template, you can create the AWS CloudFormation Stack. This stack will help you deploy services such as Amazon S3 bucket, AWS Identity & Access Management role, an AWS Key Management Service key, and 01 AWS CloudFormation StackSet.
AWS CloudFormation StackSet then uses the template above to create AWS CloudFormation Stack for different regions, and this Stack continues to be used to create resources such as S3 bucket, IAM role, and KMS key, where Amazon S3 bucket features S3 Replication Configuration.
About VTI Cloud
VTI Cloud is an Advanced Consulting Partner of AWS Vietnam with a team of over 50+ AWS certified solution engineers. With the desire to support customers in the journey of digital transformation and migration to the AWS cloud, VTI Cloud is proud to be a pioneer in consulting solutions, developing software, and deploying AWS infrastructure to customers in Vietnam and Japan.
Building safe, high-performance, flexible, and cost-effective architectures for customers is VTI Cloud’s leading mission in enterprise technology mission.
