When you create a general-purpose S3 bucket, AWS automatically redundantly saves all data across a minimum of 3 availability zones in the region where the bucket was created. This allows for 99.999999999% file durability, which ensures that you will never lose any of your data. Even with that level of data security, having the data replicated to buckets in other regions or accounts may be necessary.
- An AWS region failure or temporary outage
- The company requires files to be retained for compliance
- Stash copies of your data in multiple locations to safeguard against deletion and changes to the original data
- Have data in locations closer to customers or employees
Whatever the case may be, AWS offers a built-in solution to keep your data in as many locations as you want. By using Replication Rules, you can tell AWS what data to move and where to move it. Below are the steps to replicate to a bucket in a different account via the GUI; you can also opt to do them via the API or CLI.
1. Replication requires that both the source and destination buckets have versioning enabled. Versioning will save a copy of each object if another object of the same name replaces it. This can be set up under the Properties of each bucket. Only new objects created after enabling replication will be replicated. To replicate existing objects, you will need to create a Batch Operation.
2. Inside your bucket, go to the Management tab and select Create replication rule.
3. Give your rule a unique name. Since you can have multiple rules per bucket, be sure to use a naming convention that conveys what this rule does.
4. You can choose to limit the rule to particular objects or apply it to all objects.
- Filter by prefix – In S3 terms, objects are stored in a flat namespace. This means there are no actual directories or folders, even though there appear to be. The directory portion of each object is referred to as its prefix. Object test/logs/test.txt has a prefix of test/logs/. A prefix filter could be used here to apply only to objects under the test/ or test/logs/ prefix.
- Filter by tags – Each object in S3 can have multiple tags assigned. These key-value pairs can be considered similar to metadata for each object. Tags can help group objects and manage them. A tag filter could apply only to objects with the exact key-value pair or the key itself.
- You can also choose to use a combination of both filters.
5. Choose the Destination bucket in another account. You will need the account ID and bucket name.
- Alternatively, you may also need to check the box to Change object ownership to destination bucket owner. This option is for instances where objects in the source bucket are owned by that account, not by another account. This setting would give the destination bucket owner access to the replicated objects.
6. Choose Create new IAM role. This will allow AWS to automatically create a role with the basic permissions to complete replication. In most cases, this will be enough for standard replication. This role will be needed later to allow access to the destination bucket.
7. By default, S3 encrypts all data with Amazon S3 managed keys (SSE-S3). However, you can encrypt your replicated objects with KMS keys instead. This will require extra permissions for the source replication role and/or any role accessing the destination bucket.
- If you choose a KMS key from the source account, the destination bucket and roles will need permission to access that key to decrypt the objects. Select the option to choose from your AWS KMS keys and select the key you wish to use.
- If you choose a KMS key from the destination account, the role used by the source bucket for replication will need permissions to access the key to encrypt the objects. Select the option to enter the AWS KMS key ARN and paste the key from the destination account into the ARN.
8. Several options are also helpful.
- Destination storage class – By default, replicated objects have the same storage class as the source object. However, you can choose to set the storage class of all replicated objects to a different class.
- Replication Time Control (RTC) – If a bucket has a large amount of objects or very large objects, replication can start to lag. If you require low lag times, this can be enabled to ensure that objects are replicated within 15 minutes.
- Replication metrics – This setting will publish metrics for replication to CloudWatch. These can be found in CloudWatch -> Metrics -> S3 -> Replication Metrics. They can be used to monitor several metrics and set up alarms to be notified of potential issues. This setting does not provide actual logs, only numeric metrics.
- To enable logging, you can go to the bucket and browse to Properties -> Event Notifications -> Create event notification. Under Event types, browse the Replication section and choose which events you want to be notified of.
- Delete marker replication – If an object is deleted in the source bucket, a delete marker is added since versioning is enabled. This marker is not replicated by default, so the destination bucket will show the object as not deleted. This setting allows S3 to replicate delete markers, so the object will show as deleted in the destination bucket also.
- Replica modification sync allows metadata changes made in the destination bucket to be replicated back to the source bucket. This is the only type of bidirectional replication that replicates metadata, not new objects uploaded or deleted markers to the destination bucket.
9. Click Save, and you will be brought to the Replicate Rules page. Take note of the IAM role that was created during the setup. Click on the role’s name, and the IAM page will open with details. On that page, we need the role’s ARN value. It will be in arn:aws:iam::{Account-ID}:role/service-role/{Role-Name}. Copy that value and save it; it will be used in the destination bucket policy to allow replication access.
10. Log into the destination account and browse to the S3 destination bucket.
11. Browse to Permissions -> Bucket policy and choose Edit. Add the following policy with the values for your IAM role and destination bucket name.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::{Account-ID}:role/service-role/{Role-Name}"
},
"Action": [
"s3:ReplicateObject",
"s3:ReplicateDelete",
"s3:ReplicateTags",
"s3:GetObjectVersionForReplication",
"s3:GetObjectVersionAcl",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::{destination bucket}/*"
}
]
}12. You should now see any new object uploaded to the source bucket that has been replicated to the destination bucket. If not, enable Event Notifications for “Object failed to replicate.” Check the set destination for any errors. A list of common failures can be found here.
For more information, please contact us!