Amazon S3 as a source connection

Prerequisites

Creating a Bucket

A bucket is an object container. To store data in Amazon S3, you must create a bucket and specify a bucket name and an AWS Region. Then, upload your data as objects to the Amazon S3 bucket. Each object has a key (or key name) that serves as the object's unique identifier within the bucket.

To get started, log in to AWS and search for Buckets.

Adding an IAM policy

An IAM policy is a resource-based policy that can be attached to an IAM Role to grant permissions. Create a policy to grant the necessary permissions.

note

Make sure to replace RiveryFileZoneBucket with the name of your S3 bucket.

Here is the policy's code:

{
 "Version":"2012-10-17",
 "Statement":[
 {
 "Sid":"RiveryManageFZBucket",
 "Effect":"Allow",
 "Action":[
 "s3:GetBucketCORS",
 "s3:ListBucket",
 "s3:GetBucketLocation"
 ],
 "Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`"
 },
 {
 "Sid":"RiveryManageFZObjects",
 "Effect":"Allow",
 "Action":[
 "s3:ReplicateObject",
 "s3:PutObject",
 "s3:GetObjectAcl",
 "s3:GetObject",
 "s3:PutObjectVersionAcl",
 "s3:PutObjectAcl",
 "s3:ListMultipartUploadParts"],
 "Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`/*"
 },
 {
 "Sid":"RiveryHeadBucketsAndGetLists",
 "Effect":"Allow",
 "Action":"s3:ListAllMyBuckets",
 "Resource":"*"
 }
 ]
}

Creating a Data Integration user in AWS

To connect to the Amazon S3 Source and Target in Data Integration console, you must create an AWS Data Integration user.

Create a user for Data Integration and grant it permission to manage and read the FileZone bucket.

Procedure

1. Sign in to the AWS Management console and open the Amazon S3 console.

2. Navigate to Users > Add User.

3. In the console, set your username, and select the Access type to Programmatic access.

4. Click Next: Permissions.

5. In the Set Permissions form, select Attach Existing Policies Directly.

6. Click Create Policy.

7. Navigate to the JSON tab.

8. Copy and paste the following policy:

note

Replace RiveryFileZoneBucket with the name of your S3 bucket .

{
 "Version":"2012-10-17",
 "Statement":[
 {
 "Sid":"RiveryManageFZBucket",
 "Effect":"Allow",
 "Action":[
 "s3:GetBucketCORS",
 "s3:ListBucket",
 "s3:GetBucketAcl",
 "s3:GetBucketPolicy"
 ],
 "Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`"
 },
 {
 "Sid":"RiveryManageFZObjects",
 "Effect":"Allow",
 "Action":[
 "s3:ReplicateObject",
 "s3:PutObject",
 "s3:GetObjectAcl",
 "s3:GetObject",
 "s3:PutObjectVersionAcl",
 "s3:PutObjectAcl",
 "s3:ListMultipartUploadParts"],
 "Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`/*"
 },
 {
 "Sid":"RiveryHeadBucketsAndGetLists",
 "Effect":"Allow",
 "Action":"s3:ListAllMyBuckets",
 "Resource":"*"
 }
 ]
}

9. Select Review Policy.

10. Give the Policy a name and click Create Policy.

11. Refresh the list of policies, check the policy you just created, and select Next: Tags.

12. Click Next: Tags, Next: Review and then Create User to complete the process.

13. In the summary page, you can view the user's AWS credentials (Access key ID and Secret access key), which you can download as a CSV file (You can do this only once).

14. The User can manage and read the FZ bucket created for Data Integration. Verify that the policy you created is linked to the user you created.

AWS role chaining

This connection setup involves Role Chaining, a process where one AWS identity assumes another IAM role to access resources. AWS enforces specific session duration limits for chained roles. For more information, refer to Role chaining.

Establishing Amazon S3 as a source connection in Data Integration

Procedure

AWS keys

1. Navigate to the Data Integration Account.
2. Click Connections and select + New Connection.
3. Choose Amazon S3.
4. Enter the Connection Name.
5. From the drop-down menu, choose your Region.
6. Choose AWS Keys as the Credentials Type.
7. Enter your AWS Access key id and Secret access key.
8. Assume Role Timeout (Seconds): The default value is 3600. You can adjust this value as needed for your specific configuration.
   note

   To ensure connection stability, configure the Maximum session duration for the IAM Role in the AWS Console to 12 hours (43200 seconds).

9. Click Test Connection to verify your connection is up to the task. If the connection succeeded, you can use this connection in Data Integration.

IAM role - automatic

1. Enter the Connection Name.
2. From the drop-down menu, choose your Region.
3. Select IAM Role - Automatic credentials type.
4. To start the AWS CloudFormation Stack, click Launch Stack.
5. Replace the External ID in the Parameters section with the one provided in the Data Integration console.
6. Select I acknowledge that AWS CloudFormation may create IAM resources in the Review tab, then click Create.
7. Copy the value of RiveryAssumeRoleArn from the Output tab in the stack.
8. Paste the Role ARN Key.
9. Assume Role Timeout (Seconds): The default value is 3600. You can adjust this value as needed for your specific configuration.
   note

   To ensure connection stability, configure the Maximum session duration for the IAM Role in the AWS Console to 12 hours (43200 seconds).

10. Click Test Connection to verify your connection is up to the task. If the connection succeeded, you can use this connection in Data Integration.

IAM role - manual

1. Enter the Connection Name.
2. From the drop-down menu, choose your Region.
3. Select IAM Role - Automatic credentials type.
4. Go to the AWS IAM console.
5. Click Policies from the menu, and select Create Policy.

a. Navigate to the JSON tab.

b. Copy the following policy:

{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"RiveryManageFZBucket",
"Effect":"Allow",
"Action":[
"s3:GetBucketCORS",
"s3:ListBucket",
"s3:GetBucketAcl",
"s3:GetBucketPolicy"
],
"Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`"
},
{
"Sid":"RiveryManageFZObjects",
"Effect":"Allow",
"Action":[
"s3:ReplicateObject",
"s3:PutObject",
"s3:GetObjectAcl",
"s3:GetObject",
"s3:PutObjectVersionAcl",
"s3:PutObjectAcl",
"s3:ListMultipartUploadParts"],
"Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`/*"
},
{
"Sid":"RiveryHeadBucketsAndGetLists",
"Effect":"Allow",
"Action":"s3:ListAllMyBuckets",
"Resource":"*"
}
]
}

c. Paste the Policy it into the description, then click Review Policy.

6. Name the Policy - Data Integration-S3-Policy and click Create Policy.
7. Click Roles from the menu, and select Create Role.
8. Select Another AWS Account and change the Account ID to the one provided in the Data Integration console.
9. Check Require External ID, and set External ID to the one provided in the Data Integration console.

10. Click Next.
11. Attach the Data Integration-S3-Policy to the Attach Policy form.
12. Set Data Integration-S3-Role as the role name.
13. Copy the Role ARN from the Role's window and paste it into the field.
14. Assume Role Timeout (Seconds): The default value is 3600. You can adjust this value as needed for your specific configuration.
    note

    To ensure connection stability, configure the Maximum session duration for the IAM Role in the AWS Console to 12 hours (43200 seconds).

15. Click Test Connection to verify your connection is up to the task. If the connection succeeded, you can use this connection in Data Integration.