Amazon S3 as a source connection

This topic provides a step-by-step tutorial for creating an Amazon S3 connection.

Prerequisites

Create a Bucket

A bucket is an object container. To store data in Amazon S3, you must first create a bucket and specify a bucket name as well as an AWS Region. Then you upload your data as objects to that bucket in Amazon S3. Each object has a key (or key name) that serves as the object's unique identifier within the bucket. Let's begin by logging into AWS and searching for Buckets:

Add an IAM policy

An IAM policy is a resource-based policy that can be attached to an IAM Role to grant permissions. Let's create a policy to grant the necessary permissions.

note

Make sure to replace RiveryFileZoneBucket with the name of your S3 bucket.

Here's the policy's code:

{
 "Version":"2012-10-17",
 "Statement":[
   {
    "Sid":"RiveryManageFZBucket",
    "Effect":"Allow",
    "Action":[
    "s3:GetBucketCORS",
    "s3:ListBucket",
    "s3:GetBucketLocation"
     ],
    "Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`"
   },
   {
    "Sid":"RiveryManageFZObjects",
    "Effect":"Allow",
    "Action":[
      "s3:ReplicateObject",
      "s3:PutObject",
      "s3:GetObjectAcl",
      "s3:GetObject",
      "s3:PutObjectVersionAcl",
      "s3:PutObjectAcl",
      "s3:ListMultipartUploadParts"],
    "Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`/*"
  },
  {
     "Sid":"RiveryHeadBucketsAndGetLists",
     "Effect":"Allow",
     "Action":"s3:ListAllMyBuckets",
     "Resource":"*"
  }
 ]
}

Create a Data Integration user in AWS

Now, in order to connect to the Amazon S3 Source and Target (described in the following section) in Data Integration console, you must first create an AWS Data Integration user:

Create an AWS user for Data Integration

Create a user for Data Integration and grant it permission to manage and read the FileZone bucket by following these steps:

1. Sign in to the AWS Management Console and open the Amazon S3 console.

2. Navigate to Users > Add User.

3. In the console, set your username, and select the Access type to Programmatic access.

4. Click Next: Permissions.

5. In the Set Permissions form, select Attach Existing Policies Directly.

6. Click Create Policy.

7. Navigate to the JSON tab.

8. Copy and paste the following policy:

note

Replace RiveryFileZoneBucket with the name of your S3 bucket .

{
 "Version":"2012-10-17",
 "Statement":[
   {
    "Sid":"RiveryManageFZBucket",
    "Effect":"Allow",
    "Action":[
    "s3:GetBucketCORS",
    "s3:ListBucket",
    "s3:GetBucketAcl",
    "s3:GetBucketPolicy"
     ],
    "Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`"
   },
   {
    "Sid":"RiveryManageFZObjects",
    "Effect":"Allow",
    "Action":[
      "s3:ReplicateObject",
      "s3:PutObject",
      "s3:GetObjectAcl",
      "s3:GetObject",
      "s3:PutObjectVersionAcl",
      "s3:PutObjectAcl",
      "s3:ListMultipartUploadParts"],
    "Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`/*"
  },
  {
     "Sid":"RiveryHeadBucketsAndGetLists",
     "Effect":"Allow",
     "Action":"s3:ListAllMyBuckets",
     "Resource":"*"
  }
 ]
}

9. Select Review Policy.

10. Give the Policy a name and click Create Policy.

11. Using the rounded arrows on the upper right, refresh the list of policies, check the policy you just created, and select Next: Tags.

12. Click Next: Tags, Next: Review and then Create User to complete the process.

13. In the summary screen, you'll find the user's AWS credentials (Access key id and Secret access key), which you can download as a CSV file (this is the only time you'll be able to do so).

14. The User should now be able to manage and read the FZ bucket that was created for Data Integration. Check once again that the policy you created is linked to the user you created.

Connection procedure

AWS keys

1. Enter the Connection Name.
2. From the drop-down menu, choose your Region.
3. Select AWS Keys credentials type.
4. Enter your AWS Access key id and Secret access key.
5. Use the Test Connection function to see if your connection is up to the task. If the connection succeeded, you can now use this connection in Data Integration.

IAM role - automatic

1. Enter the Connection Name.
2. From the drop-down menu, choose your Region.
3. Select IAM Role - Automatic credentials type.
4. To initiate the AWS CloudFormation Stack, click Launch Stack.
5. Replace the External ID in the Parameters section with the one you were given in the Data Integration console.
6. Select I acknowledge that AWS CloudFormation may create IAM resources in the Review tab, then click Create.
7. Copy the value of RiveryAssumeRoleArn from the Output tab in the stack.
8. Paste the Role ARN Key.
9. Use the Test Connection function to see if your connection is up to the task. If the connection succeeded, you can now use this connection in Data Integration.

IAM role - manual

1. Enter the Connection Name.

2. From the drop-down menu, choose your Region.

3. Select IAM Role - Automatic credentials type.

4. Initiate the AWS IAM console.

5. Click Policies on the side menu, and select Create Policy.

   a. Navigate to the JSON tab.

   b. Copy the following policy:

   {
   "Version":"2012-10-17",
   "Statement":[
     {
       "Sid":"RiveryManageFZBucket",
       "Effect":"Allow",
       "Action":[
       "s3:GetBucketCORS",
       "s3:ListBucket",
       "s3:GetBucketAcl",
       "s3:GetBucketPolicy"
       ],
       "Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`"
     },
     {
       "Sid":"RiveryManageFZObjects",
       "Effect":"Allow",
       "Action":[
         "s3:ReplicateObject",
         "s3:PutObject",
         "s3:GetObjectAcl",
         "s3:GetObject",
         "s3:PutObjectVersionAcl",
         "s3:PutObjectAcl",
         "s3:ListMultipartUploadParts"],
       "Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`/*"
     },
     {
       "Sid":"RiveryHeadBucketsAndGetLists",
       "Effect":"Allow",
       "Action":"s3:ListAllMyBuckets",
       "Resource":"*"
     }
   ]
   }

   c. Paste the Policy it into the description box, then click Review Policy.

6. Name the Policy - Data Integration-S3-Policy and click Create Policy.

7. Click Roles on the side menu, and select Create Role.

8. Select Another AWS Account and change the Account ID to the one you were given in the Data Integration console.

9. Check Require External ID, and set External ID to the one you were given in the Data Integration console.

10. Click Next.
11. Attach the Data Integration-S3-Policy to the Attach Policy form.
12. Set Data Integration-S3-Role as the role name.
13. Copy the Role ARN From the Role's window and paste it into the field below.
14. Use the Test Connection function to see if your connection is up to the task. If the connection succeeded, you can now use this connection in Data Integration.