Skip to main content
Feedback

Configure custom file zone for Firebolt

Setting up a Custom File Zone in Data Integration is an optional feature, with the default option relying on the Managed File Zone provided by the platform, which requires no setup.

You can ensure that data is stored within their own file zones, as opposed to being stored in Data Integration Managed File Zone. This also enables organizations to use the Custom File Zone as a data lake, where raw data can be stored before loading into a Target cloud data warehouse. Furthermore, organizations can define their own retention policies for data stored in the Custom File Zone. Data Integration Managed File Zone (default) retains data for a period of 48 hours.

Ensure to sign up for AWS and have a console Admin user. If you do not have one of these prerequisites, refer to AWS documentation.

Create an S3 bucket

Firebolt needs the data to be loaded to an S3 bucket to serve as the basis for an external table for further ingestion.

note

For up-to-date documentation on operations, refer to Getting started with AWS topic.

Creating an S3 bucket for Data Integration

  1. Go to S3 Management in AWS Console .
  2. Click Create Bucket.

Enter the bucket name, and choose the same region your Firebolt database is in. Use the S3 wizard defaults by reviewing the page, and then click Create Bucket.

Create an AWS user for Data Integration

Data Integration uses an Amazon S3 bucket to upload your source data into it and then pull that data to your target. There is a need to create a user in the Amazon Platform console and have the relevant bucket.

Creating a user and assigning permissions for the File Zone bucket

  1. Sign in to the AWS Console.
  2. Go to IAM in AWS Console > Users > Add User.
  3. Set your username (RiveryUser), and check the Access type to Programmatic access. Click Next.
  4. In the Set Permissions page, choose Attach Policies Directly.
  5. To create a new permission for the user, click on Attach Existing Policy Directly and then on Create Policy. A new tab with AWS policy generator opens.
  6. In the policy generator page, click on JSON tab.
  7. Paste the permissions configuration below into the blank text box that opens and click Review Policy. Ensure that you replace the bold areas with your current settings.

Replace the  <RiveryFileZoneBucket> with the S3 bucket name you created. 

{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"RiveryManageFZBucket",
"Effect":"Allow",
"Action":[
"s3:GetBucketCORS",
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`"
},
{
"Sid":"RiveryManageFZObjects",
"Effect":"Allow",
"Action":[
"s3:ReplicateObject",
"s3:PutObject",
"s3:GetObjectAcl",
"s3:GetObject",
"s3:ListObjects*",
"s3:PutObjectVersionAcl",
"s3:PutObjectAcl",
"s3:ListMultipartUploadParts"],
"Resource":"arn:aws:s3:::`<RiveryFileZoneBucket>`/*"
},
{
"Sid":"RiveryHeadBucketsAndGetLists",
"Effect":"Allow",
"Action":"s3:ListAllMyBuckets",
"Resource":"*"
}
]
}
  1. Name your policy and click on Create Policy.
  2. Go to your browser tab of the user creation, Refresh the policies list using the button on the right, search and Mark the policy just created, and click on Next: Tags.
  3. Clickl on Next: Review and then on Create User.
  4. In the Complete section, you get the user’s AWS credentials (aws_access_key_id, aws_secret_access_key) and the user's ARN . Save the credentials in a safe place!
  1. The User should have permissions to manage and read the FZ bucket created for Data Integration. Ensure the policy you created is attached to the user created.

Configuring custom file zone for Firebolt in Data Integration

  1. Open your Firebolt connection by navigating to Connections > Create New Connection, and choose Firebolt.
  2. In the connection, check the "Custom File Zone".
  3. If you already have an S3 connection, you can set it here as a Custom File Zone connection, or you can create a new connection.
  4. In the form, set your Aws Access Key, Aws Access Secret Key, and Region Key
  5. Name your S3 File Zone connection and click Save.
  6. Click Test Connection .
  7. After saving, choose your default bucket for your File Zone area. Use the bucket you created.
  8. Click Save to save the Fireblot connection.
On this Page