Creating AWS private link

info

Private Link is available to users on the Pro Plus plan.

Private Link is a service AWS provides to establish a direct connection between VPCs, even if they reside in separate AWS accounts. This service does not require a VPN Peering connection.

Basic terms

Provider (client)

The service provider: An AWS account that a Consumer can connect to and access the resources it exposes.
The provider must explicitly list which IAM Principals can find it, accept new connection requests, and handle routing within the source VPC.

Consumer (Data Integration)

The service consumer: The AWS account that connects to the Provider and consumes the content it wants.

Only Consumers can access Providers (unidirectional link).

VPC endpoint service

The service that the "Provider" has to create to expose a specific load balancer. This is an NLB (Network Load Balancer) within the VPC you want to expose.

Within the Endpoint Service, you must let "IAM Principals" discover it.
Each Endpoint Service has a Service Name, which Endpoint Interfaces uses to discover it.

VPC endpoint or VPC endpoint interface

The endpoint (endpoint interface) is created on the consumer side's VPC to connect to the Endpoint Service in the Provider.
The interface creates an actual DNS Endpoint accessible to the source VPC (only the Consumer)

AWS IAM principal

An AWS account's identifier is in the standard AWS ARN format for an IAM user. Each of our production accounts has its own unique principal.

IAM Principal for {conKeyRefs.DataIntegration}: _arn:aws:iam::012922587834:root ._

Creating a private link

Before you begin

If you want to use AWS Private Link to connect Data Integration to your VPC, Data Integration is the Consumer, and you are the Provider. This process requires you to configure the endpoint service on your side.

Procedure

1. You need an IAM principal to whitelist it in your Endpoint service (depending on the region):
   • arn:aws:iam::012922587834:root – for EU (eu-west-1.console.rivery.io) and US (console.rivery.io)
   • arn:aws:iam::339713088687:root – for Israel (il-central-1)
   • arn:aws:iam::772591076045:root – for Australia (ap-southeast-2)
2. After creating the load balancer in the correct VPC, enable Private Link integration:
   • Click the Integrations tab in the "Load balancer object".
   • Click Create endpoint service under VPC endpoint services. Provide it a name, and choose the load balancer to which you are creating a private link.
   • Click Create.
   • Select the new endpoint created and navigate to the Allow principals tab.
   • Allow the principal that corresponds to your Data Integration region.
3. Open a support ticket to start the process. Provide the Service Name of your endpoint service so we can create a connection request.
4. Approve the connection request on your end.
5. We will send you the hostname to create connections to your database in Data Integration.

Supported regions

• Virginia (us-east-1)

• UAE (me-central-1)

• Sydney (ap-southeast-2)

• Stockholm (eu-north-1)

• Singapore (ap-southeast-1)

• Oregon (us-west-2)

• Ohio (us-east-2)

• Mumbai (ap-south-1)

• London (eu-west-2)

• Israel (il-central-1)

• Ireland (eu-west-1)

• Frankfurt (eu-central-1)

• Canada (ca-central-1)

Creating a proxy instance (optional)

You can use a proxy instance to reach your private data sources using the Private Link, which redirects the network from Data Integration connection via the "Private Link" into the client's database. This proxy can be used as:

• RDS proxy: You can use this option if you use RDS with MySQL/PostgreSQL. To learn more, refer to the Creating an RDS Proxy topic.

• Via SSH: This uses an SSH tunnel to connect to the DB. In this setup, the Private Link (AWS Network Load Balancer) directs the network to the SSH instance. You must configure the VPC Endpoint (Private Link) hostname - provided by Data Integration at the end of the process in the SSH Tunnel part on the connection in Data Integration. Enter the DNS of the data source in the host. Refer to the Creating an SSH Tunnel Instance.

Creating an NLB

To expose your VPC to another account, all traffic must go through a Network Load Balancer.

Procedure

1. Navigate to the EC2 Console to create an NLB using the AWS Console.

2. Click Load Balancers from the navigation pane.

3. Click Create Load Balancer and click Create under Network Load Balancer.

4. Make sure the Scheme is "Internal".

5. Network Mapping + Security Groups

   Network mapping: Make sure the VPC you select contains your data source (EC2/RDS) and select the relevant Availability Zones.
   If unsure, you can configure all available AZs and set Cross-zone load balancing to true to ensure you can reach your data source.

• Security Groups - Ensure no security groups are attached to the NLB. By default, AWS automatically attaches the NLB to your default security group.

6. The listener depends on the target.
   • If it is an RDS or RDS Proxy, create an empty target group, and we will fill it using an AWS Lambda. To learn more, refer to the https://aws.amazon.com/blogs/networking-and-content-delivery/hostname-as-target-for-network-load-balancers/

• If it is an SSH Tunnel, you must set a target group containing that EC2 instance,
  Choose the port where you want to reach the NLB and click Create target group

7. Create Target Group.

   • To connect directly to a data source hosted on an EC2 instance, or to use an SSH Tunnel, select Instances,

     • For RDS/RDS Proxy, choose IP address and do not register any IP addresses. To populate it with the destination IP addresses, refer to the guide.

     • For MySQL/MariaDB RDS instances, change the following parameter:
       skip_name_resolve - 1

   • Go to Register Targets, select the relevant instances / IPs you wish to expose, and click Add to registered.

   • Go to Review and click create. Ensure the instances' Security Groups permit ingress traffic from the chosen port.

   • Target Group attributes

     • Turn off Proxy protocol v2 and Preserve client IPs.
     • Stickiness should be off by default. To enable it, contact support.
     • Cross-Zone load balancing must be inherited from the NLB.

Creating an endpoint service

You must expose the NLB created in the last step using a VPC Endpoint Service.
To create this service, go to the VPC Console and select Endpoint Services in the navigation pane.

Procedure

1. Click on Create Endpoint Service and select the "Network Load Balancer" you created.

2. Click on Create Service (at the bottom of the page).

3. Select your newly created Endpoint Service, and choose Add principals to whitelist under Actions.

4. Enter the ARN of the Data Integration AWS account IAM Principal provided to you by our support.

5. Click Add to Whitelisted principals.

6. Go to the new service properties, copy and send the Service Name to your contact at Data Integration.

Creating an Endpoint / Endpoint Interface.

On Data Integration side - Creating an Endpoint / Endpoint Interface. After having an Endpoint Service on the Provider end, go to the Consumer's AWS account, and navigate to the VPC Console.

Procedure

1. In the navigation pane, select Endpoints and click Create Endpoint.
2. Select Find service by name under Service category and enter the Service Name the given by the Provider.
3. Click Verify.
4. Select the VPC and Subnets you want to connect to the Provider, and select the Security Group(you can choose only one).
5. Click Create endpoint.

Accepting the connection on the client's VPC end

Now a connection request is sent from Data Integration to the client's endpoint.

Procedure

1. Navigate to the Provider AWS account's VPC Console.
2. Select Endpoint Services from the navigation menu.
3. Choose your Endpoint Service, and select Endpoint Connections.
   You can view a new pending connection request.
4. Select the connection with the Consumer's AWS account ID under Owner.
5. Click Actions and select Accept endpoint connection request.

Using the endpoint interface on the consumer end

After successfully connecting to your two VPCs on different AWS accounts.

Only the Consumer can access the Provider, not vice versa.

Procedure

1. Navigate to the Consumer AWS account's VPC Console
2. Select Endpoints to use this connection from the navigation menu.
3. Find your newly created Endpoint Interface. Your DNS names are under Details. You can request these addresses (inside your VPC) to communicate with the Provider's Endpoint Service.

Connecting your AWS RDS to the exposed NLB

To connect your RDS to the Load Balancer, refer to the creating an RDS Poxy topic.