Migrate and Sync RDS PostgreSQL to Amazon Redshift Serverless using AWS Glue

Always customer prefers cost less solutions to run the business . To help them their business and the requirements we also needs to provide efficient solutions 

Some cases cloud vendors provides good solutions for analytics load but cost will be very high , most of the time we don't want to recommend that but we need to do  

Like that one of the solution in AWS , its cost much but works much faster like anything 

We are talking about Amazon Redshift Solutions only , So recently they have launched Amazon Redshift Serverless solutions for few regions .

Whatever new comes , before customer catches the features we need to find and deliver the best approach to them . So ,

What is Amazon Redshift Serverless ? 

Amazon Redshift Serverless automatically provisions data warehouse capacity and intelligently scales the underlying resources. Amazon Redshift Serverless adjusts capacity in seconds to deliver consistently high performance and simplified operations for even the most demanding and volatile workloads.

With Amazon Redshift Serverless, you can benefit from the following features:

• Access and analyze data without the need to set up, tune, and manage Amazon Redshift provisioned clusters
• Use the superior Amazon Redshift SQL capabilities, industry-leading performance, and data-lake integration to seamlessly query across a data warehouse, a data lake, and operational data sources
• Deliver consistently high performance and simplified operations for the most demanding and volatile workloads with intelligent and automatic scaling
• Use workgroups and namespaces to organize compute resources and data with granular cost controls
• Pay only when the data warehouse is in use

So , Overall no need of human interventions in the Redshift Serverless 

Everything is Fine , How to we migrate and sync Amazon RDS / EC2 Postgres / Aurora Postgres to utilize this Redshift Serverless 

What are the options available to migrate and Sync ?

• DMS - Still Redshift Target is not available to migrate the Data
• Export/Import - Yes we can perform , how to handle zero downtime migration . Syncing real-time data is not possible 
• AWS Glue - Its Good Option , We can migrate and Sync real-time data from RDS to Redshift Serverless  

Lets start sample data migrate and sync into Amazon Redshift Serverless ,

Environment Setup Go Through ,

1. RDS PostgreSQL
2. AWS Glue
3. Amazon Redshift Serverless 
4. VPC S3 Endpoint
5. IAM Role

RDS PostgreSQL : 

Amazon Redshift Serverless :

VPC S3 Endpoint :

IAM Role : 

Once Environment is completed , we can start adding connections and jobs in AWS Glue

How to add connections in AWS Glue , 

In AWS Glue Console --> Click Connections --> Create Connections 

Create source and target databases ,

For testing sample schema and data inserted into RDS PostgreSQL before creating the crawler 

Lets start to create separate crawlers for Source and Target to update catalog ,

Below data source is mapped to RDS PostgreSQL 

Also required policies updated role needs to attached ,

Choose appropriate databases for crawler , below is the for source 

Below is the for Target , 

Once its completed , we have already deployed sample schema scripts in both side to transfer the data . Lets run the crawler and check it 

So both source and target tables are updated 

Lets Create Glue Job in AWS Console ,

After all , Lets schedule the job every 5 Minutes and sync the data

Whatever data is inserted in RDS PostgreSQL will be sync into Redshift Serverless Every 5 Minutes ,

So Quickly we can migrate and save our cost 

Any Troubles and issues Please contact me immediately !!!