TechBrothersIT is a blog and YouTube channel sharing real-world tutorials, interview questions, and examples on SQL Server (T-SQL, DBA), SSIS, SSRS, Azure Data Factory, GCP Cloud SQL, PySpark, ChatGPT, Microsoft Dynamics AX, Lifecycle Services, Windows Server, TFS, and KQL. Ideal for data engineers, DBAs, and developers seeking hands-on, step-by-step learning across Microsoft and cloud platforms.
Topic: How to perform CDC from GCP MySQL Instance to BigQuery by using DataStream in Google Cloud Platform
How to perform CDC from GCP MySQL Instance to BigQuery by using DataStream in Google Cloud Platform Tutorial 2022, in this article we are going to learn How to perform CDC from GCP MySQL Instance to BigQuery by using DataStream in Google Cloud Platform | GCP Cloud SQL Tutorial
Video Demo: How to perform CDC from GCP MySQL Instance to BigQuery by using DataStream in Google Cloud Platform
Topic: Cross-database query between Google SQL instances PostgreSQL
Cross-database query between Google SQL instances PostgreSQL | GCP SQL Tutorial 2022, in this article we are going to learn Cross database query between Google SQL instances PostgreSQL.
Script used in demo:
create database sales_asia create database sales_europe create table public.AsiaSale(id int, name varchar(100), region varchar(100)) insert into public.AsiaSale values(1,'aamir','Asia') Select * From public.AsiaSalecreate table public.EuropeSale(id int, name varchar(100), region varchar(100)) insert into public.EuropeSale values(2,'lisa','Europe') Select * From public.europesale-- we want to execute union query in sales_asia database that should get data from sales_europe.public.EuropeSale table. Select * From public.europesale union all select * from public.AsiaSale1) Set up a Foreign User-- Do this on DB from which you would like to read the tables CREATE USER fdwuser WITH PASSWORD 'test123$'; GRANT USAGE ON SCHEMA PUBLIC TO fdwuser; GRANT SELECT ON europesale TO fdwuser;--Check the list of Tables select * from information_schema.tables where table_name like '%sale%'2) Create the Extension CREATE EXTENSION postgres_fdw; select * from pg_extension;3) Create the Foreign Server CREATE SERVER secondrydb_srv FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '34.67.244.181', port '5432', dbname 'sales_europe'); Select * From pg_foreign_server;4) Create User Mapping CREATE USER MAPPING FOR postgres SERVER secondrydb_srv OPTIONS(user 'fdwuser',password 'test123$'); Select * From pg_user_mappings5) Grant the Local User Access to the Foreign Data Wrapper GRANT USAGE ON FOREIGN SERVER secondrydb_srv TO postgres;6) Import the Foreign Schema or Tables IMPORT FOREIGN SCHEMA public LIMIT TO (europesale) FROM SERVER secondrydb_srv INTO public;Select * From public.europesale union all select * from public.AsiaSale
Video Demo: Cross database query between Google SQL instances PostgreSQL
Topic: How to Perform Cross Database Queries in PostgreSQL in GCP Cloud
How to Perform Cross Database Queries in PostgreSQL in GCP Cloud | GCP Cloud SQL Tutorial 2022, in this video we are going to learn How to Perform Cross Database Queries in PostgreSQL in GCP Cloud | GCP Cloud SQL Tutorial 2022
Video Demo: How to Perform Cross Database Queries in PostgreSQL in GCP Cloud
Topic: Cross database query between Google SQL instances PostgreSQL
Cross database query between Google SQL instances PostgreSQL | GCP SQL Tutorial 2022, in this video we are going to learn Cross database query between Google SQL instances PostgreSQL | GCP SQL Tutorial 2022
Video Demo: Cross database query between Google SQL instances PostgreSQL
Topic: How to Stop GCP PostgreSQL from Logging Passwords in Clear Text in Logs
How to Stop GCP PostgreSQL from Logging Passwords in Clear Text in Logs GCP Cloud SQL Tutorial | GCP SQL Tutorial 2022, in this video we are going to learnHow to Stop GCP PostgreSQL from Logging Passwords in Clear Text in Logs GCP Cloud SQL Tutorial | GCP SQL Tutorial 2022
Video Demo: How to Stop GCP PostgreSQL from Logging Passwords in Clear Text in Logs
Topic: How to Perform In place Upgrade to GCP SQL Instance | Inplace Upgrade PostgreSQL 12 to PostgreSQL 14
How to Perform In place Upgrade to GCP SQL Instance | Inplace Upgrade PostgreSQL 12 to PostgreSQL 14 | GCP SQL Tutorial 2022, in this video we are going to learn How to Perform In place Upgrade to GCP SQL Instance | Inplace Upgrade PostgreSQL 12 to PostgreSQL 14
Video Demo: How to Perform In place Upgrade to GCP SQL Instance
Topic: How to Drop User or Role in PostgreSQL Instance on Google Cloud Platform
How to Drop User or Role in PostgreSQL Instance on Google Cloud Platform | GCP SQL Tutorial 2022, in this video we are going to learn How to Drop User or Role in PostgreSQL Instance on Google Cloud Platform | GCP SQL Tutorial 2022
Script:
ERROR: role "aamir" cannot be dropped because some objects depend on it
DETAIL: owner of database test
1 object in database test
SQL state: 2BP01
Error
Invalid request: failed to delete user aamir: . role "aamir" cannot be dropped because some objects depend on it Details: owner of database test 1 object in database test.
0) Prepare the scenario
Create user aamir and then create some objects
create database Test
create table public.mytable(id int, name varchar(100));
insert into public.mytable values(1,'aamir');
Select * from public.mytable;
1) -- Let's think about a user who has left the company and you need to drop the user. If you don't know the
--password for the user, login by using postgres user and change the password of user.
--How to change the password for role
Alter role aamir LOGIN password 'Test123$';
2) -- Tried to drop the role by using postgres user session
drop role aamir -- you will get errot that objects are owened by user aamir
GRANT postgres to aamir;
--2 -- login by aamir user and run below command to assign all objects to postgres user
REASSIGN OWNED BY aamir TO postgres;
--3 login back to postgres and run below
drop role aamir
--4 Check if Objects are not dropped by dropping user.
Video Demo: How to Drop User or Role in PostgreSQL Instance on Google Cloud Platform
Topic: How to Schedule Maintenance with PostgreSQL pg cron Extension
How to Connect to GCP SQL Server Instance by using Private IP Configuration | GCP SQL Tutorial 2022, in this video we are going to learn How to Connect to GCP SQL Server Instance by using Private IP Configuration | GCP SQL Tutorial 2022
Script:
--Postgres Instance Level Settings
cloudsql.enable_pg_cron
value= on
cron.database_name
Value : Provide your Database Name
-- check if extensions are installed
select * from pg_extension;
-- use this to create extension on db, it will create two tables job and job_run_detail
Topic: How to Stop and Start SQL Instances on Schedule by using Cloud Schedule in Google Cloud Platform
How to Stop and Start SQL Instances on Schedule by using Cloud Schedule in GCP | GCP SQL Tutorial 2022, in this video we are going to learn How to Stop and Start SQL Instances on Schedule by using Cloud Schedule in GCP | GCP SQL Tutorial 2022
Topic: How to Connect to GCP SQL Server Instance by using Private IP Configuration
How to Connect to GCP SQL Server Instance by using Private IP Configuration | GCP SQL Tutorial 2022, in this video we are going to learn How to Connect to GCP SQL Server Instance by using Private IP Configuration | GCP SQL Tutorial 2022
Video Demo: How to Connect to GCP SQL Server Instance by using Private IP Configuration
Topic: How to Connect to MySQL Instance in Google Cloud Platform from Virtual Machine By Using Private IP
How to Connect to MySQL Instance in Google Cloud Platform from Virtual Machine By Using Private IP GCP Tutorial 2022, in this video, we are going to learn How to Connect to MySQL Instance in Google Cloud Platform from Virtual Machine By Using Private IP | GCP Tutorial 2022
Script:
Install MySQL Client
sudo apt-get install default-mysql-client
Connect to MySQL by using Private IP
mysql -h IP -u root -p
Connect to MySQL with private IP
mysql -h 172.21.160.7 -u root -p
Video Demo: How to Connect to MySQL Instance in Google Cloud Platform from Virtual Machine By Using Private IP
Topic: How to Connect to PostgreSQL Instance with Private IP by using GCP VM
How to Connect to PostgreSQL Instance with Private IP by using GCP VM GCP SQL Cloud Tutorial 2022, in this video we are going to learn How to Connect to PostgreSQL Instance with Private IP by using GCP VM GCP SQL Cloud Tutorial 2022
Topic: How to Install PostgreSQL 14.5 on Windows Machine and How to Connect to PostgreSQL by using pgAdmin
How to Install PostgreSQL 14.5 on Windows Machine | How to Connect to PostgreSQL by using pgAdmin GCP Tutorial 2022, in this video we are going to learn How to Install PostgreSQL 14.5 on Windows Machine | How to Connect to PostgreSQL by using pgAdmin | GCP Tutorial 2022
Video Demo: How to Install PostgreSQL 14.5 on Windows Machine and How to Connect to PostgreSQL by using pgAdmin
Topic: How to Install psql Command line on Windows Machine and Connect to PostgreSQL Instance on Google Cloud Platform
How to Install psql Command line on Windows Machine and Connect to PostgreSQL Instance on GCP Tutorial 2022, in this video we are going to learn How to Install psql Command line on Windows Machine and Connect to PostgreSQL Instance on GCP | GCP Tutorial 2022
Script: command line tools
https://www.enterprisedb.com/downloads/postgres-postgresql-downloads
commad line tools
https://www.enterprisedb.com/downloads/postgres-postgresql-downloads
Video Demo: How to Install psql Command line on Windows Machine and Connect to PostgreSQL Instance on GCP
Topic: How to Enable Slow Query Logging and General Logging for MySQL to Table or File in GCP
How to Enable Slow Query Logging and General Logging for MySQL to Table or File in GCP Tutorial 2022, in this video we are going to learn How to Enable Slow Query Logging and General Logging for MySQL to Table or File in GCP | GCP Tutorial 2022
Video Demo: How to Enable Slow Query Logging and General Logging for MySQL to Table or File in Google Cloud Platform
Topic: How to Create PostgreSQL 14 Instance on GCP How to Connect pgAdmin From Local Computer to PostgreSQL.
How to Create PostgreSQL 14 Instance on GCP How to Connect pgAdmin From Local Computer to PostgreSQL GCP Tutorial 2022, in this video we are going to learn How to Create PostgreSQL 14 Instance on GCP How to Connect pgAdmin From Local Computer to PostgreSQL
Video Demo: How to Create PostgreSQL 14 Instance on GCP How to Connect pgAdmin From Local Computer to PostgreSQL
Topic: Understanding Backups of SQL Server, MySQL, PostgreSQL in GCP Limitation of Automated and OnDemand
Understanding Backups of SQL Server, MySQL, PostgreSQL in GCP Limitation of Automated and OnDemand GCP Tutorial 2022, in this video we are going to learn Understanding Backups of SQL Server, MySQL, PostgreSQL in GCP Limitation of Automated and OnDemand.
Video Demo: Understanding Backups of SQL Server, MySQL, PostgreSQL in GCP Limitation of Automated and OnDemand
Topic: How to Upgrade SQL Server Instances from Express to Standard or Enterprise Edition in Google Cloud Platform.
How to Upgrade SQL Server Instances from Express to Standard or Enterprise Edition GCP Tutorial 2022, in this video we are going to learn How to Upgrade SQL Server Instances from Express to Standard or Enterprise Edition.
Video Demo: How to Upgrade SQL Server Instances from Express to Standard or Enterprise Edition in Google Cloud Platform
Topic: How to Migrate On Prem SQL Server Database to GCP SQL Server Instance by using BAK File & SQL File.
How to Migrate On Prem SQL Server Database to GCP SQL Server Instance by using BAK File & SQL File GCP Tutorial 2022, in this video we are going to learn How to Migrate On Prem SQL Server Database to GCP SQL Server Instance by using BAK File & SQL File | GCP Tutorial 2022.
Video Demo: How to Migrate On Prem SQL Server Database to GCP SQL Server Instance by using BAK File & SQL File
Topic: How to Edit SQL Instance Settings on Google Cloud Platform
How to Edit SQL Instance Settings on GCP | Does Restart Required if You Edit SQL Instances on GCP Tutorial 2022, in this video we are going to learn How to Edit SQL Instance Settings on GCP | Does Restart Required if You Edit SQL Instances on Google Cloud Platform.
Video Demo: How to Edit SQL Instance Settings on Google Cloud Platform.
Topic: How to Clone SQL Instances in Google Cloud Platform
How to Clone SQL Instances in Google Cloud Platform | Create Clone of SQL Server, MySQL, PostgreSQL GCP Tutorial 2022, in this video we are going to learn How to Clone SQL Instances in Google Cloud Platform | Create Clone of SQL Server, MySQL, PostgreSQL | GCP Tutorial 2022
Topic: How to Start, Stop, & Restart SQL instances on GCP by using Console and Gcloud Commands GCP Tutorial
How to Start, Stop, & Restart SQL instances on GCP by using Console and Gcloud Commands GCP Tutorial 2022, in this video we are going to learn How to Start, Stop, & Restart SQL instances on GCP by using Console and Gcloud Commands | GCP Tutorial 2022
Topic: SQL Server Error 53 Could not Open Connection On SQL Server on Google Cloud Platform
SQL Server Error 53 Could not Open Connection On SQL Server on Google Cloud Platform GCP Tutorial 2022, in this video we are going to learn about SQL Server Error 53 Could not Open Connection On SQL Server on Google Cloud Platform | GCP Tutorial 2022.
Video Tutorial: SQL Server Error 53 Could not Open Connection On SQL Server on Google Cloud Platform
Topic: Create MySQL Instance with Read Replicas in Google Cloud Platform.
Create MySQL Instance with Read Replicas in GCP | Promote MySQL Read Replica to Stand Alone Mode GCP Tutorial 2022, in this video we are going to learn Create MySQL Instance with Read Replicas in GCP | Promote MySQL Read Replica to Stand Alone Mode
Video Demo: Create MySQL Instance with Read Replicas in GCP
Topic: How to Enable Point in Time Recovery for MySQL Instance in GCP.
How to Enable Point in Time Recovery for MySQL Instance in GCP | How to Restore Database with PTR GCP Tutorial 2022, in this video we are going to learn How to Enable Point in Time Recovery for MySQL Instance in GCP | How to Restore Database with PTR | GCP Tutorial 2022.
Video Demo: How to Enable Point in Time Recovery for MySQL Instance in Google Cloud Platform
How to Create Read Replicas for SQL Server Instance in Google Cloud Platform.
How to Create Read Replicas for SQL Server Instance in GCP Availability Groups for SQL Server in GCP | GCP Tutorial 2022, in this video we are going to learn How to Create Read Replicas for SQL Server Instance in GCP Availability Groups for SQL Server in GCP.
Topic: How to Create MySQL 8.0 Instance with High Availability in Google Cloud Platform.
How to Create MySQL 8.0 Instance with High Availability in Google Cloud Platform GCP Tutorial 2022, in this video we are going to learn How to Create MySQL 8.0 Instance with High Availability in Google Cloud Platform.
Vidoe Demo: How to Create MySQL 8.0 Instance with High Availability in Google Cloud Platform
Topic: How to Setup SQL Server Instance with High Availability in Google Cloud Platform.
How to Setup SQL Server Instance with High Availability in GCP | How to Failover SQL Server Instance GCP Tutorial 2022, in this video we are going to learn How to Setup SQL Server Instance with High Availability in GCP | How to Failover SQL Server Instance | GCP Tutorial 2022, Google Cloud Platform Step by Step - GCP Tutorial 2022
Topic: How to Create SQL Instance on Google Cloud Platform and Connect by using SSMS
How to Create SQL Instance on Google Cloud Platform and Connect by using SSMS GCP Tutorial 2022, in this video we are going to learn How to Create SQL Instance on Google Cloud Platform and Connect by using SSMS from local Computer | GCP Tutorial 2022, Google Cloud Platform Step by Step - GCP Tutorial 2022 - GCP Tutorial 2022 Step by Step - Google Cloud Platform Tutorial 2022.
Video Tutorial: How to Create SQL Instance on Google Cloud Platform and Connect by using SQL Server Management Studio
Topic: How to use Append Variable activity in Azure Data Factory
In this article we are going to learn how to use append variable activity in Azure Data Factory, we are going to perform a real-time example in which what we will do, we will be emailing the list of the files which you have processed in your data factory so we will concatenate all those lists of the files and then email. let's start our demonstration.
First of all, open your Azure Data Factory studio, go to the author tab, click on the + button to create a new pipeline, then find and drag the Get Metadata activity.
Rename the Get Metadata activity for your convenience, then go to the Dataset tab, click on the + New button to create a new dataset
Select Azure Blob Storage and then click on Continue.
Select the File format and then click on continue.
Name your dataset, select the linked service if you have already created, or create a new one, provide the input file path, select the none as import schema as per your requirement, then click on ok.
Go to the dataset tab, and select the Child Items in Field list, as we are going to read the list of the files.
Find and drag the ForEach Loop Activity, connect with the Get Metadata activity, go to the settings tab then click on the Add dynamic content.
Select the Metadata activity, expressions will be shown in the above box, then add ".Childitems" and then click on Ok
Go to the variable tab, click on the + button to create a new variable, name your variable, and select the type.
Go inside ForEach activity, bring the wait activity or copy data activity as per your scenario, then find and bring the append variable activity, connect both of them then go to the Variables tab, select the name and click on Add Dynamic content.
Click on ForEach, then the expression will be shown in the above box, then add ".name" and click on Ok.
Outside the ForEach activity, find and bring the Set variable, then connect with the ForEach, then create a string type variable and then click on the set variable activity, go to the Variables tab, select the variable and then click on Add dynamic content.
Click on the variable name which we have created earlier, then add the curly parentheses around and click on ok.
Next. go to the Azure portal, find and open the Logic Apps, then click on the + Add button to create a new Logic app, in the basics tab select your Azure subscription, select the resource group, select the logic app type, name your logic app, select your region then click on Review + create and then click on the create.
Once the logic app is created, Open the resource and here we will setup the Email part, search for the "Request", then click on the request and then select http request.
Once the HTTP request is received click on the Use sample payload to generate schema.
In this window enter the expressions and then click on Done.
Click on the Method then select Post then click on Next.
.
Select the Email service, in my case it is Gmail, then provide a connection name, select the authentication type and click on Sign In, it will redirect to a new window and then you have to provide your user name and password for the email account.
Once you are signed in, it will redirect you to another window where you have to provide the email address where you want to send the email, then provide and subject and the message the select the List of files variable from the drop down window, and then finally click on Save.
Once you are done with the email part, click on the HTTP box and then copy the HTTP POST URL link.
Next, go to the Azure Data Factory studio, then find and drag the web activity, then connect with the set variable activity, go to the settings tab provide the HTTP URL link, select the method, provide the body, then publish the pipeline, and then debug your pipeline.
Video Demo: How to use Append Variable activity in Azure Data Factory