What is Virtual Network Peering in Azure?

In this tutorial, we are going to explore the use of Virtual Network Peering in Azure and why it is needed for cloud operation.

Virtual network peering allows you to combine more than one Vnet or Virtual Network in the Azure platform and it performs as an individual for connecting objects.

Microsoft backbone infrastructure is utilized for the incoming traffic from virtual machines in peered virtual networks that means traffic is routed into Microsoft private network individually.

How Virtual Network Peering Works?

In Azure, most of the resources are deployed to different virtual networks based on the needs of the services.

These networks are isolated by design, For those occasions when you need two virtual networks to be connected and the resources need to communicate, Azure virtual network peering is here to help.

In Azure With the help of peering, services in different virtual networks or VNets communicate with each other via high‑bandwidth, low‑latency Azure fiber backbones.

In our streaming scenario, setting up virtual network peering between Kaka, HDInsight, and Azure Databricks is going to establish this private communication network and allow streaming to happen.

This is done by configuring each side of the peered network and While Azure does support virtual network peering across regions known as global VNet peering.

How to Setup Virtual Network Peering?

Now let’s see how we set up virtual network peering. For our demo, peering will be established between the Azure Databricks VNet and the HD Kafka VNet. So we’ll start off in the Azure portal inside our resource group.

Step 1: One side of Peering

So we are going to go down to our Azure Databricks resource and then We will go down to Virtual Network Peerings.

We will click Add Peering so we can add our peering to this, So we are going to go ahead and call this peer spark‑peer.

And then the virtual network, notice that it has the Kafka VNet and we are going to leave that so that Allow virtual network access is enabled.

All of the other options we are simply going to leave as they are So we will go ahead and click and add that in.

Step 2: Another Side of Peering

Once that’s completed, we need to go over to the other side and complete the rest of the peering.

So we are going to do that by going over back into our resource group and go up to our virtual network, and go to Peerings, and clicking on Add here.

So on this side, we are going to call this Kafka‑peer, we are going to choose from the virtual network. We are going to choose the workers‑vnet.

Step 3: Azure Databricks cluster and worker net

So this is going to be the virtual network that was created when we created the Azure Databricks cluster.

Depending on how many demos and how many clusters you have with Azure Databricks, you might have multiple here.

So you will want to look for the name of the cluster and We see sparkcluster‑vnp in the full name of this worker net.

So we will choose that, Notice it grayed out, It shows spark‑peer so we know that we are connected properly.

We are going to leave the defaults here and we will click on OK and we will click Refresh, after that We will see that we are connected.

Step 4: Refresh and Connect Both

So just to verify this is all good to go, now we going to simply go back over to the sparkcluster and go into my virtual network peerings, and we will do a quick refresh. After we do a refresh, we see that those are both connected.

So now virtual network peering is all set for us and we are ready to start talking about connecting Azure Databricks to Kafka and producing events and consuming events.

Conclusion

You can build a peering within two virtual networks or Vnet and networks can relate to the same subscription or different subscriptions.

Similarly, it can be related to different deployment models in Azure with the same subscription.

Virtual Network peering is useful for security like You can implement network security accumulations in each virtual network to prevent access to different virtual networks or subnets.

What is Virtual Network Peering in Azure?

How Virtual Network Peering Works?