This guide is a replacement or an alternative to the published article in Github (https://github.com/mspnp/ha-nva) that I consider not very clear, and that I’m certain have discouraged many of you implementing it on production.
1- Architecture
1.1- Solution components
The following picture shows an implementation of a 2 nodes Active/Passive NVA distribution.
Image credit : Microsoft
The architecture includes the following components :
Network Virtual Appliances
2 nodes are supported. You should create two instances of your NVA, which means 2 Virtual Machines. The virtual machines have to be deployed within an Availability Set in order to ensure better availability, and for the group to achieve 99.95 % SLA. With the announcement of Availability Zones preview, you can plan to deploy each NVA in a zone, in order to ensure zone failure resiliency within the same region and achieve a 99.99% SLA. The configuration is Active/Passive, which means that even if both nodes are up and can process traffic, only one node will receive traffic on the same time.
NB : If you need an Active/Active configuration, you can use the HA Ports, which is a feature of Azure Load Balancer standard. HA ports will allow you to ‘say’ to the ‘load balancer’ to load balance any traffic to the backend pool members. More information about HA Ports : https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-ha-ports-overview
Zookeeper Nodes
Zookeeper is a centralized service that will automatically detect the failure of the active node, and switch the traffic to the passive node (by applying some configuration to your Azure resources : Public IP address, UDRs). The passive node will become active in this case. At least 3 Zookeeper nodes have to be deployed. The number 3 is the minimum required to achieve the ‘quorum’, which means that the decision made by the nodes have to be approved by at least 2 nodes. To keep it simple, 2 nodes are needed for high availability. But with 2 nodes, there are scenarios where the 2 nodes are unable to communicate with each other (whatever the reason is), and each one needs to make a decision (Node 1 wants to switch the Active/Passive nodes, Node 2 wants to keep the same configuration). In this case, we need a ‘judge’, this why we need to add a 3rd node, that will support either decision 1 or decision 2. This article related to Windows Server Failover cluster quorum can help you understand the principle (https://technet.microsoft.com/en-us/library/cc731739(v=ws.11).aspx). Zookeeper is a light-weight service, so a minimal VM size like A1 is enough.
Pubic IP Address
A Public IP address resource (Static Public IP address) is recommended to be attached to the Active NVA node, to be used for outbound internet traffic and inbound traffic coming from internet. This is the edge of your network. The Public IP address will be moved from the Active node to the passive node by zookeeper, in case of an active node failure detection.
User Defined Routes (UDRs)
This is the heart of the solution. UDRs are a collection of routes that you can apply the Subnets. They are used to force the VMs within Subnets to send traffic to selected targets instead of the default routes. You can have many routes within a UDR.
1.2- Network Virtual Appliance interface counts
This is a ‘bigger’ topic than to be discussed on few lines, but I need to mention it so that the next sections be understandable. The picture above shows 2 NICs per NVA. The first NIC will be used for external traffic (Inbound and outbound), and the Public IP address will be attached to it. The second NIC will be used for internal traffic, all internal subnets will send traffic to that NIC. You can have designs when you have more NICs per NVA, for example, you want to create an Internal Zone (Communication between Subnets within the VNET), a Cross-Premise connectivity zone (Commination to and from on-premise), a DMZ zone and an External Zone. The NICs count will affect the VM size (Each VM size have a maximum supported NIC count, so keep it in mind when choosing the VM size)
1.3- Zookeeper nodes network configuration
The zookeeper nodes must be placed within the same subnet that one of the NVA’s NICs. Do not place it in a no direct Subnet (where an UDR is applied to reach the NVA), or place it on a Subnet that have the default routes to the NVA’s subnet. The zookeeper nodes need to continually probe one of the NVA’s NICs. For example, on the above picture, the zookeeper nodes are placed on the DMZ internal subnet, and they continual probe the NVA’s NIC that is on the DMZ internal subnet. Note that zookeeper will initiate a failover when the probed NIC stops responding, even if the other NICs still alive. For example, if somehow, a firewall rule was added and prevents zookeeper from probing the NIC, it will initiate a failover even if the NVA is alive, so be careful. In addition, if somehow, all the other NICs are dead and the probed NIC is alive, zookeeper will not initiate a failover. So choose a ‘port’ that is shared by the NVA core service, and that represents the NVA health state.
The zookeeper nodes needs to exchange heart-beat and metadata information continually. Each zookeeper node will listen on a port (that you can define during the configuration), and the other nodes will communicate with it on that port. The ports should be different between the nodes (On the example below, the ports are 2181, 2182 and 2183 respectively for node 1, node 2 and node 3). If you will enable the zookeeper node’s OS firewall, do not forget to permit communication over the chosen ports.
1.4- User Defined Routes configuration
The UDRs should follow a very simple rule, to be compatible with the zookeeper initiated failover. Each UDR must contains routes that route traffic to only one interface. This will allow zookeeper to deterministically set the next hope in case of failover. You can create multiple UDRs, each UDR points to an interface, and apply them to the subnets.
The picture on the left shows a bad UDR configuration (UDR-Bad), because the routes do not have the same next hop. The picture on the right shows a good configuration(UDR-Good).
2- How it works?
This zookeeper solution implementation has a very simple concept. Let’s first see what do we need to do in case of the Active node (Node 1) fails:
- Configure all the UDRs, or to be more specify the routes inside the Route Tables to stop routing the traffic to Node 1 NICs, and send the traffic to Node 2. This is possible by changing the next hop in each route a the Node 2 specific IP
- Attach the Public IP address to Node 2 ‘external’ NIC. This is possible by un-assigning the Public IP address from Node 1 ‘external’ NIC and assign It to Node 2 ‘external’ NIC
This is what zookeeper will make:
- Continually probe the Node 1 NIC to see if it’s alive
- In case of the probe fails, it will initiate the steps mentioned above, based on a configuration file.
- Continually probe the Node 2 NIC to see if it’s alive
- In case of the probe fails, it will initiate the steps mentioned above, based on a configuration file. But this time to failover to Node 1 if it’s alive
3- Implementation
This section will show you how to implement zookeeper into your infrastructure.
3.1- Prerequisites and requirements
In order to succefully implement zookeeper, you will need to validate all the following points :
- Create an Azure AD application, to act as the identity used by zookeeper to make changes to your Azure resources during the failover. You will need to generate a certificate (pfx) during the application creation. This pfx will be converted later to a ‘jks’ format to be used by zookeeper. The Azure AD app will be assigned permissions on the resource that will be addressed by zookeepers during the failover
- The nvadaemon-remote.json file
- The log4j.properties file
- The configurezook.sh file
- An Azure Template to deploy the zookeeper VMs / configurations: 2 files: template.json and param.json
- Optional : A Deploy.ps1 that contains the powershell code to deploy the template
A- Create the Azure AD SPN
Follow this post to create an Azure AD SPN. Keep the pfx file and the password to be used later : https://buildwindows.wordpress.com/2017/12/03/create-an-azure-ad-service-principal-based-on-azure-ad-app-secured-by-a-certificate/
B- Assign the Azure AD App permissions on the Azure Resources
Zookeeper will use the Azure AD App identity to make changes on some Azure resources in order to make the failover ( The changes were discussed earlier in this post)
Give the Azure AD app the Contributor role on the following resources :
– The network Interface that Zookeeper will attach and detach the Public IP Address from
– The Route Tables that zookeeper will modify during the failover
Give the Azure AD app the Reader role on the following resources :
– The Public IP Address resource
The link in A- Create Azure AD SPN shows how to make roles assignments
C- Prepare the nvadaemon-remote.json
I highly recommend you to download and install the following tool to view and edit a JSON file : http://tomeko.net/software/JSONedit/
The following picture shows a view of the nvadaemon-remote.json file
There are 2 main sections: zookeeper and daemon
C1- General Settings
Fill the required information like the following table instructions, the red parameters must reflect your environment :
|
Parameter |
Value |
|
connectionString |
This is a comma separated string. The format is : zookpernode1:port1, zookpernode2:port2, zookpernode3:port3 Feel your zookeepers nodes names, and the ports to be used. You can keep use the default ports |
|
retrySleepTime |
This is the time interval between 2 successive probes (milliseconds) |
|
numberOfRetries |
How many retries before a zookeeper node considers the other node dead A Zookeeper node will consider the other zookeeper node dead after : retrySleepTime x numberOfRetries. In my example: 5 seconds |
|
subscriptionId |
The Azure Subscription ID |
|
clientId |
The Azure AD App client ID (Object ID). This value can be copied from the created Azure AD Application that you have created previously |
|
tenantId |
The Azure AD tenant ID. You can get this vaklue from the Azure Portal à Azure Active Directory à Properties (Directory ID) |
|
keyStorePath |
The store where the key will be stored on the zookeeper container. Keep it to default : /nvabin/nva.jks |
|
keyStorePassword |
A password to protect the access to the Key store within the zookeeper containers. |
|
certificatePassword |
The PFX certificate password defined earlier |
|
numberOfFailuresThreshold |
This is the number of probes failure, after which Zookeeper will consider the Active node dead and initiate a failover. |
|
probeConnectTimeout |
This is the maximum time interval between 2 successive probes (milliseconds) In my example, zookeeper will initiae a failover after : 3 x 3000 = 9 seconds |
C2- Routing Settings
Route Tables
The routeTables section is an array. Each line is the resource ID of a route table resource. As discussed earlier, you should have a route table for each interface. A route table should note route to different next hops.
Public IP address
This is also an array with one line. On the name, put anything that identifies the public ip address (you can keep the default). On the id, paste the Public IP address resource ID
NVAs
In this section, we will add the NVAs NICs resources. Because we have 2 NVAs, the array will contains 2 objects (0 and 1). 0 means the first NVA, 1 means the second NVA.
- For each NVA, add all the interfaces that should be addressed by the route tables configured earlier. If you have configured 2 Route Tables, then you will need to add only 2 NICs.
- Each interface have a name and id
- For the NIC where the Public IP address will be attached, use the same name that was used for the Public IP address (In my case it’s pip-fw-test). This will allow zookeeper to know which NIC will be assigned the Public IP address during the failover. On the id, type the resource id of the NIC. This NIC private IP will be used by zookeeper as the next hope for the first route table (routeTable 0 à NIC 0)
- The next route table will have as next hop the next NIC (in my case nic2) (route table 1 à NIC 1). The NIC name can be changed by must be equal between the NVA’s
- Note that in my case, nic3 is not needed, as it will be not addressed by any route table. I should have remove it.
- In the probeNetworkIntrface , type the id of the NVA’s NIC that zookeeper will probe to get the health state of the NVA.
- In the probePort, type the port that will be probed by zookeeper
D- Copy the files to the Cloud Storage
All the files that will be used during the zookeeper installation should be available for the zookeeper VMs during the deployment. The recommended way to do it is to copy the files to a new/existing Azure Storage Account. So the prerequisite is to create a new storage account or use an existing one. Note that the files can be removed after the zookeeper deployment.
The files to be copied are :
- Certificate pfx file
- nvadaemon-remote.json
- log4j.properties
- configurezook.sh
|
Step |
Screenshot |
|
Go to the Azure Portal and select your Storage Account. |
|
|
Go to Blobs Create a new Container : Name : installzook Policy Access: Container
Go to the container and upload the file : configurezook.sh |
|
|
Go to File Create a new File Share named : zookeepermonitor The quota can be set to 1 Upload the files :
|
|
E- Prepare the param.json file
The param file contains the parameters to deploy your zookeeper platform. Fill the following partamers:
- Location : The region where to deploy the resources
- adminUsername : the username for the Ubuntu zookeeper nodes admin user
- adminPassword : The password
- VmSize : the vmsize (Standard_A1_v2 is enough)
- vnetName : The VNET where the zookeeper will be deployed (Must be the same VNET where the NVAs are deployed)
- vnetResourceGroup : The VNET RG
- AvailabilitySetName : A name for the availability Set
- VmnamePrefix : The zookeeper nodes name prefix. The example shows zook0, so the VM names will be zook01, zook02 and zook03
- PrivateIPs : The Private IPs of each zookeeper node
- Subnet : The subnet for the zookeeper nodes. It must be one the NVA’s subnets
- CERTPASSWORD : The password of the pfx file created earlier
- CERTSTOREPASSWORD : the password to protect the certificate on the zookeeper nodes cert store
- CERTNAME : the name of the pfx certificate file
- SANAME : The storage account URL (where the files are stored)
- SAKEY : the Storage account key
- Customscripturl : the custom script URL
- customScriptCommandToExecute : the command to execute (leave it unchanged, unless you have changed the script file name)
3.2- Deploy
Using Powershell, CLI, you can now deploy the 3 Zookeepers nodes:
New-AzureRmResourceGroupDeployment -Name DeploymentName -ResourceGroupName ExampleResourceGroup `
-TemplateFile “template file path” TemplateParameterFile “template parameter file path”
4- Download the files
You can download the files from this link : https://www.dropbox.com/sh/hvxpzcgz6z1dg8f/AABL0ntxPT57lBaUluvlr1MUa?dl=0
buildwindows 