Before we upgrade

It is very important to validate all necessary components before NSX is upgraded.i have already shared upgrade impact in my previous post , also below checklist will help you to prepare your environment to upgrade:

• Download NSX upgrade bundle and check MD5.
• if you are upgrading from 6.2.2 to 6.2.3/6.2.4. , there is known bug which affects EDGE upgrade when incorrect ciphers are configured. VMware KB NSX Edge is unmanageable after upgrading to NSX 6.2.3 explain steps needed to check it and/or change it before the upgrade.
• Understand the  Update sequence for vSphere 6.0 and its compatible VMware products.
• Backup of components:
  • Take a backup of NSX Manager (if you are not doing it regularly) before the upgrade.
  • Before upgrading download technical support logs.
  • Export of vSphere Distributed Switches configuration.
  • Create a backup of vCenter Server database.
  • Take a snapshot of vCenter Server and vCenter Server database.
  • Take a snapshot of NSX Manager (without quiescing VMware Tools, because it is not supported and it might crash your NSX Manager).

Let’s Start…

1. Login to NSX Manager and go to Upgrade section.
2. Click on Upgrade button, specify a location of upgrade bundle and click Continue.
3. 
4. It will take some time to upload it..
5. Choose if you want to enable SSH and participate in Customer Experience Improvement Program . Click Upgrade to proceed.
6. Upgrade takes a some time , so be patient.
7. GUI reports that upgrade has been done.

NSX Controllers Upgrade

1. Go back to vSphere Web Client and go to Networking&Security and go to Installation and Click on Upgrade Available button.
2. Confirm “YES”that we want to proceed with Controllers upgrade.
3. It will take some time to upgrade controllers. In my case, it was around 10 minutes per controller.
4. 

VMware ESXi Host NSX VIBs Upgrade

1. Navigate to Installation section and click Host Preparation.
2. Click on Upgrade Available in every cluster you are using NSX and confirm Upgrade by clicking Yes.
3. In vSphere Client uninstalls and install tasks will be visible , basically process uninstalls old vibs and install new vibs. and as you are aware un-installation a vib require a Reboot, so all your hosts , will get rebooted. if you have DRS enabled cluster , NSX will do all the magic automatically , it will put a Host in maintenance mode and reboot and once this completes , it will move to other host in the cluster. once all upgraded , you will see like this…

NSX Edge Services Gateway and Distributed Logical Router Upgrade.

1. Next step is to upgrade Edge Services Gateway and Distributed Logical Router.
2. Navigate to NSX Edges and Right click on a edge and Click on “Upgrade version”
3. 
4. Please remember that there might be service interruption depending on a configuration used in environment.
5. In vSphere Web Client in current tasks, it will be visible, that two temporary new Edge Services Gateways deployment will be in progress and after some time all the edges will be upgraded to version 6.2.4.
6. Same procedure will be used to perform an upgrade of other edges.

Post-Upgrade Checklist

• Check if NSX Manager is working.
• Check if NSX Manager backup is working.
• Check if NSX VIBs are installed on ESXi host.
• Resynchronize the host message bus.
• Remove snapshot from NSX Manager.
• Remove snapshot from vCenter Server.
• Remove snapshot from vCenter Server database.

Happy learning 🙂

The NSX upgrade process can take some time, especially when upgrading ESXi hosts, because hosts must be rebooted. It is important to understand the operational state of NSX components during an upgrade, such as when some but not all hosts have been upgraded, or when NSX Edges have not yet been upgraded. VMware recommends that you run the upgrade in a single outage window to minimize downtime and reduce confusion among NSX users who cannot access certain NSX management functions during the upgrade. However, if your site requirements prevent you from completing the upgrade in a single outage window, the information below can help your NSX users understand what
features are available during the upgrade.

An NSX deployment upgrade proceeds as follows:

NSX Manager —> NSX Controller Cluster —> NSX Host Clusters —> NSX Edges

NSX Manager Upgrade

During:

NSX Manager Configuration is blocked. The NSX API service is unavailable. No changes to
the NSX configuration can be made. Existing VM communication continues to function. NewVM provisioning continues to work in vSphere, but the new VMs cannot be connected to NSX logical switches during the NSX Manager upgrade.

After:

All NSX configuration changes are allowed. At this stage, if any new NSX Controllers are
deployed, they will boot with the old version until the existing NSX Controller cluster is
upgraded. Changes to the existing NSX configuration are allowed. New logical switches, logical routers, and edge service gateways can be deployed. For distributed firewall if new features are introduced after the upgrade, those are unavailable for configuration (greyed out) in the user interface until all hosts are upgraded.

NSX Controllerr Cluster Upgrade

During:

• Logical network creation and modifications are blocked during the upgrade process. Do not make logical network configuration changes while the NSX Controller cluster upgrade is in progress.
• Do not provision new VMs during this process. Also, do not move VMs or allow DRS to move VMs during the upgrade.
• During the upgrade, when there is a temporary non-majority state, existing virtual machines do not lose networking.
• New logical network creation is automatically blocked during the upgrade.
• Do not allow dynamic routes to change during the upgrade.

After:

• Configuration changes are allowed. New logical networks can be created. Existing logical networks continue to function.

NSX Host Upgrade

During:

• Configuration changes are not blocked on NSX Manager. Upgrade is performed on a per-cluster basis. If DRS is enabled on the cluster, DRS manages the upgrade order of the hosts. Adds and changes to logical network are allowed. The host currently undergoing upgrade is in maintenance mode. Provisioning of new VMs continues to work on hosts that are not currently in maintenance mode.

When some NSX hosts in a cluster are upgraded and others are not:

• NSX Manager Configuration changes are not blocked. Controller-to-host communication is backward compatible, meaning that upgraded controllers can communicate with non-upgraded hosts. Additions and changes to logical networks are allowed. Provisioning new VMs continues to work on hosts that are not currently undergoing upgrade. Hosts currently undergoing upgrade are placed in maintenance mode, so VMs must be powered off or evacuated to other hosts. This can be done with DRS or manually.

NSX Edge Upgrade

NSX Edges can be upgraded without any dependency on the NSX Controller or host upgrades. You can upgrade an NSX Edge even if you have not yet upgraded the NSX Controller or hosts.

During:

• On the NSX Edge device currently being upgraded, configuration changes are blocked. Additions and changes to logical switches are allowed. Provisioning new VMs continues to work.
• Packet forwarding is temporarily interrupted.
• In NSX Edge 6.0 and later, OSPF adjacencies are withdrawn during upgrade if graceful restart is not enabled.

After:

• Configuration changes are not blocked. Any new features introduced in the NSX upgrade will not be configurable until all NSX Controllers and all host clusters have been upgraded to NSX version 6.2.x.
• L2 VPN must be reconfigured after upgrade.
• SSL VPN clients must be reinstalled after upgrade

Guest Introspection Upgrade

During an NSX upgrade, the NSX UI prompts you to upgrade Guest Introspection service.

 During:

• There is a loss of protection for VMs in the NSX cluster when there is a change to the VMs, such as VM additions, vMotions, or deletions.

After:

• VMs are protected during VM additions, vMotions, and deletions.

————————————————————————–

Verify the NSX Working State Before beginning and after the upgrade:

Before beginning the upgrade, it is important to test the NSX working state. Otherwise, you will not be able to determine if any post-upgrade issues were caused by the upgrade process or if they preexisted the upgrade process. Do not assume everything is working before you start to upgrade the NSX infrastructure. Make sure to check it first.

 Procedure

1. Note the current versions of NSX Manager, vCenter Server, ESXi and NSX Edges.

2. Identify administrative user IDs and passwords.

3. Verify you can log into the following components:

• vCenter Server
• NSX Manager Web UI
• Edge services gateway appliances
• Distributed logical router appliances
• NSX Controller appliances

• Verify that VXLAN segments are functional.

Make sure to set the packet size correctly and include the don’t fragment bit.

• Ping between two VMs that are on same logical switch but on two different hosts.
  1. From a Windows VM: ping -l 1472 –f <destVM>
  2. From a Linux VM: ping -s 1472 –M do <destVM>

• Ping between two hosts’ VTEP interfaces.
  • ping ++netstack=vxlan -d -s 1572 Notђ To get a host’s VTEP IP, look up the vmknicPG IP address on the host’s Manage > Networking > Virtual Switches page.

• Validate North-South connectivity by pinging out from a VM.

• Visually inspect the NSX environment to make sure all status indicators are green/normal/deployed.

1. Check Installation > Management.
2. Check Installation > Host Preparation.
3. Check Installation > Logical Network Preparation > VXLAN Transport.
4. Check Logical Switches.
5. Check NSX Edges.

• Record BGP and OSPF states on the NSX Edge devices

1. show ip ospf neighbor
2. show ip bgp neighbor
3. show ip route

• If possible, in the pre-upgrade environment, create some new components and test their functionality.

1. Create a new logical switch.
2. Create a new edge services gateway and a new distributed logical router.
3. Connect a VM to the new logical switch and test the functionality.

• Validate netcpad and vsfwd user-world agent (UWA) connections.

• On an ESXi host, run esxcli network vswitch dvs vmware vxlan network list –vds-name=<vds-name> and check the controller connection state.
• On NSX Manager, run the show tech-support save session command, and search for “5671” to ensure that all hosts are connected to NSX Manager.

• (Optional) If you have a test environment, test the upgrade and post-upgrade functionality before upgrading a production environment.

The deployment of overlay technologies has become very popular because of their capabilities in decoupling connectivity in the logical space from the physical network infrastructure. Devices connected to logical networks can leverage the entire set of network functions previously highlighted in Figure 5, independent of the underlying physical infrastructure configuration. The physical network effectively becomes a backplane used to transport overlay traffic.

This decoupling effect help solve many challenges traditional data center deployments are currently facing:

• Agile/Rapid Application Deployment: Traditional networking design is a bottleneck, preventing the rollout of new application at the pace that business is demanding. Overhead required to provision the network infrastructure in support of a new application often is counted in days if not weeks.
• Workload Mobility: Compute virtualization enables mobility of virtual workloads across different physical servers connected to the data center network. In traditional data center designs, this requires extension of L2 domains (VLANs) across the entire data center network infrastructure.This affects the overall network scalability and potentially jeopardizes the resiliency of the design.
• Large Scale Multi-Tenancy: The use of VLANs as a means of creating isolated networks limits the maximum number of tenants that can be supported (i.e., 4094 VLANs). While this value may currently be sufficient for typical enterprise deployments, it is becoming a serious bottleneck for many cloud providers.

Virtual Extensible LAN (VXLAN) has become the “de-facto” standard overlay technology and is embraced by multiple vendors; VMware in conjunction with Arista, Broadcom, Cisco, Citrix, Red Hat, and others developed it. Deploying VXLAN is key to building logical networks that provide L2 adjacency between workloads without the issues and scalability concerns found in traditional L2 technologies.

Configure VXLAN transport

The VXLAN network is used for Layer 2 logical switching across hosts and can potentially span multiple underlying Layer 3 domains. VXLAN is configured on a per-cluster basis, where each cluster that is to participate in NSX is mapped to a vSphere Distributed Switch. When mapping a cluster to a vSphere Distributed Switch, each host in that cluster is enabled for logical switches. The settings chosen here will be used in creating the VMkernel interface.

If logical routing and switching are needed, all clusters that have NSX VIBs installed on the hosts should also have VXLAN transport parameters configured. If you only plan to deploy a distributed firewall, it is not necessary to configure VXLAN transport parameters.

The teaming policy for VXLAN transport must be based on the topology of the physical switches. It is recommended that teaming policies are not mixed for different port groups on a vSphere Distributed Switch.

For certain teaming modes, VMware software creates multiple VTEPs to load balance traffic among the physical vNICs.The NSX Reference Design Guide contains a table with different teaming policy configuration options.

Now Lets configure VXLAN transport:

Log in to the vSphere Web Client and click Networking & Security.

Select Installation under the Networking & Security section and select the Host Preparation tab.

Click the Not Configured link in the VXLAN column of the appropriate cluster.

1. From the Switch drop-down menu, select the appropriate vSphere Distributed Switch.
2. In the VLAN text box, enter the VLAN number to be used for the VXLAN VTEP interfaces.
3. Enter 1600 in the MTU text box.
4. Select Use IP Pool.
5. From the IP Pool drop-down menu, select the appropriate IP pool to use for the VTEP interface addressing.
6. Select the appropriate VMKNic Teaming Policy from the drop-down menu.
   1. Note: the number of VTEPs is not editable in the UI. The VTEP number is set to match the number of dvUplinks on the vSphere Distributed Switch being prepared.
7. Click OK.

this will prepare for virtual wires

NOTE –  

• All hosts in the cluster must be in the vSphere Distributed Switch that is being leveraged by NSX.
• VMware vSphere Update Manager™ must be disabled before preparing clusters for network virtualization.

To prepare a vSphere cluster for network virtualization:

• Log in to the vSphere Web Client and click Networking & Security.
• Select Installation under the Networking & Security section and select the Host Preparation tab.
• Select the cluster you want to prepare for NSX:

1. In the Installation Status column, click Install.
2. Click OK to continue.

The Installation Status column will display In Progress for each member of the cluster.

• After the Host Preparation installation has finished, the Installation Status and Firewall columns display a green checkmark, as well as the NSX version number.

If the Installation Status column contains a red warning icon displaying Not Ready, click Resolve. Clicking Resolve might result in a reboot of the host. If the installation is still not successful, click the warning icon to display all errors. Take the required action and click Resolve again.

Happy Learning 🙂

The NSX for vSphere control plane manages logical networks and the overlay transport, and it must be configured in one of the following modes:

• Multicast Mode ‒ If multicast replication mode is chosen for a given logical switch, VMware NSX relies on the Layer 2 and Layer 3 multicast capability of the physical network to ensure VXLAN encapsulated multi-destination traffic is sent to all the VXLAN tunnel end points (VTEPs). The control plane uses multicast IP addresses on the physical network in this mode.
• Unicast Mode ‒ In this mode, the control plane is managed by the NSX Controller instances and all replication is done locally on the host. No multicast IP addresses or physical network configurations are required. This mode is very well suited for smaller deployments.
• Hybrid Mode ‒ An optimized version of unicast mode, where local traffic replication for the subnet is offloaded to the physical network. This mode requires IGMP snooping on the first hop switch, and IGMP querier must be available. However, PIM is not required.

The NSX Controller provides East-West routing by programming traffic flows on the VMware NSX Virtual Switch. If you plan to use the unicast or hybrid control plane mode for the logical switch, you must add an NSX Controller. The NSX Controller optimizes virtual machine broadcast traffic (ARP only), and it stores the learning on the host.

As stated in my previous post – NSX for vSphere 6.2 only supports controller clusters with three nodes.

Following are the resource requirement for deploying controllers….

• 4 vCPUs
• 4 GB of memory (2 GB are reserved)
• 20 GB of disk space.

To deploy NSX Controller nodes

Log in to the vSphere Web Client and click Networking & Security.

Select Installation under the Networking & Security section and select the Management tab.

In the Add Controller dialog box:

1. Select the appropriate NSX Manager from the NSX Manager drop-down menu.
2. From the Datacenter drop-down menu, select the data center where you are adding the node.
3. From the Cluster/Resource Pool drop-down menu, select the appropriate cluster or resource pool where the NSX Controller is to be deployed.
4. From the Datastore drop-down menu, select the datastore in which the NSX Controller will be deployed.
5. (Optional) From the Host drop-down menu, select the host.
6. (Optional) From the Folder drop-down menu, select the folder.
7. In the Connected To selection box, click Select to choose the logical switch, port group, or distributed port group to which the node is to be connected.
8. In the IP Pool selection box, click Select to choose the IP pool from which IP addresses are to be assigned to the node.
9. Type and re-type a password for the NSX Controller.
10. Click OK.

Deploy two additional NSX Controller nodes to provide a greater level of resiliency.

Now we have deployed all the required controllers and ready for production , next thing is to prepare our vSphere clusters.