VMware PKS, NSX-T & Kubernetes Networking & Security explained

In continuation of my last post on VMware PKS and NSX-T explaining on getting started with VMware PKS and NSX-T (Getting Started with VMware PKS & NSX-T) , here is next one explaining around behind the scene NSX-T automation for Kubernetes by VMware NSX CNI plugin and PKS.

NSX-T address all the K8s networking functions, load balancing , IPAM , Routing and Firewalling needs, it supports complete automation and dynamic provisioning of network objects required for K8s and it’s workload and this is what i am going to uncover in this blog post.

Other features  like it has support for different topology choice for POD and NODE networks (NAT or NO-NAT) , it supports network security policies for Kubernetes , Clusters , Namespaces and individual services and it also supports network traceability/visibility using NSX-T in built operational tools for kubernetes.

I will be covering the deployment procedure of PKS after some time but just want to let explain that what happens on NSX-T side when you run “#pks create cluster” on PKS command line..and then when you create K8s Namespaces and PODs

pks create cluster

So when you run #pks create cluster with some argument , it goes to vCenter and deploys Kubernetes Master and Worker VMs based on specification you have chosen during deployment and on NSX-T side a new logical switch get created for these vms and get connected to these vms. (in this Example one K8s Master and 2 Nodes has been deployed) , along with logical switch , a Tier-1 cluster router get created which get connected to your organisation’s Tier-0 router.

K8s Master and Node Logical Switches


K8s Cluster connectivity towards Tier-0


Kubernetes Namespaces and NSX-T

Now if K8s cluster deployed successfully, Kubernetes cluster by default deploys three name space:

  • Default – The default namespace for objects with no other namespace.
  • kube-system – The namespace for objects created by the Kubernetes system.
  • kube-public – The namespace is created automatically and readable by all users (including those not authenticated). This namespace is mostly reserved for cluster usage, in case that some resources should be visible and readable publicly throughout the whole cluster. The public aspect of this namespace is only a convention, not a requirement.

for each default namespace, PKS automatically deploys and configures NSX-T Logical Switchs and each logical switch will have its own Tier-1 router connected to Tier-0.


pks-infrastructure Namespace and NSX-T

in the above figure you can clearly see “default”,”kube-public” and “kube-system” Logical Switches. there is another Logical Switch “pks-infrastructure” get created which is pks specific namespace and running pks related stuff like NSX-T CNI. “pks-infrastructure” is running NSX-NCP CNI plugin to integrate NSX-T with kubernetes.


kube-system Namespace & NSX-T

Typically, this runs pod like heapster , kube-dns , kubernetes-dashboard,  monitoring db , telemetry agent and stuff like ingresses and so on if you deploy so.


on NSX-T side as explained earlier a Logical switch get created for this Namespace and for each system POD a logical port get created by PKS on NSX-T.


Default Namespace & NSX-T

This is the cluster’s default namespace which is used for holding the default set of pods, services, and deployments used by the cluster. so when you deploy a POD without creating/specifying a new name space , “default Namespace” becomes default container to hold these pods and as explained earlier this also has its own NSX-T logical switch with a uplink port to Tier-1 router.


now when you deploy a Kubernetes pod without a new namespace , since that POD will be part of “Default Namespace”, PKS create a NSX-T logical port on the default logical switch.   let’s create a simple POD:


let’s go back to NSX-T’s  “Default Namespace” logical switch:


as you can see a new logical port has been created on default logical switch.

New Namespace & NSX-T

Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces. in simple terms Namespaces are like org vdc in vCD and Kubernetes best practice is to arrange PODs in namespaces. so when we create a new Namespace , what happens in NSX-T ?

i have created a new Namespace called “demo”.


if you observe below images left image showing default switches and right image is showing logical switches after creation of new Namespace.

and as you can see a new Logical switch has been created for new Namespace.

if you are creating PODs in default Namespace then all the pods get attached to default logical switch and if you are creating Namespace ( which is K8s best-practice) then a new logical switch get created and any POD which is getting deployed in this namespace will be part of its NSX-T logical switch and this new logical switch will also have its own Tier-1 router connecting to Tier-0 router.


Expose PODs to Outer World

in this example we deployed POD and get the internal network connectivity but internal only connectivity is not going to give access to this web server to outer world and this is default forbidden in kubernetes , so we need to expose this deployment using load balancer to the public interface on the specific port. let’s do that:



Lets browse this App using EXTERNAL-IP as you might know CLUSTER-IP is internal ip.


Kubernetes Cluster and NSX-T Load Balancer

As above when we expose this deployment using service on NSX-T there is a cluster load balancer get deployed automatically when we create cluster , on this load balancer NSX-CNI go ahead and add pod to virtual servers under a new load balancer VIP.


if we drill down to pool members of this VIP , we will see our kubernetes pod ep ips.


behind the scene when you deploy a cluster a LB logical switch and LB Tier-1 router which is having logical connectivity to the Load Balancer and Tier-0 Router , so that you can access the deployment externally.


This is what your Tier-0 will look like, having connectivity to all the Tier-1 and Tier-1 is having connected to Namespace logical switches.


these all logical switchs, Tier-1 router , Tier-0 router creations , their connectivity , LB creations etc all has been done automatically by NSX-T container (CNI) plugin and PKS. i was really thrilled when i tried this first time and it was so simple , if you understood the concept.

Kubernetes and Micro-segmentation

The NSX-T container plugin helps to exposure of container “Pods”as NSX-T logical switch /ports and because of this we can easily implement micro-segmentation rules. once “Pods ” expose to the NSX ecosystem, we can use the same approach we have with Virtual Machines for implementing micro segmentation and other security measures.


or you can use security groups based on tags to achieve micro segmentation.



This is what i have tried in this post to explain what happen behind the scene on NSX-T networking stack when you deploy and expose your applications on kubernetes and how we can bring in proven NSX-T based micro-segmentation.

Enjoy learning PKS,NSX-T and Kubernetes one of the best combination for Day-1 and Day-2 operation of kubernetes 🙂 and feel free to comment and suggestions.





Getting Started with VMware PKS & NSX-T

VMware Pivotal Container Service (PKS) provides a Kubernetes based container service for deploying and operating modern applications across private and public clouds. basically it is Managed kubernetes for multiple kubernetes cluster and aimed at Day 2 operations. K8S is designed with focus on high availability, auto-scaling and supports rolling upgrades.

PKS integrates with VMware NSX-T for advanced container networking, including micro-segmentation, ingress controller, load balancing, and security policy and also by using VMware Harbor, PKS secures container images through vulnerability scanning, image signing, and auditing.A PKS deployment consists of multiple VM instances classified into 2 different categories:

PKS Management Plane –

PKS management plane consist of below VMs:


  • PCF Ops Manager

Pivotal Operations Manager (Ops Manager) is a graphical interface for deploying and managing Pivotal BOSH, PKS Control Plane, and VMware Harbor application tiles. Ops Manager also provides a programmatic interface for performing lifecycle management of Ops Manager and application tiles.

  • VMware BOSH Director

Pivotal BOSH is an open-source tool for release engineering for the deployment and lifecycle management of large distributed systems. By using BOSH, developers can version, package, and deploy software in a consistent and reproducible manner.

BOSH is the first component, that’s installed by Ops Manager. BOSH is a primary PKS tile.BOSH was originally designed to deploy open source Cloud Foundry.Internally BOSH has below components:

  1. Director: This holds the role of core orchestration engine controls the provisioning of vms , required softwares and service life cycle events.
  2. Blobstore: The Blobstore stores the source forms of releases and the compiled images of releases. An operator uploads a release using the CLI, and the Director inserts the release into the Blobstore. When you deploy a release, BOSH orchestrates the compilation of packages and stores the result in the Blobstore.
  3. Postgres DB: Bosh director uses a postgres database to store information about the desired state of deployment including information about stemcells, releases and deployments. DB is internal to the Director VM.
  • Pivotal Container Service (PKS Control Plane)

PKS Control Plane is the self service API for on-demand deployment and Life cycle management of K8s clusters. API submit the request to BOSH which automates the creation , deletion and updates of kubernetes clusters.

  • VMware Harbor

VMwware harbor is an open-source, enterprise-class container registry service that stores and distributes container images in a private, on-premises registry. In addition to providing Role-Based Access Control (RBAC), Lightweight Directory Access Protocol (LDAP), and Active Directory (AD) support, Harbor provides container image vulnerability scanning, policy-based image replication, notary and auditing service.

PKS Data Plane

  • Kubernetes

K8s is an open-source container orchestration framework. Containers package applications and their dependencies in container images. A container image is a distributable artifact that provides portability across multiple environments, streamlining the development and deployment of software. Kubernetes orchestrates these containers to manage and automate resource use, failure handling, availability, configuration, scalability, and desired state of the application.

Integration with NSX-T

VMware NSX-T helps simplify networking and security for Kubernetes by automating the implementation of network policies, network object creation, network isolation, and micro-segmentation. NSX-T also provides flexible network topology choices and end-to-end network visibility.

PKS integrates with VMware NSX-T for production-grade container networking and security. A new capability introduced in NSX-T 2.2 allows you to perform workload SSL termination using Load Balancing services. PKS can leverage this capability to provide better security and workload protection.

Major benefit of using NSX-T with PKS and K8s is automation that is  dynamic provisioning and association of network objects for unified VM and pod networking. The automation includes the following:

  • On-demand provisioning of routers and logical switches for each Kubernetes cluster
  • Allocation of a unique IP address segment per logical switch
  • Automatic creation of SNAT rules for external connectivity
  • Dynamic assignment of IP addresses from an IPAM IP block for each pod
  • On-demand creation of load balancers and associated virtual servers for HTTPS and HTTP
  • Automatic creation of routers and logical switches per Kubernetes namespace, which can isolate environments for production, development, and test.

PKS Networking

  • PKS Management Network

This network will be used to deploy PKS Management components. this could be a dvSwitch or NSX-T logical switch since in my Lab i will be using no NAT topologies with virtual switch. in my Lab i will be using dvs with network segment of 192.168.110.x/24.

  • Kubernetes Node Network

This Network will be used for kubernetes management nodes. it is allocated to master and worker nodes. these nodes embed Node Agent to monitor the liveness of the cluster.

  • Kubernetes Pod Network

This network is used when an application will be deployed on to a new kubernetes namespace. A /24 network is taken from IP Block and is allocated to a specific Kubernetes namespace allowing for network isolation and policies to be applied between name spaces. The NSX-T Container Plugin automatically creates the NSX-T logical switch and Tier-1 router for each name spaces.

  • Load Balancer and NAT Subnet

This network pool, also known as the Floating IP Pool, provides IP addresses for load balancing and NAT services which are required as a part of an application deployment in Kubernetes.

PKS deployment Network Topologies – Refer Here

PKS Deployment Planning

Before you install PKS on vSphere with NSX-T integration, you must prepare your vSphere and NSX-T environment and ensure vCenter, NSX-T components, and ESXi hosts must be able to communicate with each other, ensure we have adequate resources.

  • PKS Management VM Sizing

When you size the vSphere resources, consider the compute and storage requirements for each PKS management component.

VM Name vCPU Memory Storage No. of VMs
Ops Manager 1 8 160 GB 1
BOSH 2 8 103 GB 1
PKS Control VM 2 8 29 GB 1
Compilation VMs 4 4 10 GB 4
Client VM 1 2 8 GB 1
VMware Harbor 2 8 169 GB 1

Compilation vms get created when an initial K8s cluster is deployed, software packages are compiled and four additional service VMs are automatically deployed as a process as a single task and these vms get deleted once compilation process completes. To manage and configure PKS,  PKS and Kubernetes CLI command-line utilities are required , these utilities can be installed locally on a workstation called Client VM.

  • Plan your CIDR block

Before you install PKS on vSphere with NSX-T, you should plan for the CIDRs and IP blocks that you are using in your deployment as explained above. these are the CIDR blocks that we need to plan:

  • Pods IP Block
  • Nodes IP Block

Below are the CIDR blocks that you can’t use because:

The Docker daemon on the Kubernetes worker node uses the subnet in the following CIDR range:


If PKS is deployed with Harbor, Harbor uses the following CIDR ranges for its internal Docker bridges:


Each Kubernetes cluster uses the following subnet for Kubernetes services,Do not use the following IP block for the Nodes IP Block:


In this blog post series i will be deploying NO-NAT topology and will walk you through step by step process of PKS deployment with NSX-T integration.

Next post on this series is VMware PKS, NSX-T & Kubernetes Networking & Security explained , this will help you understand what happens behind the scene in networking and security stack when PKS and NSX-T deploys kubernetes and its networking stack.



Upgrade NSX-T 2.1 to NSX-T 2.3

I am working on PKS deployment and will soon sharing my deployment procedure on PKS but before proceeding with PKS deployment, i need to upgrade my NSX-T lab environment to support latest PKS as per below compatibility matrix.

PKS Version Compatible NSX-T Versions Compatible Ops Manager Versions
v1.2 v2.2, v2.3 v2.2.2+, v2.3.1+
v1.1.6 v2.1, v2.2 v2.1.x, 2.2.x
v1.1.5 v2.1, v2.2 v2.1.x, v2.2.x
v1.1.4 v2.1 v2.1.x, 2.2.x
v1.1.3 v2.1 v2.1.0 – 2.1.6
v1.1.2 v2.1 v2.1.x, 2.2.x
v1.1.1 v2.1 – Advanced Edition v2.1.0 – 2.1.6

In this post i will be covering the procedure to upgrade NSX-T 2.1 to NSX-T 2.3.

So Before proceeding for upgrade , lets check the health of current deployment which is very important because if we start upgrading the environment and once upgrade is completed and after upgrade if some thing is not working , we will not come to know whether before upgrade it was working or not , so lets get in to validation of health and version checks.

Validate Current Version Components Health

First thing to check the Management Cluster and Controller connectivity and ensure they are up.

Next is to Validate host deployment status and connectivity.


Check the Edge health


Lets check the Transport Node Health


Upgrade Procedure

Now Download the upgrade bundle


Go to NSX Manager and browse to Upgrade


Upload the downloaded upgrade bundle file in NSX Manager


Since upgrade bundle is very big in size , it will take lots of time in upload, extraction and verification.Once the package has uploaded, click to “BEGIN UPGRADE”.


The upgrade coordinator will then check the install for any potential issues. In my environment there is one warnings for the Edge that the connectivity is degraded – this is because of i have disconnected 4 th nic which is safe to ignore, so when you are doing for your environment , please access all the warnings and take necessary actions before proceeding with upgrade.


Click Next will take you to view the Hosts Upgrade page. Here you can define the order and method of upgrade for each host, and define host groups to control the order of upgrade. I’ve gone with the defaults, serial (one at a time) upgrades over the parallel because i have two hosts in each clusters.

Click START to begin the upgrade, and the hosts will be put in maintenance mode, then upgraded and rebooted if necessary. ensure you need to have DRS enabled and the VMs on the hosts must be able to vMotion off of the host being put in maintenance mode. Once the host has upgraded, and the Management Plane Agent has reported back to the Manager, the Upgrade Coordinator will move on to the next host in the group.


Once the hosts are upgraded, click next to move to the Edge Upgrade page. Edge Clusters can be upgraded parallel if you have multiple edge clusters, but the Edges which has formed the Edge Clusters and upgraded serially to ensure connectivity is maintained. In my lab , i have a single Edge Cluster with two Edge VMs, so this will be upgraded one Edge at a time.Click on the “START” to start the edge upgrade process.



Once the Edge Cluster has been upgraded successfully, click NEXT to move to the Controller Node Upgrade Page. here you can’t change the sequence of upgrade of the controllers, controllers are done in parallel by default. (in my Lab i am running a single controller because of resource constraint but in production you will see three controllers deployed in a cluster). Click on “START” to begin the upgrade process.


Once the controller upgrade has been completed, click NEXT to move to the NSX Manager upgrade page. The NSX Manager will become unavailable for about 5 minutes after you click START and it might take 15 to 20 minutes to upgrade the manager.


Once the Manager upgrade has completed. review the upgrade cycle.


you can re-validate the installation as we did at the start of the upgrade, checking that we have all the green lights on, and the version of components have increased.