Automate Cloud Director Edge Firewall Rules using API

Edge Firewall Rules:

Tenant can use the edge gateway Firewall tab to add firewall rules for that edge gateway. You can add multiple NSX Edge interfaces and multiple IP address groups as the source and destination for these firewall rules.

The Firewall rules will already have a few entries pre-built in as part of preconfigured services, which you should not need to change in most cases:


When Provider/Tenant creates firewall rules from GUI, the user is allowed to create one firewall rule at a time and at times provider/tenant want to automate firewall rule creation specially if provider/tenant has many rules to create or want to add multiple ports in to firewall rule which reduces their manual efforts..



To overcome with this issue, Cloud Director offers NSX proxy API , which will help provider/tenant to automate rules. Here is process of creating firewall rule using API:

  • Find NSX Edge ID using API
  • Get Edge FW Configuration
  • Post Firewall rules

Find NSX Edge ID:

To find the NSX Edge ID , you need to fire “Get” to this API “https://<vCD>/network/edges&#8221;, here is “Get” to my API’s headers information , body will be empty.


Here is API Content:

This call will return All the edges and their related information, note on which edge you want to apply firewall rule and its ID.


Get Edge FW Configuration:

Though this step is not required but if you still want to see what all firewall rules has been configured “get” against “Edge Id”  -“https://<vCD>/network/edges/8417a9fc-c1df-4c03-befd-c79f60d5d0ab/firewall/config&#8221;


Post Firewall Rule:

Header Information:

To add firewall rule, we need to do “Post” against URL “https://<vCD>/network/edges/8417a9fc-c1df-4c03-befd-c79f60d5d0ab/firewall/config/rules&#8221; with following parameters:

  • 417a9fc-c1df-4c03-befd-c79f60d5d0ab – this is Edge ID which we got from Step-1
  • Content-Type – Application/xml
  • Authorization – Bearer Token


Body Content:

Here is body content which is self explanatory. few important items are as below:

  • Entire body must be within <firewallRules></firewallRules>
  • Within<firewallRules></firewallRules> you can write various rule within <firewallRule></firewallRule> section.
  • <service></service>  – In this section you will define “protocol” & “port”.
<name>New Rule</name>
Screenshot of my API Call body which should be in “application/xml” format.7

if you need to write multiple rules in to single API call, then you can create multiple sections of <firewallRule></firewallRule> with single section of <firewallRules></firewallRules>


Once Header and body is ready, do a post and you should get a valid response of “201 Created”


vCD GUI reflects what you put in to the body of the API call.


I hope this will help Cloud providers/tenants to automate rules which needs to be automated or there are rules which providers need to create by default when onboarding a tenant.

Onboard Tenants on Cloud Director in less than 5 Minutes using vCD Terraform Provider

In continuation to my last post, In this post we are going to onboard a tenant using vCloud Director Terraform provider , there are five things that we are going to do:


  • Create a new Organisation for the Tenant
  • Create a new Organisation Administrator for this Tenant
  • Create a new Organisation VDC for the Tenant
  • Deploy a new Edge gateway for the Tenant
  • Create a new routed Network for the Tenant

Code for New Organisation:

So in this section , we are going to create a new organisation names “T3” which is enabled to use, This section creates a new vCloud Organisation by specifying the name, full name, and description.

#Create a new org names "T3"
resource "vcd_org" "org-name" {
  name             = "T3"
  full_name        = "My organization"
  description      = "The pride of my work"
  is_enabled       = "true"
  delete_recursive = "true"
  delete_force     = "true"

Code for Creating Organisation Administrator:

Once as a provider you created Org, this org need an admin, below code will create local org admin. In this code everything is self explanatory but few important parameters explained here:

  • Resource Type -> “vcd_org_user”
  • org & name -> these are variable, referred in variable file.
  • role -> role assigned to this user
  • password -> initial password assigned
  • depends_on -> Explicit dependencies that this resource has. These dependencies will be created before this resource
#Create a new Organization Admin
resource "vcd_org_user" "org-admin" {
org = var.org_name #variable referred in variable file 
name = var.org_admin #variable referred in variable file
description = "a new org admin"
role = "Organization Administrator"
password = "change-me"
enabled = true
email_address = ""
depends_on = []

Code for Creating new Organisation VDC:

So till now we created Org and Org admin , next is to create a organisation virtual data center , so that tenant can provision VMs, Containers and Applications. few important configuration parameters to consider:

  • name -> T3-vdc
  • Org -> T3
  • Allocation Pool -> Pay as you go (represented as “AllocationVApp”).
  • network_pool_name -> Network pool name as defined during provider config.
  • provider_vdc_name -> Name of Provider VDC name.
  • Compute & Storage -> Define compute and storage allocation.
  • VM_quota -> Maximum no. of vms can be provisioned in to this VDC
  • network_quota -> Maximum no of networks can be created.
# Create Org VDC for above org
resource "vcd_org_vdc" "vdc-name" {
  name        = var.vdc_name
  description = "The pride of my work"
  org         = var.org_name #variable referred in variable file
  allocation_model = "AllocationVApp"
  network_pool_name = "PVDC-A-VXLAN-NP"
  provider_vdc_name = "PVDC-A"
  compute_capacity {
    cpu {
      limit = 0
    memory {
      limit = 0
  storage_profile {
    name     = "*"
    limit    = 10240
    default  = true    
  metadata = {
    role    = "For Customer T3"
    env     = "staging"
    version = "v1"
  vm_quota                 = 10 #Max no. of VMs 
  network_quota            =  100
  enabled                  = true
  enable_thin_provisioning = true
  enable_fast_provisioning = true
  delete_force             = true
  delete_recursive         = true
depends_on = []

Code for Creating Edge Gateway for Tenant

This next section creates a new vCloud Organisation Edge Gateway by specifying the name, full name, and description. Provider configures an edge gateway to provide connectivity to one or more external networks.

  • Configuration -> compact
  • Advanced -> this will be an advance edge
  • distributed_routing -> distributed routing is enabled
  • external_network ->  uplink information towards DC exit.
  • You will notice there is a ‘depends_on’ setting. This means that this resource depends on the resource specified before executing.
resource "vcd_edgegateway" "egw" {
  org = var.org_name #variable referred in variable file
  vdc = var.vdc_name #variable referred in variable file
  name                    = var.edge_name
  description             = "T3 new edge gateway"
  configuration           = "compact"
  advanced                = true
  distributed_routing     = true
  external_network {
    name = "SiteA-ExtNet"
    subnet {
      ip_address            = ""
      gateway               = ""
      netmask               = ""
      use_for_default_route = true
depends_on = [vcd_org_vdc.vdc-name]

Code for Creating Organisation Routed Network

An organization VDC network with a routed connection provides controlled access to machines and networks outside of the organization VDC. System administrators (Providers) and organization administrators can configure network address translation (NAT) and firewall settings on the network’s Edge Gateway to make specific virtual machines in the VDC accessible from an external network. Things to consider:

  • resource -> must be of type “vcd_network_routed”
  • Define other networking information
resource "vcd_network_routed" "net" {
org = var.org_name #variable referred in variable file
vdc = var.vdc_name #variable referred in variable file
name = "T3-Routed-net"
edge_gateway = var.edge_name 
gateway = ""
dhcp_pool {
start_address = ""
end_address = ""
static_ip_pool {
start_address = ""
end_address = ""
depends_on = [vcd_edgegateway.egw]

Putting it all together:

So i have put all this code in to a single file and also created a variable file, which will allow providers to on-board a new Tenant less then “5 minute” , provider admin just need to update few parameters in to the variable file like:

  • org_name -> Tenant organisation name
  • vcd_name -> Tenant Org VDC Name
  • edge_name -> Tenant N/S router name
  • org_admin -> Org Admin name


Once you input the parameters, run terraform plan and Apply the plan, this enitre process should not take more than 5 minutes to complate.

  • Terraform Plan -out f1.tfplan
    • 16
  • Terraform apply “f1.tfplan”
    • 17


As described above all five tasks related to a Tenant on-boarding got successfully completed and if you notice highlighted boxes , everything is over in less than 2 minutes, isn’t it awesome ?


Here i am attaching variable and code file , which you can use it in your environment by just changing variable file contents like , org_name , vdc_name etc..which i explained above. pls try these files in to a non-prod environment and make your self comfortable before doing it in production.Here is the Code file to download – Please share feedback , suggestion any in the comment section…


Automate vCloud Director with Terraform Provider

The new refreshed Terraform vCloud Director provider enables administrators and DevOps engineers to define vCloud Director “infrastructure as code” inside Terraform configuration files. This makes it an efficient automation and integration tool and this project is fully open-source and available on GitHub and also HashiCorp is hosting it in the “terraform-providers” namespace together with all the other official Terraform providers.

If you’d like to contribute with a feature request, report an issue or propose a code improvement please visit the project’s site below. There you can also see current activity and what’s in the works.

Terraform Installation & Configuration:

Terraform installation is very simple, it is just a single file. if you are running Linux OS system it is “terraform” and if it is windows based systems then it is terraform.exe.You can download the latest version of Terraform from the HashiCorp website using this direct link:

I am using Windows in my Lab so for my Windows based system, I simply downloaded the Terraform Windows x64 file and put into a folder called “c:\tf” then added this folder into my PATHS variable so that I could run terraform.exe from anywhere. This can be done by going to System Properties –> Advanced –> Environment Variable


now go to “System variables” and add :terraform.exe” folder location.

  • Select Path and Click on “Edit”
  • at the end of the line put symbol “;” and add “terraform.exe” directory path.


Download Terraform vCloud Director Plugin and VS Code:

Download the latest vCloud director terraform plugin from here  and put in to a directory and we will use this during terraform initialization.

Now a days i start using Microsoft Visual Studio Code to write my automation scripts. It has inbuilt terraform plugin to highlight/validate code which makes it way more simple, then working with different text editors and multiple windows. It’s a free download, check it out or you are free to use any other text editor of your choice.

Create Terraform files:

  1. Create a new folder
  2. Create two terraform files (terraform.tfvars & as below and put the below content in to the files
  3. Save both the files in to the folder which we created in to step1
  4. terraform.tfvars:
    • This is where we give value to the variables. For example vcd_url = “https://vcd-01a.corp.local/api”, this means that anytime var.vcd_url is referenced in the “ file” it will reference back to . Most variables below are self-explanatory.
    • # vCloud Director Connection Variables
      vcd_user = "administrator"
      vcd_pass = "VMware1!"
      vcd_url = "https://vcd-01a.corp.local/api"
      vcd_max_retry_timeout = "60"
      vcd_allow_unverified_ssl = "true"
    • 10
  5. – This will have actual actionable code which will perform action on to vCloud Director.We do this by using a provider and multiple resources. The provider we are using in this demonstration is “vcd”. The resources are then responsible for different parts of vCloud Director. In this example “vcd_org” is going to create, modify or delete an Org. we are created new VCD Organisation.
    • variable "vcd_user" {
      description = "vCloud user"
      variable "vcd_pass" {
      description = "vCloud pass"
      variable "vcd_allow_unverified_ssl" {
      default = true
      variable "vcd_url" {}
      variable "vcd_max_retry_timeout" {
      default = 60
      # Connect VMware vCloud Director Provider
      provider "vcd" {
      user = var.vcd_user
      password = var.vcd_pass
      org = "System"
      url = var.vcd_url
      max_retry_timeout = var.vcd_max_retry_timeout
      allow_unverified_ssl = var.vcd_allow_unverified_ssl
      #Create a new org names "T3"
      resource "vcd_org" "org-name" {
      name = "T3"
      full_name = "My organization"
      description = "The pride of my work"
      is_enabled = "true"
      delete_recursive = "true"
      delete_force = "true"
    • 11

Initialize terraform plugin:

The terraform “init” command is used to initialize a working directory containing Terraform configuration files. This is the first command that you should be run after writing a new Terraform configuration or cloning an existing one from version control. It is safe to run this command multiple times.this command will never delete your existing configuration or state.

During init, Terraform searches the configuration for both direct and indirect references to providers and attempts to load the required plugins. For providers distributed by HashiCorp, init will automatically download and install plugins if necessary.

Since on my windows init didn’t downloaded plugin for some reason , so i have downloaded vCD plugin in above steps and during initialization i am pointing towards my directory and it says terraform will use vCloud Director Plug-in version 2.6.


Terraform Plan:

The terraform “plan” command is a convenient way to check whether the execution plan for a set of changes matches your expectations without making any changes to real resources or to the state.

“Terraform plan” command will check above created files and check what changes it needs to do on the environment and will give you summary of tasks.


Terraform Apply

The “terraform apply” command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a terraform plan execution plan.


Org Created Successfully:

above step created an organisation in to my vCloud Director environment and it took only few minutes.


lots of development is happening on terraform  provider for vCloud Director which will help VMware Cloud provider to automate repeated tasks. i will continue to add few more blog articles on this topic, stay tuned….


Configure HCX for Cloud to Cloud Migrations

VMware HCX is an application mobility platform that is designed for simplifying application migration across cloud, workload rebalancing, and business continuity across data centers and across clouds.

Application migration

You can schedule and migrate thousands of vSphere virtual machines within and across data centers or Clouds without requiring a reboot.

Change platforms or upgrade vSphere versions

With HCX, you can migrate workloads from vSphere 5.x and non-vSphere (KVM and Hyper-V) environments within and across data centers or clouds to current vSphere versions without requiring an upgrade.

Workload rebalancing

Workload rebalancing provides a mobility platform across cloud regions and cloud providers to allow customers to move applications and workloads at any time to meet scale, cost management, compliance, and vendor neutrality goals.

HCX Deployment Types

  • Legacy vSphere to SDDC

    • In this deployment type, the HCX Manager at the Legacy site initiates Site Pairing, and the Service Mesh appliances initiate the Interconnect tunnels. The HCX Manager and Service Mesh appliances at the SDDC site are the receivers.
  • Legacy vSphere to Public Cloud

    • In this deployment type, the HCX Manager at the Legacy site initiates Site Pairing, and the Service Mesh appliances initiate the Interconnect tunnels. The HCX Manager and Service Mesh appliances at the Public Cloud are the receivers.
  • Cloud-to-Cloud

    • In this deployment type, the HCX Manager at the SDDC or Public Cloud can initiate or receive Site Pairing requests and act as the initiator or receiver during the HCX Interconnect tunnel creation.

In this post we are going to configure HCX cloud to cloud between two SDDC deployed on VMware Cloud on AWS


  • Deploy SDDC on Site A
  • Deploy SDDC on Site B
  • Deploy HCX on Site A
  • Deploy HCX on Site B

Connectivity Options

  • Over EIP
  • Over Direct Connect

In this post i will be configuring HCX cloud to cloud migration over EIP as HCX sets up its own secure tunnel with military grade encryption, so it is secure to migrate over EIP but you can also setup using VPN connection for management network or you can setup using direct connect between two clouds also.

Open Firewall

Once both the SDDC and HCX has been deployed , go ahead inside “Network & Security” section and create firewall rules for HCX as below inside management gateway.


with above firewall rules , you should be able to access HCX over the public IP.

Site Pairing

  • On HCX console , go to “Site Pairing” and enter the second site HCX server name and credential.

This slideshow requires JavaScript.

  • if you want to do reverse migration then again do pairing from second site to primary site.

This slideshow requires JavaScript.

Create Service Mesh

  • Click Create Service Mesh
  • Select Sites:
    • Click each drop-down and select a source and destination site. Only connected Site Pairs are displayed
    • Click Continue.
  • Select Compute Profiles: 
    • Click the Select Source Compute Profile drop-down and select a Compute Profile.
    • Click the Select Remote Compute Profile drop-down and select a Compute Profile.
    • Click Continue.

NOTE – you can pickup existing compute profile or you can create a new profile.

  • By default, the HCX interconnect uses the Uplink Network Profiles defined in the Compute Profile for the source and destination sites. You can override the default As an example, an override can be useful in vCloud Director-based deployments where an uplink network that deviates from a common configuration created for an Organization to consume during the Service Mesh creation.
    • Click the Select Source Uplink Network Profile drop-down.
    • In VMC case we will choose “External Network” which is already having two public IPs.
    • Click Close.
    • Click Continue.
  • Configure the Network Extension appliances deployed per switch and click Continue.
  • Optionally Configure HCX Traffic Engineering features such as Traffic bandwidth control etc..
  • Review the topology and name the service Mesh and click Finish.
  • Wait for few minutes , this will deploy all the appliances and show connectivity across site is up.

This slideshow requires JavaScript.

if everything is done correctly , you should show all the HCX services are up and running.


Configure Network Extension

  • In the HCX dashboard, select Network Extension.
  • At the top of the page, select Extend Network.
  • Select one or more Distributed Port Groups or NSX Logical Switches.
  • Specify network segment which need to be extended.

This slideshow requires JavaScript.

Live Migrate a VM from one Cloud to another Cloud

  • Choose a VM which you want to migrate from one Cloud to another Cloud
  • Specify destination resource pool , storage , network etc and click on Migrate
  • check the progress of migration in vCenter , your application user even will not come to know that their application has been migrated to another cloud/sddc.

This slideshow requires JavaScript.

If you see with this HCX can easily migrate application based on requirement across cloud like AWS, IBM, Oracle or Azure or VCPP based or back to on-prem or hosted  clouds. Most important is that HCX solves problem of Cloud lock in. it gives freedom to customer to move VM across cloud with maximum security , most faster and without impact to applications.


please share feedback 🙂




vCloud Director 10 – NSX-T – Tenant Configuration

In continuation of my previous post , in this post i will be covering tenant side configuration of vCloud Director 10 along with NSX-T.

Create OrgVDC

To provide resources to an Tenant organization, you create one or more organization virtual data centers for tenant organization.To Create an OrgVDC , you need to go to “Cloud Resources” then “Organization VDCs” and Click on New:

  1. Name Tenant OrgVDC appropriately
  2. Select the Organisation
  3. Select the PVDC which is NSX-T backed.
  4. Choose appropriate allocation model (flex)
  5. Configure reservation pool related settings
  6. Choose appropriate storage policy
  7. enable “Network Pool” and select correct network pool and specify max networks
  8. Review and click on Finish.

This slideshow requires JavaScript.

Create Org Edges

To connect tenants networks created inside org vDC to out side , we need to create edge gateways, which internally automatically create T1 router, here are the steps to create edge:

  1. Login to tenant by clicking on “Open in Tenant Portal” and go to Edges & click  New
    • 29.png
  2. Name Tenant edge appropriately
  3. Select IP segment and reserve few IPs to talk to external world.
  4. Review configuration and submit

This slideshow requires JavaScript.

If you look back in to NSX-T , this will create a Tier-1 router automatically and connect it to Tier-0 router.


Org Edge supported Tenant Operation:

Currently the following T1 GW networking services are available to tenants:

  • Firewall
  • NAT
  • DHCP (without binding and relay)
  • DNS forwarding
  • IPSec VPN with API only and only apply Policy based with pre share key.


Create Org Networks

The first network to create for tenant is an organization Virtual Datacenter network, An orgVDC network allows virtual machines in the orgVDC to communicate with each other and to access other networks, including orgVDC networks and external networks, either directly or through an Edge Gateway (T0) that can provide firewall and NAT services as of now. There are three type of org Networks:


You can add an isolated orgVDC network, which is accessible only by this organization. This network provides no connectivity to virtual machines outside this organization. Virtual machines outside of this organization have no connectivity to the virtual machines in the organization.


Routed network control the access to an external network, System administrators and organization administrators can configure network address translation (NAT), firewall, and VPN settings to make specific virtual machines accessible from the external network.


You can import existing NSX-T overlay switch in to org , for this networking type all networking need to be configured and managed out side of vCloud Director.

This slideshow requires JavaScript.

Tenant VM External Access:

As i said tenant networks are not advertised , we need to create SNAT rules to allow external access:



NOTE – Tenant can only self service Isolated and routed networks, there are few options like DFW and Load Balancer still has not been exposed to tenants.



vCloud Director 10 – NSX-T – Provider Configuration

As you may be aware vCloud Director from its inception initially was relying on vCNS and after that on NSX-V to provide on-demand , self service cloud networking capabilities and now since VMware is moving towards newly re-written networking platform called NSX-T and with every new version , it is getting mature and feature rich , vCloud Director with version 10 brings many of its capabilities in to it to offer more and more self service capabilities to tenant and ease of implementation and operation for providers, in this post i am covering how to integrate NSX-T with vCD from Provider prospective.


As you may be aware that NSX-T is no more coupled/dependent on vCenter ,so to integrate NSX-T with vCloud Director you must install and configure NSX-T Data Center. Here are the high level steps:

  • Deploy and configure the NSX-T Manager virtual appliances.
  • Create transport zones based on your networking requirements.
  • Deploy and configure Edge nodes and an Edge cluster.
  • Configure the ESXi host transport nodes, these will become PVDC resources of NSX-T based tenants.
  • Create a tier-0 gateway , this will work as “External Network” for vCloud Director.

Register NSX-T Manager

Once NSX-T setup is done, login to vCloud Director with administrator credential and  Click on “vSphere Resources” and go to NSX-T Managers to add NSX-T manager.

Create Network Pool

A network pool is a group of undifferentiated networks that is available for use in an organization virtual datacenter to create vApp networks and certain types of organization virtual datacenter networks.
so once NSX-T manager is added , next thing is we need to create network pool and to create network pool  , go back to “Cloud Resources” , go to “Network Pools” and Click on new:
Here is the creation of Network pool steps:
  1. Name it appropriately.
  2. Select “Geneve Backed” type Network pool
  3. Select Appropriate NSX-T Providers (you can have multiple NSX-T Providers)
  4. Select Appropriate Overlay Transport Zone
  5. review and submit.

This slideshow requires JavaScript.

Configure External Networks

External networks helps providing a connection to the outside the world (internet). external networks are backed up by NSX-T Tier-0 router.

As i said in pre-requisite section , you need to manually create Tier0 in NSX-T, this T0 router will provide external network access to your tenant and should be routable from Internet. Create an Active-Active T0 with ECMP mode is recommended practice.


Once T0 is created , you will then import T0 in to vCloud Director 10. you will also need to define IP pool , which will be used to sub-allocate IPs to Tenants.

Below is the process to create vCloud Director 10 external network by importing Tier0  router created in side NSX-T.

  1. Choose Backing Type as “NSX-T Resources (Tier-0 Router)” and select registered NSX-T
  2. Provide Name
  3. Select Tier-0 router
  4. Add a “Network Pool” with Gateway details.
  5. review and complete , which will import T0 in to vCloud Director construct.

This slideshow requires JavaScript.

Create Provider VDC

Now you can create Provider VDC (PVDC) which is basically mapped to a vSphere cluster or a resource pool. PVDC to successfully work you need to ensure that vSphere cluster has been prepared with NSX-T and part of a transport zone.When creating NSX-T backed PVDC you will have to specify the Geneve Network Pool created in the previous step.

Go to “Cloud Resources” – “Provider VDCs” and Click on “NEW” to create new PVDC backed by NSX-T based networks.

  1. Name your PVDC
  2. Select vCenter which is having NSX-T backed Cluster
  3. Select appropriate Cluster and VM Hardware version
  4. Select appropriate Storage policy
  5. Select NSX-T manager and Network Pool ( as created above – Geneve backed pool )
  6. Review configuration and finish.

This slideshow requires JavaScript.

if everything is configured properly, PVDC get created successfully.


This completes vCloud Director configuration from provider prospective. In the next post i will be covering tenant onboarding process on NSX-T based Network.


vCloud Director 10 : T-Shirt Sizing

In vCloud Director 9.7 compute policies were introduced to offer/manage the T-shirt sizing of the VMs which i have covered in detail in my this post,  in vCloud Director 10 similar concept has been brought in to GUI , which is now easy to implement & manage and in vCD 10 this is being called “Sizing Policy”

So from Cloud Provider prospective VM sizing policy defines the compute resource allocation for virtual machines within an organization VDC. Sizing policy allow provider to control the following aspects of compute resources consumption at the virtual machine level:

  • Number of vCPU, vCPU clock speed, reservations, limits and shares
    • 1.png
  • Amount of memory allocated to the virtual machine , reservation, limits and shares.
    • 2.png

Create T-Shirt Sizes:

Let’s create few example T-Shirt sizing policies:

  • Policy Name – X1
    • Description: Small-size VM policy Memory: 1024 Number of vCPUs: 1
    • Name: X1
    • Memory: 1024
    • Number of vCPUs: 1
  • Policy Name – X2
    • Description: Medium-size VM policy Memory: 2048 Number of vCPUs: 2
    • Name: X2
    • Memory: 2048
    • Number of vCPUs: 2
  • Policy Name – X3
    • Description: Large-size VM policy Memory: 4096 Number of vCPUs: 4
    • Name: X3
    • Memory: 4096
    • Number of vCPUs: 4
  • Policy Name – X4
    • Description: X-Large-size VM policy Memory: 8192 Number of vCPUs: 8
    • Name: X4
    • Memory: 8192
    • Number of vCPUs: 8

Create T-Shirt Sizing policies:

  1. Cloud Provider Administrator, logins to vCloud Director and go to “VM Sizing Policies” and Click on “New” to create new policy
    • 7.png
  2. Name and describe the policy as per above example and move to Next.
    • 3.png
  3. In next section enter CPU related parameters , in this example i am choosing “vCPU Count” , providers can choose based on their requirement and leave it all blank as none of the fields are mandatory.
    • 4.png
  4. In next section enter Memory related parameters , in this example i am choosing only “Memory”, providers can choose based on their requirement and leave it all blank as none of the fields are mandatory.
    • 5.png

that’s it , so simple to create policies , follow the same step to create multiple policies as per above example.

Publish Created Policies:

vCloud Director system administrators create and manage VM sizing policies at a global level and can publish individual policies to one or more organization VDCs.

so above step we have created polices , we need to publish these policies to organisation VDC’s.

  1. Select Cloud Resources then click on Organization VDCs and go inside an organization VDC
    • 8.png
  2. Inside VDC , go to VM Sizing Policies and click on Add
    • 9.png
  3. Select the policies that you want to make available for a Particular oVDC/Tenant
    • 10.png
  4. You can set policies as default policies, which will make policy appear as the default choice for the tenants during a VM and vApp creation and VM edit.
    • 11.png

Once polices published to organisation’s VDC, when tenant user logins and try to deploy a new VM , he/she now see options to chose T-Shirt sizes with their descriptions and if user does not choose any policy , it will pickup default policy and i showed you how to setup default policy.12.png

Tag Template with T-Shirt sizes

So while cloud providers can control sizing of new virtual machines, how about Templates ?

vCloud Director helps providers to achieve this by associating  the VMs of a vApp template with specific VM sizing policies, Providers/tenant  can tag individual VMs of a vApp template with the policies you want to assign.

To Tag template to  a particular sizing policy, you need to login to org and then go to Libraries, and select vApp Templates from the left panel.


Click on particular template/highlight the template and select Tag with Compute Policies.


“TAG WITH COMPUTE POLICIES” gives two options to tag with:

  • VM Placement Policies – which allows VM to deploy in to particular cluster.
  • VM Sizing Policy – As explained in this Post, so when user will try to deploy a VM from template, it will get deployed according to “VM Sizing Policy”


This completes the process , gain control of your cloud offerings.



vCloud Director 10 : VM Placement Policies

vCloud Director 10 has introduced a new concept called VM placement policies which helps Cloud Provider to control the virtual machine (VM) placement on a specific cluster or host.VM placement policies give cloud providers various options to allocate resources to various use cases like:

  1. Deploy VM’s to specific cluster based on performance requirement
  2. Deploy VM’s to Specific cluster based on resource requirements
  3. Deploy VM’s based on Licensing requirement as a part of Oracle/SQL licenses optimisation
  4. Allocate specific hosts to specific Tenants
  5. Deploy container/special use case specific VMs to a specific host/cluster
  6. Restrict elastic VDC pools to deploy VMs to a specific cluster

vCD Provider administrator create and manage VM placement policies and placement policies are created and managed for each provider VDC, because a VM placement policy is scoped at the provider VDC level.

Create a VM Placement Policy

Before we create VM Placement policies, provider need to perform few steps on vCenter , so lets go and login to vCenter which is providing resource to vCloud Director and go to Cluster -> Configure -> VM/Host Groups


In this case i want to limit deployment of Oracle and MS SQL VM’s to specific hosts due to licensing, so let’s create Hosts groups and VM Groups:

Host Groups: 

To create Host Groups , Click on Add inside VM/Host Groups:

  1. Enter  Host Group Name
  2. Select Type as “Host Group”
  3. Click on Add to add Host/Hosts of the cluster.


VM Groups

To create VM Groups , Click on Add inside VM/Host Groups

  1. Enter  VM Group Name
  2. Select Type as “VM Group”
  3. Click on Add to add VM/VMs of the cluster. (select any dummy VM as of now)


once both the groups has been created go to VM/Host rules in the cluster and create a rule.


VM/Host Rules

To create VM/Host Rules, Click on Add inside VM/Host Rules

  1. Enter  Rule Name
  2. Ensure “Enable rule”
  3. Select rule type as “Virtual Machine to Hosts”
  4. VM Group: Select VM Group that we have created above
  5. Here you have four choices: (In my case i have choose Must rule)
    • Must run on host in group
    • Should run on host in group
    • Must not run on host in group
    • Should not run on host in group
  6. Host Group: Select Host Group that we have created above


From vCenter prospective we are done, we have multiple choice to create VM to Hosts affinity/anti-affinity rules , once we have created rules , vCloud director picks up only “VM Groups” which provider will expose to tenants.

Create VM Placement Policies in vCloud Director

  1. Go to Provider VDCs.
  2. Click on a provider VDC from the list , in my case it was “nsxtpvdc”
  3. Click on “VM Placement Policies”
  4. Click the VM Placement Policies tab and click New.


New Policy Creation Wizard

  1. First Page , click on Next
    1. 7
  2. Enter a name for the VM placement policy and description and click Next
    1. 8.png
  3. Select the VM groups or logical VM groups to which you want the VM to be linked and click Next.
    1. 9.png

  4. Review the VM placement policy settings and click Finish.
    1. 10.png

Publish VM Placement Policies to Org VDC

When provider creates a VM placement policy, it is not visible to tenants. Provider need to publish a VM placement policy to an org VDC to make it available to tenants and publishing a VM placement policy to an org VDC makes the policy visible to tenants. The tenant can select the policy when they:

  • Create a new standalone VM
  • Create a VM from a template,
  • edit a VM
  • add a VM to a vApp
  • Create a vApp from a vApp template. 

To publish this newly created policy to tenants , go to:

  1. Organization VDCs and Select an organization VDC
    1. 11.png
  2. Click the VM Placement Policies tab and Click Add.
    1. 12.png
  3. Select the VM placement policies that you want to add to the organization VDC and click OK.
    1. 13.png
  4. Provider can make certain policies as “Default” when customer does not choose any policy , system will automatically use “Default”.
    1. 14.png

Policy Usage by Tenant

Once policies has been created and exposed to tenant organisation, tenant can use those policies while provisioning VMs. like here i have created two policies “Oracle” and “SQL” and tenant can choose based on workload requirement.


NOTE –  Placement Policies are optional and a provider can continue to use the default policy that is created during installation and only one policy can be assigned to a VM.

This completes the creation of placement policies and their exposure to tenants. please feel free to share/comment.




Connect AWS Transit Gateway to VMware Cloud on AWS

This post is to deploy AWS transit Gateway and connect with VMware Cloud on AWS.

AWS Transit Gateway 

AWS Transit Gateway is a service that helps customers to connect their AWS VPC and their on-premises networks to a single gateway. As customers grow the number of workloads running on Native AWS or VMware Cloud on AWS , Customer need to be able to scale your networks across multiple accounts and Amazon VPCs/VMC to keep up with the growth.

With AWS TGW, you only have to create and manage a single connection from the central gateway in to each Amazon VPC , VMware Cloud on AWS , on-premises data center or even remote office across your network. Transit Gateway acts as a hub that controls how traffic is routed among all the connected networks which act like spokes

Now to setup Transit Gateway let’s go to VPC Dashboard inside your region where you want to deploy Transit Gateway and Click on create Transit Gateway:


Enter Required details like:

  • Name & Description
  • Amazon side ASN ( in between 64512 to 65535)
  • leave other as default or select/unselect based on your requirement.


This is will create a TGW, once TGW is created, wait for few minutes , it will show “available” in AWS console.


Connect TGW to VMware Cloud on AWS

Pervious step we created TGW and to attach to VMware Cloud on AWS or any other VPC , you need to go to “Transit Gateway  Attachment” and Click on “Create Transit Gateway Attachment”


On the new Transit Gateway Attachment page , input parameters as below:

  1. Transit Gateway ID – Choose TGW which you have created in previous step
  2. Attachment Type – VPN
  3. IP Address – get Public IP address from your VMC SDDC
  4. ASN – get ASN from your VM SDDC
  5. you can leave other things “Default” or enter based on specific requirement


Once created attachment , it will look like this:8.png

Once attachment is created , you can see it under “Site-to-Site VPN Connections” , from there follow below steps to download VPN config file:

  1. Go to Site-to-Site VPN Connections
  2. Select VPN Attachment which we created in previous step
  3. Click on “Download Configuration”
  4. Select “Generic”
  5. Click Download


Open downloaded config file and go to VMware Cloud on AWS SDDC and create a route based tunnel by input information from config file which we have downloaded in previous step.

  1. IKE Version – match in SDDC as per config file
  2. Copy the “Pre-shared Key” and paste in to SDDC “Preshared Key”
  3. Enter “Virtual Private Gateway” IP as “Remote Public IP” in side SDDC VPN config.
  4. Enter “Customer Gateway” as “BGP Local IP/Prefix Length” inside SDDC VPN config.
  5. Enter “Neighbor IP address” as “BGP Remote IP” inside SDDC VPN config.
  6. Enter “Virtual Private Gateway ASN” inside “BGP Remote ASN” inside SDDC VPN.


If every thing entered correctly , you will see , Tunnel and BGP is up and if tunnel is not up ensure Compute gateway firewall is configured appropriate as default Firewall rule for VPN in VMware cloud on AWS SDDC is “Drop”.


So tunnel and BGP is up. you can check connectivity between a VPC attached to TGW and SDDC, this should be up if you have populated proper routes in AWS route table.



Using vCD-CLI for vCloud Director

VMware vCloud Director vCD-CLI  is a command line interface for vCloud Director using short, easy-to-remember commands to administer vCD. it also allows tenants to perform certain operations for convenience and automation.

vCD-CLI is Python based and fully open source and licensed under the Apache 2.0 license. Installation process is very easy and can be installed on various platforms .  pls check, which has detailed installation instructions for Mac OS X, Linux, and Windows.

if you are using VMware’s container service extension , you need to add extension to vCD-CLI.


Here are steps which is followed for the installation on Photon OS v2 , Photon OS minimal installs lack standard tools like pip3 and even ping, so you need to install a number of packages using tdnf.

  • #tdnf install -y build-essential python3-setuptools python3-tools python3-pip python3-devel
  • 1.png
  • #pip3 install –user vcd-cli
  • 2.png
  • Set PATH  using  #PATH=$PATH:~/.local/bin
  • 3.png
  • Run #vcd (if everything goes well , you should see as below)
  • 4.png

Command Use

  • Login to vcd #vcd login cloud.corp.local system administrator –password <********> -w -i  -> this will login to vCD system.
    • 51.png
  • Let’s create a PVDC using vcd-cli , to create Provider VDC run this:
    • 7.png
    • 8.png
    • VC NAME and -t NSX-T name should be as per vCD Console
    • -r – Resource Pool – Name should be as per VC cluster name
    • -e to enable the PVDC.
    • -s – storage profile name, i am choosing all.
    • PVDC get created successfully.
    • 9.png
  • Let’s now create an organisation
    • to create an organisation run this: #vcd org create T1 Tenant1 -e
    • 10.png

so if you see this is very easy way to create object in vCD using command lines , an script can be written to automate some of the routine tasks and jobs. Refer here for more command syntax.



Upgrade NSX-T 2.3 to NSX-T 2.4

This blog is about how I upgraded my homelab from NSX-T from version 2.3 to version 2.4.First downloaded the NSX-T 2.4.x upgrade bundle (the MUB-file) from the My VMware download portal.

Checking prerequisites

before starting the upgrade , check the compatibly and ensure that the vCenter and ESXi versions are supported.

Upgrade NSX Manager Resources

In my lab the NSX-Manager VM was running with 2vCPU and 8 GB memory, with this new version minimum requirements went up to 4 vCPUS and to 16 GB of RAM.So in my case I had to shut down the NSX-T Manager and upgrade the specs to 6 vCPUs and 16 GB of memory as i was seeing 4 vCPUs were 100% getting utilised.

The Upgrade Process

Upload the MUB file to the NSX Manager1.png

After uploading the MUB file to the NSX Manager which takes some time, the file is validated and extracted which again takes little extra time. so in my Lab downloading, uploading, validating and extracting the upgrade bundle took around 40 – 50 minutes. so have patience.

Begin upgrade and accept EULA.23

After you click on “Begin Upgrade” , accept EUPA, it will throw a message , upgrade “upgrade corrdinator” , click on “Yes” and wait for some time , post “upgrade coordinator” update session will logout , login again , you will see a new upgrade interface: NOTE – Do not initiate multiple simultaneous upgrade processes for the upgrade coordinator.


Host Upgrade

Select the hosts and click on Start to start the hosts upgrade process.


While upgrade is going on continue to check you vCenter , at times ESXi host goes not go in maintenance mode due to some rule/restriction created by you , so help your ESXi server to in to maintenance mode.



Continue to monitor the progress once done successfully , move to “Next”.

Edge Upgrade

Clicking on “NEXT” will take you to Edge section , click on “Start” and continue to monitor the progress..


very simple and straight forward process , once upgrade completed , click on “Next”


Controller Upgrade

In NSX2.4 onwards , controllers has been merged in to NSX Manager ,so no need to upgrade controller , move ahead and upgrade NSX Manager.


NSX Manager Upgrade

upgrade of NSX Manager gives to two options:

  • Allow Transport Node connection after a single node cluster is formed.
  • Allow Transport Node connections only after three node cluster is formed.

choose option which is configured in your environment.


Accept the upgrade notification. You can safely ignore any upgrade related errors such as, HTTP service disruption that appears at this time. These errors appear because the Management plane is rebooting during the upgrading. Wait until the reboot finishes and the services are reestablished.

You can get in to CLI, log in to the NSX Manager to verify that the services have started run: #get service When the services start, the Service state appears as running. Some of the services include, SSH, install-upgrade, and manager.

Finally after around one hour in my Lab , Manager is also get updated successfully.


so this completes the upgrade process, check the health of every NSX-T component and if everything is green , you can now go ahead and shutdown and delete the NSX controller.Hope this help you in your NSX-T upgrade planning.





Setup RabbitMQ Server on Ubuntu for vCloud Director

I am working on a Lab which require messaging queue server , so i setup this and thought of sharing the steps, so here it is..

AMQP is an open standard for message queuing that supports flexible messaging for enterprise systems. vCloud Director uses the RabbitMQ AMQP broker to provide the message bus used by extension services, object extensions, and notifications. we will be setting up this on Ubuntu System , so download Ubuntu and install it on a VM and then follow below steps

Update Ubuntu System

Before starting, you will need to update Ubuntu repository with the latest one.You can do so by running the following commands:

  • #sudo apt-get update -y
  • #sudo apt-get upgrade -y

Installing Erlang on Ubuntu

Before installing Rabbitmq, we will need to install erlang as a prerequisite of rabbitmq. You can install by running the following commands:

Once we are done with Erlang installation, we can continue with installation of RabbitMQ.

Installing RabbitMQ on Ubuntu

First we will need to add Rabbitmq repository to apt and to do that run the following command:

Once the repository has been added, Add the RabbitMQ public key to our trusted key list to avoid any warnings about unsigned packages:

Next step is to update the apt repository with the following command:

  • #sudo apt-get update

Once the repository is updated, go ahead and  install rabbitmq server by running the following command:

  • #sudo apt-get install rabbitmq-server

Once installation is complete, start the rabbitmq server and enable it to start on boot by running the following command:

  • #sudo systemctl start rabbitmq-server
  • #sudo systemctl enable rabbitmq-server

You can check the status of rabbitmq server with the following command:

  • #sudo systemctl status rabbitmq-server 

To enable RabbitMQ Management Console, run the following:

  • #sudo rabbitmq-plugins enable rabbitmq_management

Login to URL using IP address with port number 1567, here it is you have successfully installed RabbitMQ.


Change default admin user (For security hardening)

By default the admin user for RMQ installation is guest/guest. we can change the default admin account by using below commands

  • #rabbitmqctl add_user vmware vmware  (first vmware is admin username and second vmware is password , you change it based on your requirement)
  • #rabbitmqctl set_user_tags vmware administrator (taging user with Admin priveledge)
  • #rabbitmqctl set_permissions -p / vmware “.*” “.*” “.*”
  • Now you can login with new admin user.
  • 4.png

Go ahead and configure it in your vCD instance.


This completes the installation and configuration process.



vCloud Director VM Maximum vCPU&RAM Size limits

As you know vCloud Director 9.7 comes with a default compute policy for VDC , that does provide options for custom vm sizing and this can go out of control from provides point of view as tenant can try to deploy any size of VM which might impact many things and to control this behaviour we need to limit the VM’s maximum number of vCPU and vRAM of a customer VDC can have and with vCloud Director 9.7, this is now easily can be achieved using few API calls , here is the step by step procedure to set the maximum limits:

NOTE – This gets applied on the policy like default policy which has cpuCount and memory fields as null values.

Step-1 – Create a MAX compute Policy

Let’s suppose we want to setup MAX vCPU = 32 and MAX RAM = 32 GB , so to setup this max , let’s first create a compute policy.

Procedure: Make an API call with below content to create MAX VDC compute policy:

  • POST:  https://<vcd-hostname>/cloudapi/1.0.0/vdcComputePolicies
  • Payload:  ( i kept payload short , you can create based on sample section)
    • {
      “description”:”Max sized vm policy”,
  • Header
    • 1.png
  • Post to create compute Policy
    • 2.png

Step-2: Create a Default Policy for VDC

Publish MAX policy to VDC.


  • Get VDC using below API Call
  • Take the entire output of above GET call and put in to body of new call with PUT as below screenshot and inside body add below line after DefaultComputePolicy element

Now if you go back and try to provision a virtual machine with more than 32GB memory , it will through the error as below:


Simple two API calls , will complete the much awaited feature now.


vCloud Director T-Shirt Sizing

Many of my customer with whom i directly interact has been asking this feature from quite some time , few of them says that T-shirt sized based offering matches of what hyper scalars offer, so with the release of vCloud Director 9.7, we can now control the resource allocation and the VM placement much better by using compute policies. As you know traditionally vCloud Director has two type of scope one is Provider VDC and another one Organisation VDC, similarly based on the scope and the function, there are two types of compute policies – provider virtual data center (VDC) compute policies and VDC compute policies.

In this post i will discusses VDC compute Policies and how you can leverage VDC compute policies to offer T-Shirt size option to your Best in class VMware vCloud director based cloud.

Provider VDC Compute Policies

Provider VDC compute policies applies to provider VDC level. A provider VDC compute policy defines VM-host affinity rules that decides the placement of tenant workloads. as you know Provider VDC level configuration is not visible to Tenant users and same applies to PVDC policies.

VDC Compute Policies

VDC compute policies control the compute characteristics of a VM at the organization VDC level and using VDC compute policies. A VDC compute policy groups attributes that define the compute resource allocation for VMs within an organization VDC. The compute resource allocation includes CPU and memory allocation, reservations, limits, and shares. here is the sample configuration:

  • {
    “description”:”2vCPU and 2 GB RAM”,
    “name”:”X2 Policy”,
    “config1″:”value1”  – Key Value Pair

For More detailed description of there parameters , please refer here we will going to create few policies which will reflect your cloud’s T-Shirt sizing options for your tenants/customers.

Step-1: Create VDC Compute Policy

Let’s first Create a VDC compute policy, which should be matching to your T-Shirt Sizes that you want to offer , for example here i am creating four T-shirt sizes as below:

  • X1 – 1 vCPU and 1024 MB Memory
  • X2 – 2 vCPU and 2048 MB Memory
  • X3 – 4 vCPU and 4096 MB Memory
  • X4 – 8 vCPU and 8192 MB Memory


Make an API call with below content to create VDC compute policy:

  • POST:  https://<vcd-hostname>/cloudapi/1.0.0/vdcComputePolicies
  • Payload:  ( i kept payload short , you can create based on sample section)
    • {
      “description”:”8vCPU & 8GB RAM”,
  • Header:
    • 5.png
  • Here is my one of four API call. similarly you make other 3 calls for other three T-Shirt sizes.
    • 1.png
  • After each successful API call , you will get a return like above , here note down the “id” of each T-Shirt size policy , which we will use in subsequent steps. you can also see the compatibility of policy for VDC type.
    • X8 – “id”: “urn:vcloud:vdcComputePolicy:b209edac-10fc-455e-8cbc-2d720a67e812”
    • X4 – “id”: “urn:vcloud:vdcComputePolicy:69548b08-c9ff-411a-a7d1-f81996b9a4bf”
    • X2 – “id”: “urn:vcloud:vdcComputePolicy:c71f0a47-d3c5-49fc-9e7e-df6930660817”
    • X1 – “id”: “urn:vcloud:vdcComputePolicy:1c87f0c1-ffa4-41d8-ac5b-9ec3fab211bb”

Step-2: Get VDC Id to Assign  VDC Compute Policies

Make an API call to your vCloud Director with below content to get the VDC ID:


  • Get:  https://<vcd-hostname>/api/query?type=adminOrgVdc
  • Use Header as in below screenshot:
    • 6.png
  • and write down the VDC ID ( as highlighted in above screenshot in return body) , this we will use in other calls. you can also get VDC id from vCD GUI.

Step-3: Get current Compute Policies Applied to the VDC

Using VDC identifier from step2 , Get the current compute policies applied on this VDC using below API Call:


  • Get: https://<vcd-hostname>/api/admin/vdc/443e0c43-7a6d-43f0-9d16-9d160e651fa8/computePolicies
    • 443e0c43-7a6d-43f0-9d16-9d160e651fa8 – got from step2
  • use Header as per below image
    • 8.png
  • Since this is an Get call , so no body.
    • 7.png
  • Copy the output of this Get and paste in to a new postman window to make a new API Call as this is going to be body for the our next API call.

Step-4: Publish the T-Shirt Size Compute Policies to VDC

In this step we will publish the policies to VDC , let’s create a new API call with below content:


  • PUT: https://<vcd-hostname>/api/admin/vdc/443e0c43-7a6d-43f0-9d16-9d160e651fa8/computePolicies
  • Header as below image:  ensure correct “Content-Type” – application/vnd.vmware.vcloud.vdcComputePolicyReferences+xml
    • 9.png
  • Payload:
    • paste the output of step3 in the body
    • copy full line starting with <VdcComputePolicyReference ******** /> and paste number of times as your policies. in my case i have four policies , i pasted four times.
    • in each line (underline RED) replace policy identifier with identifier we captured in step1 (compute policy identifier).
    • 10.png
  • Here is API call which will associate VDC compute policies to your Tenant’s VDC.


Now go back and login to tenant portal and click on “New VM” and see under compute policy , now you can see all your compute policy which is nothing but your T-shirt size virtual machine offerings..


Once tenant chooses a policy , he can’t choose CPU and Memory parameters..


Step-5: Create a Default Policy for VDC

With Every VDC , there is default policy which is auto generated and  has empty parameters. Now since we have published our four sizing policies to this VDC, we will make one of them default policy of the VDC. This means that if user does not provide any policy during VM creation then the default policy of the vDC would be applied on the VM.


  • Get VDC using below API Call
  • 15.png
  • Take the entire output of above GET call and put in to body of new call with PUT as below screenshot and inside body within <DefaultComputePolicy section , change the id of the Policy.
  • 16

Step-6: Delete System Default Policy

There is “System Default” policy which when selected , give options like “Pre-defined Sizing Options” and “Custom Sizing Options” , and will allow your tenants to define sizes of their choice , to restrict this , we need to un-publish this policy from VDC.

  • ab .png


To disable this policy , follow the procedure in Step-5

  • Query VDC and copy the return Body
  • Make a PUT and inside body paste body copied in above step and remove the “system Default” policy , only keep policy , which you want to offer for this particular VDC.
  • policy_remove.png
  • After above call if you see , there is no “System Default” policy.
  • 17.png

NOTE – Ensure that non of the VM and catalogs are associated with this “System Default” policy , ideally after creation of VDC , you must create and assign policy before these policies are consumed by VM/catalogs.

Extra-Step: Update the Policy

if you want to update the policy make an “PUT” api call to policy with updated body content , see below my policy update API call for reference.


Compute policies cannot be edited except for name and description ,I hope this helps providers now offer various T-Shirt size options to their customers.





Deploy VMware PKS – Part3

In  continuation to  my PKS installation, we are now going to install and configure the PKS Control Plane which provides a frontend API that will be used by Cloud Operators and Platform Operators to very easily interact with PKS for provisioning and managing (create, delete, list, scale up/down) Kubernetes Clusters.

Once a Kubernetes cluster has been successfully provisioned through PKS by Cloud Operations , the operators will need to  provide  the external hostname of the K8S Cluster and the Kubectl configuration file to their developers and then developers can  start consuming this newly provisioned K8s clusters and deploying applications without knowing simplicity/complexity of PKS/NSX-T.

Previous Posts of this series for your reference is here:

Download PKS

First of all download PKS from Pivotal Network , file will have extension .pivotal.


Installation Procedure

To import the PKS Tile, go to the home page of Ops Manager and click “Import a Product” and select the PKS package to begin the import process in to ops manager , it takes some time since this is a 4+GB appliance.


Once the PKS Tile has been successfully imported, go ahead and click on the “plus” sign to add the PKS Tile which will make it available for us to start configuring. After that, Click the orange Pivotal Container Service tile to start the configuration process.


Assign AZ and Networks

  • Here we will place the PKS API vm in the Management AZ and on the PKS Management Network that we have created on dvs in previous posts.
  • Choose Network which PKS API VM will use to connect to Network , in our case it is management network.
  • First time installation of PKS does not apply “Service Network” but we need to choose a network , for this installation i have created a NSX-T LS called “k8s” for Service network and i can use this in future, you can also create or specify “pks-mgmt” as this does not apply on new installation.
  • 3

Configure PKS API

  • Generate a wild card certificate for PKS API by selecting Generate RSA Certificate and create a DNS record.
  • Worker VM Max in Flight:  This makes sure how many instances of a component (non-canary worker) can start simultaneously when a cluster is created or resized. The variable defaults to 1 , which means that only one component starts at a time.
  • 4

Create Plans

Basically a plan defines a set of resource types used for deploying clusters. You can configure up to three plans in GUI. You must configure Plan 1.

  • Create multiple plans based on your needs like you can have master either 1 or 3.
  • you can choose to deploy number of worker VMs for each cluster and as per documentation worker nodes upto 200 has been tested but this number can go beyond 200 but sizing needs to be planned based on the other factors (like application and its requirement etc)
  • Availability Zone – Select one or more AZs for the Kubernetes clusters deployed by PKS for Master and same setting you need to configure for worker nodes and if you choose multiple AZ , then equal number of worker node will get deployed across AZs
  • Errand VM Type – select the size of the VM that contains the errand. The smallest instance may be sufficient, as the only errand running on this VM is the one that applies the Default Cluster App YAML configuration.
  • To allow users to create pods with privileged containers, select the Enable Privileged Containers – Use with caution because privileged containers is a container running as privileged essentially disables the security mechanisms provided by Docker and allows code to run on the underlying system.
  • Disable DenyEscalatingExec – This will disable Admission Control.
    • 56

Create Plan 2 and Plan3 or just choose Inactive and create them later but remember PKS does not support changing the number of master/etcd nodes for plans with existing deployed clusters.

Configure Kubernetes Cloud Provider (IAAS Provider)

Here you will configure your IAAS where all these VMs will get deployed and in my case this is vSphere based cloud but now PKS supports forAWS, GCP and Azure.

  • Enter vCenter Details like Name , Credentials , data store names etc..


Configure PKS Logging

  • Logging is optional and can be configured with vRealize Log Insight , for my Lab i am leaving it default.
  • To enable clusters to drain app logs to sinks using SYSLOG://, select the Enable Sink Resources checkbox.
  • 9

Configure Networking for Kubernetes Clusters

NSX Manager Super User Principal Identity Certificate – As per NSX-T documentation , a principal can be an NSX-T component or a third-party application such open stack or PKS. With a principal identity, a principal can use the identity name to create an object and ensure that only an entity with the same identity name can modify or delete the object (except Enterprise Admin). A principal identity can only be created or deleted using the NSX-T API. However, you can view principal identities through the NSX Manager UI.

We will have to create a user id and that user, id PKS API uses the NSX Manager superuser principal identity to communicate with NSX-T to create, delete, and modify networking resources for Kubernetes cluster nodes. Follow the steps here to create it.

  • Choose NSX-T as  Networking Interface
  • Specify NSX Manager hostname and generate the certificate as per above step.
  • 10
  • Pods IP Block ID – Here enter the UUID of the IP block to be used for Kubernetes pods. PKS allocates IP addresses for the pods when they are created in Kubernetes. every time a namespace is created in Kubernetes, a subnet from this IP block is allocated.
  • Nodes IP Block ID – Here enter the UUID of the IP block to be used for Kubernetes nodes. PKS allocates IP addresses for the nodes when they are created in Kubernetes. The node networks are created on a separate IP address space from the pod networks.
  • 11.png
  • T0 Router ID – Here enter the  T0 router UUID.
  • 12.png
  • Floating IP Pool ID – Here enter the ID that you created for load balancer VIPs. PKS uses these floating IP pool to allocate IP addresses to the load balancers created for each of the clusters. The load balancer routes the API requests to the master nodes and the data plane.
  • 13.png
  • Node DNS – Specify Node DNS Server Name , ensure Nodes are reachable to DNS servers.
  • vSphere Cluster Names – Here enter a comma-separated list of the vSphere clusters where you will deploy Kubernetes clusters. The NSX-T pre-check errand uses this field to verify that the hosts from the specified clusters are available in NSX-T
  • HTTP/HTTPS Proxy – Optional
  • 14.png

Configure User Account and Authentication (UAA)

Before users can log in and use the PKS CLI, you must configure PKS API access with UAA. You use the UAA Command Line Interface (UAAC) to target the UAA server and request an access token for the UAA admin user. If your request is successful, the UAA server returns the access token. The UAA admin access token authorizes you to make requests to the PKS API using the PKS CLI and grant cluster access to new or existing users.

  • Leaving setting default with some timer changes
  • 15


You can monitor kubernetes cluster and pods using VMware Wavefront , which i will be covering in a separate post.

  • For now leave it default.

Usage Data

VMware’s Customer Experience Improvement Program (CEIP) and the Pivotal Telemetry Program (Telemetry)  program.

  • choose based in your preference.


Errands are scripts that run at designated points during an installation.

  • Since we are running PKS with NSX-T , we must need to verify our NSX-T configuration.
  • 16.png

Resource Config for PKS VM

Edit resources used by the Pivotal Container Service job and if there are timeouts while accessing PKS API VM , use high resource VM Type , for this LAB i am going with Default.

  • Leave it default.
  • 17.png

Missing Stemcell

A stemcell is a versioned Operating System image wrapped with IaaS specific packaging.A typical stemcell contains a bare minimum OS skeleton with a few common utilities pre-installed, a BOSH Agent, and a few configuration files to securely configure the OS by default. For example: with vSphere, the official stemcell for Ubuntu Trusty is an approximately 500MB VMDK file.

Click on missing stemcell link which will take you to StemCell Library. Here you can see PKS requires stemcell 170.15 , since i have already downloaded thats the reason it is showing 170.25 in the deployed section but in new installation cases it will show none  deployed. Click IMPORT STEMCELL and choose a stemcell which can be downloaded from Here to import.


Apply Changes

Return to Ops Manager installation Dashboard and click on “Review Pending Changes” and finally “Apply Changes” , this will go ahead and deploy PKS API VM at your IAAS location which you have chosen while configuring PKS tile.


and if the configuration of the tile is correct , around after 30 minute , you will see a successful message that deployment has been completed , which gives very nice feeling that your hard work  and dedication resulting success (for me it failed couple of time because of storage/network and resource issues).

To identify which VM has been deployed , you can check custom attributes or go back to  the Installation Dashboard, click the PKS tile then go to the Status tab. Here we can see the IP address of our PKS API , also notice CID which is VM name in vCenter inventory. also you can see the health of the PKS VM.


This completes the PKS VM deployment procedure. in the next post we will deploy kubernetes Cluster.
















vCloud Director 9.7 Portal Custom Branding

Much awaited feature for cloud provider to match thier corporate  standards and to create a fully custom cloud experience, now with release on vCloud Director 9.7 you can set the logo and the theme for your vCloud Director Service Provider Admin Portal and also now you can customize the vCloud Director Tenant Portal of each tenants . In addition, you can modify and add custom links to the two upper right menus in the vCloud Director provider and tenant portals.

Provider Portal Branding

vCloud Director 9.7 UI can be modified for the following elements:

  • Portal name
  • Portal color
  • Portal theme (vCloud Director contains two themes – default and dark.)
  • Logo & Browser icon

Customize Portal Name ,Portal Color and Portal Theme

To configure the Cloud Provider Portal Branding , make a PUT request to vCloud Director end point as below:

  • PUThttps://<vCD Url>/cloudapi/branding
  • BODY – {
    “portalName”: “string”,
    “portalColor”: “string”,
    “selectedTheme”: {
    “themeType”: “string”,
    “name”: “string”
    “customLinks”: [
    “name”: “string”,
    “menuItemType”: “link”,
    “url”: “string”
  • Headers
    • 2.png

Here is my API call using Postman client:


Customize Logo

To change the Logo, here is the procedure for API

  • Headers
    • 4.png
  • PUT
  • Body – This is bit tricky since we need to upload an image as a body.
    • In Postman client inside “Body” click on “Binary” which will allow you to choose file as body. select your logo.
    • 5.png

Customize Icon

To customize the icon, follow this API call and procedure.

  • Headers
    • 9.png
  • PUT
  • Body – same as above section , choose a image
    • 10.png

so after running above API calls , here is what my vCloud Director provider portal looks like.


Tenant Portal Branding

As we did above similarly we can now fully customize Tenant Portal

Customize Portal Name ,Portal Color and Portal Theme

To configure the Cloud Provider Portal Branding , make a PUT request to vCloud Director end point in to tenant organisation as below: ( T1 is my org Name)

  • PUThttps://<vCD Url>/cloudapi/branding/tenant/T1
  • BODY – {
    “portalName”: “string”,
    “portalColor”: “string”,
    “selectedTheme”: {
    “themeType”: “string”,
    “name”: “string”
    “customLinks”: [
    “name”: “string”,
    “menuItemType”: “link”,
    “url”: “string”
  • Headers
    • 11.png

Here is my API call using Postman client:


Customize Logo

To change the Logo, here is the procedure for API

  • Headers
    • 4.png
  • PUT
  • Body – As said above ,this is bit tricky since we need to upload an image as a body.
    • In Postman client inside “Body” click on “Binary” which will allow you to choose file as body, select your logo.
    • 14.png

Once i have done with above API calls, this is how my Tenant portal look like for “T1” organisation.


For a particular tenant, you can selectively override any combination of the portal name, background color, logo, icon, theme, and custom links. Any value that you do not set uses the corresponding system default value.

This completes feature walk through of Provider and Tenant custom branding options available now with vCD9.7.

Upgrade Postgres SQL 9 to 10 for vCloud Director

Since vCloud Director 9.7 has dropped support for Postgres SQL9.5 , so i had to upgrade my postgres to 10 , then i have updated my vCloud Director to versions 9.7 , i followed below steps to upgrade the DB , basically at High level steps are as below:

  • You need to backup the existing database and data directory.
  • Uninstall old version of Postgres SQL.
  • Install Postgres10
  • Restore Backup


  • Create database backup using:
    • su – postgres
    • pg_dumpall > /tmp/pg9dbbackup
    • exit
    • 1
  • Check and Stop the service using
    • #chkconfig
    • #service postgresql-9.5 stop
    • 2
  • Move current data file as .old to /tmp directory using below command.
    • #mv /var/lib/pgsql/9.5/data/ /tmp/data.old
  • Uninstall 9.5 version of Postgres SQL using :
    • yum remove postgresql*
  • Install  PostgreSQL v10:
  • Initialise the database
    • service postgresql-10 initdb
    • as suggested by my friend miguel if above step is not working then use this (/usr/pgsql-10/bin/postgresql-10-setup initdb)
  • Copy the pg_hba.conf and postgresql.conf from old backed up directory to new directory , this will save some time or you can go ahead and edit existing files with required settings.
    • cp /data.old/pg_hba.conf /var/lib/pgsql/10/data/
    • cp /data.old/postgresql.conf /var/lib/pgsql/10/data/
    • service postgresql-10 start
  • Restore backup using below commands:
    • su – postgres
    • psql -d postgres -f /tmp/pg9dbbackup

you can run the reconfigure-database command and that’s it. (change your environment variable accordingly)


This will complete the database upgrade and database migration procedure.




Deploy VMware PKS – Part2

In this part I will begin PKS installation by deploying Pivotal Ops Manager which basically provides a management interface (UI/API) for Platform Operators to manage the complete lifecycle of both BOSH and PKS starting from install then going to patch and upgrade.

To refer other posts of this series are here:

Getting Started with VMware PKS & NSX-T

Deploy VMware PKS – Part1

In addition, you can also deploy new application services using Ops Manager Tiles like adding an Enterprise-class Container Registry like VMware Harbor which can then be configured to work with PKS.

Installing OpsManager


  • Once Downloaded , Log into vCenter using the vSphere Web Client or HTML5 Client to deploy the Ops Manager OVA.
  • Choose your Management cluster , appropriate network and other OVA deployment options , i am not going to cover OVA deployment procedure here. Only at customize template , enter below details:
    • Admin Password: A default password for the user “ubuntu”.
      • If you do not enter a password, Ops Manager will not boot up.
    • Custom hostname: The hostname for the Ops Manager VM, in My example opsmgr.corp.local.
    • DNS: One or more DNS servers for the Ops Manager VM.
    • Default Gateway: The default gateway for Ops Manager.
    • IP Address: The IP address of the Ops Manager network interface.
    • NTP Server: The IP address of one or more NTP servers for Ops Manager.
    • Netmask: The network mask for Ops Manager.1.png
  • Create a DNS entry for the IP address that you used for Ops Manager ,which we will use in next steps. use this DNS NAME/IP address and browse on a browser , which will take you to Authentication System for initial authentication setup and for our setup, i will use “Internal Authentication” for this Lab. Click on “Internal Authentication”
    • 2
  • Next, you will be prompted to create a new admin user which we will use to manage BOSH. Once you have successfully created the user, go ahead and login with the new user account
    • 3.png
  • Once you are logged into Ops Manager, you can see that the BOSH Tile is already there but is showing as un-configured (orange colour denotes un-configured) which means BOSH has not yet been deployed yet. Go ahead and click on the tile to begin the configuration to deploy BOSH.
    • 4.png

Before starting Bosh Tile configuration , we need to prepare NSX Manager, listed below procedure for:

Generating and Registering the NSX Manager Certificate for PKS

The NSX Manager CA certificate is used to authenticate PKS with NSX Manager. You create an IP-based, self-signed certificate and register it with the NSX Manager.By default, the NSX Manager includes a self-signed API certificate with its hostname as the subject and issuer. PKS Ops Manager requires strict certificate validation and expects the subject and issuer of the self-signed certificate to be either the IP address or fully qualified domain name (FQDN) of the NSX Manager. That’s the reason, we need to regenerate the self-signed certificate using the FQDN of the NSX Manager in the subject and issuer field and then register the certificate with the NSX Manager using the NSX API.

  • Create a file for the certificate request parameters named “nsx-cert.cnf” in a linux VM where openssl tool is installed.
  • Write below content in to the file which we create in above step.
    • 3.png
  • Export the NSX_MANAGER_IP_ADDRESS and NSX_MANAGER_COMMONNAME environment variables using the IP address of your NSX Manager and the FQDN of the NSX Manager host.
    • 4.png
  • Using openssl tool generate the certificate by running below command:
    • 56
  • Verify the certificate by running command as below:
    • ~$ openssl x509 -in nsx.crt -text -noout and ensure SAN has DNS name and IP addresses.
    • 7.png
  • Import this Certificate in NSX Manager , go to System -> Trust -> Certificates and click on Import -> Import Certificate
    • 8.png
  • Ensure that Certificate looks like this in your NSX Manager.
    • 9.png
  • Next is to Register the certificate with NSX Manager using below procedure , first the ID of the certificate from gui.
    • 10.png
  • Run the below command (in to your API client ) to register the certificate using below command replace “CERTIFICATE-ID” with your certificate ID.

Now let’s configure BOSH tile , which will deploy BOSH based on our input parameters.

Configure BOSH Tile to Deploy BOSH Director

Click on the tile. It will open the tile’s setting tab with the vCenter Config parameters page.

  • vCenter Config 
    • Name: a unique meaning full name
    • vCenter Host: The hostname of the vCenter.
    • vCenter Username: Username for above VC with create and delete privileges for virtual machines (VMs) and folders.
    • vCenter Password: the password of above VC.
    • Datacenter Name: Exact name of data center object in vCenter
    • Virtual Disk Type: Select “Thin” or “Thick”
    • Ephemeral Datastore Names: The names of the data stores that store ephemeral VM disks deployed by Ops Manager , you can specify many data stores by using comma.
    • Persistent Datastore Names (comma delimited): The names of the datastores that store persistent VM disks deployed by Ops Manager.
    • To deploy BOSH as well as the PKS Control Plane VMs, Ops Manager will go ahead and upload a Stemcell VM ( A VM Template that PKS) and it will clone from that image for both PKS Management VMs as well as base K8S VMs.
    • 11
  • NSX-T Config 
    • Choose NSX Networking and Select NSX-T 
    • NSX Address: IP/DNS Name for NSX-T Manager.
    • NSX Username and NSX Password: NSX-T credential
    • NSX CA Cert: Open the NSX CA Cert that you generated in above section and copy/paste its content to this field.
    • 12.png
  •  Other Config
    • VM Folder: The vSphere datacenter folder where Ops Manager places VMs.
    • Template Folder: The vSphere folder where Ops Manager places stemcells(templates).
    • Disk path Folder: The vSphere datastore folder where Ops Manager creates attached disk images. You must not nest this folder.
    • And Click on “Save”.
    • 13
  • Director Config
    • For Director config , i have put in few details like:
      • NTP Server Details
      • Enable VM Resurrector Plugin
      • Enable Post Deploy Scripts
      • Recreate all VMs (optional)
    • 14
  • Availability Zone    
    • Availability zones are defined at a vSphere Cluster level. These AZs will then be used by BOSH director to determine where to deploy the PKS Management VMs as well as the Kubernetes VMs.Multiple Availability Zones allow you to provide high-availability across data centers. for this demonstration i have created two AZs, one for Management and one for my Compute.15.png
  • Create Network
    • Since i am using dvs for my PKS management component , we need to specify those details in to this segment and make sure you select the Management AZ which is the vSphere Cluster where BOSH and PKS Control Plane VM will be deployed.


  • Assign AZs and Networks 
    • In this section,  Define the AZ and networking placement settings for the PKS Management VM.Singleton Availability Zone – The Ops Manager Director installs in this Availability Zone.


  • Security & Syslog
    • This section i am leaving default , if required for your deployment , pls refer documentation.
  • Resource Config
    • As per in part 1 sizing , the BOSH director vm by default allocates 2 vCPUs, 8GB memory, 64GB disk and also has a persistent disk of 50GB and Each of the four Compilation VMs uses 4 vCPUs, 4GB memory, 16GB disk each. For my lab deployment i have changed it to suite to my lab resources.Bosh director needs minimum of 8 GB Memory to run , so choose options accordingly.


  • Review Pending Changes and Apply Changes 
    • With all the input completed, return to the Installation Dashboard view by clicking Installation Dashboard at the top of the window. The BOSH Director tile will now have a green bar indicating all the required parameters have been entered. Next we click REVIEW PENDING CHANGES and Apply Changes


  • Monitor Installation and Finish
    • 8.png
    • If all the inputs are right then you will see that your installation is successful


After you login to your vCenter , you will see a new powered on VM in your  inventory that starts with vm-UUID which is the BOSH VM. Ops Manager uses vSphere Custom Attributes to add additional metadata fields to identify the various VMs it can deploy, you can check what type of VM this is by simply looking at the deployment, instance_group or job property. In this case, we can see its been noted as p-bosh.


and this completes Ops Manager and BOSH deployment , next post we will install PKS tile.



Deploy VMware PKS – Part1

In Continuation of my previous blog post here , where i have explained PKS component and sizing details , in this post i will be covering PKS component deployment.

Previous Post in this Series:

Getting Started with VMware PKS & NSX-T


  • Install a New or Existing server which has DNS role installed and configured , which we will use in this deployment.
  • Install vCenter and ESXi , for this Lab i have created two vSphere Cluster:
    • Management Cluster + Edge Cluster – Three Nodes
    • Compute Cluster – Two Nodes
  • Create a Ubuntu server , where you will need to install client utilities like:
    • PKS CLI
      • The PKS CLI is used to create, manage, and delete Kubernetes clusters.
      • To deploy workloads/applications to a Kubernetes cluster created using the PKS CLI, use the Kubernetes CLI called “kubectl“.
    • UAAC
      • To manage users in Pivotal Container Service (PKS) with User Account and Authentication (UAA). Create and manage users in UAA with the UAA Command Line Interface (UAAC).

    • BOSH
      • BOSH CLI used to manage PKS management components deployments and provides information about the VMs using its Cloud Provider Interface (CPI) which is vSphere in my Lab and could be AZURE , AWS and GCP also.
    • OM
      • Bosh Operations Manager command line interface.
  • Prepare NSX-T

    For this Deployment make sure NSX-T is deployed and configured, high level steps are as below:

    • Install NSX Manager
    • Deploy NSX Controllers
    • Register Controllers with Managers as well as other controller with Master controller.
    • Deploy NSX Edge Nodes

    • Register NSX Edge Nodes with NSX Manager

    • Enable Repository Service on NSX Manager

    • Create TEP IP Pool

    • Create Overlay Transport Zone

    • Create VLAN Transport Zone

    • Create Edge Transport Nodes

    • Create Edge Cluster

    • Create T0 Logical Router and configure BGP routing with physical device

    • Configure Edge Nodes for HA

    • Prepare ESXi Servers for the PKS Compute Cluster

My PKS deployment topology is look like below:


  • PKS Deployment Topology – PKS management stack running out of NSX-T
    • PKS VMs (Ops Manager, BOSH, PKS Control Plane, Harbor) are deployed to a VDS backed portgroup
    • Connectivity between PKS VMs, K8S Cluster Management and T0 Router is through a physical router
    • NAT is only configured on T0 to provide POD networks access to associated K8S Cluster Namespace
  • Create a IP Pool
    • Create a new IP Pool which will be used to allocate Virtual IPs for the exposed K8S Services The network also provides IP addresses for Kubernetes API access. Go to Inventory->Groups->IP Pool and enter the following configuration:
      • IP Range: –
      • CIDR:
    • 5.png
  • Create POD-IP-BLOCK
    • We need to create a new POD IP Block which will by used by PKS on-demand to create smaller /24 networks and assigned those to each K8S namespace. This IP block should be sized sufficiently to ensure that you do not run out of addresses. To create POD-IP-BLOCK , go to NETWORKING->IPAM and enter the following:
    • 7
  •  Create NODEs-IP-BLOCK
    • We need to create new NODEs IP Block which will be used by PKS to assign IP address to Kubernetes master and worker nodes.Each Kubernetes cluster owns the /24 subnet , so to deploy multiple Kubernetes clusters, plan for larger than /24 subnet. (recommendation is /16)
    • 6

Prepare Client VM

  • Create and install a small Ubuntu VM with default configuration. you can use the latest server version and insure that the VM has internet connectivity either by proxy or direct.
  • Once the Ubuntu VM is ready , download PKSCLI and KUBECTL from


and copy both the PKS (pks-linux-amd64-1.3.0-build.126 or latest) and Kubectl                      (kubectl-linux-amd64-1.12.4 or latest) CLI to VM.

  • Now SSH to the Ubuntu VM and run the following commands to make binaries executable and renaming/relocating them to /usr/local/bin directory:
    • chmod +x pks-linux-amd64-1.3.0-build.126
    • chmod +x kubectl-linux-amd64-1.12.4
    • mv pks-linux-amd64-1.3.0-build.126 /usr/local/bin/pks
    • mv kubectl-linux-amd64-1.12.4 /usr/local/bin/kubectl
    • Check version using – pks -v and kubectl version
  • Next is to install Cloud Foundry UAAC , run this command
    • apt -y install ruby ruby-dev gcc build-essential g++
    • gem install cf-uaac
    • Check version using – uaac -v
  • Next is to install
  • 34

This completes this part , in the next part we will start deploying PKS management VMs and their configuration.


VMware Container Service Extension Upgrade

With the release of new Container Service Extension (CSE) version 1.2.7 due to vulnerability related to docker (CVE-2019-5736 ) for both Ubuntu and Photon OS templates , it is very important to update the CSE ASAP , here is the procedure to help you to upgrade the CSE easily.


  • Check the release notes Here for version compatibility.

Upgrade procedure for Cloud Admins:

  • Update CSE to 1.2.7 ( follow procedure below)
  • Update the templates (follow procedure below)

Upgrading CSE Server Software

  •  Stop CSE Server services gracefully.
    • #vcd cse system stop -y
    • 2.png
  • Reinstall container-service-extension using Python Package Index:
    • #pip3 install –user –upgrade container-service-extension
    • 3.png
  • Review the configuration file for any new options introduced or deprecated in the new version. cse sample  can be used to generate a new sample config file as well.
    • 3.png
    • Follow the steps listed here , to edit your environment variable for CSE to use.
  • If the previously generated templates are no longer supported by the new version, delete the templates and re-generate new ones using below command.
    • cse install -c mysample.yaml –update
    • 12
  • If running CSE as a service, start the new version of the service with
    • $systemctl start cse
    • 4.png

Upgrade procedure for Tenant Users:

  • Delete clusters that were created with older templates. Recreate clusters with new templates
  • Alternatively, tenant-users can update docker version manually on their existing clusters.

This completes the upgrade procedure , go ahead and let the customer consume Kubernetes as a Service from your platform.