vCloud Director VM Maximum vCPU&RAM Size limits

As you know vCloud Director 9.7 comes with a default compute policy for VDC , that does provide options for custom vm sizing and this can go out of control from provides point of view as tenant can try to deploy any size of VM which might impact many things and to control this behaviour we need to limit the VM’s maximum number of vCPU and vRAM of a customer VDC can have and with vCloud Director 9.7, this is now easily can be achieved using few API calls , here is the step by step procedure to set the maximum limits:

NOTE – This gets applied on the policy like default policy which has cpuCount and memory fields as null values.

Step-1 – Create a MAX compute Policy

Let’s suppose we want to setup MAX vCPU = 32 and MAX RAM = 32 GB , so to setup this max , let’s first create a compute policy.

Procedure: Make an API call with below content to create MAX VDC compute policy:

  • POST:  https://<vcd-hostname>/cloudapi/1.0.0/vdcComputePolicies
  • Payload:  ( i kept payload short , you can create based on sample section)
    • {
      “description”:”Max sized vm policy”,
      “name”:”MAX_SIZE”,
      “memory”:32768,
      “cpuCount”:32
      }
  • Header
    • 1.png
  • Post to create compute Policy
    • 2.png

Step-2: Create a Default Policy for VDC

Publish MAX policy to VDC.

Procedure

  • Get VDC using below API Call
  • Take the entire output of above GET call and put in to body of new call with PUT as below screenshot and inside body add below line after DefaultComputePolicy element

Now if you go back and try to provision a virtual machine with more than 32GB memory , it will through the error as below:

7

Simple two API calls , will complete the much awaited feature now.

 

vCloud Director T-Shirt Sizing

Many of my customer with whom i directly interact has been asking this feature from quite some time , few of them says that T-shirt sized based offering matches of what hyper scalars offer, so with the release of vCloud Director 9.7, we can now control the resource allocation and the VM placement much better by using compute policies. As you know traditionally vCloud Director has two type of scope one is Provider VDC and another one Organisation VDC, similarly based on the scope and the function, there are two types of compute policies – provider virtual data center (VDC) compute policies and VDC compute policies.

In this post i will discusses VDC compute Policies and how you can leverage VDC compute policies to offer T-Shirt size option to your Best in class VMware vCloud director based cloud.

Provider VDC Compute Policies

Provider VDC compute policies applies to provider VDC level. A provider VDC compute policy defines VM-host affinity rules that decides the placement of tenant workloads. as you know Provider VDC level configuration is not visible to Tenant users and same applies to PVDC policies.

VDC Compute Policies

VDC compute policies control the compute characteristics of a VM at the organization VDC level and using VDC compute policies. A VDC compute policy groups attributes that define the compute resource allocation for VMs within an organization VDC. The compute resource allocation includes CPU and memory allocation, reservations, limits, and shares. here is the sample configuration:

  • {
    “description”:”2vCPU and 2 GB RAM”,
    “name”:”X2 Policy”,
    “cpuSpeed”:1000,
    “memory”:2048,
    “cpuCount”:2,
    “coresPerSocket”:1
    “memoryReservationGuarantee”:0.5,
    “cpuReservationGuarantee”:0.5,
    “cpuLimit”:1000,
    “memoryLimit”:1000,
    “cpuShares”:1000,
    “memoryShares”:1000,
    “extraConfigs”:{
    “config1″:”value1”  – Key Value Pair
    },
    “pvdcComputePolicy”:null
    }

For More detailed description of there parameters , please refer here we will going to create few policies which will reflect your cloud’s T-Shirt sizing options for your tenants/customers.

Step-1: Create VDC Compute Policy

Let’s first Create a VDC compute policy, which should be matching to your T-Shirt Sizes that you want to offer , for example here i am creating four T-shirt sizes as below:

  • X1 – 1 vCPU and 1024 MB Memory
  • X2 – 2 vCPU and 2048 MB Memory
  • X3 – 4 vCPU and 4096 MB Memory
  • X4 – 8 vCPU and 8192 MB Memory

Procedure:

Make an API call with below content to create VDC compute policy:

  • POST:  https://<vcd-hostname>/cloudapi/1.0.0/vdcComputePolicies
  • Payload:  ( i kept payload short , you can create based on sample section)
    • {
      “description”:”8vCPU & 8GB RAM”,
      “name”:”X8″,
      “memory”:8192,
      “cpuCount”:8
      }
  • Header:
    • 5.png
  • Here is my one of four API call. similarly you make other 3 calls for other three T-Shirt sizes.
    • 1.png
  • After each successful API call , you will get a return like above , here note down the “id” of each T-Shirt size policy , which we will use in subsequent steps. you can also see the compatibility of policy for VDC type.
    • X8 – “id”: “urn:vcloud:vdcComputePolicy:b209edac-10fc-455e-8cbc-2d720a67e812”
    • X4 – “id”: “urn:vcloud:vdcComputePolicy:69548b08-c9ff-411a-a7d1-f81996b9a4bf”
    • X2 – “id”: “urn:vcloud:vdcComputePolicy:c71f0a47-d3c5-49fc-9e7e-df6930660817”
    • X1 – “id”: “urn:vcloud:vdcComputePolicy:1c87f0c1-ffa4-41d8-ac5b-9ec3fab211bb”

Step-2: Get VDC Id to Assign  VDC Compute Policies

Make an API call to your vCloud Director with below content to get the VDC ID:

Procedure:

  • Get:  https://<vcd-hostname>/api/query?type=adminOrgVdc
  • Use Header as in below screenshot:
    • 6.png
  • and write down the VDC ID ( as highlighted in above screenshot in return body) , this we will use in other calls. you can also get VDC id from vCD GUI.

Step-3: Get current Compute Policies Applied to the VDC

Using VDC identifier from step2 , Get the current compute policies applied on this VDC using below API Call:

Procedure:

  • Get: https://<vcd-hostname>/api/admin/vdc/443e0c43-7a6d-43f0-9d16-9d160e651fa8/computePolicies
    • 443e0c43-7a6d-43f0-9d16-9d160e651fa8 – got from step2
  • use Header as per below image
    • 8.png
  • Since this is an Get call , so no body.
    • 7.png
  • Copy the output of this Get and paste in to a new postman window to make a new API Call as this is going to be body for the our next API call.

Step-4: Publish the T-Shirt Size Compute Policies to VDC

In this step we will publish the policies to VDC , let’s create a new API call with below content:

Procedure:

  • PUT: https://<vcd-hostname>/api/admin/vdc/443e0c43-7a6d-43f0-9d16-9d160e651fa8/computePolicies
  • Header as below image:  ensure correct “Content-Type” – application/vnd.vmware.vcloud.vdcComputePolicyReferences+xml
    • 9.png
  • Payload:
    • paste the output of step3 in the body
    • copy full line starting with <VdcComputePolicyReference ******** /> and paste number of times as your policies. in my case i have four policies , i pasted four times.
    • in each line (underline RED) replace policy identifier with identifier we captured in step1 (compute policy identifier).
    • 10.png
  • Here is API call which will associate VDC compute policies to your Tenant’s VDC.

3.png

Now go back and login to tenant portal and click on “New VM” and see under compute policy , now you can see all your compute policy which is nothing but your T-shirt size virtual machine offerings..

11

Once tenant chooses a policy , he can’t choose CPU and Memory parameters..

12

Step-5: Create a Default Policy for VDC

With Every VDC , there is default policy which is auto generated and  has empty parameters. Now since we have published our four sizing policies to this VDC, we will make one of them default policy of the VDC. This means that if user does not provide any policy during VM creation then the default policy of the vDC would be applied on the VM.

Procedure

  • Get VDC using below API Call
  • 15.png
  • Take the entire output of above GET call and put in to body of new call with PUT as below screenshot and inside body within <DefaultComputePolicy section , change the id of the Policy.
  • 16

Step-6: Delete System Default Policy

There is “System Default” policy which when selected , give options like “Pre-defined Sizing Options” and “Custom Sizing Options” , and will allow your tenants to define sizes of their choice , to restrict this , we need to un-publish this policy from VDC.

  • ab .png

Procedure

To disable this policy , follow the procedure in Step-5

  • Query VDC and copy the return Body
  • Make a PUT and inside body paste body copied in above step and remove the “system Default” policy , only keep policy , which you want to offer for this particular VDC.
  • policy_remove.png
  • After above call if you see , there is no “System Default” policy.
  • 17.png

NOTE – Ensure that non of the VM and catalogs are associated with this “System Default” policy , ideally after creation of VDC , you must create and assign policy before these policies are consumed by VM/catalogs.

Extra-Step: Update the Policy

if you want to update the policy make an “PUT” api call to policy with updated body content , see below my policy update API call for reference.

policy_update.png

I hope this helps providers now offer various T-Shirt size options to their customers.

 

 

 

 

Deploy VMware PKS – Part3

In  continuation to  my PKS installation, we are now going to install and configure the PKS Control Plane which provides a frontend API that will be used by Cloud Operators and Platform Operators to very easily interact with PKS for provisioning and managing (create, delete, list, scale up/down) Kubernetes Clusters.

Once a Kubernetes cluster has been successfully provisioned through PKS by Cloud Operations , the operators will need to  provide  the external hostname of the K8S Cluster and the Kubectl configuration file to their developers and then developers can  start consuming this newly provisioned K8s clusters and deploying applications without knowing simplicity/complexity of PKS/NSX-T.

Previous Posts of this series for your reference is here:

Download PKS

First of all download PKS from Pivotal Network , file will have extension .pivotal.

1

Installation Procedure

To import the PKS Tile, go to the home page of Ops Manager and click “Import a Product” and select the PKS package to begin the import process in to ops manager , it takes some time since this is a 4+GB appliance.

2.png

Once the PKS Tile has been successfully imported, go ahead and click on the “plus” sign to add the PKS Tile which will make it available for us to start configuring. After that, Click the orange Pivotal Container Service tile to start the configuration process.

12

Assign AZ and Networks

  • Here we will place the PKS API vm in the Management AZ and on the PKS Management Network that we have created on dvs in previous posts.
  • Choose Network which PKS API VM will use to connect to Network , in our case it is management network.
  • First time installation of PKS does not apply “Service Network” but we need to choose a network , for this installation i have created a NSX-T LS called “k8s” for Service network and i can use this in future, you can also create or specify “pks-mgmt” as this does not apply on new installation.
  • 3

Configure PKS API

  • Generate a wild card certificate for PKS API by selecting Generate RSA Certificate and create a DNS record.
  • Worker VM Max in Flight:  This makes sure how many instances of a component (non-canary worker) can start simultaneously when a cluster is created or resized. The variable defaults to 1 , which means that only one component starts at a time.
  • 4

Create Plans

Basically a plan defines a set of resource types used for deploying clusters. You can configure up to three plans in GUI. You must configure Plan 1.

  • Create multiple plans based on your needs like you can have master either 1 or 3.
  • you can choose to deploy number of worker VMs for each cluster and as per documentation worker nodes upto 200 has been tested but this number can go beyond 200 but sizing needs to be planned based on the other factors (like application and its requirement etc)
  • Availability Zone – Select one or more AZs for the Kubernetes clusters deployed by PKS for Master and same setting you need to configure for worker nodes and if you choose multiple AZ , then equal number of worker node will get deployed across AZs
  • Errand VM Type – select the size of the VM that contains the errand. The smallest instance may be sufficient, as the only errand running on this VM is the one that applies the Default Cluster App YAML configuration.
  • To allow users to create pods with privileged containers, select the Enable Privileged Containers – Use with caution because privileged containers is a container running as privileged essentially disables the security mechanisms provided by Docker and allows code to run on the underlying system.
  • Disable DenyEscalatingExec – This will disable Admission Control.
    • 56

Create Plan 2 and Plan3 or just choose Inactive and create them later but remember PKS does not support changing the number of master/etcd nodes for plans with existing deployed clusters.

Configure Kubernetes Cloud Provider (IAAS Provider)

Here you will configure your IAAS where all these VMs will get deployed and in my case this is vSphere based cloud but now PKS supports forAWS, GCP and Azure.

  • Enter vCenter Details like Name , Credentials , data store names etc..

8

Configure PKS Logging

  • Logging is optional and can be configured with vRealize Log Insight , for my Lab i am leaving it default.
  • To enable clusters to drain app logs to sinks using SYSLOG://, select the Enable Sink Resources checkbox.
  • 9

Configure Networking for Kubernetes Clusters

NSX Manager Super User Principal Identity Certificate – As per NSX-T documentation , a principal can be an NSX-T component or a third-party application such open stack or PKS. With a principal identity, a principal can use the identity name to create an object and ensure that only an entity with the same identity name can modify or delete the object (except Enterprise Admin). A principal identity can only be created or deleted using the NSX-T API. However, you can view principal identities through the NSX Manager UI.

We will have to create a user id and that user, id PKS API uses the NSX Manager superuser principal identity to communicate with NSX-T to create, delete, and modify networking resources for Kubernetes cluster nodes. Follow the steps here to create it.

  • Choose NSX-T as  Networking Interface
  • Specify NSX Manager hostname and generate the certificate as per above step.
  • 10
  • Pods IP Block ID – Here enter the UUID of the IP block to be used for Kubernetes pods. PKS allocates IP addresses for the pods when they are created in Kubernetes. every time a namespace is created in Kubernetes, a subnet from this IP block is allocated.
  • Nodes IP Block ID – Here enter the UUID of the IP block to be used for Kubernetes nodes. PKS allocates IP addresses for the nodes when they are created in Kubernetes. The node networks are created on a separate IP address space from the pod networks.
  • 11.png
  • T0 Router ID – Here enter the  T0 router UUID.
  • 12.png
  • Floating IP Pool ID – Here enter the ID that you created for load balancer VIPs. PKS uses these floating IP pool to allocate IP addresses to the load balancers created for each of the clusters. The load balancer routes the API requests to the master nodes and the data plane.
  • 13.png
  • Node DNS – Specify Node DNS Server Name , ensure Nodes are reachable to DNS servers.
  • vSphere Cluster Names – Here enter a comma-separated list of the vSphere clusters where you will deploy Kubernetes clusters. The NSX-T pre-check errand uses this field to verify that the hosts from the specified clusters are available in NSX-T
  • HTTP/HTTPS Proxy – Optional
  • 14.png

Configure User Account and Authentication (UAA)

Before users can log in and use the PKS CLI, you must configure PKS API access with UAA. You use the UAA Command Line Interface (UAAC) to target the UAA server and request an access token for the UAA admin user. If your request is successful, the UAA server returns the access token. The UAA admin access token authorizes you to make requests to the PKS API using the PKS CLI and grant cluster access to new or existing users.

  • Leaving setting default with some timer changes
  • 15

Monitoring

You can monitor kubernetes cluster and pods using VMware Wavefront , which i will be covering in a separate post.

  • For now leave it default.

Usage Data

VMware’s Customer Experience Improvement Program (CEIP) and the Pivotal Telemetry Program (Telemetry)  program.

  • choose based in your preference.

Errands

Errands are scripts that run at designated points during an installation.

  • Since we are running PKS with NSX-T , we must need to verify our NSX-T configuration.
  • 16.png

Resource Config for PKS VM

Edit resources used by the Pivotal Container Service job and if there are timeouts while accessing PKS API VM , use high resource VM Type , for this LAB i am going with Default.

  • Leave it default.
  • 17.png

Missing Stemcell

A stemcell is a versioned Operating System image wrapped with IaaS specific packaging.A typical stemcell contains a bare minimum OS skeleton with a few common utilities pre-installed, a BOSH Agent, and a few configuration files to securely configure the OS by default. For example: with vSphere, the official stemcell for Ubuntu Trusty is an approximately 500MB VMDK file.

Click on missing stemcell link which will take you to StemCell Library. Here you can see PKS requires stemcell 170.15 , since i have already downloaded thats the reason it is showing 170.25 in the deployed section but in new installation cases it will show none  deployed. Click IMPORT STEMCELL and choose a stemcell which can be downloaded from Here to import.

18.png

Apply Changes

Return to Ops Manager installation Dashboard and click on “Review Pending Changes” and finally “Apply Changes” , this will go ahead and deploy PKS API VM at your IAAS location which you have chosen while configuring PKS tile.

14.png

and if the configuration of the tile is correct , around after 30 minute , you will see a successful message that deployment has been completed , which gives very nice feeling that your hard work  and dedication resulting success (for me it failed couple of time because of storage/network and resource issues).

To identify which VM has been deployed , you can check custom attributes or go back to  the Installation Dashboard, click the PKS tile then go to the Status tab. Here we can see the IP address of our PKS API , also notice CID which is VM name in vCenter inventory. also you can see the health of the PKS VM.

1920

This completes the PKS VM deployment procedure. in the next post we will deploy kubernetes Cluster.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

vCloud Director 9.7 Portal Custom Branding

Much awaited feature for cloud provider to match thier corporate  standards and to create a fully custom cloud experience, now with release on vCloud Director 9.7 you can set the logo and the theme for your vCloud Director Service Provider Admin Portal and also now you can customize the vCloud Director Tenant Portal of each tenants . In addition, you can modify and add custom links to the two upper right menus in the vCloud Director provider and tenant portals.

Provider Portal Branding

vCloud Director 9.7 UI can be modified for the following elements:

  • Portal name
  • Portal color
  • Portal theme (vCloud Director contains two themes – default and dark.)
  • Logo & Browser icon

Customize Portal Name ,Portal Color and Portal Theme

To configure the Cloud Provider Portal Branding , make a PUT request to vCloud Director end point as below:

  • PUThttps://<vCD Url>/cloudapi/branding
  • BODY – {
    “portalName”: “string”,
    “portalColor”: “string”,
    “selectedTheme”: {
    “themeType”: “string”,
    “name”: “string”
    },
    “customLinks”: [
    {
    “name”: “string”,
    “menuItemType”: “link”,
    “url”: “string”
    }
    ]
    }
  • Headers
    • 2.png

Here is my API call using Postman client:

1.png

Customize Logo

To change the Logo, here is the procedure for API

  • Headers
    • 4.png
  • PUT
  • Body – This is bit tricky since we need to upload an image as a body.
    • In Postman client inside “Body” click on “Binary” which will allow you to choose file as body. select your logo.
    • 5.png

Customize Icon

To customize the icon, follow this API call and procedure.

  • Headers
    • 9.png
  • PUT
  • Body – same as above section , choose a image
    • 10.png

so after running above API calls , here is what my vCloud Director provider portal looks like.

678.png

Tenant Portal Branding

As we did above similarly we can now fully customize Tenant Portal

Customize Portal Name ,Portal Color and Portal Theme

To configure the Cloud Provider Portal Branding , make a PUT request to vCloud Director end point in to tenant organisation as below: ( T1 is my org Name)

  • PUThttps://<vCD Url>/cloudapi/branding/tenant/T1
  • BODY – {
    “portalName”: “string”,
    “portalColor”: “string”,
    “selectedTheme”: {
    “themeType”: “string”,
    “name”: “string”
    },
    “customLinks”: [
    {
    “name”: “string”,
    “menuItemType”: “link”,
    “url”: “string”
    }
    ]
    }
  • Headers
    • 11.png

Here is my API call using Postman client:

12.png

Customize Logo

To change the Logo, here is the procedure for API

  • Headers
    • 4.png
  • PUT
  • Body – As said above ,this is bit tricky since we need to upload an image as a body.
    • In Postman client inside “Body” click on “Binary” which will allow you to choose file as body, select your logo.
    • 14.png

Once i have done with above API calls, this is how my Tenant portal look like for “T1” organisation.

15.png

For a particular tenant, you can selectively override any combination of the portal name, background color, logo, icon, theme, and custom links. Any value that you do not set uses the corresponding system default value.

This completes feature walk through of Provider and Tenant custom branding options available now with vCD9.7.

Upgrade Postgres SQL 9 to 10 for vCloud Director

Since vCloud Director 9.7 has dropped support for Postgres SQL9.5 , so i had to upgrade my postgres to 10 , then i have updated my vCloud Director to versions 9.7 , i followed below steps to upgrade the DB , basically at High level steps are as below:

  • You need to backup the existing database and data directory.
  • Uninstall old version of Postgres SQL.
  • Install Postgres10
  • Restore Backup

Procedure

  • Create database backup using:
    • su – postgres
    • pg_dumpall > /tmp/pg9dbbackup
    • exit
    • 1
  • Check and Stop the service using
    • #chkconfig
    • #service postgresql-9.5 stop
    • 2
  • Move current data file as .old to /tmp directory using below command.
    • #mv /var/lib/pgsql/9.5/data/ /tmp/data.old
  • Uninstall 9.5 version of Postgres SQL using :
    • yum remove postgresql*
  • Install  PostgreSQL v10:
  • Initialise the database
    • service postgresql-10 initdb
    • as suggested by my friend miguel if above step is not working then use this (/usr/pgsql-10/bin/postgresql-10-setup initdb)
  • Copy the pg_hba.conf and postgresql.conf from old backed up directory to new directory , this will save some time or you can go ahead and edit existing files with required settings.
    • cp /data.old/pg_hba.conf /var/lib/pgsql/10/data/
    • cp /data.old/postgresql.conf /var/lib/pgsql/10/data/
    • service postgresql-10 start
  • Restore backup using below commands:
    • su – postgres
    • psql -d postgres -f /tmp/pg9dbbackup

you can run the reconfigure-database command and that’s it. (change your environment variable accordingly)

7.png

This will complete the database upgrade and database migration procedure.

 

 

 

Deploy VMware PKS – Part2

In this part I will begin PKS installation by deploying Pivotal Ops Manager which basically provides a management interface (UI/API) for Platform Operators to manage the complete lifecycle of both BOSH and PKS starting from install then going to patch and upgrade.

To refer other posts of this series are here:

Getting Started with VMware PKS & NSX-T

Deploy VMware PKS – Part1

In addition, you can also deploy new application services using Ops Manager Tiles like adding an Enterprise-class Container Registry like VMware Harbor which can then be configured to work with PKS.

Installing OpsManager

2.png

  • Once Downloaded , Log into vCenter using the vSphere Web Client or HTML5 Client to deploy the Ops Manager OVA.
  • Choose your Management cluster , appropriate network and other OVA deployment options , i am not going to cover OVA deployment procedure here. Only at customize template , enter below details:
    • Admin Password: A default password for the user “ubuntu”.
      • If you do not enter a password, Ops Manager will not boot up.
    • Custom hostname: The hostname for the Ops Manager VM, in My example opsmgr.corp.local.
    • DNS: One or more DNS servers for the Ops Manager VM.
    • Default Gateway: The default gateway for Ops Manager.
    • IP Address: The IP address of the Ops Manager network interface.
    • NTP Server: The IP address of one or more NTP servers for Ops Manager.
    • Netmask: The network mask for Ops Manager.1.png
  • Create a DNS entry for the IP address that you used for Ops Manager ,which we will use in next steps. use this DNS NAME/IP address and browse on a browser , which will take you to Authentication System for initial authentication setup and for our setup, i will use “Internal Authentication” for this Lab. Click on “Internal Authentication”
    • 2
  • Next, you will be prompted to create a new admin user which we will use to manage BOSH. Once you have successfully created the user, go ahead and login with the new user account
    • 3.png
  • Once you are logged into Ops Manager, you can see that the BOSH Tile is already there but is showing as un-configured (orange colour denotes un-configured) which means BOSH has not yet been deployed yet. Go ahead and click on the tile to begin the configuration to deploy BOSH.
    • 4.png

Before starting Bosh Tile configuration , we need to prepare NSX Manager, listed below procedure for:

Generating and Registering the NSX Manager Certificate for PKS

The NSX Manager CA certificate is used to authenticate PKS with NSX Manager. You create an IP-based, self-signed certificate and register it with the NSX Manager.By default, the NSX Manager includes a self-signed API certificate with its hostname as the subject and issuer. PKS Ops Manager requires strict certificate validation and expects the subject and issuer of the self-signed certificate to be either the IP address or fully qualified domain name (FQDN) of the NSX Manager. That’s the reason, we need to regenerate the self-signed certificate using the FQDN of the NSX Manager in the subject and issuer field and then register the certificate with the NSX Manager using the NSX API.

  • Create a file for the certificate request parameters named “nsx-cert.cnf” in a linux VM where openssl tool is installed.
  • Write below content in to the file which we create in above step.
    • 3.png
  • Export the NSX_MANAGER_IP_ADDRESS and NSX_MANAGER_COMMONNAME environment variables using the IP address of your NSX Manager and the FQDN of the NSX Manager host.
    • 4.png
  • Using openssl tool generate the certificate by running below command:
    • 56
  • Verify the certificate by running command as below:
    • ~$ openssl x509 -in nsx.crt -text -noout and ensure SAN has DNS name and IP addresses.
    • 7.png
  • Import this Certificate in NSX Manager , go to System -> Trust -> Certificates and click on Import -> Import Certificate
    • 8.png
  • Ensure that Certificate looks like this in your NSX Manager.
    • 9.png
  • Next is to Register the certificate with NSX Manager using below procedure , first the ID of the certificate from gui.
    • 10.png
  • Run the below command (in to your API client ) to register the certificate using below command replace “CERTIFICATE-ID” with your certificate ID.

Now let’s configure BOSH tile , which will deploy BOSH based on our input parameters.

Configure BOSH Tile to Deploy BOSH Director

Click on the tile. It will open the tile’s setting tab with the vCenter Config parameters page.

  • vCenter Config 
    • Name: a unique meaning full name
    • vCenter Host: The hostname of the vCenter.
    • vCenter Username: Username for above VC with create and delete privileges for virtual machines (VMs) and folders.
    • vCenter Password: the password of above VC.
    • Datacenter Name: Exact name of data center object in vCenter
    • Virtual Disk Type: Select “Thin” or “Thick”
    • Ephemeral Datastore Names: The names of the data stores that store ephemeral VM disks deployed by Ops Manager , you can specify many data stores by using comma.
    • Persistent Datastore Names (comma delimited): The names of the datastores that store persistent VM disks deployed by Ops Manager.
    • To deploy BOSH as well as the PKS Control Plane VMs, Ops Manager will go ahead and upload a Stemcell VM ( A VM Template that PKS) and it will clone from that image for both PKS Management VMs as well as base K8S VMs.
    • 11
  • NSX-T Config 
    • Choose NSX Networking and Select NSX-T 
    • NSX Address: IP/DNS Name for NSX-T Manager.
    • NSX Username and NSX Password: NSX-T credential
    • NSX CA Cert: Open the NSX CA Cert that you generated in above section and copy/paste its content to this field.
    • 12.png
  •  Other Config
    • VM Folder: The vSphere datacenter folder where Ops Manager places VMs.
    • Template Folder: The vSphere folder where Ops Manager places stemcells(templates).
    • Disk path Folder: The vSphere datastore folder where Ops Manager creates attached disk images. You must not nest this folder.
    • And Click on “Save”.
    • 13
  • Director Config
    • For Director config , i have put in few details like:
      • NTP Server Details
      • Enable VM Resurrector Plugin
      • Enable Post Deploy Scripts
      • Recreate all VMs (optional)
    • 14
  • Availability Zone    
    • Availability zones are defined at a vSphere Cluster level. These AZs will then be used by BOSH director to determine where to deploy the PKS Management VMs as well as the Kubernetes VMs.Multiple Availability Zones allow you to provide high-availability across data centers. for this demonstration i have created two AZs, one for Management and one for my Compute.15.png
  • Create Network
    • Since i am using dvs for my PKS management component , we need to specify those details in to this segment and make sure you select the Management AZ which is the vSphere Cluster where BOSH and PKS Control Plane VM will be deployed.

16

  • Assign AZs and Networks 
    • In this section,  Define the AZ and networking placement settings for the PKS Management VM.Singleton Availability Zone – The Ops Manager Director installs in this Availability Zone.

17

  • Security & Syslog
    • This section i am leaving default , if required for your deployment , pls refer documentation.
  • Resource Config
    • As per in part 1 sizing , the BOSH director vm by default allocates 2 vCPUs, 8GB memory, 64GB disk and also has a persistent disk of 50GB and Each of the four Compilation VMs uses 4 vCPUs, 4GB memory, 16GB disk each. For my lab deployment i have changed it to suite to my lab resources.Bosh director needs minimum of 8 GB Memory to run , so choose options accordingly.

18.png

  • Review Pending Changes and Apply Changes 
    • With all the input completed, return to the Installation Dashboard view by clicking Installation Dashboard at the top of the window. The BOSH Director tile will now have a green bar indicating all the required parameters have been entered. Next we click REVIEW PENDING CHANGES and Apply Changes

22

  • Monitor Installation and Finish
    • 8.png
    • If all the inputs are right then you will see that your installation is successful

9.png

After you login to your vCenter , you will see a new powered on VM in your  inventory that starts with vm-UUID which is the BOSH VM. Ops Manager uses vSphere Custom Attributes to add additional metadata fields to identify the various VMs it can deploy, you can check what type of VM this is by simply looking at the deployment, instance_group or job property. In this case, we can see its been noted as p-bosh.

24.png

and this completes Ops Manager and BOSH deployment , next post we will install PKS tile.

 

 

Deploy VMware PKS – Part1

In Continuation of my previous blog post here , where i have explained PKS component and sizing details , in this post i will be covering PKS component deployment.

Previous Post in this Series:

Getting Started with VMware PKS & NSX-T

Pre-requisite:

  • Install a New or Existing server which has DNS role installed and configured , which we will use in this deployment.
  • Install vCenter and ESXi , for this Lab i have created two vSphere Cluster:
    • Management Cluster + Edge Cluster – Three Nodes
    • Compute Cluster – Two Nodes
  • Create a Ubuntu server , where you will need to install client utilities like:
    • PKS CLI
      • The PKS CLI is used to create, manage, and delete Kubernetes clusters.
    • KUBECTL
      • To deploy workloads/applications to a Kubernetes cluster created using the PKS CLI, use the Kubernetes CLI called “kubectl“.
    • UAAC
      • To manage users in Pivotal Container Service (PKS) with User Account and Authentication (UAA). Create and manage users in UAA with the UAA Command Line Interface (UAAC).

    • BOSH
      • BOSH CLI used to manage PKS management components deployments and provides information about the VMs using its Cloud Provider Interface (CPI) which is vSphere in my Lab and could be AZURE , AWS and GCP also.
    • OM
      • Bosh Operations Manager command line interface.
  • Prepare NSX-T

    For this Deployment make sure NSX-T is deployed and configured, high level steps are as below:

    • Install NSX Manager
    • Deploy NSX Controllers
    • Register Controllers with Managers as well as other controller with Master controller.
    • Deploy NSX Edge Nodes

    • Register NSX Edge Nodes with NSX Manager

    • Enable Repository Service on NSX Manager

    • Create TEP IP Pool

    • Create Overlay Transport Zone

    • Create VLAN Transport Zone

    • Create Edge Transport Nodes

    • Create Edge Cluster

    • Create T0 Logical Router and configure BGP routing with physical device

    • Configure Edge Nodes for HA

    • Prepare ESXi Servers for the PKS Compute Cluster

My PKS deployment topology is look like below:

8.png

  • PKS Deployment Topology – PKS management stack running out of NSX-T
    • PKS VMs (Ops Manager, BOSH, PKS Control Plane, Harbor) are deployed to a VDS backed portgroup
    • Connectivity between PKS VMs, K8S Cluster Management and T0 Router is through a physical router
    • NAT is only configured on T0 to provide POD networks access to associated K8S Cluster Namespace
  • Create a IP Pool
    • Create a new IP Pool which will be used to allocate Virtual IPs for the exposed K8S Services The network also provides IP addresses for Kubernetes API access. Go to Inventory->Groups->IP Pool and enter the following configuration:
      • Name: PKS-FLOATING-POOL
      • IP Range: 172.26.0.100 – 172.26.255.254
      • CIDR: 172.26.0.0/16
    • 5.png
  • Create POD-IP-BLOCK
    • We need to create a new POD IP Block which will by used by PKS on-demand to create smaller /24 networks and assigned those to each K8S namespace. This IP block should be sized sufficiently to ensure that you do not run out of addresses. To create POD-IP-BLOCK , go to NETWORKING->IPAM and enter the following:
    • 7
  •  Create NODEs-IP-BLOCK
    • We need to create new NODEs IP Block which will be used by PKS to assign IP address to Kubernetes master and worker nodes.Each Kubernetes cluster owns the /24 subnet , so to deploy multiple Kubernetes clusters, plan for larger than /24 subnet. (recommendation is /16)
    • 6

Prepare Client VM

  • Create and install a small Ubuntu VM with default configuration. you can use the latest server version and insure that the VM has internet connectivity either by proxy or direct.
  • Once the Ubuntu VM is ready , download PKSCLI and KUBECTL from https://network.pivotal.io/products/pivotal-container-service

2

and copy both the PKS (pks-linux-amd64-1.3.0-build.126 or latest) and Kubectl                      (kubectl-linux-amd64-1.12.4 or latest) CLI to VM.

  • Now SSH to the Ubuntu VM and run the following commands to make binaries executable and renaming/relocating them to /usr/local/bin directory:
    • chmod +x pks-linux-amd64-1.3.0-build.126
    • chmod +x kubectl-linux-amd64-1.12.4
    • mv pks-linux-amd64-1.3.0-build.126 /usr/local/bin/pks
    • mv kubectl-linux-amd64-1.12.4 /usr/local/bin/kubectl
    • Check version using – pks -v and kubectl version
  • Next is to install Cloud Foundry UAAC , run this command
    • apt -y install ruby ruby-dev gcc build-essential g++
    • gem install cf-uaac
    • Check version using – uaac -v
  • Next is to install
  • 34

This completes this part , in the next part we will start deploying PKS management VMs and their configuration.

 

VMware Container Service Extension Upgrade

With the release of new Container Service Extension (CSE) version 1.2.7 due to vulnerability related to docker (CVE-2019-5736 ) for both Ubuntu and Photon OS templates , it is very important to update the CSE ASAP , here is the procedure to help you to upgrade the CSE easily.

Pre-requisite:

  • Check the release notes Here for version compatibility.

Upgrade procedure for Cloud Admins:

  • Update CSE to 1.2.7 ( follow procedure below)
  • Update the templates (follow procedure below)

Upgrading CSE Server Software

  •  Stop CSE Server services gracefully.
    • #vcd cse system stop -y
    • 2.png
  • Reinstall container-service-extension using Python Package Index:
    • #pip3 install –user –upgrade container-service-extension
    • 3.png
  • Review the configuration file for any new options introduced or deprecated in the new version. cse sample  can be used to generate a new sample config file as well.
    • 3.png
    • Follow the steps listed here , to edit your environment variable for CSE to use.
  • If the previously generated templates are no longer supported by the new version, delete the templates and re-generate new ones using below command.
    • cse install -c mysample.yaml –update
    • 12
  • If running CSE as a service, start the new version of the service with
    • $systemctl start cse
    • 4.png

Upgrade procedure for Tenant Users:

  • Delete clusters that were created with older templates. Recreate clusters with new templates
  • Alternatively, tenant-users can update docker version manually on their existing clusters.

This completes the upgrade procedure , go ahead and let the customer consume Kubernetes as a Service from your platform.

VMware CSE Upgrade Error – Missing keys in config file ‘service’ section: {‘enforce_authorization’}

Trying to upgrade CSE to latest version of CSE 1.2.7 and during upgrade process facing error , like this: Missing keys in config file ‘service’ section: {‘enforce_authorization’}

error.png

with this new release there are many new options has been added in to configuration file considering PKS integration , so to resolve this issue , there are two options:

  • Create a new sample config.yaml file using command:
    • cse sample > myconfig.yaml  – and reconfigure it.
  • If don’t need PKS integration as of now and edit the existing config.yaml file and add “enforce_authorization: false” in to service section
    • 7.png

and once you done the changes , re-run the command and it should now successfully complete the process.

8.png

this new process has not been documented properly in the CSE git page 🙂

 

VMware Container Service Extension Installation – Part-1

In continuation of my last post on Kubernetes as a service on vCloud Director , here is the next post on installation of Container Server Extension on vCloud Director.

This post applies to CSE version 1.2.5

CSE Installation

This installation procedure applies to Client VM as well as CSE Server VM. For this installation i will leverage a Photon OS 2.0 VM based on the official OVA which is available here. deploy OVA following the standard OVA deployment procedure.Once deployed, make sure you configure static IP and configure networking correctly based on your environment and ensure that this machine can reach to internet to download necessary binaries.

Configure Static IP on Photon OS

Edit file 99-dhcp-en.network inside directory /etc/systemd/network  and change as below.

IP.png

By default ping is disabled on this , so open firewall using below commands:

fw.png

Now Install Python related binaries using below command:

root@photon-machine [ ~ ]# tdnf install -y build-essential python3-setuptools python3-tools python3-pip python3-devel

root@photon-machine [ ~ ]# pip3 install –upgrade pip

Install CSE Software:

Now install and verify the installation CSE:

root@photon-machine [ ~ ]# pip3 install container-service-extension

version.png

This completes installation of CSE , now we need to enable CSE client on this VM.

Enable CSE Client:

Go and edit ~/.vcd-cli/profiles.yaml  file to include this section: (exactly like in Image)

yaml.png

vCD Prerequisites:

There are many important requirements that must be fulfilled to install CSE successfully on vCD.

  • Catalog Organization creation:
  • Create a VDC within the org that has an external org network in which vApps may be instantiated and sufficient storage to create vApps and publish them as templates. The external network connection is required to enable template VMs to download packages during configuration. The process as follows:
    • CSE server will upload base OS image to vCloud Director in a CSE Catalog
    • CSE server will deploy the template as a VM on a Org VDC Network that requires internet access and will download and install required kubernetes and docker binaries.
    • CSE will then validate the VM and capture as vApp template and add it back to the CSE Catalog as a valid item for deploying container hosts.
  • Create a user in the org with privileges necessary to perform operations like configuring AMQP, creating public catalog entries, and managing vApps.
  • A good network connection from the host running CSE installation to vCD as well as the Internet. This avoids intermittent failures in OVA upload/download operations.

CSE Server Config File:

The CSE server is controlled by a yaml configuration file that must be filled out prior to installation. Once vCD pre-requisites are ready,  You can generate a sample file using below command:

#cse sample > config.yaml  ( cse sample generates sample config yaml)

Run above command on above VM which we have prepared for our CSE server , This file is having five sections , which i am going to cover one by one.

AMQP Section:

  • During CSE Server installation, CSE will configure AMQP to ensure communication between vCD and the running CSE server. if vCD has already been configured then skip this section while running install command , if vCD has not been configured with AMQP configuration then enter information in this section which will automatically go and configure this for you in vCD. Configure this section as described below:

 

1 copy

vCD Section:

  • This section is self explanatory , you need to specify vCD related details (ensure API version is related to vCD version):

2.png

vCS Section:

  • In this section provide vCenter information like VC name and credential.

3.png

 Service Section:

  • The service section specifies the number of threads to run in the CSE server process.

4

Broker Section:

  • The broker section contains properties to define resources used by the CSE server including org and VDC as well as template definitions. The following Image summarise key parameters. More Details can be found here

5

  • Sample Config.yaml file can be downloaded from config.

CSE SERVER INSTALLATION:

  • Once your are ready with file run CSE install command to start the installation. ( as said earlier we need to create a VM on which CSE server must be installed by the vCloud Director System/Cloud Administrator.The CSE appliance must be reachable to vCenter , vCD and AMQP servers. i am installing on the VM which i have prepared in first section)
  • #cse install -c config.yaml –ssh-key=$HOME/.ssh/id_rsa.pub –ext config -amqp skip
  • I am skipping amqp configuration as “AMQP” is already configured in my vCD.

14.png

15

  • it failed due to some issue , so i have to rerun the command after fixing the issue and same can be done multiple times.

16

  • Once installation is completed , check the installation status using:
  • #cse check –config config.yaml –check-install

17

  • Now to validate that CSE has been registered in vCD Use “vcd-cli” command line, check that the extension has been registered in vCD:

181920

Running CSE Server as a Service:

  • create a file named “cse.sh”  inside directory /home/vmware with following content:
    • 7.png
  • create file name cse.service inside directory /etc/systemd/system with following content:
    • 6.png
  • Once installed you can start the CSE service daemon using #systemctl start cse . To enable, disable, and stop the CSE service, use CSE client.
    • 23.png

Setting the API Extension Timeout

  • The API extension timeout is the number of seconds that vCD waits for a response from the CSE server extension. The default value is 10 seconds, which may be too short for some environments. To change the time follow the steps :

    • On the vCloud Director cell run:

    • Go to Cd /opt/vmware/vcloud-director/bin and run below commands -l to list -v to Set.2122

Enable CSE

  • Login to vCD and enable the CSE using below commands…

8.png

This completes the installation of Container Server Extension and allow providers to offer Kubernetes as a Service to their customers. feel free to share your experience on this installation.

What is VMware Cloud Provider Pod

There are lots of partners looking for a solution which can automate the entire deployment of vCD based cloud once racking , stacking and cabling is done for their infrastructure that is where VMware Cloud Provider Pod helps…Basically Cloud Provider Pod automate the deployment of VMware-based clouds. A Cloud Provider Pod-deployed stack adheres to VMware Validated Design principles and is thoroughly tested for interoperability and performance. It is also tested for cloud-scale and is built to handle rigorous Cloud Provider workloads. It deploys technologies with core provider capabilities such as data center extension, cloud migration, and multi-tenancy and chargeback, and helps achieve the fastest path to VMware-based cloud services delivery. Cloud provider POD help cloud providers time to market and help in improving service delivery.

Features:

2          3     4

Cloud Provider Pod 1.1 : Supported Interoperable Version for Deployment

vSphere 6.7u1
vSAN 6.7u1
NSX 6.4.4
vCloud Director Extender 1.1.0.2
vRealize Orchestrator 7.5
vRealize Operations 7.0, including Multi-Tenant App 2.0
vRops – Cloud Pod Management Pack
vRealize Log Insight 4.7
vRealize Network Insight 4.0
Usage Meter 3.6.1

POD Designer walk through

The Cloud Provider Pod Designer offers Providers the choice to start with a VMware Validated Design (CONFIGURE YOUR CLOUD) or the Advanced Design which is custom designer based on your environment specific requirement not as per VMware validated design.

5.png

The main difference between the VMware Validated Design and the Advanced Configuration modes is that you can choose to use NFS, iSCSI or Fibre Channel as your storage options. The setup of BGP AS and other options is also not required, but can be done.VVD designer start with asking basic details about your cloud environment that you want to build , Click on “Configure Your Cloud” which will take you to below screen where you need to fill in “General Parameters”

6

Next will take you the screen where you need to choose optional packages that you want to add/exclude from your deployment.

7

Next will take you to Resource cluster selection , where you need to choose how many resource cluster your deployment will have and within that resource cluster how many host that you would have.

8

In the next screen , Enter your environment variable like DNS ,NTP  etc…

9

Enter your Management Cluster’s Networking and public facing ip addressing in “External/DMZ IP Assignment” and in MAC Addresses , you can add the MAC addresses for the hosts during the Cloud Provider Pod Designer workflow, or later during the deployment workflow. The number of available MAC addresses text boxes depends on how many hosts have been configured on the Sizing page

10.png

11.png

Enter your resource cluster details like VXLAN Segment etc…

13

Choose Hypervisor’s NIC allocation.

14

Enter License Keys now or post deployment also licenses can be assigned.

15.png

“Generate all Documentation Files”  –  This is very important and all the providers will like it , it basically automate the creation of design document and configuration work book of your environment , which was the biggest pain where Architects/consultants used to spend lots of time writing design document with visio’s etc.. this is all automatically get generated using CPod.

16.png

Once you click on “Generate Configuration” , it will generate your deployment bundle and documentation and email it to you then you can use “Cloud Provider Pod Deployer” to start the deployment. here is overall flow of the entire process

18.png

Cloud Provider Pod Deployer

Use Cloud provider deployer to deploy entire infrastructure on a click of a button. Detailed documentation and step-by-step instructions on how to use the Cloud Provider Pod Deployer to create a new environment are available in the Cloud Provider Operations guide. This guide is delivered by the Cloud Provider Pod Designer as part of generated documentation by an Email , which you registered at the start of designer.

Deployer Video is here for your reference – https://www.youtube.com/watch?v=5xOiToL2o94&feature=youtu.be&list=PLunwH0gjkUBi7Mu18nNXxUl6FgzpU3iyd

 

 

Kubernetes-as-a-Service on vCloud Director

VMware’s Container Service Extension (CSE)  on vCloud Director is a VMware vCloud Director extension that helps Cloud Providers to Offer Kubernetes-as-a-Service to their tenants , who can easily create and work with Kubernetes clusters. basically it means  using CSE a Service Provider can offer compute resources to tenants secured through a multi-tenant IaaS vCloud Director deployment , and tenants/end users will have the ability to deploy & manage their kubernetes clusters from a self service portal

CSE brings Kubernetes-as-a-service to vCD by creating customized VM templates and enabling tenant/organization administrators to deploy fully functional Kubernetes clusters in self-contained vApps.

CSE Components:

  • CSE Client

    • Tenant Organization Administrators and users can use CSE client to handle Kubernetes cluster management. This includes deploying clusters, adding worker nodes, configuring NFS storage etc…
    • CSE client running on a Virtual Machine runs as an extension of vcd-cli which leverages CSE/vCD public API to manage and administer the service.
    • CSE Client which is extension of  vcd-cli offers easy way to manage life cycle of the kubernetes cluster by the Tenant.
    • From this VM CSE commands are getting issued to vCloud Director , which takes these instructions using AMQP message bus to CSE server.
  • vCloud Director Based Cloud

    • Service Provider’s cloud administrators will setup vCD, Org Network , catalog etc.
    • vCD will be the platform which will provider compute , network , security and multi-tenancy on which kubernetes clusters will be deployed.
    • CSE will use vCloud Directors Extensibility framework to deploy Kubernetes cluster , kubernetes cluster scaling operations like scale up/down , scale In/out etc..
  • CSE Server

    • Service Provider’s cloud administrators will setup CSE config file, CSE Server, and VM templates.
    • You install CSE Server on a new VM and it works in conjunction with vCD extensibility framework.
    • CSE automatically downloads and installs required binaries like Kubernetes , docker , weave etc on a template.
    • Handles CSE Client request for creation and deletion of K8s Cluster and nodes.

User Accessibility of Kubernetes cluster

  • Kubectl

    • Developers and other Kubernetes users interact with CSE Kubernetes clusters using kubernetes native “Kubectl” command line tool, For any tenant  users, Kubernetes clusters work like any other Kubernetes cluster implementation. No special knowledge of vCloud Director or CSE administration is required. Such users do not even need a vCloud Director account.

Below figure clearly lists out the required component and their owners , this picture and more details can be accessed from here

1.png

Installation Type:

Installation Type dependent on the type of the user as stated in above figure:

Kubernetes User – Install Kubectl on your laptop/desktop.

Tenant Administrator – Install CSE and configure CSE Client on a VM.

Service Provider – Install CSE , Install Messaging Bus , configure and register with vCloud Director.

In the Next series of posts i will be covering installation and configuration of CSE.

Whats new with VMware PKS v1.3

Last week VMware announced release of PKS 1.3 , which has some of the much awaited features  like enhance multi-cloud support, additional networking and security options, ease of management and operations. Few features i am going to discusses here:

Microsoft Azure support as IAAS

VMware PKS already support VMware vSphere , Google Cloud Platform and Amazon EC2 as supported platform for PKS deployment , in this new VMware PKS 1.3 release introduces support for Microsoft Azure. so now you can deploy production grade kubernetes from a single console to your choice of IAAS. Here is the list of features supported by PKS on different IAAS.

1.png

Kubernetes 1.12 Support

if you see kubernetes 1.12 release notes around 60+ enhancement and features has been introduced so it make all the sense to upgrade to Kubernetes 1.12.4.

Backup and Recovery of Kubernetes Clusters

This release supports backup and recovery of Kubernetes clusters when they are deployed in a single master mode. You can recover Kubernetes clusters and stateless workloads by using the BOSH Backup and Restore (BBR) toolset.

Smoke Tests

Smoke tests let you assess the impact of an upgrade before actually upgrading running clusters.The smoke tests create an ephemeral Kubernetes cluster after each upgrade of VMware PKS, but before applying upgrades to running Kubernetes clusters. This ensures that a test cluster can be provisioned and basic Kubernetes functionality validated with the upgraded software before applying the upgrade to the running clusters. Upon successful completion of the smoke test, the test cluster is deprovisioned to reduce resource consumption, and upgrades then proceed on the running clusters.

2.png

Support for Multiple Tier 0 and Selectable Tier 0 Routers

As you know NSX-T Tier 0 edges connects the physical and virtual networks. A single VMware NSX-T instance can support multiple Tier 0 routers. By deploying Kubernetes clusters across multiple Tier 0 routers service providers get better network isolation between tenants and additionally  service providers can use multiple Tier 0 routers which allows them to use overlapping IP address ranges, providing greater autonomy to tenants in choosing IP address ranges for their services.

With this VMware PKS 1.3 release, now provider/customer can specify a Tier 0 router using the network profile when you create a cluster (pks create cluster). The Kubernetes clusters and all networking objects that are created or configured as part of the cluster such as a load balancer, Tier 1 routers, and SNAT rules are created on this Tier 0 router. Given that a single Tier 0 router can support a finite set of such networking objects, use of multiple Tier 0 routers allows much greater scale.

3.png

Support for Larger Load Balancers

Previous versions of VMware PKS, we can only specify small or medium load balancers. now with VMware PKS 1.3 , it adds support for large load balancers. large load balancers provides higher scale in areas like number of services, number of backend pods per service, and throughput per service.

Routable CIDR blocks for Pod Networks

Routable IP addresses assigned to pods provide traceability of workloads making egress requests. also routable IP addresses provide direct ingress access to pods for some of the specialized workloads. With VMware PKS 1.3, at the time of Kubernetes cluster creation, you can specify whether you need the pods to be routable or non-routable (NAT’ed) by using the network profile.

Specific IP Address Range and Subnet Size for Pod IP Addresses

VMware PKS 1.3 allow you to override the global pod IP address block configured for VMware PKS with a custom IP address block range along with a custom subnet size. This feature helps in where your global IP address range for pods is reaching capacity and you need to deploy new Kubernetes clusters or you need a larger or smaller size subnet for each namespace being created within a cluster.

Multiple VMware PKS Control Planes across a Single NSX-T Instance

With this new release, multiple instances of VMware PKS can be deployed on a single shared NSX-T instance. Each instance of the VMware PKS control plane can be deployed on a dedicated NSX-T Tier 0 router to provide complete end-to-end isolation. With this feature, users can dedicate separate VMware PKS instances to their development, staging, and production environments or cloud provider can offer dedicated PKS as a Service to their customer.

4.png

Harbor 1.7

Harbor is an VMware’s contribution to open source community , Harbor is open source cloud native registry that stores, signs, and scans container images for vulnerabilities. Harbor solves common challenges by delivering trust, compliance, performance, and interoperability. with PKS 1.3 , Harbor 1.7 has been shipped and offers below enhancements like:

  • Support deploy Harbor with Helm Chart, enables the user to have high availability of Harbor services.
  • Support on-demand Garbage Collection, enables the admin to configure run docker registry garbage collection manually or automatically with a cron schedule.
  • Support Image Retag, enables the user to tag image to different repositories and projects, this is particularly useful in cases when images need to be retagged programmatically in a CI pipeline.
  • Support Image Build History, makes it easy to see the contents of a container image.
  • Improve user experience of Helm Chart Repository:
    • Chart searching included in the global search results
    • Show chart versions total number in the chart list
    • Mark labels to helm charts
    • The chart can be deleted by deleting all the versions under it

Monitoring with vRealize Operation Manager

With the integration of cAdvisor, vRops can be used to monitor entire cloud native infrastructure with the help of vRops Management Pack for Containers.

Sinks

Sink resources include both pod logs as well as events from the Kubernetes API. These events are combined in a shared format that provides operators with a robust set of filtering and monitoring options. Now inbuilt Support for creating sink resources with the PKS Command Line Interface.

Workers Scale up and down

with this version kubernetes cluster’s worker node can easily be scaled up and down with a single command like:

5

These are the some of the important features which i like to share , for details feature list check Release note here.

 

How to Prepare for Certified Kubernetes Administration (CKA) Exam

Finally my last two months of preparation for the CKA exam is paid off , when i got this:

CKA.png

so after getting certified , i got lots of message from friends , colleagues around how to prepare for the exam , so this post is all about how to prepare for the exam , one of my friend in his blog (Blog Link) shared that this is manageable exam not as tough as people talk about and i totally agree with him that it is achievable with lots of practice and hard work on understanding the product.

About CKA Exam

The Certified Kubernetes Administrator (CKA) program was created by the Cloud Native Computing Foundation (CNCF), in collaboration with The Linux Foundation, to help develop the Kubernetes ecosystem. Kubernetes is one of the highest velocity open source projects and is exploding. CNCF offers a certification program that allows users to demonstrate their competence in a hands-on, command-line environment. The purpose of the Certified Kubernetes Administrator (CKA) program is to provide assurance that CKAs have the skills, knowledge, and competency to perform the responsibilities of Kubernetes administrators.

CKA is an online, proctored, performance-based test that requires solving multiple issues from a command line.this  certification focuses on the skills required to be a successful Kubernetes Administrator in industry today. This includes these general domains and their weights on the exam:

  • Application Lifecycle Management 8%
  • Installation, Configuration & Validation 12%
  • Kubernetes Core Concepts 19%
  • Kubernetes Networking 11%
  • Kubernetes Scheduling 5%
  • Kubernetes Security 12%
  • Kubernetes Cluster Maintenance 11%
  • Kubernetes Logging / Monitoring 5%
  • Kubernetes Storage 7%
  • Kubernetes Troubleshooting 10%

How to Prepare – My Way of Preparation 

First to prepare for the exam , ensure that you deploy a LAB , without LAB and practice you can not pass the exam no way!!. for Lab i followed below:

You need three virtual machines , so deploy a Lab with three nodes, can be easily setup on a laptop/desktop with virtulization software like VMware Workstation or Virtual Box etc..

  • Deploy Kubernetes using “Kubernetes the Hard Way” , this will help you understand communication between nodes and what all components makes kubernetes.
  • Understand Kubernetes Architecture Components in Detail.

you can also choose above or can deploy on one of your favourite public or private cloud environments. once Lab is ready , i would suggest kubernetes.io is the best resource for preparation of this exam, since in exam you are allowed to open “kubernetes.io” , so preparation from this website is going to help you in the exam also. I prepared using “kubernetes.io” website. all though i  would suggest that you follow each and every page of this website and here are few links on which i focused more for the exam:

Time management is the Key

time management is very important during this exam. CKA exam is  3 hour exam, its is very important to be careful and to pace yourself on the questions so as not get stuck on one question for too long. The exam environment (i.e. ssh session) runs in Google Chrome with a specific extension and can be laggy and slow at times. In addition, I noticed myself spending way too much time trying to select text and running into various UI issues , be prepared and have lots of practice in your labs. there is no way you can successfully clear the exam if not practicing a lot.

The day of the exam

Here are the few Tips for the Exam:

  • Before the exam, the examiner will ask you to clean your desk completely.

  • The place should be quiet because even if you work with headphones you will not be allowed to use them and for next three hours no body will be allowed in the room.

  • The examiner will ask you to see all the room, even under the desk.
  • The examiner will not talk to you by voice, only by chat. He/she will hear you because you will need to share the screen and micro phone.
  • The exam happens in a Chrome tab, the left side will show you the questions and the percentage marks of the questions.The right side is for the shell, I tried to use tmux  ( i would suggest not to use ) there, but it was pretty difficult inside a browser terminal. You can also have a popup with notes.
  • You can only open a tab with kubernetes.io and use its search box, no Google or anything .
  • It’s ok to request a brake, but be very careful because the time doesn’t stop.
  • You have three hours to finish the exam, if you get blocked it’s better to skip that question for now and retake it later.

So, that’s it from me. If you are interested in Kubernetes and you work with it go ahead and prepare for it and once you are certified , you will be proud on you because this certification really does carry the weight it implies and the real-world, live cluster examination is a nail-biter. i like the way CNCF measuring this competency using a Live Lab exam than a multiple choice exam. Best of Luck!!!

 

VMware PKS, NSX-T & Kubernetes Networking & Security explained

In continuation of my last post on VMware PKS and NSX-T explaining on getting started with VMware PKS and NSX-T (Getting Started with VMware PKS & NSX-T) , here is next one explaining around behind the scene NSX-T automation for Kubernetes by VMware NSX CNI plugin and PKS.

NSX-T address all the K8s networking functions, load balancing , IPAM , Routing and Firewalling needs, it supports complete automation and dynamic provisioning of network objects required for K8s and it’s workload and this is what i am going to uncover in this blog post.

Other features  like it has support for different topology choice for POD and NODE networks (NAT or NO-NAT) , it supports network security policies for Kubernetes , Clusters , Namespaces and individual services and it also supports network traceability/visibility using NSX-T in built operational tools for kubernetes.

I will be covering the deployment procedure of PKS after some time but just want to let explain that what happens on NSX-T side when you run “#pks create cluster” on PKS command line..and then when you create K8s Namespaces and PODs

pks create cluster

So when you run #pks create cluster with some argument , it goes to vCenter and deploys Kubernetes Master and Worker VMs based on specification you have chosen during deployment and on NSX-T side a new logical switch get created for these vms and get connected to these vms. (in this Example one K8s Master and 2 Nodes has been deployed) , along with logical switch , a Tier-1 cluster router get created which get connected to your organisation’s Tier-0 router.

K8s Master and Node Logical Switches

13.png

K8s Cluster connectivity towards Tier-0

12.png

Kubernetes Namespaces and NSX-T

Now if K8s cluster deployed successfully, Kubernetes cluster by default deploys three name space:

  • Default – The default namespace for objects with no other namespace.
  • kube-system – The namespace for objects created by the Kubernetes system.
  • kube-public – The namespace is created automatically and readable by all users (including those not authenticated). This namespace is mostly reserved for cluster usage, in case that some resources should be visible and readable publicly throughout the whole cluster. The public aspect of this namespace is only a convention, not a requirement.

for each default namespace, PKS automatically deploys and configures NSX-T Logical Switchs and each logical switch will have its own Tier-1 router connected to Tier-0.

14.png

pks-infrastructure Namespace and NSX-T

in the above figure you can clearly see “default”,”kube-public” and “kube-system” Logical Switches. there is another Logical Switch “pks-infrastructure” get created which is pks specific namespace and running pks related stuff like NSX-T CNI. “pks-infrastructure” is running NSX-NCP CNI plugin to integrate NSX-T with kubernetes.

15.png

kube-system Namespace & NSX-T

Typically, this runs pod like heapster , kube-dns , kubernetes-dashboard,  monitoring db , telemetry agent and stuff like ingresses and so on if you deploy so.

16.png

on NSX-T side as explained earlier a Logical switch get created for this Namespace and for each system POD a logical port get created by PKS on NSX-T.

17.png

Default Namespace & NSX-T

This is the cluster’s default namespace which is used for holding the default set of pods, services, and deployments used by the cluster. so when you deploy a POD without creating/specifying a new name space , “default Namespace” becomes default container to hold these pods and as explained earlier this also has its own NSX-T logical switch with a uplink port to Tier-1 router.

18.png

now when you deploy a Kubernetes pod without a new namespace , since that POD will be part of “Default Namespace”, PKS create a NSX-T logical port on the default logical switch.   let’s create a simple POD:

19.png

let’s go back to NSX-T’s  “Default Namespace” logical switch:

20.png

as you can see a new logical port has been created on default logical switch.

New Namespace & NSX-T

Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces. in simple terms Namespaces are like org vdc in vCD and Kubernetes best practice is to arrange PODs in namespaces. so when we create a new Namespace , what happens in NSX-T ?

i have created a new Namespace called “demo”.

23.png

if you observe below images left image showing default switches and right image is showing logical switches after creation of new Namespace.

and as you can see a new Logical switch has been created for new Namespace.

if you are creating PODs in default Namespace then all the pods get attached to default logical switch and if you are creating Namespace ( which is K8s best-practice) then a new logical switch get created and any POD which is getting deployed in this namespace will be part of its NSX-T logical switch and this new logical switch will also have its own Tier-1 router connecting to Tier-0 router.

24.png

Expose PODs to Outer World

in this example we deployed POD and get the internal network connectivity but internal only connectivity is not going to give access to this web server to outer world and this is default forbidden in kubernetes , so we need to expose this deployment using load balancer to the public interface on the specific port. let’s do that:

26.png

27.png

Lets browse this App using EXTERNAL-IP as you might know CLUSTER-IP is internal ip.

33.png

Kubernetes Cluster and NSX-T Load Balancer

As above when we expose this deployment using service on NSX-T there is a cluster load balancer get deployed automatically when we create cluster , on this load balancer NSX-CNI go ahead and add pod to virtual servers under a new load balancer VIP.

28.png

if we drill down to pool members of this VIP , we will see our kubernetes pod ep ips.

29.png

behind the scene when you deploy a cluster a LB logical switch and LB Tier-1 router which is having logical connectivity to the Load Balancer and Tier-0 Router , so that you can access the deployment externally.

3031

This is what your Tier-0 will look like, having connectivity to all the Tier-1 and Tier-1 is having connected to Namespace logical switches.

32.png

these all logical switchs, Tier-1 router , Tier-0 router creations , their connectivity , LB creations etc all has been done automatically by NSX-T container (CNI) plugin and PKS. i was really thrilled when i tried this first time and it was so simple , if you understood the concept.

Kubernetes and Micro-segmentation

The NSX-T container plugin helps to exposure of container “Pods”as NSX-T logical switch /ports and because of this we can easily implement micro-segmentation rules. once “Pods ” expose to the NSX ecosystem, we can use the same approach we have with Virtual Machines for implementing micro segmentation and other security measures.

3435

or you can use security groups based on tags to achieve micro segmentation.

3637

 

This is what i have tried in this post to explain what happen behind the scene on NSX-T networking stack when you deploy and expose your applications on kubernetes and how we can bring in proven NSX-T based micro-segmentation.

Enjoy learning PKS,NSX-T and Kubernetes one of the best combination for Day-1 and Day-2 operation of kubernetes 🙂 and feel free to comment and suggestions.

 

 

 

Getting Started with VMware PKS & NSX-T

VMware Pivotal Container Service (PKS) provides a Kubernetes based container service for deploying and operating modern applications across private and public clouds. basically it is Managed kubernetes for multiple kubernetes cluster and aimed at Day 2 operations. K8S is designed with focus on high availability, auto-scaling and supports rolling upgrades.

PKS integrates with VMware NSX-T for advanced container networking, including micro-segmentation, ingress controller, load balancing, and security policy and also by using VMware Harbor, PKS secures container images through vulnerability scanning, image signing, and auditing.A PKS deployment consists of multiple VM instances classified into 2 different categories:

PKS Management Plane –

PKS management plane consist of below VMs:

tiles.png

  • PCF Ops Manager

Pivotal Operations Manager (Ops Manager) is a graphical interface for deploying and managing Pivotal BOSH, PKS Control Plane, and VMware Harbor application tiles. Ops Manager also provides a programmatic interface for performing lifecycle management of Ops Manager and application tiles.

  • VMware BOSH Director

Pivotal BOSH is an open-source tool for release engineering for the deployment and lifecycle management of large distributed systems. By using BOSH, developers can version, package, and deploy software in a consistent and reproducible manner.

BOSH is the first component, that’s installed by Ops Manager. BOSH is a primary PKS tile.BOSH was originally designed to deploy open source Cloud Foundry.Internally BOSH has below components:

  1. Director: This holds the role of core orchestration engine controls the provisioning of vms , required softwares and service life cycle events.
  2. Blobstore: The Blobstore stores the source forms of releases and the compiled images of releases. An operator uploads a release using the CLI, and the Director inserts the release into the Blobstore. When you deploy a release, BOSH orchestrates the compilation of packages and stores the result in the Blobstore.
  3. Postgres DB: Bosh director uses a postgres database to store information about the desired state of deployment including information about stemcells, releases and deployments. DB is internal to the Director VM.
  • Pivotal Container Service (PKS Control Plane)

PKS Control Plane is the self service API for on-demand deployment and Life cycle management of K8s clusters. API submit the request to BOSH which automates the creation , deletion and updates of kubernetes clusters.

  • VMware Harbor

VMwware harbor is an open-source, enterprise-class container registry service that stores and distributes container images in a private, on-premises registry. In addition to providing Role-Based Access Control (RBAC), Lightweight Directory Access Protocol (LDAP), and Active Directory (AD) support, Harbor provides container image vulnerability scanning, policy-based image replication, notary and auditing service.

PKS Data Plane

  • Kubernetes

K8s is an open-source container orchestration framework. Containers package applications and their dependencies in container images. A container image is a distributable artifact that provides portability across multiple environments, streamlining the development and deployment of software. Kubernetes orchestrates these containers to manage and automate resource use, failure handling, availability, configuration, scalability, and desired state of the application.

Integration with NSX-T

VMware NSX-T helps simplify networking and security for Kubernetes by automating the implementation of network policies, network object creation, network isolation, and micro-segmentation. NSX-T also provides flexible network topology choices and end-to-end network visibility.

PKS integrates with VMware NSX-T for production-grade container networking and security. A new capability introduced in NSX-T 2.2 allows you to perform workload SSL termination using Load Balancing services. PKS can leverage this capability to provide better security and workload protection.

Major benefit of using NSX-T with PKS and K8s is automation that is  dynamic provisioning and association of network objects for unified VM and pod networking. The automation includes the following:

  • On-demand provisioning of routers and logical switches for each Kubernetes cluster
  • Allocation of a unique IP address segment per logical switch
  • Automatic creation of SNAT rules for external connectivity
  • Dynamic assignment of IP addresses from an IPAM IP block for each pod
  • On-demand creation of load balancers and associated virtual servers for HTTPS and HTTP
  • Automatic creation of routers and logical switches per Kubernetes namespace, which can isolate environments for production, development, and test.

PKS Networking

  • PKS Management Network

This network will be used to deploy PKS Management components. this could be a dvSwitch or NSX-T logical switch since in my Lab i will be using no NAT topologies with virtual switch. in my Lab i will be using dvs with network segment of 192.168.110.x/24.

  • Kubernetes Node Network

This Network will be used for kubernetes management nodes. it is allocated to master and worker nodes. these nodes embed Node Agent to monitor the liveness of the cluster.

  • Kubernetes Pod Network

This network is used when an application will be deployed on to a new kubernetes namespace. A /24 network is taken from IP Block and is allocated to a specific Kubernetes namespace allowing for network isolation and policies to be applied between name spaces. The NSX-T Container Plugin automatically creates the NSX-T logical switch and Tier-1 router for each name spaces.

  • Load Balancer and NAT Subnet

This network pool, also known as the Floating IP Pool, provides IP addresses for load balancing and NAT services which are required as a part of an application deployment in Kubernetes.

PKS deployment Network Topologies – Refer Here

PKS Deployment Planning

Before you install PKS on vSphere with NSX-T integration, you must prepare your vSphere and NSX-T environment and ensure vCenter, NSX-T components, and ESXi hosts must be able to communicate with each other, ensure we have adequate resources.

  • PKS Management VM Sizing

When you size the vSphere resources, consider the compute and storage requirements for each PKS management component.

VM Name vCPU Memory Storage No. of VMs
Ops Manager 1 8 160 GB 1
BOSH 2 8 103 GB 1
PKS Control VM 2 8 29 GB 1
Compilation VMs 4 4 10 GB 4
Client VM 1 2 8 GB 1
VMware Harbor 2 8 169 GB 1

Compilation vms get created when an initial K8s cluster is deployed, software packages are compiled and four additional service VMs are automatically deployed as a process as a single task and these vms get deleted once compilation process completes. To manage and configure PKS,  PKS and Kubernetes CLI command-line utilities are required , these utilities can be installed locally on a workstation called Client VM.

  • Plan your CIDR block

Before you install PKS on vSphere with NSX-T, you should plan for the CIDRs and IP blocks that you are using in your deployment as explained above. these are the CIDR blocks that we need to plan:

  • PKS MANAGEMENT CIDR
  • PKS LB CIDR
  • Pods IP Block
  • Nodes IP Block

Below are the CIDR blocks that you can’t use because:

The Docker daemon on the Kubernetes worker node uses the subnet in the following CIDR range:

  • 172.17.0.1/16
  • 172.18.0.1/16
  • 172.19.0.1/16
  • 172.20.0.1/16
  • 172.21.0.1/16
  • 172.22.0.1/16

If PKS is deployed with Harbor, Harbor uses the following CIDR ranges for its internal Docker bridges:

  • 172.18.0.0/16
  • 172.19.0.0/16
  • 172.20.0.0/16
  • 172.21.0.0/16
  • 172.22.0.0/16

Each Kubernetes cluster uses the following subnet for Kubernetes services,Do not use the following IP block for the Nodes IP Block:

  • 10.100.200.0/24

In this blog post series i will be deploying NO-NAT topology and will walk you through step by step process of PKS deployment with NSX-T integration.

Next post on this series is VMware PKS, NSX-T & Kubernetes Networking & Security explained , this will help you understand what happens behind the scene in networking and security stack when PKS and NSX-T deploys kubernetes and its networking stack.

 

 

Upgrade NSX-T 2.1 to NSX-T 2.3

I am working on PKS deployment and will soon sharing my deployment procedure on PKS but before proceeding with PKS deployment, i need to upgrade my NSX-T lab environment to support latest PKS as per below compatibility matrix.

PKS Version Compatible NSX-T Versions Compatible Ops Manager Versions
v1.2 v2.2, v2.3 v2.2.2+, v2.3.1+
v1.1.6 v2.1, v2.2 v2.1.x, 2.2.x
v1.1.5 v2.1, v2.2 v2.1.x, v2.2.x
v1.1.4 v2.1 v2.1.x, 2.2.x
v1.1.3 v2.1 v2.1.0 – 2.1.6
v1.1.2 v2.1 v2.1.x, 2.2.x
v1.1.1 v2.1 – Advanced Edition v2.1.0 – 2.1.6

In this post i will be covering the procedure to upgrade NSX-T 2.1 to NSX-T 2.3.

So Before proceeding for upgrade , lets check the health of current deployment which is very important because if we start upgrading the environment and once upgrade is completed and after upgrade if some thing is not working , we will not come to know whether before upgrade it was working or not , so lets get in to validation of health and version checks.

Validate Current Version Components Health

First thing to check the Management Cluster and Controller connectivity and ensure they are up.

Next is to Validate host deployment status and connectivity.

34

Check the Edge health

5.png

Lets check the Transport Node Health

6

Upgrade Procedure

Now Download the upgrade bundle

7.png

Go to NSX Manager and browse to Upgrade

8.png

Upload the downloaded upgrade bundle file in NSX Manager

9.png

Since upgrade bundle is very big in size , it will take lots of time in upload, extraction and verification.Once the package has uploaded, click to “BEGIN UPGRADE”.

11

The upgrade coordinator will then check the install for any potential issues. In my environment there is one warnings for the Edge that the connectivity is degraded – this is because of i have disconnected 4 th nic which is safe to ignore, so when you are doing for your environment , please access all the warnings and take necessary actions before proceeding with upgrade.

12

Click Next will take you to view the Hosts Upgrade page. Here you can define the order and method of upgrade for each host, and define host groups to control the order of upgrade. I’ve gone with the defaults, serial (one at a time) upgrades over the parallel because i have two hosts in each clusters.

Click START to begin the upgrade, and the hosts will be put in maintenance mode, then upgraded and rebooted if necessary. ensure you need to have DRS enabled and the VMs on the hosts must be able to vMotion off of the host being put in maintenance mode. Once the host has upgraded, and the Management Plane Agent has reported back to the Manager, the Upgrade Coordinator will move on to the next host in the group.

13.png

Once the hosts are upgraded, click next to move to the Edge Upgrade page. Edge Clusters can be upgraded parallel if you have multiple edge clusters, but the Edges which has formed the Edge Clusters and upgraded serially to ensure connectivity is maintained. In my lab , i have a single Edge Cluster with two Edge VMs, so this will be upgraded one Edge at a time.Click on the “START” to start the edge upgrade process.

14

15

Once the Edge Cluster has been upgraded successfully, click NEXT to move to the Controller Node Upgrade Page. here you can’t change the sequence of upgrade of the controllers, controllers are done in parallel by default. (in my Lab i am running a single controller because of resource constraint but in production you will see three controllers deployed in a cluster). Click on “START” to begin the upgrade process.

16.png

Once the controller upgrade has been completed, click NEXT to move to the NSX Manager upgrade page. The NSX Manager will become unavailable for about 5 minutes after you click START and it might take 15 to 20 minutes to upgrade the manager.

17.png

Once the Manager upgrade has completed. review the upgrade cycle.

19.png

you can re-validate the installation as we did at the start of the upgrade, checking that we have all the green lights on, and the version of components have increased.

vCloud Availability Cloud-to-Cloud Design and Deploy Guide

a.pngvCloud Architecture Toolkit white paper that I have written now has been  published on the cloudsolutions.vmware.com website – this design and deploy guide helps cloud providers to design and deploy vCloud Availability Cloud-to-Cloud DR solution.  This guide is based on real life example and helps cloud providers to successfully plan , design and deploy vCloud Availability Cloud-to-Cloud DR based on version 1.5.

White Paper Download Link

This white paper includes the following chapters to plan your deployment:

  • Introduction
  •  Use Cases
  • vCloud Availability Cloud-to-Cloud DR Components
  • vCloud Availability Cloud-to-Cloud DR Node Types and Sizing
  • vCloud Availability Cloud-to-Cloud DR Deployment Requirements
  • vCloud Availability Cloud-to-Cloud DR Architecture Design
  • Physical Design
  • Certificate
  • Network Communication and Firewalls
  • Deployment
  • Replication Policy
  • Services Management Interface Addresses
  • Log Files
  • Configuration Files
  •  References

I hope this helps in your plan, design  and deployment of vCloud Availability Cloud-to-Cloud DR version 1.5. please feel free to share the feedback to make this white paper more effective and helpful.

What is VMware vCloud Availability Cloud-to-Cloud DR

The VMware vCloud Availability for Cloud-to-Cloud DR solution extends existing hybrid cloud offerings of VMware Cloud Providers™ on top of VMware vCloud Director with disaster recovery and application continuity between vCloud Director Virtual Data Centers or Cloud Environments. vCloud Availability Cloud-to-Cloud brings in a much-needed in providing native Disaster Recovery between vCloud Director instances. VMware vCloud Availability for Cloud-to-Cloud DR can help VMware Cloud Providers enable further monetization of existing VMware vCloud Director multi-tenant cloud environments with DR services, including replication and failover capabilities for workloads at both VM and vApp level.

1.png

Features:

  • vCloud Availability Cloud-to-Cloud DR has capability of each deployment to serve as both source and recovery sites. There are no dedicated source and destination sites. Same set of appliances works as Sources or Destination.
  • Replication and recovery of vApps (VMs) between organization Virtual Data Centers (orgVDC) as well as two instances of vCloud Director for migration, DR, and planned migration.
  • it offers complete self-serviceability for the provider and tenant administrator via a unified HTML5 portal that can be used alongside vCloud Director. Replication, migration, and failover can be managed completely by the tenant or provided as a managed service by the provider.
  • Symmetrical replication flow that can be started from either the source or the recovery vCD instance.
  • Built-in encryption or encryption and compression of replication traffic.
  • Enhanced control with white-listing of DR-enabled vCloud Director organizations, enforcement of specific min Recovery Point Objective (RPO) at an organization (org) level, maximum snapshots per org and max replications per tenant.
  • Provide Non-disruptive, on-demand disaster recovery testing.
  • Policies that allow service provider administrators to control the following system attributes for one or multiple vCloud Director organizations:
    • Limit the number of replications at the vCloud Director organization level
    • Limit the minimum Recovery Point Objective (RPO)
    • Limit number of retained snapshots per VM replication
    • Limit the total number of VM replications

Use Cases:

Though the most obvious use case for VMware vCloud Availability Cloud-to-Cloud DR is disaster recovery from one cloud availability zone to another cloud availability zone, it can handle a number of different use cases and provide significant capability and flexibility to service providers. For all use cases and situations, VMware vCloud Availability Cloud-to-Cloud DR supports non-disruptive testing of protected cloud workload in network and storage isolated environments. This provides the ability to test disaster recovery, disaster avoidance, or planned migrations as frequently as desired to ensure confidence in the configuration and operation of recovery on cloud. The use cases are as below:

Migration:

A tenant or provider administrator can utilize C2C to migrate workloads from one organization VDC to another with minimal disruption from a self-service portal. End benefit is re-organizing workloads from an easy to use workflow.

  • Easy to use workflow mechanism
  • Organize workloads in different orgVDCs
  • Ability to migrate between vCD instances or within the same vCD instance

Disaster Recovery:

A service provider has multiple sites with vCD based multi-tenant environment. Customer like to do DR from one cloud provider site to another cloud site. Disaster recovery or a planned/unplanned failover is what VMware vCloud Availability Cloud-to-Cloud DR was specifically designed to accomplish for cloud providers. This helps providers and customers to achieve:

  • Fastest RTO
  • Recover from unexpected failure
  • Full or partial site recovery

Disaster Avoidance:

Preventive failover is another common use case for VMware vCloud Availability Cloud-to-Cloud DR. This can be anything from an oncoming storm to the threat of power issues.

VMware vCloud Availability Cloud-to-Cloud DR allows for the graceful shutdown of virtual machines at the protected site, full replication of data, and startup of virtual machines and applications at the recovery site ensuring app-consistency and zero data loss. Solution helps Providers and Customer in recovering from:

  • Anticipate outages
  • Preventive failover
  • Graceful shutdown ensuring no data loss

Upgrade and Patch Testing:

The VMware vCloud Availability Cloud-to-Cloud DR test environment provides a perfect location for conducting operating system and application upgrade and patch testing. Test environments are complete copies of production environments configured in an isolated network segment which ensures that testing is as realistic as possible while at the same time not impacting production workloads or replication.

This will give you basic idea of what vCloud Availability Clout-to-Cloud DR solves for the providers.

 

Features of VMware Cloud on AWS

VMware Cloud on AWS enables operational consistency for customers of all sizes whether their workloads operate on-premises or in the public cloud. here i would be covering some of the great feature which i like most and will give you opportunity to understand and explore more..

Automated Cluster Remediation:

Let’s suppose in our on-prem environment we have 8 node cluster , one of the node goes down because of hardware failure , that’s where our struggle start to get required hardware from hardware vendor etc.. but most importantly we loose one host in our HA cluster and if this cluster was highly utilised then your application VM might start facing resource crunch and in my experience this might go for at least 3-4 days by the time you get hardware fix and put back the host in to the cluster.

Now see the power of VMware Cloud on AWS – failed hosts in a VMware SDDC are automatically detected by VMware and replaced with healthy hosts and process runs as below:

  • VMware Team detects Host failure or problem identified
  • New Host will be added in to the cluster and data from problematic host will be either rebuild or migrated.
  • Old host evacuated from the cluster and replaced by new host.

Scale as per your convenience:

One of the major challenges in traditional data centers is finding the right balance between hardware and workload utilization.

VMware Cloud on AWS enables you to quickly scale up to ensure that you always have enough capacity to run your workloads during volume spikes and quickly scale down to ensure that you are not paying for hardware that is not being used. This feature provides higher availability with lower overall costs.

aws4

you have option to add and remove cluster as well as Host or you can enable Elastic Distributed Resources Scheduler (EDRS) , which is a policy-based solution that automatically scales a vSphere Cluster in VMware Cloud on AWS based on utilization. EDRS monitors CPU, memory, and storage resources for scaling operations. EDRS monitors the vSphere cluster continuously, and each 5 minutes EDRS runs the algorithm to determine if scale-out or scale-in operations is required.

vCenter Hybrid Linked Mode:

Hybrid Linked Mode allows you to link your VMware Cloud on AWS vCenter Server instance with an on-premises vCenter Single Sign-On domain and If you link your cloud vCenter Server to a domain that contains multiple vCenter Server instances linked using Enhanced Linked Mode, all of those instances are linked to your cloud SDDC.

You have two options for configuring Hybrid Linked Mode. You can use only one of these options at a time.

  • You can install the Cloud Gateway Appliance and use it to link from your on-premises data center to your cloud SDDC. In this case, Active Directory groups are mapped from your on-premises environment to the cloud.

  • you can link from your cloud SDDC to your on-premises data center. In this case, you must add Active Directory as an identity source to the cloud vCenter Server.

Using Hybrid Linked Mode, you can:

  • View and manage the inventories of both your on-premises and VMware Cloud on AWS data centers from a single vSphere Client interface, accessed using your on-premises credentials.

  • Migrate workloads between your on-premises data center and cloud SDDC.

  • Share tags and tag categories across vCenter Server instances.

Well Defined Separation of Duty for VMware and Customer Teams:

Amazon in discussion with VMware performs the following  tasks:

Hardware refresh , failed component replacement , bios upgrade and underline firmware patching will be done by AWS based on VMware compatibility list and this allow customer not to worry about this tedious exercise, compatibility issues and dedicated skill resources.

VMware Experts perform the following maintenance tasks:

  • Backup and restore of VMware appliances and infrastructure  like vCenter, NSX Manager,PSC etc…
  • Patching VMware Cloud on AWS components like vSphere, ESXi drivers, vSAN, NSX, SDDC console etc…this helps customers to just focus of App VM and their business , leave their virtual infrastructure maintenance to experts.
  • Providing VMware Tools patches through vSphere and will be available to your virtual machines , now customer is free to
  • Host and infrastructure VM monitoring

Customer’s Administrator are responsible for the following tasks:

  • Customer administrator manages backup and restoration of your workload VMs and applications.
  • Patching inside VM like guest OS, applications etc..
  • Upgrading VMware Tools installed on workload VMs
  • Monitoring of the your workload VMs and applications
  • Keeping VM templates and content library files updated so that new vms are deployed with latest/updated/patched updated master templates.
  • Manage and monitoring user access and monitoring of resource utilization and charges of integrated AWS if consuming.

Outages, Scheduled Maintenance, and Health Service Information:

VMware has hosted a separate website to display the current status of VMware Cloud services at https://status.vmware-services.io/ , you can subscribe to updates.

Apart from VMware Cloud on AWS service, this website reports for below services also:

  • VMware AppDefense
  • VMware Cost Insight
  • VMware Discovery
  • VMware Kubernetes Engine
  • Log Intelligence
  • VMware Network Insight

NSX Hybrid Connect

NSX Hybrid Connect enables cloud on-boarding without retrofitting source infrastructure and supports migration from vSphere 5.1 or later to VMware Cloud on AWS without introducing application risk and complex migration assessments.NSX Hybrid Connect includes:

  • vSphere vMotion
  • bulk migration
  • high throughput network extension
  • WAN optimization
  • traffic engineering
  • load balancing
  • automated VPN with strong encryption
  • secured data center interconnectivity with built-in hybrid abstraction and hybrid interconnects.

aws1.png

VMware Site Recovery

VMware Site Recovery for VMware Cloud on AWS is separately purchased item that communicates with separately licensed VMware Site Recovery Manager and VMware vSphere Replication instances. Recovery can occur from on-premises to AWS or AWS SDDC to AWS SDDC. VMware Site Recovery can protect vCenter Server version 6.7, 6.5, and 6.0 U3.

aws2.png

Consumption of AWS Native Services with VMware Cloud on AWS

The partnership between VMware and Amazon increases the catalog of solutions readily available to all VMware Cloud on AWS users. Some of the popular AWS solutions are listed below:

  • Simple Storage Service (S3): Highly available, highly durable object storage service.
  • Glacier: Highly durable, high latency archive storage used mostly for backup.
  • EC2: AWS flagship compute platform.
  • VPC: Networking solution of AWS solutions both internal and external.
  • CloudWatch: Monitoring for AWS solutions.
  • IAM: Identity and Access Management solution of AWS.
  • AWS Database Services: Wide range of  DB service like: Relational Database Service (RDS), DynamoDB (NoSQL Database Service), RedShift (data warehouse for data from relational databases for analytics)
  • Simple Queue Service (SQS): Fully managed message queues for microservices, distributed systems, and server-less applications.
  • Route 53: (DNS) Domain name provider and services.
  • Elasti-Cache: Managed, in-memory data store services.

Simple and feature-rich Web Interface for Network Services

Customer can easily consume Network services with few clicks , you need not to be network expert and strong command line hands-on experience. just few clicks and your IPsec VPN, L2 VPN , NAT , Edge FW rules , getting public IP from amazon all are ready to consume.

aws3.png

i have covered few features of VMware Cloud on AWS , if you wants to dirty your hands , go ahead and login to http://labs.hol.vmware.com  and if your organisation wants to test the feature and ease of consumption , there is one host option is there , By deploying a 1-node SDDC, you will be able to test out the features and functionality of VMware Cloud on AWS at a fraction of the cost. These 1-node SDDC’s are fully self-service, paid for by credit card (or HPP/SPP credits), and deployed in just under two hours.

Hope this helps you in understanding feature of VMware Cloud on AWS  better 🙂