Kubernetes-as-a-Service on vCloud Director

VMware’s Container Service Extension (CSE)  on vCloud Director is a VMware vCloud Director extension that helps Cloud Providers to Offer Kubernetes-as-a-Service to their tenants , who can easily create and work with Kubernetes clusters. basically it means  using CSE a Service Provider can offer compute resources to tenants secured through a multi-tenant IaaS vCloud Director deployment , and tenants/end users will have the ability to deploy & manage their kubernetes clusters from a self service portal

CSE brings Kubernetes-as-a-service to vCD by creating customized VM templates and enabling tenant/organization administrators to deploy fully functional Kubernetes clusters in self-contained vApps.

CSE Components:

  • CSE Client

    • Tenant Organization Administrators and users can use CSE client to handle Kubernetes cluster management. This includes deploying clusters, adding worker nodes, configuring NFS storage etc…
    • CSE client running on a Virtual Machine runs as an extension of vcd-cli which leverages CSE/vCD public API to manage and administer the service.
    • CSE Client which is extension of  vcd-cli offers easy way to manage life cycle of the kubernetes cluster by the Tenant.
    • From this VM CSE commands are getting issued to vCloud Director , which takes these instructions using AMQP message bus to CSE server.
  • vCloud Director Based Cloud

    • Service Provider’s cloud administrators will setup vCD, Org Network , catalog etc.
    • vCD will be the platform which will provider compute , network , security and multi-tenancy on which kubernetes clusters will be deployed.
    • CSE will use vCloud Directors Extensibility framework to deploy Kubernetes cluster , kubernetes cluster scaling operations like scale up/down , scale In/out etc..
  • CSE Server

    • Service Provider’s cloud administrators will setup CSE config file, CSE Server, and VM templates.
    • You install CSE Server on a new VM and it works in conjunction with vCD extensibility framework.
    • CSE automatically downloads and installs required binaries like Kubernetes , docker , weave etc on a template.
    • Handles CSE Client request for creation and deletion of K8s Cluster and nodes.

User Accessibility of Kubernetes cluster

  • Kubectl

    • Developers and other Kubernetes users interact with CSE Kubernetes clusters using kubernetes native “Kubectl” command line tool, For any tenant  users, Kubernetes clusters work like any other Kubernetes cluster implementation. No special knowledge of vCloud Director or CSE administration is required. Such users do not even need a vCloud Director account.

Below figure clearly lists out the required component and their owners , this picture and more details can be accessed from here


Installation Type:

Installation Type dependent on the type of the user as stated in above figure:

Kubernetes User – Install Kubectl on your laptop/desktop.

Tenant Administrator – Install CSE and configure CSE Client on a VM.

Service Provider – Install CSE , Install Messaging Bus , configure and register with vCloud Director.

In the Next series of posts i will be covering installation and configuration of CSE.


Whats new with VMware PKS v1.3

Last week VMware announced release of PKS 1.3 , which has some of the much awaited features  like enhance multi-cloud support, additional networking and security options, ease of management and operations. Few features i am going to discusses here:

Microsoft Azure support as IAAS

VMware PKS already support VMware vSphere , Google Cloud Platform and Amazon EC2 as supported platform for PKS deployment , in this new VMware PKS 1.3 release introduces support for Microsoft Azure. so now you can deploy production grade kubernetes from a single console to your choice of IAAS. Here is the list of features supported by PKS on different IAAS.


Kubernetes 1.12 Support

if you see kubernetes 1.12 release notes around 60+ enhancement and features has been introduced so it make all the sense to upgrade to Kubernetes 1.12.4.

Backup and Recovery of Kubernetes Clusters

This release supports backup and recovery of Kubernetes clusters when they are deployed in a single master mode. You can recover Kubernetes clusters and stateless workloads by using the BOSH Backup and Restore (BBR) toolset.

Smoke Tests

Smoke tests let you assess the impact of an upgrade before actually upgrading running clusters.The smoke tests create an ephemeral Kubernetes cluster after each upgrade of VMware PKS, but before applying upgrades to running Kubernetes clusters. This ensures that a test cluster can be provisioned and basic Kubernetes functionality validated with the upgraded software before applying the upgrade to the running clusters. Upon successful completion of the smoke test, the test cluster is deprovisioned to reduce resource consumption, and upgrades then proceed on the running clusters.


Support for Multiple Tier 0 and Selectable Tier 0 Routers

As you know NSX-T Tier 0 edges connects the physical and virtual networks. A single VMware NSX-T instance can support multiple Tier 0 routers. By deploying Kubernetes clusters across multiple Tier 0 routers service providers get better network isolation between tenants and additionally  service providers can use multiple Tier 0 routers which allows them to use overlapping IP address ranges, providing greater autonomy to tenants in choosing IP address ranges for their services.

With this VMware PKS 1.3 release, now provider/customer can specify a Tier 0 router using the network profile when you create a cluster (pks create cluster). The Kubernetes clusters and all networking objects that are created or configured as part of the cluster such as a load balancer, Tier 1 routers, and SNAT rules are created on this Tier 0 router. Given that a single Tier 0 router can support a finite set of such networking objects, use of multiple Tier 0 routers allows much greater scale.


Support for Larger Load Balancers

Previous versions of VMware PKS, we can only specify small or medium load balancers. now with VMware PKS 1.3 , it adds support for large load balancers. large load balancers provides higher scale in areas like number of services, number of backend pods per service, and throughput per service.

Routable CIDR blocks for Pod Networks

Routable IP addresses assigned to pods provide traceability of workloads making egress requests. also routable IP addresses provide direct ingress access to pods for some of the specialized workloads. With VMware PKS 1.3, at the time of Kubernetes cluster creation, you can specify whether you need the pods to be routable or non-routable (NAT’ed) by using the network profile.

Specific IP Address Range and Subnet Size for Pod IP Addresses

VMware PKS 1.3 allow you to override the global pod IP address block configured for VMware PKS with a custom IP address block range along with a custom subnet size. This feature helps in where your global IP address range for pods is reaching capacity and you need to deploy new Kubernetes clusters or you need a larger or smaller size subnet for each namespace being created within a cluster.

Multiple VMware PKS Control Planes across a Single NSX-T Instance

With this new release, multiple instances of VMware PKS can be deployed on a single shared NSX-T instance. Each instance of the VMware PKS control plane can be deployed on a dedicated NSX-T Tier 0 router to provide complete end-to-end isolation. With this feature, users can dedicate separate VMware PKS instances to their development, staging, and production environments or cloud provider can offer dedicated PKS as a Service to their customer.


Harbor 1.7

Harbor is an VMware’s contribution to open source community , Harbor is open source cloud native registry that stores, signs, and scans container images for vulnerabilities. Harbor solves common challenges by delivering trust, compliance, performance, and interoperability. with PKS 1.3 , Harbor 1.7 has been shipped and offers below enhancements like:

  • Support deploy Harbor with Helm Chart, enables the user to have high availability of Harbor services.
  • Support on-demand Garbage Collection, enables the admin to configure run docker registry garbage collection manually or automatically with a cron schedule.
  • Support Image Retag, enables the user to tag image to different repositories and projects, this is particularly useful in cases when images need to be retagged programmatically in a CI pipeline.
  • Support Image Build History, makes it easy to see the contents of a container image.
  • Improve user experience of Helm Chart Repository:
    • Chart searching included in the global search results
    • Show chart versions total number in the chart list
    • Mark labels to helm charts
    • The chart can be deleted by deleting all the versions under it

Monitoring with vRealize Operation Manager

With the integration of cAdvisor, vRops can be used to monitor entire cloud native infrastructure with the help of vRops Management Pack for Containers.


Sink resources include both pod logs as well as events from the Kubernetes API. These events are combined in a shared format that provides operators with a robust set of filtering and monitoring options. Now inbuilt Support for creating sink resources with the PKS Command Line Interface.

Workers Scale up and down

with this version kubernetes cluster’s worker node can easily be scaled up and down with a single command like:


These are the some of the important features which i like to share , for details feature list check Release note here.


How to Prepare for Certified Kubernetes Administration (CKA) Exam

Finally my last two months of preparation for the CKA exam is paid off , when i got this:


so after getting certified , i got lots of message from friends , colleagues around how to prepare for the exam , so this post is all about how to prepare for the exam , one of my friend in his blog (Blog Link) shared that this is manageable exam not as tough as people talk about and i totally agree with him that it is achievable with lots of practice and hard work on understanding the product.

About CKA Exam

The Certified Kubernetes Administrator (CKA) program was created by the Cloud Native Computing Foundation (CNCF), in collaboration with The Linux Foundation, to help develop the Kubernetes ecosystem. Kubernetes is one of the highest velocity open source projects and is exploding. CNCF offers a certification program that allows users to demonstrate their competence in a hands-on, command-line environment. The purpose of the Certified Kubernetes Administrator (CKA) program is to provide assurance that CKAs have the skills, knowledge, and competency to perform the responsibilities of Kubernetes administrators.

CKA is an online, proctored, performance-based test that requires solving multiple issues from a command line.this  certification focuses on the skills required to be a successful Kubernetes Administrator in industry today. This includes these general domains and their weights on the exam:

  • Application Lifecycle Management 8%
  • Installation, Configuration & Validation 12%
  • Kubernetes Core Concepts 19%
  • Kubernetes Networking 11%
  • Kubernetes Scheduling 5%
  • Kubernetes Security 12%
  • Kubernetes Cluster Maintenance 11%
  • Kubernetes Logging / Monitoring 5%
  • Kubernetes Storage 7%
  • Kubernetes Troubleshooting 10%

How to Prepare – My Way of Preparation 

First to prepare for the exam , ensure that you deploy a LAB , without LAB and practice you can not pass the exam no way!!. for Lab i followed below:

You need three virtual machines , so deploy a Lab with three nodes, can be easily setup on a laptop/desktop with virtulization software like VMware Workstation or Virtual Box etc..

  • Deploy Kubernetes using “Kubernetes the Hard Way” , this will help you understand communication between nodes and what all components makes kubernetes.
  • Understand Kubernetes Architecture Components in Detail.

you can also choose above or can deploy on one of your favourite public or private cloud environments. once Lab is ready , i would suggest kubernetes.io is the best resource for preparation of this exam, since in exam you are allowed to open “kubernetes.io” , so preparation from this website is going to help you in the exam also. I prepared using “kubernetes.io” website. all though i  would suggest that you follow each and every page of this website and here are few links on which i focused more for the exam:

Time management is the Key

time management is very important during this exam. CKA exam is  3 hour exam, its is very important to be careful and to pace yourself on the questions so as not get stuck on one question for too long. The exam environment (i.e. ssh session) runs in Google Chrome with a specific extension and can be laggy and slow at times. In addition, I noticed myself spending way too much time trying to select text and running into various UI issues , be prepared and have lots of practice in your labs. there is no way you can successfully clear the exam if not practicing a lot.

The day of the exam

Here are the few Tips for the Exam:

  • Before the exam, the examiner will ask you to clean your desk completely.

  • The place should be quiet because even if you work with headphones you will not be allowed to use them and for next three hours no body will be allowed in the room.

  • The examiner will ask you to see all the room, even under the desk.
  • The examiner will not talk to you by voice, only by chat. He/she will hear you because you will need to share the screen and micro phone.
  • The exam happens in a Chrome tab, the left side will show you the questions and the percentage marks of the questions.The right side is for the shell, I tried to use tmux  ( i would suggest not to use ) there, but it was pretty difficult inside a browser terminal. You can also have a popup with notes.
  • You can only open a tab with kubernetes.io and use its search box, no Google or anything .
  • It’s ok to request a brake, but be very careful because the time doesn’t stop.
  • You have three hours to finish the exam, if you get blocked it’s better to skip that question for now and retake it later.

So, that’s it from me. If you are interested in Kubernetes and you work with it go ahead and prepare for it and once you are certified , you will be proud on you because this certification really does carry the weight it implies and the real-world, live cluster examination is a nail-biter. i like the way CNCF measuring this competency using a Live Lab exam than a multiple choice exam. Best of Luck!!!