Assess Your Sovereign Cloud Stack for Compliance

Featured

Posted on April 26, 2023 by vmtechie

VMware vRealize (ARIA) Operations Compliance Pack for Sovereign Cloud is a management pack available in the VMware Marketplace. You can download and install this management pack on an instance of vRealize (ARIA) Operations to automatically assess a Sovereign Cloud stack for compliance.

VMware vRealize (ARIA) Operations Compliance Pack for Sovereign Cloud is intended to be used by the VMware Cloud Service Partners who are part of the Sovereign Cloud Initiative. The following products in the Sovereign Cloud stack are currently supported for compliance assessment:

vSphere
NSX-T
VMware Cloud Director
VMware Cloud Director Availability

For every Sovereign Cloud instance, providers need one instance of vRealize (ARIA) Operations with the VMware vRealize (ARIA) Operations Compliance Pack for Sovereign Cloud installed and configured. The compliance score card is available in the Optimize > Compliance screen of vRealize (ARIA) Operations.

Compliance Pack for Sovereign Cloud Controls Rules

The compliance rules are based on a checklist that VMware vRealize (ARIA) Operations Compliance Pack for Sovereign Cloud utilizes to monitor the products in the Sovereign Cloud stack. The checklist is based on the Sovereign Cloud Framework which takes into consideration the following key principles:

Data Sovereignty and Jurisdictional Control

Data should reside locally.
The cloud should be managed and governed locally, and all data processing including API calls should happen within the country/geography.
Data should be accessible only to residents of the same country, and the data should not be accessible under foreign laws or from any outside geography.

Data Access and Integrity

Two data center locations.
File, Block, and Object store options
Backup services, Disaster Recovery
Low-latency connectivity, Micro segmentation

Data Security and Compliance

Industry recognized Security Controls (minimum ISO/IEC 27001 or equivalent)

Additional relevant industry or governmental certifications

Third-party audits & Zero Trust Security & Encryption

Catalog of trusted images using the sovereign repository

Support for air gapped zones/regions

Operating personnel requirements and security clearance

Data Independence and Interoperability

Workload migration with bi-directional workload portability

Modern application architecture using containers

Support for hybrid cloud deployments

Control Rules and Product Control Set for vSphere

The vSphere control set is available for version 7, and 6.5/6.7 separately and details can be Here

Control Rules and product control set for nsx-t

Control rules and product control set for NSX-T and details can be found Here. The NSX-T version supported is greater than or equal to 3.2.x.

Controls Rules and Product Control Set for VMware Cloud Director

Control rules and product control set for VMware Cloud Director and details can be found Here. The supported version of the VMware Cloud Directory management pack is 8.10.2.

Control Rules and Product Control Set for VMware Cloud Director Availability

Controls rules and product control set for VMware Cloud Director Availability and details can be found Here. The version of the VMware Cloud Director Availability management pack supported is 1.2.1.

Install the VMware vRealize (ARIA) Operations Compliance Pack for Sovereign Cloud

The VMware vRealize (ARIA) Operations Compliance Pack for Sovereign Cloud consists of a PAK file that contains default contains views, reports, alerts and symptoms for the VMware software in the Sovereign Cloud stack.

Download the PAK file for VMware vRealize Operations Compliance Pack for Sovereign Cloud from the VMware Marketplace, and save the file to a temporary folder on your local system.

Log in to the vRealize (ARIA) Operations user interface with administrator privileges. Installation of this management pack is to be done by VMware Cloud Provider Program Partners.
In the left pane of vRealize (ARIA) Operations, click Integrations under Data Sources.
In the Repository tab, click the ADD button.

The Add Solution dialog box opens , Click BROWSE to locate the temporary folder on your system, and select the PAK file.
Read and select the checkboxes if required, Click Upload. The upload might take several minutes

Read and accept the EULA, and click Next. Installation details appear in the window during the process.
When the installation is completed, click Finish.

in the vROps instance, go to Optimize > Compliance. In the VMware Cloud tab, you can see the VMware Sovereign Cloud Compliance card in the VMware Sovereign Cloud Benchmarks section. Click Enable.

When you click Enable, a list of policies appears. You must select a policy that you want to apply. i am selecting default policy here

With the vRealize (ARIA) Operations reporting functions, you can generate a report to view the compliance status of your Sovereign Cloud. You can download the report in a PDF or CSV file format for future and offline needs.

Accessing Compliance Reports

In the VMware vRealize (ARIA) Operations Compliance Pack for Sovereign Cloud two kinds of reports are available:

At a VMware Cloud Provider Program Partner level – This report considers all the infrastructure level resources and generates a report at the org level, showing non-compliance. The compliance data is for the org and associated child hierarchy. Includes compliance details for org, org-vdc, virtual machines, logical switches, and logical routers as per the hierarchy.

From the left menu, click Visualize > Reports and run the report – [vCloud Director] – VMware Sovereign Cloud – Non-Compliance Report.

From the Reports panel, click Generated Reports, To select a generated report from the list, click the vertical ellipsis against the [vCloud Director] – VMware Sovereign Cloud – Non-Compliance Report report and select options such as run and delete.

At a tenant level – The VMware Cloud Provider Program Partner generates a filtered report that the tenant can access. This report is at an org-VDC level. The compliance data is for the child hierarchy only. Includes compliance details for virtual machines, logical switches, and logical routers as per the hierarchy and Tenants can view the generated reports in the VMware Chargeback console by logging in and navigating to Reports > Tenant Reports and clicking the Generated Reports tab.

Custom Benchmarks

In case the CSP partner/customer requires they can create a custom compliance benchmark to ensure that objects comply with compliance alerts available in vRealize (ARIA) Operations, or custom compliance alert definitions. When a compliance alert is triggered on your vCenter instance, hosts, virtual machines, distributed port groups, or distributed switches, you investigate the compliance violation. You can add up to five custom compliance scorecards

This is the first version of the sovereign cloud compliance pack for ARIA Operations for our Cloud Providers brings a comprehensive compliance checklist that encompasses the sovereign framework as a benchmark and continuously validates applications and infrastructure to help partners maintain their sovereignty posture. This brings extended capabilities to ARIA Operations, which, now addition to the capacity, cost, and performance monitoring, will monitor and report compliance drifts to the right stakeholders to better manage their compliance structure.

Cloud Director Container Service Extension – Tanzu Contour, Prometheus and Grafana Install Guide

Featured

Posted on February 15, 2023 by vmtechie

This post explains how to install and access Tanzu Contour, Promethous and Grafana on Tanzu clusters deployed by Cloud Director Container Service extension. so to get started first ensure TANZU CLI is installed on your local machine, if not then you can install by following documentation given Here

Next thing you need is the kubeconfig file of your target TKG cluster which is reachable from your local client machine on which you have installed Tanzu CLI, also make sure you run:

# tanzu init

Installation Steps

NOTE: CSE4 provisioned TKG cluster, cert-manager, kapp-controller, secretgren-controller and tanzu-standard package repository already have been installed. so you can skip step1,2 and 3.

Step:1- Install kapp-controller

kapp-controller gives us a flexible way to fetch, template, and deploy our applications to Kubernetes. It will also keep our apps continuously up to date when the configuration in our source repository changes. Install kapp-controller in the cluster using:

#kubectl apply -f https://github.com/vmware-tanzu/carvel-kapp-controller/releases/latest/download/release.yml

Step:2- Install secretgren-controller

In order to manage secrets across namespaces, Tanzu utilizes the carvel secret-gen-controller. you can install secretgren-controller in the cluster using:

#kubectl apply -f https://github.com/vmware-tanzu/carvel-secretgen-controller/releases/latest/download/release.yml

Step:3- Install cert-manager

Install cert-controller in the cluster using:

#kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml

Verify Tanzu Packages

Using the Tanzu CLI, you can install packages from the built-in tanzu-standard package repository or from other package repositories that you add to your target cluster, such as the Tanzu Application Platform Repository. Install tanzu-standard package repository v1.6.0.

#tanzu package repository add tanzu-standard --url projects.registry.vmware.com/tkg/packages/standard/repo:v1.6.0

Verify that the Prometheus package is available in your Tanzu K8s cluster as well as retrieve the version of the available package:

#tanzu package available list prometheus.tanzu.vmware.com -A

Verify that the Contour package is available in your Tanzu K8s cluster as well as retrieve the version of the available package:

#tanzu package available list contour.tanzu.vmware.com -A

verify that the Grafana package is available in your Tanzu K8s cluster as well as retrieve the version of the available package:

#tanzu package available list grafana.tanzu.vmware.com -A

Step4:- Implement Ingress Control with Contour

Contour is a Kubernetes ingress controller that uses the Envoy edge and service proxy. Tanzu Kubernetes Grid includes signed binaries for Contour and Envoy, which you can deploy into Tanzu Kubernetes (workload) clusters to provide ingress control services in those clusters.

You must first create the configuration file that will be used when you install the Contour package and then install the package. you can generate config file using:

#tanzu package available get contour.tanzu.vmware.com/1.20.2+vmware.1-tkg.1 --values-schema

#tanzu package available get contour.tanzu.vmware.com/PACKAGE-VERSION -generate-default-values-file

I am using below using data-values.yaml for contour

envoy:
  service:
    type: LoadBalancer
  hostPorts:
    enable: false
  hostNetwork: false
certificates:
  useCertManager: true

Install the package as below:

#tanzu package install contour --package-name contour.tanzu.vmware.com --version 1.20.2+vmware.1-tkg.1 --values-file contour-data-values.yaml

Step 5:- Deploy Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit. Tanzu Kubernetes Grid includes signed binaries for Prometheus that you can deploy on workload clusters to monitor cluster health and services.verify the configuration file using below commands, this file configures the Prometheus package.

#tanzu package available get prometheus.tanzu.vmware.com/2.36.2+vmware.1-tkg.1 --values-schema
#tanzu package available get prometheus.tanzu.vmware.com/PACKAGE-VERSION --generate-default-values-file

This command lists configuration parameters of the Grafana package and their default values. You can use the output to update your prometheus-data-values.yaml file, I have used below config file which is hosted on git, if you want you can download and use and in my config file ingress is enabled in the yaml which means it works with ingress.

https://raw.githubusercontent.com/avnish80/prometheus/main/prometheus-data-values.yaml

Install/update/delete prometheus pkg using below commands..

#tanzu package install prometheus --package-name prometheus.tanzu.vmware.com --version 2.36.2+vmware.1-tkg.1 --values-file prometheus-data-values.yaml

#tanzu package installed update prometheus --values-file prometheus-data-values.yaml
 
#tanzu package installed delete prometheus

Step 6:- Deploy Grafana

Grafana is open-source software that allows you to visualize and analyze metrics data collected by Prometheus on your clusters. Tanzu Kubernetes Grid includes a Grafana package that you can deploy on your Tanzu Kubernetes clusters. verify the configuration file, this file configures the Grafana package..

#tanzu package available get grafana.tanzu.vmware.com/7.5.16+vmware.1-tkg.1 --values-schema

##tanzu package available get grafana.tanzu.vmware.com/PACKAGE-VERSION --generate-default-values-file

This command lists configuration parameters of the Grafana package and their default values. You can use the output to update your grafana-data-values.yml file, I have used below config file which is hosted on git, if you want you can download and use and in my config file ingress is enabled in the yaml which means it works with ingress.

https://raw.githubusercontent.com/avnish80/grafana/main/grafana-data-values.yaml

#tanzu package install grafana --package-name grafana.tanzu.vmware.com --version 7.5.16+vmware.1-tkg.1 --values-file grafana-data-values.yaml
 
#tanzu package installed update grafana --values-file grafana-data-values.yaml
 
#tanzu package installed delete grafana

Access the Grafana Dashboard

After Grafana is deployed, the grafana package creates a Contour HTTPProxy object with a Fully Qualified Domain Name (FQDN) of grafana.system.tanzu. To use this FQDN to access the Grafana dashboard, Use the IP address of the LoadBalancer for the Envoy service in the tanzu-system-ingress namespace.

In case FQDN to access the Grafana dashboard does not work

Create an entry in your local /etc/hosts file that points an IP address to this FQDN:
Use the IP address of the LoadBalancer for the Envoy service in the tanzu-system-ingress namespace.
Navigate to https://grafana.system.tanzu.

Another issue is because the site uses self-signed certificates, you might need to navigate through a browser-specific security warning before you are able to access the dashboard.

Getting Started with VMware Cloud Director Container Service Extension 4.0

Featured

Posted on December 8, 2022 by vmtechie

VMware Cloud Director Container Service Extension brings Kubernetes as a service to VMware Cloud Director, offering multi-tenant, VMware supported, production ready, and compatible Kubernetes services with Tanzu Kubernetes Grid. As a service provider administrator, you can add the service to your existing VMware Cloud Director tenants. By using VMware Cloud Director Container Service Extension, customers can also use Tanzu products and services such as Tanzu^® Mission Control to manage their clusters.

Pre-requisite for Container Service Extension 4.0

Provider Specific Organization – Before you can configure VMware Cloud Director Container Service Extension server, it is must to create an organization to hosts VMware Cloud Director Container Service Extension server
Organization VDC within Organization – Container Service extension Appliance will be deployed in this organization virtual data center
Network connectivity – Network connectivity between the machine where VMware Cloud Director Container Service Extension is installed, and the VMware Cloud Director server. VMware Cloud Director Container Service Extension communicates with VMware Cloud Director using VMware Cloud Director public API endpoint
CSE 4.0 CPI automatically creates Load balancer, you must ensure that you have configured NSX Advanced Load Balancer, NSX Cloud, and NSX Advanced Load Balancer Service Engine Group for tenants who need to create Tanzu Kubernetes Cluster.

Provider Configuration

With the release of VMware Cloud Director Container Service Extension 4.0, service providers can use the CSE Management tab in the Kubernetes Container Clusters UI plug-in, which demonstrate step by step process to configure the VMware Cloud Director Container Service Extension server.

Install Kubernetes Container Clusters UI plug-in for VMware Cloud Director

You can download the Kubernetes Container Clusters UI plug-in for the VMware Cloud Director Download Page and upload the plug-in to VMware Cloud Director.

NOTE: If you have previously used the Kubernetes Container Clusters plug-in with VMware Cloud Director, it is necessary to deactivate it before you can activate a newer version, as only one version of the plug-in can operate at one time in VMware Cloud Director. Once you activate a new plug-in, it is necessary to refresh your Internet browser to begin using it.

Once partner has installed plugin, The Getting Started section with in CSE Management page help providers to learn and set up VMware Cloud Director Container Service Extension in VMware Cloud Director through the Kubernetes Container Clusters UI plug-in 4.0. At very High Level this is Six Step process:

Lets start following these steps and deploy

Step:1 – This section links to the locations where providers can download the following two types of OVA files that are necessary for VMware Cloud Director Container Service Extension configuration:

NOTE- Do not download FIPS enabled templates

Step:2 – Create a catalog in VMware Cloud Director and upload VMware Cloud Director Container Service Extension OVA files that you downloaded in the step:1 into this catalog

Step:3 – This section initiates the VMware Cloud Director Container Service Extension server configuration process. In this process, you can enter details such as software versions, proxy information, and syslog location. This workflow automatically creates a Kubernetes Clusters rights bundle, CSE Admin Role role, Kubernetes Cluster Author role, and VM sizing policies. In this process, the Kubernetes Clusters rights bundle and Kubernetes Cluster Author role are automatically published to all tenants as well as following Kubernetes resource versions will be deployed

Kubernetes Resources	Supported Versions
Cloud Provider Interface (CPI)	1.2.0
Container Storage Interface (CSI)	1.3.0
CAPVCD	1.0.0

Step:4 – This section links to the Organization VDCs section in VMware Cloud Director, where you can assign VM sizing policies to customer organization VDCs. To avoid resource limit errors in clusters, it is necessary to add Tanzu Kubernetes Grid VM sizing policies to organization virtual data centers.The Tanzu Kubernetes Grid VM sizing policies are automatically created in the previous step. Policies created are as below:

Sizing Policy	Description	Values
TKG small	Small VM sizing policy	2 CPU, 4 GB memory
TKG medium	Medium VM sizing policy	2 CPU, 8 GB memory
TKG large	Large VM sizing policy	4 CPU, 16 GB memory
TKG extra-large	X-large VM sizing policy	8 CPU, 32 GB memory

NOTE: Providers can create more policies manually based on requirement and publish to tenants

In VMware Cloud Director UI, select an organization VDC, and from the left panel, under Policies, select VM Sizing and Click Add and then from the data grid, select the Tanzu Kubernetes Grid sizing policy you want to add to the organization, and click OK.

Step:5 – This section links to the Users section in VMware Cloud Director, where you can create a user with the CSE Admin Role role. This role grants administration privileges to the user for VMware Cloud Director Container Service Extension administrative purposes. You can use this user account as OVA deployment parameters when you start the VMware Cloud Director Container Service Extension server.

Step:6 – This section links to the vApps section in VMware Cloud Director where you can create a vApp from the uploaded VMware Cloud Director Container Service Extension server OVA file to start the VMware Cloud Director Container Service Extension server.

Create a vApp from VMware Cloud Director Container Service Extension server OVA file.
Configure the VMware Cloud Director Container Service Extension server vApp deployment lease
Power on the VMware Cloud Director Container Service Extension server.

Container Service Extension OVA deployment

Enter a vApp name, optionally a description, runtime lease and storage lease (should be no lease so that it does not suspend automatically), and click Next.

Select a virtual data center and review the default configuration for resources, compute policies, hardware, networking, and edit details where necessary.

In the Custom Properties window, configure the following settings:
- VCD host: VMware Cloud Director URL
- CSE service account’s username The username: CSE Admin user in the organization
- CSE service account’s API Token: to generate API Token, login to provider session with CSE user which you created in Step:5 and then go to “User Preferences” and click on “NEW” in “ACCESS Tokens” section (When you generate an API Access Token, you must copy the token, because it appears only once. After you click OK, you cannot retrieve this token again, you can only revoke it. )

CSE service account’s org: The organization that the user with the CSE Admin role belongs to, and that the VMware Cloud Director Container Service Extension server deploys to.
CSE service vApp’s org: Name of the provider org where CSE app will be deployed

In the Virtual Applications tab, in the bottom left of the vApp, click Actions > Power > Start. this completes the vApp creation from the VMware Cloud Director Container Service Extension server OVA file. This task is the final step for service providers to perform before the VMware Cloud Director Container Service Extension server can operate and start provisioning Tanzu Kubernetes Clusters.

CSE 4.0 is packed with capabilities that address and attract developer personas with an improved feature set and simplified cluster lifecycle management. Users can now build and upgrade versions, resize, and delete K8s clusters directly from the UI making it simpler and faster to accomplish tasks than before. This completes provider section of Container Service extension in next blog post i will write about Tenant workflow.

VMware Cloud Director Charge Back Explained

Featured

Posted on November 22, 2022 by vmtechie

VMware Chargeback not only enables metering and chargeback capabilities, but also provides visibility into infrastructure usage through performance and capacity dashboards for the Cloiud Providers as well as tenants.

To help Cloud Providers and tenants realise more value for every dollar they spend on infrastructure (ROI) (and in turn provide similar value to their tenants), our focus is to not only expand the coverage of services that can be priced in VMware Chargeback, but also to provide visibility into the cost of infrastructure to providers, and billing summary to organizations, clearly highlighting the cost incurred by various business units. but before we dive in further to know what’s new with this release, please note:

vRealize Operations Tenant App is now rebranded to VMware Chargeback.
VMware Chargeback is also now available as a SaaS offering, The Software-as-a-Service (SaaS) offering will be available as early access, with limited availability, with the purchase or trial of the VMware Cloud Director™ service. See, Announcing VMware Chargeback for Managed Service Providers Blog.

Creation of pricing policy based on chargeback strategy

Provider administrator can create one or more pricing policies based on how they want to chargeback their tenants. Based on the vCloud Director allocation models, each pricing policy is of the type, Allocation pool, Reservation pool, or Pay-As-You-Go

NOTE – The pricing policies apply to VMs at a minimum granularity of five minutes. The VMs that are created and deleted within the short span of five minutes will still be charged.

CPU Rate

Provider can charge the CPU rate based on GHz or vCPU Counts

Charge Period which indicates the frequency of charging and values are: Hourly, Daily Monthly
Charge Based on Power State indicates the pricing model based on which the charges are applied and values are: Always, Only when powered on, Powered on at least once
Default Base Rate any base rate that provider want to charge
Add Slab providers can optionally charge different rates depending on the number of vCPUs used
Fixed Cost Fixed costs do not depend on the units of charging

Memory Rate

Charge Period which indicates the frequency of charging and values are: Hourly, Daily Monthly
Charge Based on indicates the pricing model based on which the charge is applied, values are: Usage, Allocation and Maximum from usage and allocation
Charge Based on Power State indicates the pricing model based on which the charges are applied and values are: Always, Only when powered on, Powered on at least once
Default Base Rate any base rate that provider want to charge
Add Slab providers can optionally charge different rates depending on the memory allocated
Fixed Cost Fixed costs do not depend on the units of charging

Storage Rate

You can charge for storage either based on storage policies or independent of it.

This way of setting rates will be deprecated in the future release and it is advisable to instead use the Storage Policy option.

Select the Storage Policy Name from the drop-down menu.
Charge Period indicates the frequency of charging and values are: Hourly, Daily Monthly
Charge Based on indicates the pricing model based on which the charge is applied. You can charge for used storage or configured storage of the VMs
Charge Based on Power State This decides if the charge should be applied based on the power state of the VM and values are: Always, Only when powered on, Powered on at least once
Add Slab you can optionally charge different rates depending on the storage allocated

Network Rate

Enter the External Network Transmit and External Network Receive rates.

Note: If your network is backed by NSX-T, you will be charged only for the network data transmit and network data receive.

Network Transmit Rate select the Change Period and enter the Default Base Rate as well as using slabs, you can optionally charge different rates depending on the network data consumed
Network Receive Rate select the Change Period and enter the Default Base Rate. as well as using slabs, you can optionally charge different rates depending on the network data consumed. Enter valid numbers for Base Rate Slab and click Add Slab.

Advanced Network Rate

Under Edge Gateway Size, enter the base rates for the corresponding edge gateway sizes

Charge Period indicates the frequency of charging and values are: Hourly, Daily Monthly
Enter the Base Rate

Guest OS Rate

Use the Guest OS Rate to charge differently for different operating systems

Enter the Guest OS Name
Charge Period indicates the frequency of charging and values are: Hourly, Daily Monthly
Charge Based on Power State This decides if the charge should be applied based on the power state of the VM and values are: Always, Only when powered on, Powered on at least once
Enter the Base Rate

Cloud Director Availability

Cloud Director Availability is to set pricing for replications created from Cloud Director Availability

Replication SLA Profile – enter a replication policy name
Charge Period indicates the frequency of charging and values are: Hourly, Daily Monthly
Enter the Base Rate

You can also charge for the storage consumed by replication objects in the Storage Usage Charge section.This is used to set additional pricing for storage used by Cloud Director Availability replications in Cloud Director. Please note that the storage usage defined in this tab will be added additionally to the Storage Policy Base Rate

vCenter Tag Rate

This section is used for Any additional charges to be applied on the VMs based on their discovered Tags from vCenter. (Typical examples are Antivirus=true, SpecialSupport=true etc)

Enter the Tag Category and Tag Value
Charge based on Fixed Rate or
Charge based on Alternate Pricing Policy – Select the appropriate Pricing Policy
Charge Period indicates the frequency of charging and values are: Hourly, Daily Monthly
Charge Based on Power State This decides if the charge should be applied based on the power state of the VM and values are: Always, Only when powered on, Powered on at least once
Enter the Base Rate

VCD Metadata Rate

Use the VCD Metadata Rate to charge differently for different metadata set on vApps

NOTE- Metadata based prices are available in bills only if Enable Metadata option is enabled in vRealize Operations Management Pack for VMware Cloud Director.

Enter the Tag Category and Tag Value
Charge based on Fixed Rate or
Charge based on Alternate Pricing Policy – Select the appropriate Pricing Policy
Charge Period indicates the frequency of charging and values are: Hourly, Daily Monthly
Charge Based on Power State This decides if the charge should be applied based on the power state of the VM and values are: Always, Only when powered on, Powered on at least once
Enter the Base Rate

One Time Fixed Cost

One time fixed cost used to charge for One time incidental charges on Virtual machines, such as creation/Setup charges, or charges for one off incidents like installation of a patch. These costs do not repeat on a recurring basis.

For values follow VCD METADATA and vCenter Tag section.

Rate Factors

Rate factors are used to either bump up or discount the prices either against individual resources consumed by the Virtual Machines, or whole charges against the Virtual Machine. Some examples are:

Increase CPU rate by 20% (Factor 1.2) for all VMs tagged with CPUOptimized=true
Discount overall charge on VM by 50% (Factor 0.5) for all Vms tagged with PromotionalVM=True

VCD Metadata
- enter the Tag Key and Tag Value
  - Change the price of – Total, vCPU, Memory and Storage
  - By applying a factor of – increase or decrease the price by entering a valid number
vCenter Tag
- enter the Tag Key and Tag Value
  - Change the price of – Total, vCPU, Memory and Storage
  - By applying a factor of – increase or decrease the price by entering a valid number

Tanzu Kubernetes Clusters

This section will be used to charge for Tanzu K8s clusters and objects.

Cluster Fixed Cost
Charge Period indicates the frequency of charging and values are: Hourly, Daily Monthly
- Fixed Cost Fixed costs do not depend on the units of charging
Cluster CPU Rate
- Charge Period indicates the frequency of charging and values are: Hourly, Daily Monthly
- Charge Based on this decides if the charge should be applied based on Usage or Allocation
- Default Base Rate(per ghz)
Cluster Memory Rate
- Charge Period indicates the frequency of charging and values are: Hourly, Daily Monthly
- Charge Based on this decides if the charge should be applied based on Usage or Allocation
- Default Base Rate(per gb)

Additional Fixed Cost

You can use Additional Fixed Cost section to charge at the Org-VDC level. You can use this for charges such as overall tax, overall discounts, and so on. The charges can be applied to selective Org-VDCs based on Org-VDC metadata.

Fixed Cost
- Charge Period indicates the frequency of charging and values are: Hourly, Daily Monthly
- Fixed Cost
VCD Metadata – enter the Tag Key and Tag Value
VCD Metadata One Time – enter the Tag Key and Tag Value

Apply Policy

Cloud Director Charge Back provides flexibility to the Service Providers to map the created pricing policies with specific organization vDC. By doing this, the service provider can holistically define how each of their customers can be charged based on resource types.

Bills

Every tenant/customer of service provider can see/review their bills using the Cloud Director Charge Back app. Service Provider administrator can generate bills for a tenant by selecting a specific resource and a pricing policy that must be applied for a defined period and can also log in to review the bill details.

This completes the feature demonstration available with Cloud Director Charge back. Go ahead and deploy and add native charge back power to your Cloud.

NFS DataStore on VMware Cloud on AWS using Amazon FSx for NetApp

Featured

Posted on September 15, 2022 by vmtechie

Amazon FSx for NetApp ONTAP integration with VMware Cloud on AWS is an AWS-managed external NFS datastore built on NetApp’s ONTAP file system that can be attached to a cluster in your SDDC. It provides customers with flexible, high-performance virtualized storage infrastructure that scales independently of compute resources.

PROCESS

Make sure SDDC has been deployed on VMware Cloud on AWS with version 1.20
The SDDC is added to an SDDC Group. While creating the SDDC Group, a VMware Managed Transit Gateway (vTGW) is automatically deployed and configured
A Multi-AZ file system powered by Amazon FSx for NetApp ONTAP is deployed across two AWS Availability Zones (AZs). (You can also deploy in single AZ but not recommended for production)

DEPLOY VMWARE MANAGED TRANSIT GATEWAY

To use FSx for ONTAP as an external datastore, an SDDC must be a member of an SDDC group so that it can use the group’s vTGW and to configure you must be logged into the VMC console as a user with a VMC service role of Administrator and follow below steps:

Log in to the VMC Console and go on the Inventory page, click SDDC Groups
On the SDDC Groups tab, click ACTIONS and select Create SDDC Group
Give the group a Name and optional Description, then click NEXT
On the Membership grid, select the SDDCs to include as group members.The grid displays a list of all SDDCs in your organization. To qualify for membership in the group, an SDDC must meet several criteria:
- It must be at SDDC version 1.11 or later. Members of a multi-region group must be at SDDC version 1.15 or later.
- Its management network CIDR block cannot overlap the management CIDR block of any other group member.
- It cannot be a member of another SDDC Group.
When you have finished selecting members, click NEXT. You can edit the group later to add or remove members.
Acknowledge that you understand and take responsibility for the costs you incur when you create an SDDC group, then click CREATE GROUP to create the SDDC Group and its VMware Transit Connect network.

ATTACH VPC TO VMWARE MANAGED TRANSIT GATEWAY

After the SDDC Group is created, it shows up in your list of SDDC Groups. Select the SDDC Group, and then go to the External VPC tab and click on ADD ACCOUNT button, then provide the AWS account that will be used to provision the FSx file system, and then click Add.

Now it’s time for you to go back to the AWS console and sign in to the same AWS account where you will create Amazon FSx file system. Here navigate to the Resource Access Manager service page and

click on the Accept resource share button.

Next, we need to attach VMC Transit Gateway to the FSX VPC, for that you need to go to:

ATTACH VMWARE MANAGED TRANSIT GATEWAY TO VPC

Open the Amazon VPC console and navigate to Transit Gateway Attachments.
Choose Create transit gateway attachment
For Name tag, optionally enter a name for the transit gateway attachment.
For Transit gateway ID, choose the transit gateway for the attachment, make sure you choose a transit gateway that was shared with you.
For Attachment type, choose VPC.
For VPC ID, choose the VPC to attach to the transit gateway.This VPC must have at least one subnet associated with it.
For Subnet IDs, select one subnet for each Availability Zone to be used by the transit gateway to route traffic. You must select at least one subnet. You can select only one subnet per Availability Zone.
Choose Create transit gateway attachment.

Accept the Transit Gateway attachment as follows:

Navigating back to the SDDC Group, External VPC tab, select the AWS account ID used for creating your FSx NetApp ONTAP, and click Accept. This process takes some time..

Next, you need to add the routes so that the SDDC can see the FSx file system. This is done on the same External VPC tab, where you will find a table with the VPC. In that table, there is a button called Add Routes. In the Add Route section, add the CIDR of your VPC where the FSX will be deployed.

In the AWS console, create the route back to the SDDC by locating VPC on the VPC service page and navigating to the Route Table as seen below.

also ensure that you have the correct inbound rules for the SDDC Group CIDR to allow the inbound rules for SDDC Group CIDR. it this case i am using entire SDDC CIDR, Further to this Security Group, the ENI Security Group also needs the NFS port ranges adding as inbound and outbound rules to allow communication between VMware Cloud on AWS and the FSx service.

Deploy FSx for NetApp ONTAP file system in your AWS account

Next step is to create an FSx for NetApp ONTAP file system in your AWS account. To connect FSx to VMware cloud on AWS SDDC, we have two options:

Either create a new Amazon VPC under the same connected AWS account and connect it using VMware Transit Connect.
or Create a new AWS account in the same region as well as VPC, connect it using VMware Transit Connect.

In this blog, i am deploying in the same connected VPC and for it to deploy, Go to Amazon FSx service page, click on Create File System and on the Select file system type page, select Amazon FSx for NetApp ONTAP,

On Next page, select the Standard create method and enter require details like:

Select Deployment type (Multi-AZ) and Storage capacity
Select correct VPC, Security group and Subnet

After the file system is created, check the NFS IP address under the Storage virtual machines tab. The NFS IP address is the floating IP that is used to manage access between file system nodes, and this IP we will use to configuring in VMware Transit Connect to allow access volume from SDDC.

we are done with creating the FSx for NetApp ONTAP file system.

MOUNT NFS EXTERNAL STORAGE TO SDDC Cluster

Now it’s time for you to go back to the VMware Cloud on AWS console and open the Storage tab of your SDDC. Click ATTACH DATASTORE and fill in the required values.

Select a cluster. Cluster-1 is preselected if there are no other clusters.
Choose Attach a new datastore
The NFS IP address shown in the Endpoints section of the FSx Storage Virtual Machine tab. Click VALIDATE to validate the address and retrieve the list of mount points (NFS exports) from the server.

Pick one from the list of mount points exported by the server at the NFS server address. Each mount point must be added as a separate datastore
AWS FSx ONTAP
Give the datastore a name. Datastore names must be unique within an SDDC.
- Click on ATTACH DATASTORE

VMware Cloud on AWS supports external storage starting with SDDC version 1.20. To request an upgrade to an existing SDDC, please contact VMware support or notify your Customer Success Manager.

Cross-Cloud Disaster Recovery with VMware Cloud on AWS and Azure VMware Solution

Featured

Posted on August 23, 2022 by vmtechie

Disaster Recovery is an important aspect of any cloud deployment. It is always possible that an entire cloud data center or region of the cloud provider goes down. This has already happened to most cloud providers like Amazon AWS, Microsoft Azure, Google Cloud and will surely happen again in future. Cloud providers like Amazon AWS, Microsoft Azure and Google Cloud will readily suggest that you have a Disaster Recovery and Business Continuity strategy that spans across multiple regions, so that if a single geographic region goes down, business can continue to operate from another region. This only sounds good in theory, but there are several issues in the methodology of using the another region of a single cloud provider. Some of the key reasons which I think that single cloud provider’s Cross-Region DR will not be that effective.

A single Cloud Region failure might cause huge capacity issues for other regions used as DR
Cloud regions are not fully independent , like AWS RDS allows read replicas in other regions but one wrong entry will get replicated across read replicas which breaks the notion of “Cloud regions are independent“
Data is better protected from accidental deletions when stored across clouds. For Example what if any malicious code or an employee or cloud providers employee runs a script which deletes all the data but in most cases this will not impact cross cloud.

In this blog post we will see how VMware cross cloud disaster recovery solution can help customers/partners to overcome BC/DR challenges.

Deployment Architecture

Here is my deployment architecture and connectivity:

One VMware Cloud on AWS SDDC
One Azure VMware Solution SDDC
Both SDDC’s are connected over MegaPort MCR

Activate VMware Site Recovery on VMware Cloud on AWS

To configure site recovery on VMware Cloud on AWS SDDC, go to SDDC page, click on the Add Ons tab and under the Site Recovery Add On, Click the ACTIVATE button

In the pop up window Click ACTIVATE again

This will deploy SRM on SDDC, wait for it to finish.

Deploy VMware Site Recovery Manager on Azure VMware Solution

In your Azure VMware Solution private cloud, under Manage, select Add-ons > Disaster recovery and click on “Get Started”

From the Disaster Recovery Solution drop-down, select VMware Site Recovery Manager (SRM) and provide the License key, select agree with terms and conditions, and then select Install

After the SRM appliance installs successfully, you’ll need to install the vSphere Replication appliances. Each replication server accommodates up to 200 protected VMs. Scale in or scale out as per your needs.

Move the vSphere server slider to indicate the number of replication servers you want based on the number of VMs to be protected. Then select Install

Once installed, verify that both SRM and the vSphere Replication appliances are installed.After installing VMware SRM and vSphere Replication, you need to complete the configuration and site pairing in vCenter Server.

Sign in to vCenter Server as cloudadmin@vsphere.local.
Navigate to Site Recovery, check the status of both vSphere Replication and VMware SRM, and then select OPEN Site Recovery to launch the client.

Configure site pairing in vCenter Server

Before starting site pair, make sure firewall rules between VMware cloud on AWS and Azure VMware solution has been opened as described Here and Here

To start pairing select NEW SITE PAIR in the Site Recovery (SR) client in the new tab that opens.

Enter the remote site details, and then select FIND VCENTER SERVER INSTANCES and select then select Remote vCenter and click on NEXT, At this point, the client should discover the VRM and SRM appliances on both sides as services to pair.

Select the appliances to pair and then select NEXT.

Review the settings and then select FINISH. If successful, the client displays another panel for the pairing. However, if unsuccessful, an alarm will be reported.

After you’ve created the site pairing, you can now view the site pairs and other related details as well as you are ready to plan for Disaster Recovery.

Planning

Mappings allow you to specify how Site Recovery Manager maps virtual machine resources on the protected site to resources on the recovery site, You can configure site-wide mappings to map objects in the vCenter Server inventory on the protected site to corresponding objects in the vCenter Server inventory on the recovery site.

Network Mapping
IP Customization
Folder Mapping
Resource Mapping
Storage Policy Mapping
Placeholder Datastores

Creating Protection Groups

A protection group is a collection of virtual machines that the Site Recovery Manager protects together. Protection group are per SDDC configuration and needs to be created on each SDDC if VMs are replicated in bi-directionally.

Recovery Plan

A recovery plan is like an automated run book. It controls every step of the recovery process, including the order in which Site Recovery Manager powers on and powers off virtual machines, the network addresses that recovered virtual machines use, and so on. Recovery plans are flexible and customizable.

A recovery plan runs a series of steps that must be performed in a specific order for a given workflow such as a planned migration or re-protection. You cannot change the order or purpose of the steps, but you can insert your own steps that display messages and run commands.

A recovery plan includes one or more protection groups. Conversely, you can include a protection group in more than one recovery plan. For example, you can create one recovery plan to handle a planned migration of services from the protected site to the recovery site for the whole SDDC and another set of plans per individual departments. Thus, having multiple recovery plans referencing one protection group allows you to decide how to perform recovery.

Steps to add a VM for replication:

there are multiple ways, i am explaining here one:

Choose VM and right click on it and select All Site Recovery actions and click on Configure Replication
Choose Target site and replication server to handle replication
VM validation happens and then choose Target datastore
under Replication setting , choose RPO, point in time instances etc..
Choose protection group to which you want to add this VM and check summary and click Finish

Cross-cloud disaster recovery ensures one of the most secure and reliable solutions for service availability, reason cross-cloud disaster recovery is often the best route for businesses is that it provides IT resilience and business continuity. This continuity is of most important when considering how companies operate, how customers and clients rely on them for continuous service and when looking at your company’s critical data, which you do not want to be exposed or compromised.

Frankly speaking IT disasters happen and happens everywhere including public clouds and much more frequently than you might think. When they occur, they present stressful situations which require fast action. Even with a strategic method for addressing these occurrences in place, it can seem to spin out of control. Even when posed with these situations, IT leaders must keep face, remain calm and be able to fully rely on the system they have in place or partner they are working with for disaster recovery measures.

Customer/Partner with VMware Cloud on AWS and Azure VMware Solution can build cross cloud disaster recovery solution to simplify disaster recovery with the only VMware-integrated solution that runs on any cloud. VMware Site Recovery Manager (SRM) provides policy-based management, minimizes downtime in case of disasters via automated orchestration, and enables non-disruptive testing of your disaster recovery plans.

Persistent Volumes for Tanzu on VMware Cloud on AWS using Amazon FSx for NetApp ONTAP

Featured

Posted on June 23, 2022 by vmtechie

Amazon FSx for NetApp ONTAP provides fully managed shared storage in the AWS Cloud with the popular data access and management capabilities of ONTAP and this blog post we are going to use these volumes mount as Persistent Volumes on Tanzu Kubernetes Clusters running on VMware Cloud on AWS

With Amazon FSx for NetApp ONTAP, you pay only for the resources you use. There are no minimum fees or set-up charges. There are five Amazon FSx for NetApp ONTAP components to consider when storing and managing your data: SSD storage, SSD IOPS, capacity pool usage, throughput capacity, and backups.

The Amazon FSx console has two options for creating a file system – Quick create option and Standard create option. To rapidly and easily create an Amazon FSx for NetApp ONTAP file system with the service recommended configuration, I use the Quick create option.

The Quick create option creates a file system with a single storage virtual machine (SVM) and one volume. The Quick create option configures this file system to allow data access from Linux instances over the Network File System (NFS) protocol.

In the Quick configuration section, for File system name – optional, enter a name for your file system.

For Deployment type choose Multi-AZ or Single-AZ.

Multi-AZ file systems replicate your data and support failover across multiple Availablity Zones in the same AWS Region.
Single-AZ file systems replicate your data and offer automatic failover within a single Availability Zone, for this post i am creating in Single AZ
SSD storage capacity, specify the storage capacity of your file system, in gibibytes (GiBs). Enter any whole number in the range of 1,024–196,608.
For Virtual Private Cloud (VPC), choose the Amazon VPC that is associate with your VMware Cloud on AWS SDDC.

Review the file system configuration shown on the Create ONTAP file system page. For your reference, note which file system settings you can modify after the file system is created.

Choose Create file system.

Quick create creates a file system with one SVM (named fsx) and one volume (named vol1). The volume has a junction path of /vol1 and a capacity pool tiering policy of Auto.

For us to use this SVM, we need to get the IP address of SVM for NFS , Click on SVM ID and take a note of this IP, we will use this IP in our NFS configurations for Tanzu.

Kubernetes NFS-Client Provisioner

NFS subdir external provisioner is an automatic provisioner that use your existing and already configured NFS server to support dynamic provisioning of Kubernetes Persistent Volumes via Persistent Volume Claims. Persistent volumes are provisioned as ${namespace}-${pvcName}-${pvName}.

More Details – Explained here in detail https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner

I am deploying this on my Tanzu Kubernetes cluster which is deployed on VMware Cloud on AWS.

Add the helm repo –

#helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

Install using as below:

#helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
    --set nfs.server=<IP address of Service> \
    --set nfs.path=/<Volume Name>
#My command will be like this#
#helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
    --set nfs.server=172.31.1.234 \
    --set nfs.path=/vol1

Post installation of chart, you can check the status of Pod, it is not in running state then describe and see where it stuck

Finally, Test Your Environment!

Now we’ll test your NFS subdir external provisioner by creating a persistent volume claim and a pod that writes a test file to the volume. This will make sure that the provisioner is provisioning and that the Amazon FSx for NetApp ONTAP service is reachable and writable.

As you can see deployed application created an PV and PVC successfully on Amazon FSx for NetApp ONTAP

Describe the Persistent Volume to see the source of it, as you can see below it is created on NFS running on SVM having IP – 172.31.1.234

This is the power of VMware Cloud on AWS and AWS native services, customers can use any AWS native service without worrying about egress charges as well as security as everything is being configured and accessed over the private connections.

Building Windows Custom Machine Image for Creating Tanzu Workload Clusters

Featured

Posted on June 17, 2022 by vmtechie

If your organisation is building an application based on Windows components (such as .NET Framework) and willing to deploy Windows containers on VMware Tanzu, this blog post is on how to build a Windows custom machine image and deploy windows Kubernetes cluster.

Windows Image Prerequisites

vSphere 6.7 Update 3 or greater
A macOS or Linux workstation, Docker Desktop and Ansible must be installed on workstation
Tanzu Kubernetes Grid v1.5.x or greater
Tanzu CLI
A Recent Image of Windows 2019 (newer than April 2021) and must be downloaded from Microsoft Developer Network (MSDN) or Volume Licensing (VL) account.
The latest VMware Tools Windows ISO image. Download from VMware Tools
on vCenter, Inside a data store create a folder such as iso and upload windows ISO and VMware Tools iso

Build a Windows Image

Deploy Tanzu Management Cluster with Ubuntu 2004 Kubernetes v1.22.9 OVA
Create a YAML file named builder.yaml with the following configuration, On my local system I have saved this yaml as builder.yaml

apiVersion: v1
kind: Namespace
metadata:
 name: imagebuilder
---
apiVersion: v1
kind: Service
metadata:
 name: imagebuilder-wrs
 namespace: imagebuilder
spec:
 selector:
   app: image-builder-resource-kit
 type: NodePort
 ports:
 - port: 3000
   targetPort: 3000
   nodePort: 30008
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: image-builder-resource-kit
 namespace: imagebuilder
spec:
 selector:
   matchLabels:
     app: image-builder-resource-kit
 template:
   metadata:
     labels:
       app: image-builder-resource-kit
   spec:
     nodeSelector:
       kubernetes.io/os: linux
     containers:
     - name: windows-imagebuilder-resourcekit
       image: projects.registry.vmware.com/tkg/windows-resource-bundle:v1.22.9_vmware.1-tkg.1
       imagePullPolicy: Always
       ports:
         - containerPort: 3000

Connect the Kubernetes CLI to your management cluster by running:

#kubectl config use-context MY-MGMT-CLUSTER-admin@MY-MGMT-CLUSTER

Apply the builder.yaml file as below:

To ensure the container is running run below command:

List the cluster’s nodes, with wide output and take note of Internal IP address value of the node with ROLE listed as control-plane,master

#kubectl get nodes -o wide

Retrieve the containerd component’s URL and SHA, Query the control plane’s nodePort endpoint:

#curl http://CONTROLPLANENODE-IP:30008

Take note of containerd.path and containerd.sha256 values. The containerd.path value ends with something like containerd/cri-containerd-v1.5.9+vmware.2.windows-amd64.tar.

Create a JSON file in an empty folder named windows.json with the following configuration:

{
 "unattend_timezone": "WINDOWS-TIMEZONE",
 "windows_updates_categories": "CriticalUpdates SecurityUpdates UpdateRollups",
 "windows_updates_kbs": "",
 "kubernetes_semver": "v1.22.9",
 "cluster": "VSPHERE-CLUSTER-NAME",
 "template": "",
 "password": "VCENTER-PASSWORD",
 "folder": "",
 "runtime": "containerd",
 "username": "VCENTER-USERNAME",
 "datastore": "DATASTORE-NAME",
 "datacenter": "DATACENTER-NAME",
 "convert_to_template": "true",
 "vmtools_iso_path": "VMTOOLS-ISO-PATH",
 "insecure_connection": "true",
 "disable_hypervisor": "false",
 "network": "NETWORK",
 "linked_clone": "false",
 "os_iso_path": "OS-ISO-PATH",
 "resource_pool": "",
 "vcenter_server": "VCENTER-IP",
 "create_snapshot": "false",
 "netbios_host_name_compatibility": "false",
 "kubernetes_base_url": "http://CONTROLPLANE-IP:30008/files/kubernetes/",
 "containerd_url": "CONTAINERD-URL",
 "containerd_sha256_windows": "CONTAINERD-SHA",
 "pause_image": "mcr.microsoft.com/oss/kubernetes/pause:3.5",
 "prepull": "false",
 "additional_prepull_images": "mcr.microsoft.com/windows/servercore:ltsc2019",
 "additional_download_files": "",
 "additional_executables": "true",
 "additional_executables_destination_path": "c:/k/antrea/",
 "additional_executables_list": "http://CONTROLPLANE-IP:30008/files/antrea-windows/antrea-windows-advanced.zip",
 "load_additional_components": "true"
}

update the values in file as below:

Add the XML file that contains the Windows settings by following these steps:

Go to the autounattend.xml file on VMware {code} Sample Exchange.
Select Download.
If you are using the Windows Server 2019 evaluation version, remove <ProductKey>...</ProductKey>.
Name the file autounattend.xml.
Save the file in the same folder as the windows.json file and change permission of file to 777.

From your client VM run following command from folder containing your windows.json and autounattend.xml file:

#docker run -it --rm --mount type=bind,source=$(pwd)/windows.json,target=/windows.json --mount type=bind,source=$(pwd)/autounattend.xml,target=/home/imagebuilder/packer/ova/windows/windows-2019/autounattend.xml -e PACKER_VAR_FILES="/windows.json" -e IB_OVFTOOL=1 -e IB_OVFTOOL_ARGS='--skipManifestCheck' -e PACKER_FLAGS='-force -on-error=ask' -t projects.registry.vmware.com/tkg/image-builder:v0.1.11_vmware.3 build-node-ova-vsphere-windows-2019

NOTE: Before you run below command, make sure your workstation is running “Docker Desktop” as well “Ansible”

To ensure the Windows image is ready to use, select your host or cluster in vCenter, select the VMs tab, then select VM Templates to see the Windows image listed.

Use a Windows Image for a Workload Cluster

Use a Windows Image for a Workload Cluster, below yaml shows you how to deploy a workload cluster that uses your Windows image as a template. (This windows cluster is using NSX Advance LB)

#! ---------------------------------------------------------------------
#! non proxy env configs
#! ---------------------------------------------------------------------
CLUSTER_CIDR: 100.96.0.0/11
CLUSTER_NAME: tkg-workload02
CLUSTER_PLAN: dev
ENABLE_CEIP_PARTICIPATION: 'true'
IS_WINDOWS_WORKLOAD_CLUSTER: "true"
VSPHERE_WINDOWS_TEMPLATE: windows-2019-kube-v1.22.5
ENABLE_MHC: "false"

IDENTITY_MANAGEMENT_TYPE: oidc

INFRASTRUCTURE_PROVIDER: vsphere
SERVICE_CIDR: 100.64.0.0/13
TKG_HTTP_PROXY_ENABLED: false
DEPLOY_TKG_ON_VSPHERE7: 'true'
VSPHERE_DATACENTER: /SDDC-Datacenter
VSPHERE_DATASTORE: WorkloadDatastore
VSPHERE_FOLDER: /SDDC-Datacenter/vm/tkg-vmc-workload
VSPHERE_NETWORK: /SDDC-Datacenter/network/tkgvmc-workload-segment01
VSPHERE_PASSWORD: <encoded:T1V3WXpkbStlLUlDOTBG>
VSPHERE_RESOURCE_POOL: /SDDC-Datacenter/host/Cluster-1/Resources/Compute-ResourcePool/Tanzu/tkg-vmc-workload
VSPHERE_SERVER: 10.97.1.196
VSPHERE_SSH_AUTHORIZED_KEY: ssh-rsa....loudadmin@vmc.local

VSPHERE_USERNAME: cloudadmin@vmc.local
WORKER_MACHINE_COUNT: 3
VSPHERE_INSECURE: 'true'
ENABLE_AUDIT_LOGGING: 'true'
ENABLE_DEFAULT_STORAGE_CLASS: 'true'
ENABLE_AUTOSCALER: false
AVI_CONTROL_PLANE_HA_PROVIDER: 'true'
OS_ARCH: amd64
OS_NAME: photon
OS_VERSION: 3

WORKER_SIZE: small
CONTROLPLANE_SIZE: large
REMOVE_CP_TAINT: "true"

if your cluster yaml file is correct, you should see that new windows cluster has been started to deploy.

and after some time if should deploy cluster sucessfully.

In case if you are using NSX-ALB AKO or Pinniped and see that those pods are not running, please refer Here

NOTE – if you see this error during image build process : Permission denied: ‘./packer/ova/windows/windows-2019/autounattend.xml, check the permission of file autounattend.yaml

Cloud Director OIDC Configuration using OKTA IDP

Featured

Posted on April 5, 2022 by vmtechie

OpenID Connect (OIDC) is an industry-standard authentication layer built on top of the OAuth 2.0 authorization protocol. The OAuth 2.0 protocol provides security through scoped access tokens, and OIDC provides user authentication and single sign-on (SSO) functionality. For more refer here (https://datatracker.ietf.org/doc/html/rfc6749). There are two main types of authentication that you can perform with Okta:

The OAuth 2.0 protocol controls authorization to access a protected resource, like your web app, native app, or API service.
The OpenID Connect (OIDC) protocol is built on the OAuth 2.0 protocol and helps authenticate users and convey information about them. It’s also more opinionated than plain OAuth 2.0, for example in its scope definitions.

So If you want to import users and groups from an OpenID Connect (OIDC) identity provider to your Cloud Director system (provider) or Tenant organization, you must configure provider/tenant organization with this OIDC identity provider. Imported users can log in to the system/tenant organization with the credentials established in the OIDC identity provider.

We can use VMware Workspace ONE Access (VIDM) or any public identity providers, but make sure OAuth authentication endpoint must be reachable from the VMware Cloud Director cells.in this blog post we will use OKTA OIDC and configure VMware Cloud to use this OIDC for authentication.

Step:1 – Configure OKTA OIDC

For this blog post, i created an developer account on OKTA at this url –https://developer.okta.com/signup and once account is ready, follow below steps to add cloud director as an application in OKTA console:

In the Admin Console, go to Applications > Applications.
Click Create App Integration.

To create an OIDC app integration, select OIDC – OpenID Connect as the Sign-in method.
Choose what type of application you plan to integrate with Okta, in Cloud Director case Select Web Application.

App integration name: Specify a name for Cloud Director
Logo (Optional): Add a logo to accompany your app integration in the Okta org
Grant type: Select from the different grant type options
Sign-in redirect URIs: The Sign-in redirect URI is where Okta sends the authentication response and ID token for the sign-in request, in our case for provider https://<vcd url>/login/oauth?service=provider and incase if you are doing it for tenant then use https://<vcd url>/login/oauth?service=tenant:<org name>
Sign-out redirect URIs: After your application contacts Okta to close the user session, Okta redirects the user to this URI.

Assignments: Controlled access: The default access option assigns and grants login access to this new app integration for everyone in your Okta org or you can choose to Limit access to selected groups

Click Save. This action creates the app integration and opens the settings page to configure additional options.

The Client Credentials section has the Client ID and Client secret values for Cloud Director integration, Copy both the values as we enter these in Cloud Director.

The General Settings section has the Okta Domain, for Cloud Director integration, Copy this value as we enter these in Cloud Director.

Step:2 – Cloud Director OIDC Configuration

Now I am going to configure OIDC authentication for provider side of cloud provider and with very minor changes (tenant URL) it can be configured for tenants too.

Let’s go to Cloud Director and from the top navigation bar, select Administration and on the left panel, under Identity Providers, click OIDC and click CONFIGURE

General: Make sure that OpenID Connect Status is active, and enter the client ID and client secret information from the OKTA App registration which we captured above.

To use the information from a well-known endpoint to automatically fill in the configuration information, turn on the Configuration Discovery toggle and enter a URL, for OKTA the URL would look this – https://<domain.okta.com>/.well-known/openid-configuration and click on NEXT

Endpoint: Clicking on NEXT will populate “Endpoint” information automatically, it is however, essential that the information is reviewed and confirmed.

Scopes: VMware Cloud Director uses the scopes to authorize access to user details. When a client requests an access token, the scopes define the permissions that this token has to access user information.enter the scope information, and click Next.

Claims: You can use this section to map the information VMware Cloud Director gets from the user info endpoint to specific claims. The claims are strings for the field names in the VMware Cloud Director response

This is the most critical piece of configuration. Mapping of this information is essential for VCD to interpret the token/user information correctly during the login process.

For OKTA developer account, user name is email id, so i am mapping Subject to email as below

Key Configuration:

OIDC uses a public key cryptography mechanism.A private key is used by the OIDC provider to sign the JWT Token and it can be verified by a 3rd party using the public keys published on the OIDC provider’s well-known URL.These keys form the basis of security between the parties. For security to be maintained, this is required to keep the private keys protected from any cyber-attacks.One of the best practices that has been identified to secure the keys from being compromised is known as key rollover or key Refresh.

From VMware Cloud Director 10.3.2 and above, if you want VMware Cloud Director to automatically refresh the OIDC key configurations, turn on the Automatic Key Refresh toggle.

Key Refresh Endpoint should get populated automatically as we choose auto discovery.
Select a Key Refresh Strategy.
- Add – Preferred option, add the incoming set of keys to the existing set of keys. All keys in the merged set are valid and usable.
- Replace – Replace the existing set of keys with the incoming set of keys.
- Expire After – You can configure an overlap period between the existing and incoming sets of keys. You can configure the overlapping time using the Expire Key After Period, which you can set in hourly increments from 1 hour up to 1 day.

If you did not use Configuration Discovery in Step 6, upload the private key that the identity provider uses to sign its tokens and click on SAVE

Now go to Cloud Director, under Users, Click on IMPORT USERS and choose Source as “OIDC” and add user which is there in OKTA and Assign Role to that user, thats it.

Now you can logout from the vCD console and try to login again, Cloud Director automatically redirects to OKTA and asks for credential to validate.

Once the user is authenticated by Okta, they will be redirected back to VCD and granted access per rights associated with the role that was assigned when the user was provisioned.

Verify that the Last Run and the Last Successful Run are identical. The runs start at the beginning of the hour. The Last Run is the time stamp of the last key refresh attempt. The Last Successful Run is the time stamp of the last successful key refresh. If the time stamps are different, the automatic key refresh is failing and you can diagnose the problem by reviewing the audit events. (This is only applicable if Automatic Key Refresh is enabled. Otherwise, these values are meaningless)

Bring on your Own OIDC – Tenant Configuration

For tenant configuration, i have created a video, please take a look here, Tenant can bring their own OIDC and self service in cloud director tenant portal.

This concludes the OIDC configuration with VMware Cloud Director. I would like to Thank my colleague Ankit Shah, for his guidance and review of this document.

Tanzu Service on VMware Cloud on AWS – Installing Tanzu Application Platform

Featured

Posted on March 25, 2022 by vmtechie

VMware Tanzu Application Platform is a modular, application detecting platform that provides a rich set of developer tools and a paved path to production to build and deploy software quickly and securely on any compliant public cloud or on-premises Kubernetes cluster.

Tanzu Application Platform delivers a superior developer experience for enterprises building and deploying cloud-native applications on Kubernetes. It enables application teams to get to production faster by automating source-to-production pipelines. It clearly defines the roles of developers and operators so they can work collaboratively and integrate their efforts.

Operations teams can create application scaffolding templates with built-in security and compliance guardrails, making those considerations mostly invisible to developers. Starting with the templates, developers turn source code into a container and get a URL to test their app in minutes.

Pre-requisite

You should have created an account on Tanzu Network to download Tanzu Application Platform packages.
Servers should have Network access to https://registry.tanzu.vmware.com
A container image registry and access from K8s cluster, in my case i have installed “Harbor” with let’s encrypt certificate.
Registry credentials with read and write access made available to Tanzu Application Platform to store images.
Git repository for the Tanzu Application Platform GUI’s software catalogs, along with a token allowing read access.

Kubernetes cluster requirements

Installation requires Kubernetes cluster v1.20, v1.21, or v1.22 on Tanzu Kubernetes Grid Service on VMware Cloud on VMC as well as pod security policies must be configured so that Tanzu Application Platform controller pods can run as root. To set the pod security policies, run:

#kubectl create clusterrolebinding default-tkg-admin-privileged-binding --clusterrole=psp:vmware-system-privileged --group=system:authenticated

Install Cluster Essentials for VMware Tanzu

The Cluster Essentials for VMware Tanzu package simplifies the process of installing the open-source Carvel tools on your cluster. It includes a script that uses the Carvel CLI tools to download and install the server-side components kapp-controller and secretgen-crontroller on the targeted cluster. Currently, only MacOS and Linux are supported for Cluster Essentials for VMware Tanzu.

Sign in to Tanzu Network.
Navigate to Cluster Essentials for VMware Tanzu on VMware Tanzu Network.
on Linux, download tanzu-cluster-essentials-linux-amd64-1.0.0.tgz.
Unpack the TAR file into the tanzu-cluster-essentials directory by running:

#mkdir $HOME/tanzu-cluster-essentials
#tar -xvf tanzu-cluster-essentials-linux-amd64-1.0.0.tgz -C $HOME/tanzu-cluster-essentials

Configure and run install.sh using below commands:

#export INSTALL_BUNDLE=registry.tanzu.vmware.com/tanzu-cluster-essentials/cluster-essentials-bundle@sha256:82dfaf70656b54dcba0d4def85ccae1578ff27054e7533d08320244af7fb0343
#export INSTALL_REGISTRY_HOSTNAME=registry.tanzu.vmware.com
#export INSTALL_REGISTRY_USERNAME=TANZU-NET-USER Name
#export INSTALL_REGISTRY_PASSWORD=TANZU-NET-USER PASSWORD
#cd $HOME/tanzu-cluster-essentials
#./install.sh

now Install kapp & imgpkg CLI onto your $PATH using below commands:

sudo cp $HOME/tanzu-cluster-essentials/kapp /usr/local/bin/kapp
sudo cp $HOME/tanzu-cluster-essentials/imgpkg /usr/local/bin/imgpkg

For Linux Client VM: Install the Tanzu CLI and Plugins

To install the Tanzu Tanzu command line interface (CLI) on a Linux operating system, Create a directory named Tanzu and download tanzu-framework-bundle-linux from Tanzu Net and unpack the TAR file into the Tanzu directory and install using below commands:

#mkdir $HOME/tanzu 
#tar -xvf tanzu-framework-linux-amd64.tar -C $HOME/tanzu
#export TANZU_CLI_NO_INIT=true
#cd $HOME/tanzu 
#sudo install cli/core/v0.11.1/tanzu-core-linux_amd64 /usr/local/bin/tanzu
#tanzu version
#cd $HOME/tanzu
#tanzu plugin install --local cli all
#tanzu plugin list

Ensure that you have the accelerator, apps, package, secret, and services plug-ins installed. You need these plug-ins to install and interact with the Tanzu Application Platform.

Installing the Tanzu Application Platform Package and Profiles

VMware recommends install of Tanzu Application Platform packages by relocating the images to your registry from VMware Tanzu Network registry, this will ease the deployment process, so lets do it by logging in to Tanzu Net Registry, setting some env variables and relocate images.

#docker login registry.tanzu.vmware.com
#export INSTALL_REGISTRY_USERNAME=MY-REGISTRY-USER
#export INSTALL_REGISTRY_PASSWORD=MY-REGISTRY-PASSWORD
#export INSTALL_REGISTRY_HOSTNAME=MY-REGISTRY
#export TAP_VERSION=VERSION-NUMBER
#imgpkg copy -b registry.tanzu.vmware.com/tanzu-application-platform/tap-packages:1.0.2 --to-repo ${INSTALL_REGISTRY_HOSTNAME}/TARGET-REPOSITORY/tap-packages

This completes the download and upload on images to local registry.

Create a registry secret by running below command:

#tanzu secret registry add tap-registry \
  --username ${INSTALL_REGISTRY_USERNAME} --password ${INSTALL_REGISTRY_PASSWORD} \
  --server ${INSTALL_REGISTRY_HOSTNAME} \
  --export-to-all-namespaces --yes --namespace tap-install

Add the Tanzu Application Platform package repository to the cluster by running:

#tanzu package repository add tanzu-tap-repository \
  --url ${INSTALL_REGISTRY_HOSTNAME}/TARGET-REPOSITORY/tap-packages:$TAP_VERSION \
  --namespace tap-install

Get the status of the Tanzu Application Platform package repository, and ensure the status updates to Reconcile succeeded by running:

#tanzu package repository get tanzu-tap-repository --namespace tap-install

Tanzu Application Platform profile

The tap.tanzu.vmware.com package installs predefined sets of packages based on your profile settings. This is done by using the package manager you installed using Tanzu Cluster Essentials.Here is my full profile sample file:

buildservice:
  descriptor_name: full
  enable_automatic_dependency_updates: true
  kp_default_repository: harbor.tkgsvmc.net/tbs/build-service
  kp_default_repository_password: <password>
  kp_default_repository_username: admin
  tanzunet_password: <password>
  tanzunet_username: tripathiavni@vmware.com
ceip_policy_disclosed: true
cnrs:
  domain_name: tap01.tkgsvmc.net
grype:
  namespace: default
  targetImagePullSecret: tap-registry
learningcenter:
  ingressDomain: learningcenter.tkgsvmc.net
metadata_store:
  app_service_type: LoadBalancer
ootb_supply_chain_basic:
  gitops:
    ssh_secret: ""
  registry:
    repository: tap
    server: harbor.tkgsvmc.net/tap
profile: full
supply_chain: basic
tap_gui:
  app_config:
    app:
      baseUrl: http://tap-gui.tap01.tkgsvmc.net
    backend:
      baseUrl: http://tap-gui.tap01.tkgsvmc.net
      cors:
        origin: http://tap-gui.tap01.tkgsvmc.net
    catalog:
      locations:
        - target: https://github.com/avnish80/tap/blob/main/catalog-info.yaml
          type: url
  ingressDomain: tap01.tkgsvmc.net
  ingressEnabled: "true"
  service_type: LoadBalancer

save this file with modified values as per your environment, for more details about details of settings, check Here.

Install Tanzu Application Platform

finally lets install TAP, to install the Tanzu Application Platform package run below commands:

#tanzu package install tap -p tap.tanzu.vmware.com -v $TAP_VERSION --values-file tap-values.yml -n tap-install

to verify the packages installed, you can go to TMC and check there

or you an run below command to verify too

#tanzu package installed get tap -n tap-install

This completes the installation of Tanzu Application platform, now developer can: Develop and promote an application, Create an application accelerator, Add testing and security scanning to an application, Administer, set up, and manage supply chains.

Tanzu Service on VMware Cloud on AWS – Kubernetes Cluster Operations

Featured

Posted on March 16, 2022 by vmtechie

Tanzu Kubernetes Grid is a managed service offered by VMware Cloud on AWS. Activate Tanzu Kubernetes Grid in one or more SDDC clusters to configure Tanzu support in the SDDC vCenter Server.In my previous post (Getting Started with Tanzu Service on VMware Cloud on AWS),in this i walked you through how to enable Tanzu Service on VMware Cloud on AWS.

In this post i will deploy Tanzu Kubernetes Cluster by GUI (from Tanzu Mission Control) and as well as CLI but this CLI is updated API V2 version, so lets get started.

Deploy Tanzu Kubernetes Cluster using Tanzu Mission Control

Go to Tanzu Mission Control and validate that VMC supervisor cluster is registered and healthy by going to Tanzu Mission Control, Click on Administration, to go “management cluster” and check the status

Now on Tanzu Mission Control, click on “Clusters” and then click on “CREATE CLUSTER”

Select your VMC Tanzu Management Cluster and click on “CONTINUE TO CREATE CLUSTER”

on the next screen choose “Provisioner” (namespace name”). you add a provisioner by creating a vSphere namespace in the Supervisor Cluster, which you can do in VMC vCenter.

Next is select Kubernetes Version, latest supported version is preselected for you, Pod CIDR, and Service CIDR. You can also optionally select the default storage class for the cluster and allowed storage classes.The list of storage classes that you can choose from is taken from your vSphere namespace.

Select the type of cluster you want to create. the primary difference between the two is that the highly available cluster is deployed with multiple control plane nodes.

You can optionally select a different instance type for the cluster’s control plane node and its storage class as well as you can optionally additional storage volumes for your control plane.

To configure additional volumes, click Add Volume and then specify the name, mount path, and capacity for the volume. To add another, click Add Volume again.

Next is you can define the default node pool and create additional node pools for your cluster. specify the number of worker nodes to provision also select the instance type for workload clusters and select the storage class

When you ready to provision the new cluster, click Create Cluster and wait for few minutes

you can also view vCenter activities about creation of Tanzu Kubernetes cluster.

Once the cluster is fully created and TMC agent reported back, you should see below status on TMC console, which shows that cluster has been successfully created.

This complates Tanzu Kubernetes Cluster deployment using GUI.

Deploy Tanzu Kubernetes Grid Service using v1alpha2 API yaml

The Tanzu Kubernetes Grid Service v1alpha2 API provides a robust set of enhancements for provisioning Tanzu Kubernetes clusters. there is an YAML specification which i am using for provisioning a Tanzu Kubernetes Cluster Using the Tanzu Kubernetes Grid Service v1alpha2 API

apiVersion: run.tanzu.vmware.com/v1alpha2
kind: TanzuKubernetesCluster
metadata:
  name: tkgsv2
  namespace: wwmca
spec:
  topology:
    controlPlane:
      replicas: 1
      vmClass: guaranteed-medium
      storageClass: vmc-workload-storage-policy-cluster-1
      volumes:
        - name: etcd
          mountPath: /var/lib/etcd
          capacity:
            storage: 4Gi
      tkr:  
        reference:
          name: v1.21.2---vmware.1-tkg.1.ee25d55
    nodePools:
    - name: worker-nodepool-a1
      replicas: 2
      vmClass: best-effort-large
      storageClass: vmc-workload-storage-policy-cluster-1
      tkr:  
        reference:
          name: v1.21.2---vmware.1-tkg.1.ee25d55
  settings:
    storage:
      defaultClass: vmc-workload-storage-policy-cluster-1
    network:
      cni:
        name: antrea
      services:
        cidrBlocks: ["198.53.100.0/16"]
      pods:
        cidrBlocks: ["192.0.5.0/16"]
      serviceDomain: managedcluster.local
      trust:
        additionalTrustedCAs:
          - name: CompanyInternalCA-1
            data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tDQpNSUlG

Two key parameters which i am using for cluster provistioning

#tkr.reference.name is the TKR NAME #to be used by control plane nodes; supported format is “v1.21.2—vmware.1-tkg.1.ee25d55”
#trust configures additional certificates for the cluster #if omitted no additional certificate is configured

You can run below command to check the status of cluster provustioning:

#kubectl get tkc

Scale a Tanzu Kubernetes cluster

Publish the service Internally/Externally

Before we can make our service available over the Internet, it should be accessible from within the VMware Cloud on AWS instance. Platform operators can publish applications through a Kubernetes Service of type LoadBalancer. This ability is made possible through the NSX-T Container Plugin (NCP) functionality built into Tanzu Kubernetes Grid. lets deploy a basic container and exposed it as type “LoadBalancer”

#kubectl run nginx1 --image=nginx
#kubectl expose pod nginx1 --type=LoadBalancer --port=80

Now you can access the application internally by accessing internal

Access application from Internet

To make it publicly available, we must assign a public IP address, and configure a Destination NAT, let do it request an Public IP on VMC console and create a NAT rule on Internet Tab to access the application from internet.

Now access the application from Internet and you should be able to successfully access it using provided public ip.

Exposing a Kubernetes service to the Internet takes a couple of more steps to complete than exposing it to your internal networks, but the VMware Cloud Console makes those steps simple enough. After exposing the Kubernetes service using an NSX-T Load Balancer, you can request a new Public IP Address and then configure a NAT rule to send that traffic to the virtual IP address of the load balancer.

Getting Started with Tanzu Service on VMware Cloud on AWS

Posted on March 1, 2022 by vmtechie

VMware Tanzu Kubernetes Grid (TKG) is a multi-cloud Kubernetes footprint that customers/partners can run both on-premises in vSphere, VMware Cloud on AWS and the public cloud on Amazon EC2 and Microsoft Azure VMs.

TKG provides a Container orchestration through Kubernetes is now built into the vSphere 7 platform.As a VMware Cloud on AWS customer you can take advantage of this new functionality to build Kubernetes clusters in the same platform you’ve grown accustomed to using to manage your virtual infrastructure.

Take control of Cloud Resources and give freedom to Developers based on Personas

Virtualization Administrator: They will be able to define resource allocations and permissions for your users to create their own Kubernetes clusters according to their own specifications.Define access policies, storage policies, memory and CPU restrictions for teams needing Kubernetes access.

Developer or Platform Administrator: They can create new Kubernetes clusters within the defined access policies, upgrade those clusters and scale clusters within the approved resource allocations.

VMware recognizes that not all environments are running on top of vSphere. Tanzu Kubernetes Grid(TKG) leverages the same ClusterAPI engine as VMware Tanzu to manage cluster lifecycles, and can run on any infrastructure. VMware provides three variants of the TKG:

Tanzu Kubernetes Grid Multi-Cloud (TKGm): Installer driven wizard to set up Kubernetes environment to run across multi clouds for example: on AWS EC2 or Azure Native VMs
Tanzu Kubernetes grid Service (TKGS) aka vSphere With Tanzu: Natively integrated with vSphere7+ and available to customers at no extra cost for basic version on VCF on-prem as well as VMware Cloud on AWS
Tanzu Kubernetes Grid Integrated Edition: VMware Tanzu Kubernetes Grid Integrated Edition (formerly known as VMware Enterprise PKS) is a Kubernetes-based container solution with advanced networking, a private container registry, and life cycle management.

Enable Tanzu Service on VMware Cloud on AWS

Pre-requisite:

Make sure we have at-least three node SDDC is deployed and running with enough available resources (at least 112 GB of available memory, and has sufficient free resources to support 16 vCPUs)
Get Three CIDR blocks for the deployment. These three needs to be ranges that does not overlap with the Management CIDR or any other networks used on-prem or in the VMware Cloud on AWS SDDC.
You can activate Tanzu Kubernetes Grid in any SDDC at version 1.16 and later.
If Edge cluster has been configured with medium configuration, then a SDDC cluster requires a minimum of three hosts for activation.
If Edge cluster has been configured with Large configuration, then a SDDC cluster requires a minimum of four hosts for activation.

Once pre-requisites are ready, go to VMware Cloud on AWS SDDC and click on “Activate the Tanzu Kubernetes Service”

Activation process will check required resources and will only move ahead if you have pre-requisite completed.

on the next screen:

Leave the Service CIDR as default or pick of your choice but non-overlapping and used for Tanzu supervisor services for the cluster
Enter the “namespace Network CIDR”, non-overlapping
Enter an ‘Ingress CIDR”, non-overlapping
Enter an “Egress CIDR”, non-overlapping
next Click on “Validate and Proceed”

NOTE: CIDR blocks of size 16, 20, 23, or 26 are supported, and must be in one of the “private address space” blocks defined by RFC 1918 (10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16).

and finally once validation is done, click on Activate Tanzu Kubernetes Grid

this will start activation process and you should be seeing “Activating Tanzu Kubernetes Grid” on your SDDC tile.This process should get completed within 20-30 minutes.

Such an easy process to make your SDDC enabled for running VMs and Containers together. When activation is completed, login to SDDC vCenter and click on Workload Management

Persona (Virtualization/vSphere Administrator) – vSphere Administrator create a vSphere Namespace on the Supervisor Cluster, sets resource limits to the namespace and permissions so that DevOps engineers can access it. he/she provide the URL of the Kubernetes control plane to DevOps engineers where they can run create their own Kubernetes clusters and run their workloads.

Step -1: Set permissions so that DevOps engineers can access the namespace.

From the Permissions pane, select Add Permissions.

Select an identity source, a user or a group, and a role, and click OK.

Step-2: Set persistent storage to the namespace.Storage policies that you assign to the namespace control how persistent volumes and Tanzu Kubernetes cluster nodes are placed within datastores in the SDDC environment.

From the Storage pane, select Add Storage.

Select a storage policy to control datastore placement of persistent volumes and click OK

The VM class is a VM specification that can be used to request a set of resources for a VM. The VM class is controlled and managed by a vSphere administrator, and defines such parameters as the number of virtual CPUs, memory capacity, and reservation settings. The defined parameters are backed and guaranteed by the underlying infrastructure resources of a Supervisor Cluster.

Workload Management offers several default VM classes. Generally, each default class type comes in two editions: guaranteed and best effort. A guaranteed edition fully reserves resources that a VM specification requests. A best effort class edition does not and allows resources to be overcommitted. Typically, a guaranteed type is used in a production environment.

vSphere Administrator can setup additional limits based on use cases and requirements.

Copy NameSpace URL by clicking on “Copy link” and give it to your DevOps/Platfrom admin)

Persona (DevOps/Platform Administrator)

How to Access and Work ?

Install a new VM (clientvm) or from their desktop/laptop, he/she can access this newly created “Namespace” and then create new Kubernetes cluster. When the new VM is provisioned, power it on and and ssh to it and Download the command line tools from vCenter, make sure the item below in red box is changed to your supervisor cluster address that you copied earlier by running:

#wget https://k8s.Cluster-1.vcenter.sddc-18-139-9-54.vmwarevmc.com/wcp/plugin/linux-amd64/vsphere-plugin.zip

Unzip using below command

Now lets login to the supervisor cluster by entering the following :

kubectl vsphere login --vsphere-username cloudadmin@vmc.local --server=https://k8s.Cluster-1.vcenter.sddc-18-139-9-54.vmwarevmc.com

enter the password for cloudadmin or any other user to complete the login

From here onwards, Devops can create their own K8s clusters and deploy applications, they can also utlize VMware’s multi-cloud mamagement platfrom to spin up kubernetes clusters using GUI.

For Devops to use GUI, vSphere Administrator need to Register VMware Cloud on AWS management cluster with Tanzu Mission Control. lets do that:

Register This Management Cluster with Tanzu Mission Control

Tanzu service ships with a namespace for Tanzu Mission Control. This namespace exists on the Supervisor Cluster where you install the Tanzu Mission Control agent.

The vSphere Namespace provided for Tanzu Mission Control is identified as svc-tmc-cXX

To integrate the Tanzu Kubernetes Grid Service with Tanzu Mission Control, install the agent on the Supervisor Cluster.

Register the Supervisor Cluster with Tanzu Mission Control and obtain the Registration URL. See Register a Management Cluster with Tanzu Mission Control.

On the client-vm, create a .yaml file with below content:

apiVersion: installers.tmc.cloud.vmware.com/v1alpha1
kind: AgentInstall
metadata:
  name: tmc-agent-installer-config
  namespace: <NAMESPACE captured in above step>
spec:
  operation: INSTALL
  registrationLink: <TMC-REGISTRATION-URL captured from TMC console>

Run this yaml file on using:

#kubectl create -f tmc.yaml

you can also check the status of TMC registration by running below command:

#kubectl get pods -n <ns name>

Now go back to Tanzu Mission Control and after some time you should see your Supervisor cluster ready

Devops/Platform admins are now ready to deploy your TKC clustes as well they can deploy containers, this completes this part of blog, in the next part i will write how to create TKC clusters, run applications within containers and how to expose to internet.

Load Balancer as a Service with Cloud Director

Featured

Posted on October 19, 2020 by vmtechie

NSX Advance Load Balancer’s (AVI) Intent-based Software Load Balancer provides scalable application delivery across any infrastructure. AVI provides 100% software load balancing to ensure a fast, scalable and secure application experience. It delivers elasticity and intelligence across any environments. It scales from 0 to 1 million SSL transactions per second in minutes. It achieves 90% faster provisioning and 50% lower TCO than traditional appliance-based approach.

With the release of Cloud Director 10.2 , NSX ALB is natively integrated with Cloud Director to provider self service Load Balancing as a Service (LBaaS) where providers can release load balancing functionality to tenants and tenants consume load balancing functionality based on their requirement. In this blog post we will cover how to configure LBaaS.

Here is High Level workflow:

Deploy NSX ALB Controller Cluster
Configure NSX-T Cloud
Discover NSX-T Inventory,Logical Segments, NSGroups (ALB does it automatically)
Discover vCenter Inventory,Hosts, Clusters, Switches (ALB does it automatically)
Upload SE OVA to content library (ALB does it automatically, you just need to specify name of content library)
Register NSX ALB Controller, NSX-T Cloud and Service Engines to Cloud Director and Publish to tenants (Provider Controlled Configuration)
Create Virtual Service,Pools and other settings (Tenant Self Service)
Create/Delete SE VMs & connect to tenant network (ALB/VCD Automatically)

Deploy NSX ALB (AVI) Controller Cluster

The NSX ALB (AVI) Controller provides a single point of control and management for the cloud. The AVI Controller runs on a VM and can be managed using its web interface, CLI, or REST API but in this case Cloud Director.The AVI Controller stores and manages all policies related to services and management. To ensure AVI controllers High Availability we need to deploy 3 AVI Controller nodes to create a 3-node AVI Controller cluster.

Deployment Process is documented Here & Cluster creation Process is Here

Create NSX-T Cloud inside NSX ALB (AVI) Controller

NSX ALB (AVI) Controller which uses APIs to interface with the NSX-T manager and vCenter to discover the infrastructure.here is high level activities to configure NSX-T Cloud in NSX ALB management console:

Configure NSX-T manager IP/URL (One per Cloud)
Provide admin credentials
Select Transport zone (One to One Mapping – One TZ per Cloud)
Select Logical Segment to use as SE Management Network
Configure vCenter server IP/URL (One per Cloud)
Provide Login username and password
Select Content Library to push SE OVA into Content Library

Service Engine Groups & Configuration

Service Engines are created within a group, which contains the definition of how the SEs should be sized, placed, and made highly available. Each cloud will have at least one SE group.

SE Groups contain sizing, scaling, placement and HA properties
A new SE will be created from the SE Group properties
SE Group options will vary based upon the cloud type
An SE is always a member of the group it was created within in this case NSX-T Cloud
Each SE group is an isolation domain
Apps may gracefully migrate, scale, or failover across SEs in the groups

Service Engine High Availability:

Active/Standby

VS is active on one SE, standby on another
No VS scaleout support
Primarily for default gateway / non-SNAT app support
Fastest failover, but half of SE resources are idle

Elastic N + M

All SEs are active
N = number of SEs a new Virtual Service is scaled across
M = the buffer, or number of failures the group can sustain
SE failover decision determined at time of failure
Session replication done after new SE is chosen
Slower failover, less SE resource requirement

Elastic Active / Active

All SEs are active
Virtual Services must be scaled across at least 2 Service engines
Session info proactively replicated to other scaled service engines
Faster failover, require more SE resources

Cloud Director Configuration

Cloud Director Configuration is two fold, Provider Config and Tenant Config, lets first cover provider Config…

Provider Configuration

Register AVI Controller: Provider administrator login as a admin and register AVI Controller with Cloud Director. provider has option to add multiple AVI controllers.

NOTE – incase if you are registering with NSX ALB’s default self sign certificate and if it throws error while registering , then regenerate self sign certificate in NSX ALB.

Register NSX-T cloud

Now next thing is we need to register NSX-T cloud with Cloud Director, which we had configured in ALB controller:

Selecting one of the registered AVI Controller
Provide a meaning full name to the controller
Select the NSX-T cloud which we had registered in AVI
Click on ADD.

Assign Service Engine groups

Now register service engine groups either “Dedicated” or “Reserved” based on tenant requirement or provider can have both type of groups and assign to tenant based on requirements.

Select NSX-T Cloud which we had registered above
Select the “Reservation Model”
1. Dedicated Reservation Model:- For each tenant Organization VDC Edge gateway, AVI will create two Service Engine nodes for each LB enabled Org VDC Edge GW.
2. Shared Reservation Model:- Shared is elastic and shared among all tenants. AVI will create pool of service engines that are going to be shared across tenant. Capacity allocation is managed in VCD, Avi elastically deploys and un-deploys service engines based on usage

Provider Enables and Allocates resources to Tenant

Provider enables LB functionality in the context of Org VCD Edge by following below steps:

Click on Edges
Choose Edge on which he want to enable load balancing
Go to “Load Balancer” and click on “General Settings”
Click on “Edit”
Toggle on to Activate to activate the load balancer
Select Service Specification

Next step is to assign Service Engines to tenant based on requirement, for that go to Service Engine Group and Click on “ADD” and add one of the SE group which we had registered previously to customer’s one of the Edge.

Provider can restrict usage of Service Engines by configuring:

Maximum Allowed: The maximum number of virtual services the Edge Gateway is allowed to use.
Reserved: The number of guaranteed virtual services available to the Edge Gateway.

Tenant User Self Service Configuration

Pools: Pools maintain the list of servers assigned to them and perform health monitoring, load balancing, persistence.

Inside General Settings some of the key settings are:
1. Provide Name of the Pool
2. Load Balancing Algorithm
3. Default Server Port
4. Persistence
5. Health Monitor
Inside Members section:
1. Add Virtual Machine IP addresses which needs to be load balanced
2. Define State, Port and Ratio
3. SSL Settings allow SSL offload and Common Name Check

Virtual Services: A virtual service advertises an IP address and ports to the external world and listens for client traffic. When a virtual service receives traffic, it may be configured to:

Proxy the client’s network connection.
Perform security, acceleration, load balancing, gather traffic statistics, and other tasks.
Forward the client’s request data to the destination pool for load balancing.

Tenant choose Service Engine Group which provider has assigned to tenant, then choose Load Balancer Pool which we created in above step and most important Virtual IP This IP address can be from External IP range of the Org VDC or if you want Internal IP , then you can use any IP.

So in my example, i am running two virtual machines having Org VDC Internal IP addresses and VIP is from external public IP address range, so if I browse VIP , i can reach to web servers sucessfully using VCD/AVI integration.

This completes basic integration and configuration of LBaaS using Cloud Director & NSX Advance Load Balancer. feel free to share feedback.

Assess Your Sovereign Cloud Stack for Compliance

by vmtechie April 26, 2023April 26, 2023

Cloud Director Container Service Extension – Tanzu Contour, Prometheus and Grafana Install Guide

This post explains how to install and access Tanzu Contour, Promethous and Grafana on Tanzu clusters deployed by Cloud Director Container Service extension. so to get started first ensure TANZU CLI is installed on your local machine, if not then you can install by following documentation given Here Next thing you need is the kubeconfig…

by vmtechie February 15, 2023

Security VDC Architecture with VMware Cloud Director

Cloud Director VDCs come with all the features you’d expect from a public cloud, Virtual Data Center is a logical representation of a physical data center, created using virtualization technologies and a virtual data center allows IT administrators to create, provision, and manage virtualized resources such as servers, storage, and networking in a flexible and…

by vmtechie January 25, 2023

Getting Started with VMware Cloud Director Container Service Extension 4.0

by vmtechie December 8, 2022

Multi-Tenant Tanzu Data Services with VMware Cloud Director

VMware Cloud Director extension for VMware Data Solutions is a plug-in for VMware Cloud Director (VCD) that enables cloud providers expand their multi-tenant cloud infrastructure platform to deliver a portfolio of on-demand caching, messaging and database software services at massive scale. This brings in new opportunity for our Cloud Providers to offer additional cloud native developer services in addition to…

by vmtechie November 29, 2022

Compliance Pack for Sovereign Cloud Controls Rules

Data Sovereignty and Jurisdictional Control

Data Access and Integrity

Data Security and Compliance

Data Independence and Interoperability

Control Rules and Product Control Set for vSphere

Control Rules and product control set for nsx-t

Controls Rules and Product Control Set for VMware Cloud Director

Control Rules and Product Control Set for VMware Cloud Director Availability

Install the VMware vRealize (ARIA) Operations Compliance Pack for Sovereign Cloud

Accessing Compliance Reports

Custom Benchmarks

Share this:

Like this:

Installation Steps

Step:1- Install kapp-controller

Step:2- Install secretgren-controller

Step:3- Install cert-manager

Verify Tanzu Packages

Step4:- Implement Ingress Control with Contour

Step 5:- Deploy Prometheus

Step 6:- Deploy Grafana

Access the Grafana Dashboard

Share this:

Like this:

Pre-requisite for Container Service Extension 4.0

Provider Configuration

Install Kubernetes Container Clusters UI plug-in for VMware Cloud Director

Container Service Extension OVA deployment

Share this:

Like this:

Creation of pricing policy based on chargeback strategy

CPU Rate

Memory Rate

Storage Rate

Network Rate

Advanced Network Rate

Guest OS Rate

Cloud Director Availability

vCenter Tag Rate

VCD Metadata Rate

One Time Fixed Cost

Rate Factors

Tanzu Kubernetes Clusters

Additional Fixed Cost

Apply Policy

Bills

Share this:

Like this:

PROCESS

DEPLOY VMWARE MANAGED TRANSIT GATEWAY

ATTACH VPC TO VMWARE MANAGED TRANSIT GATEWAY

ATTACH VMWARE MANAGED TRANSIT GATEWAY TO VPC

Deploy FSx for NetApp ONTAP file system in your AWS account

MOUNT NFS EXTERNAL STORAGE TO SDDC Cluster

Share this:

Like this:

Deployment Architecture

Activate VMware Site Recovery on VMware Cloud on AWS

Deploy VMware Site Recovery Manager on Azure VMware Solution

Configure site pairing in vCenter Server

Planning

Creating Protection Groups

Recovery Plan

Steps to add a VM for replication:

Share this:

Like this:

Kubernetes NFS-Client Provisioner

Finally, Test Your Environment!

Share this:

Like this:

Windows Image Prerequisites

Build a Windows Image

Use a Windows Image for a Workload Cluster

Share this:

Like this:

Step:1 – Configure OKTA OIDC

Step:2 – Cloud Director OIDC Configuration

Bring on your Own OIDC – Tenant Configuration

Share this:

Service Engine High Availability:

Elastic N + M