VMmark User’s Guide

VMmark 4.0.2
October 11, 2024
VMmark User’s Guide

You can find the most up-to-date technical documentation on the VMware website at:
http://www.vmware.com/support/
The VMware Web site also provides the latest product updates.
If you have comments about this document, submit your feedback to:
[email protected]

VMware by Broadcom
3401 Hillview Ave.
Palo Alto, CA 94304
www.vmware.com

Copyright © 2007-2024 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.
For more information, go to https://www.broadcom.com. All trademarks, trade names, service marks, and logos referenced
herein belong to their respective companies.
Revision: 20241011

2 Broadcom
Contents

Quick Start Guide 7

Overview 11
Why Virtualization? 11
Why a Multi-Host Virtual Machine Benchmark? 11
Why a new version of VMmark? 12
Intended Audience 12
Legal Notice 12

What is the VMmark Benchmark? 13

Overview of the VMmark 4.x Benchmark 13
VMmark Benchmark Workloads 15
Scalable Web Simulation: Weathervane 16
E-Commerce Simulation: DVD Store 3.5 16
Social Network Simulation: DeathStarBench 17
Database Workload: NoSQLBench 17
Virtual Machine Cloning and Deployment 17
Dynamic Virtual Machine Relocation Between Servers 18
Dynamic Virtual Machine Relocation Across Storage 18
Simultaneous Server and Storage Virtual Machine Relocation 18
Automated Load Balancing 18
VMmark Client Systems 20
VMmark Harness 20
Contents of VMmark Package 20
VMmark Run and Reporting Rules 20
VMmark 4.x Benchmark Scoring Methodology 21
Example One: Single-Tile Benchmark Scoring 23
Example Two: Multiple-Tile Benchmark Scoring 24
Reference Scores 26
VMmark Version Notes 28
VMmark 4.0.2 28
VMmark 4.0.1 28
VMmark 4.0 28
VMmark 3.1.1 28
VMmark 3.1 28
VMmark 3.0 28
VMmark 2.5.2 28
VMmark 2.5.1 29
VMmark 2.5 29
VMmark 2.1.1 29
VMmark 2.1 29
VMmark 2.0 29
VMmark 1.1.1 29
VMmark 1.1 29
VMmark 1.0 29

VMmark Benchmark Requirements 31

VMmark Version and Settings Requirements 31

Broadcom 3
VMmark User’s Guide

VMmark Hardware Requirements 31

System Under Test Hardware Requirements 31
System Under Test CPU Requirements 32
System Under Test Memory Requirements 32
System Under Test Storage Requirements 32
Other System Under Test Requirements 33
Client Hardware Requirements 33
CPU Requirements for VMmark Clients 33
Memory Requirements for VMmark Clients 33
Storage Requirements for VMmark Clients 33
vCenter Hardware Requirements 34
Network Requirements 34
Network Performance 34
Network Topology 34

Prepare the Infrastructure for VMmark 4.0 Benchmark Tests 37

Potential Security Vulnerabilities 38
Create the Virtualization Infrastructure 39
Install vSphere vCenter Server 40
Install VMware vCenter Server using the OVF Tool 40
Configure an Existing VMware vCenter Server 41
Install and Configure VMware ESXi 41
Configure vCenter Server 41
Configure Networking 43
Configure Time Synchronization on the vCenter Server and the ESXi Hosts 44
Configure Time Synchronization for ESXi Hosts 44
Configure Time Synchronization for vCenter Server 44
Non-VMmark Virtual Machines in a VMmark Cluster 44
Obtain the VMmark Template and Create the Prime Client 46
Obtain the VMmark Template 46
Deploy the VMmark Template 46
Use ovftool to Deploy the VMmark Template 46
Use the vSphere Client to Deploy the VMmark Template 46
Configure the VMmark Template 47
Create the Prime Client 47
Configure the Prime Client 48
Configure Passwordless SSH 49
Deploy and Run VMmark 50
Use Quick Start to Deploy and Run the Benchmark 50
Optional Quick Start Parameters 51
Interpreting Quick Start Results 52
Perform a Full One-Tile VMmark Run 52
Configure Your VMmark Environment for Multiple Tiles 53
Add Tiles to the VMmark Environment 54
Calculate the Number of Simultaneous Infrastructure Operations 54
Make Copies of the VMmark Template for Infrastructure Operations 55
Optional Configuration Changes to the VMmark Environment 55
Update the Prime Client 55
Update VMmark Workload Virtual Machines 55
Update Virtual Hardware Version 56
Update VMmark Tools Version 56
Make One-for-One Replacements of Virtual Hardware 56
Prohibited Changes to Workload Virtual Machines 56
Disk Types for Provisioned VMs 56

4 Broadcom
 Contents

Run the VMmark Benchmark 57

Prepare for a VMmark Run 57
Start a VMmark Run 57
Start a VMmark Run Through the GUI 57
Start a VMmark Run Without a GUI 58
VMmark Results Files 60
Sample VMmark Results File 61
Create and Submit a Benchmark Results Package 63
Verify the Reporter is enabled and set the Disclosure Settings in the VMmark4.properties file, then
run a compliant test 63
Run the disclosure_creator Tool to Generate the disclosure.html File 63
Edit the disclosure.html File 64
Submit the Benchmark Results for Review 65

Optional Configurations and Settings 67

Add an NFS Datastore for Infrastructure Operations 67
Manually Provision the Benchmark 69
Load Parameters from a VMmark4.properties file 69
Set the runtime_seconds Parameter 69
Perform a Partial Tile Run 70
Perform a Benchmark Run Without Infrastructure Workloads 70
Upgrade from Previous Versions of VMmark 70

Troubleshooting 71
Benchmark Provisioning Issues 71
CNS Naming in vCenter 8.0 Update 3 71
Error: “Requested number of processors 4 is more than 1 processors this virtual machine is configured
for” 71
Warning: “Line -1: Unsupported value 'pciBridgeN.present'” 71
Benchmark Execution Issues 72
Keyboard and mouse become unresponsive in a VM 72
Error “All required properties are not set in /root/VMmark4/VMmark4.properties, provisioning failed.
Exiting...” 72
Error Similar to “Could not properly start the data services for run 1-0. Exiting.” 72
Delete Weathervane PVCs Before Removing Kubernetes Cluster 72
Empty “Please wait” Popup Box When Opening STAX 73
No Score in the Score_n_Tile_Test.txt file / “Error: Turbo mode disabled but Duration (Runtime) = 180
seconds. Rerun with -T/--turbo” 73
Warning “Failed to find principal: VSPHERE.LOCAL\WorkloadStorage” 73
Error: “Failed - Feature ‘cpuid.mwait’ was 0, but must be 1.” 73
Cancel a Benchmark Run 73
Manually Cancel a Benchmark Run 74
Automatically Cancel on Error 74
Delete VMmark 4 Tiles 74
Using the VMmark4-ConfigChecker Script 75
Manually Running the VMmark Reporter Scripts 76
Manually Running the Reporter Script 76
How to Enable and Analyze esxtop Performance Data 77
Obtain a Tool to View the esxtop Performance Data 77
Enable esxtop Performance Data Collection in VMmark 77
Capture the esxtop Performance Data 77
Open the esxtop Performance Data for Analysis 77
Example esxtop Performance Data Analysis: CPU Utilization 78

Potential Security Vulnerabilities 79

Broadcom 5
VMmark User’s Guide

6 Broadcom
Quick Start Guide

This quick start guide outlines the steps typically required to install, configure, and run the VMmark
benchmark. This is only an outline; most of the steps in this section include a link to detailed instructions
provided later in the book.

Quick Start Guide

NOTE This section does not include enough detail for a first-time VMmark 4 user to completely set up the
benchmark. We strongly recommend that first-time users follow the more detailed instructions in “Prepare the
Infrastructure for VMmark 4.0 Benchmark Tests” on page 37.

Setting up a VMmark environment for the first time typically takes about 30 minutes of hands-on time and
several hours of hands-off time, while VMmark finishes creating application workloads.

Configuring a Single-Tile VMmark Deployment

This section describes how to configure a single-tile VMmark deployment. Once this is complete, see
“Configure Additional VMmark Tiles” on page 9 for instructions to configure additional tiles.

1 Make sure your environment will meet the minimum requirements detailed in Chapter 2, “VMmark
Benchmark Requirements.”.

2 Install a version of vSphere vCenter Server supported by VMmark 4 (for details and configuration options
see “Install vSphere vCenter Server” on page 40).

3 Install and configure VMware ESXi on all hosts in your VMmark environment, add those hosts to your
vCenter Server, and make sure they can all reach the same shared storage (see “Install and Configure
VMware ESXi” on page 41).
4 Configure vSphere vCenter (see “Configure vCenter Server” on page 41):

a Create a datacenter.

b Create a VMmark systems under test (SUT) cluster in the datacenter, activate DRS for the cluster, set
it to Fully Automated, and set the Migration Threshold to 4, and add your SUT hosts to the cluster.

c Create a VMmark client cluster in the datacenter and add your client host(s) to the cluster.

d If the vCenter Server instance that will be used for the VMmark benchmark isn’t already on a host in
the client cluster, migrate it there.

5 Configure two networks on the ESXi hosts in each cluster. You can achieve this with two physical NICs in
each ESXi host and two vSwitches or with a single NIC with two port groups (see “Configure
Networking” on page 43).

 The first network is the external network; this will generally be VM Network, the default network
configured when installing vSphere.

 The second network is a “private” network; all VMmark workload traffic will use this network.

Broadcom 7
VMmark User’s Guide

6 Configure the vCenter server and the ESXi hosts in both the SUT and client clusters to an NTP server (or
server pool) external to the SUT (for detailed requirements see “Configure Time Synchronization on the
vCenter Server and the ESXi Hosts” on page 44).

7 Obtain the latest VMmark template from http://www.vmware.com/products/vmmark.html.

8 Use ovftool to deploy the VMmark template to the SUT cluster using a command similar to the
following, but customized for your environment:
ovftool --diskMode=thin --noSSLVerify --acceptAllEulas --datastore=[datastore_name]
/path/to/VMmark-4.*.*-nnn.ova vi://[email protected]:[VC
password]@[VC IP]/[Datacenter name]/host/[SUT cluster name]/

Alternately, use the vSphere Client to deploy the VMmark template (see “Use the vSphere Client to
Deploy the VMmark Template” on page 46).

9 Verify VMware Tools time synchronization is activated (i.e., that there are check marks for both the
Synchronize at startup and resume (recommended) and Synchronize time periodically options) and
change the default passwords if desired (both tasks described in “Configure the VMmark Template” on
page 47).

Quick Start Guide

10 Clone the VMmark template to a new VM, named PrimeClient, in the Client cluster, then edit the new
VM’s virtual hardware to add two disks (one 50GB, one 50GB or larger for benchmark results) and a
second NIC attached to the private network (see “Create the Prime Client” on page 47).

11 Power on the prime client VM and use the remote console to log in (credentials: root and vmmark) and
perform the following tasks (as detailed in “Configure the Prime Client” on page 48):

a Connect the first NIC to the external network and configure it as needed for your network
environment.

b Connect the second NIC to the private network and configure it as needed.

NOTE To use the default private IP address for the prime client, select Method Manual, then click
Add. Use Address 198.18.4.251 and Netmask 255.255.0.0; leave Gateway blank. Using these
values will reduce the amount of input required later for provisioning and running.

c Restart the virtual machine.

d Convert this virtual machine into the prime client by running the make-prime script:
make-prime.sh primeclient_private_ip_address
(replace primeclient_private_ip_address with the private IP address for the prime client).

e Make sure the prime client is configured for the correct time zone.

f Restart the virtual machine.

12 Configure your vSphere environment to allow passwordless SSH from the prime client to each ESXi host
in the two clusters and the vCenter Server (as detailed in “Configure Passwordless SSH” on page 49).

13 Deploy and run the benchmark using the new Quick Start mode in interactive mode, which prompts for
the minimum set of parameters required:
vmmark4service --mode quick_start --vcenter_ip <IP> --vcenter_password '<password>'
(for more details and other options, see “Use Quick Start to Deploy and Run the Benchmark” on page 50).

14 Update the relevant parameters in /root/VMmark4/VMmark4.properties with the contents of

/root/VMmark4/VMmark4-quickstart.properties., as described in “Perform a Full One-Tile
VMmark Run” on page 52.

15 Use the GUI to start a VMmark run (for alternate methods or more detail see “Perform a Full One-Tile
VMmark Run” on page 52):

a Double click on the VMmark4-STAX icon on the desktop.

b From the within the STAX window:

 Click on Submit New Job.

8 Broadcom
 Quick Start Guide

 Point XML Job File > Local Machine > Filename to:
/root/VMmark4/source/vmmark4_main.xml

 Add a job name.

 Click Submit New Job.

The run should launch and complete. Troubleshooting and additional information can be found
throughout the rest of this book and in Appendix B, “Troubleshooting.”

Configure Additional VMmark Tiles

Once you’ve configured and tested a single VMmark tile, as described above, this section describes how
to configure additional tiles, if desired.

1 Create additional tiles, as described in “Add Tiles to the VMmark Environment” on page 54:

a Power off your tile 0 VMs.

b Copy the vmmark4service command you used for the Quick Start process, but with the following

Quick Start Guide

changes:

 Replace --mode quick_start with --mode provision

 Change the --tile_number parameter to your desired number of tiles.

For example, to add one more tile to the single tile you already have, replace --tile_number 1
with --tile_number 2.

For example:
OUTPUT=provision-second-tile.txt
nohup vmmark4service --mode provision --vcenter_ip <ip_addr> \
--vcenter_password '<password>' --datacenter <datacenter> --cluster <cluster> \
--client_cluster <cluster2> --tile_number 2 --provisioning_source <template-VM> \
--datastore <datastore_name> --client_datastore <client_datastore> \
--infra_datastore '<different_datastore>' --network_label 'testbed' > $OUTPUT 2>&1 &

c Double check the /etc/hosts file on your prime client to ensure everything is configured correctly.

d Copy VMmark4-quickstart.properties to VMmark4.properties, then edit

VMmark4.properties file, changing Tiles to your desired number.

e Run the benchmark as normal.

2 If your system under test has four or more hosts:

a Calculate the number of simultaneous infrastructure operations (see “Calculate the Number of
Simultaneous Infrastructure Operations” on page 54).

b Make copies of the VMmark template for infrastructure operations (see “Make Copies of the
VMmark Template for Infrastructure Operations” on page 55).

3 Use the GUI to start a VMmark run (for alternate methods or more detail see “Start a VMmark Run” on
page 57):

a Double click on the VMmark4-STAX icon on the desktop.

b From the within the STAX window:

 Click on Submit New Job.

 Point XML Job File > Local Machine > Filename to:
/root/VMmark4/source/vmmark4_main.xml

 Add a job name.

 Click Submit New Job.

Broadcom 9
VMmark User’s Guide

Quick Start Guide

10 Broadcom
Overview

This document describes the VMmark® virtual infrastructure benchmarking utility, it provides instructions for
configuring a system for VMmark testing, it details the steps required to perform such a test, and it discusses
the interpretation of the data acquired.

Why Virtualization?
Server virtualization is the process of running multiple virtual computer systems (called “virtual machines”)
on a single physical server.

Virtualization can increase IT agility, flexibility, and scalability while creating significant cost savings.
Workloads get deployed faster, performance and availability increase, and operations can be automated,
resulting in IT that’s simpler to manage and less costly to own and operate.

Why a Multi-Host Virtual Machine Benchmark?

Traditional single-workload performance and scalability benchmarks for non-virtualized environments were
developed with neither virtual machines nor server consolidation in mind. Their metrics focus on achieving
maximum system performance for a single workload by driving at least one of the underlying hardware
resources to saturation. These types of benchmarks don’t provide sufficient insight into the scalability of
virtual environments supporting multiple simultaneous workloads. VMmark first introduced the concept of
a multi-workload server consolidation benchmark with version 1. Then VMmark 2 and 3 built on the
multi-workload concept and added multi-host requirements. As a multi-host virtual machine benchmark,
VMmark continues to address the increasing virtualization of bursty and heavy workloads, dynamic virtual
machine relocation (VMware vSphere® vMotion®), dynamic datastore relocation (VMware vSphere® Storage
vMotion®), and the automation of many provisioning and administrative tasks across large-scale multi-host
environments. Load balancing across multiple hosts can also affect application performance. While still
focusing on user-centric application performance, these versions of the benchmark also accounted for the
effects of this infrastructure activity on overall platform performance.

In the years since the introduction of VMmark 2 and VMmark 3, the demands on modern data centers have
increased. New technologies have become commonplace in the data center, while more mature technologies
continue to evolve. Traditional relational database workloads coexist with NoSQL applications and
workloads. Virtual machine based architectures coexist with containerized workloads and Kubernetes
orchestration. In addition, social network style interactions are commonplace for employee communication for
everything from Human Resources tasks to IT Services, and even employee networks. VMmark 4 was created
with this innovation and data center evolution in mind. However, although VMmark 4 represents and
incorporates the increased complexity seen in today’s data centers, we’ve simplified the benchmarker
experience. You’ll see easier initial installation, faster initial deployment times, and an “automation first”
mindset for faster time-to-value in your performance characterization endeavors.

Broadcom 11
VMmark User’s Guide

Why a new version of VMmark?

In today’s dynamic environment, it is imperative for users to have meaningful and precise metrics in order to
effectively compare the suitability and performance of different platforms for virtual environments. VMmark
4 builds on the legacy of earlier VMmark versions and includes new workloads to better represent the
complexity of today’s enterprise data centers.

Intended Audience
This document is written for users with a relatively advanced understanding of system administration.
However, familiarity with virtualization software and with benchmarking methodology is not assumed.

Legal Notice
This documentation contains information including but not limited to the installation and operation of the
Software. Modifications, additions, deletions or other updates (“Modifications”) to the information may be
incorporated in future releases.

VMware®, Inc., its affiliates or subsidiaries (“VMware”) are not responsible for any Modifications made to the
published version of this documentation unless performed by VMware. All information is provided “as is”
and is believed to be accurate at the time of publication. VMware shall not be liable for any damages arising
out of or in connection with the information and recommended actions provided herein (if any), including
direct, indirect, consequential damages, loss of business profits or special damages, even if VMware has been
advised of the possibility of such damages.

12 Broadcom
What is the VMmark Benchmark? 1
This chapter provides an overview of the VMmark Benchmark and describes how it works. The chapter
consists of the following sections:

 “Overview of the VMmark 4.x Benchmark” on page 13

 “VMmark Benchmark Workloads” on page 15

 “VMmark Client Systems” on page 20

 “VMmark Harness” on page 20

 “Contents of VMmark Package” on page 20

 “VMmark 4.x Benchmark Scoring Methodology” on page 21

 “Reference Scores” on page 26

 “VMmark Version Notes” on page 28

 “VMmark Run and Reporting Rules” on page 20

Overview of the VMmark 4.x Benchmark

Enterprise server consolidation typically collects several diverse workloads onto a virtualization platform—a
collection of physical servers accessing shared storage and network resources. Dynamic load balancing
techniques ensure that all system resources such as CPU, network, and disk are more efficiently utilized. In
fact, virtual environments tend to function more smoothly when resource demands are balanced across
multiple resources.

The unit of work for a benchmark of virtualized consolidation environments can be naturally defined as a
collection of virtual machines executing a set of diverse workloads. The VMmark 4.x Benchmark follows the
convention of previous VMmark versions and refers to this unit of work as a tile. The total number of VMmark
tiles a multi-host platform can accommodate gives a coarse-grain measure of that platform's consolidation
capacity. This concept is similar to some server benchmarks, such as TPC-C (see http://www.tpc.org/tpcc), that
scale the workload in a step-wise fashion to increase the system load.

Tiles are relatively heavyweight objects that cannot by themselves capture small variations in platform
performance. To address this, both the number of tiles and the performance of each individual workload
determine the overall benchmark score.

Each workload within a tile is constrained to execute at less than full utilization of its virtual machine.
However the performance of each workload can vary to a degree with the speed and capabilities of the
underlying platform. The addition of a fast disk array, for example, might result in disk-centric workloads
producing a more favorable score. These variations can capture system improvements that do not warrant the
addition of another tile. However the workload throttling forces the use of additional tiles for large jumps in
platform performance.

Broadcom 13
VMmark User’s Guide

When a tile is added, the performance of the workloads in existing tiles might decrease. If the system has not
been overcommitted, however, and the minimum quality-of-service metrics are met, the aggregate score,
including the new tile, should increase. The result is a flexible benchmark metric that provides a measure of
the total number of workloads that can be supported by a particular multi-host platform as well as the overall
performance level within the workload virtual machines.

VMmark 4.x generates a realistic measure of platform performance by incorporating a variety of

platform-level workloads in addition to traditional application-level workloads. Live migration both of virtual
machines and of their underlying disk files is a powerful and commonplace tool in virtualized data centers.
Likewise, the ability to quickly clone and deploy virtual machines onto available resources has transformed
traditional server provisioning. Each of these operations place non-trivial resource demands on the underlying
infrastructure and must be included in order to accurately characterize platform performance.

Virtual machine migration, clone and deploy, storage migration, and vMotion without shared storage
operations are repeatedly performed on a set of the workload virtual machines to simulate the additional
resource demands typical in production data centers. Additionally, automated load balancing is enabled to
ensure application-level workloads are relocated to satisfy their resource needs as the computational loads
vary among the individual hosts over time.

VMmark is designed to benchmark the performance of the virtualization software and hardware and is not
designed as a benchmark of any other software component.

Figure 1-1. Conceptual Example of a Cluster of Hosts Running a VMmark Tile

14 Broadcom
 Chapter 1 What is the VMmark Benchmark?

VMmark Benchmark Workloads

A meaningful tiled consolidation benchmark is based on a set of relevant data center workloads. A survey of
data center applications led to the inclusion of the following application workloads:

 Standby system

 Scalable web simulation

 E-Commerce simulation

 Social network simulation

 Database workload

Rather than develop workloads from scratch, existing workloads or benchmarks were used where possible.
This reduces implementation effort and provides a well-understood foundation upon which to build.

The workloads chosen for use in VMmark are representative of popular applications commonly run by
VMware customers.

VMmark leverages components of these application workload virtual machines to perform some common
virtualization procedures, which we call infrastructure workloads, including:

 Virtual machine cloning and deployment

 Dynamic virtual machine relocation between servers

 Dynamic virtual machine relocation across storage

 Simultaneous server and storage virtual machine relocation

 Automated load balancing

These application and infrastructure workloads, along with their virtual machine names and virtual
hardware, are summarized in Table 1-1.

Table 1-1. VMmark 4 Workloads and Virtual Machine Summary

Workload Benchmark VM Names Virtual Machine Platform

Scalable Web Weathervane AuctionWebA Ubuntu 22.04.2 64-bit, 4 vCPU, 12GB RAM, 50GB disk
Simulation (VMs) AuctionWebB
AuctionWebC

AuctionWebDNosql Ubuntu 22.04.2 64-bit, 4 vCPU, 20GB RAM, 50GB & 83GB
disks

AuctionWebEDB Ubuntu 22.04.2 64-bit, 4vCPU, 16GB RAM, 50GB & 24GB
disks

AuctionWebF Ubuntu 22.04.2 64-bit, 2 vCPU, 8GB RAM, 50GB disk

AuctionAppA Ubuntu 22.04.2 64-bit, 4 vCPU, 8GB RAM, 50GB disk

AuctionAppB
AuctionAppC

Scalable Web Weathervane AuctionKA Ubuntu 22.04.2 64-bit, 8 vCPU, 32GB RAM, 50GB disk
Simulation (containers) AuctionKB Distributed across the Kubernetes cluster: 100GB PVC, 20GB
AuctionKC PVC, six 6GB PVCs
AuctionKD

E-Commerce DVD Store 3.5 DS35WebA Ubuntu 22.04.2 64-bit, 6 vCPU, 0.5GB RAM, 50GB disk
Simulation DS35WebB
DS35WebC

DS35DB Ubuntu 22.04.2 64-bit, 24 vCPU, 48GB RAM, 50GB base disk,
250GB data disk

Social DeathStarBench SocialNetwork Ubuntu 22.04.2 64-bit, 36 vCPU, 16GB RAM, 50GB base disk,
Network 50GB data disk

Broadcom 15
VMmark User’s Guide

Table 1-1. VMmark 4 Workloads and Virtual Machine Summary (Continued)

Workload Benchmark VM Names Virtual Machine Platform

Database NoSQLBench NoSQLBenchA Ubuntu 22.04.2 64-bit, 8 vCPU, 8GB RAM, 50GB disk
Simulation NoSQLBenchB
NoSQLBenchC

Standby None Standby Ubuntu 22.04.2 64-bit, 1 vCPU, 2GB RAM, 50GB disk

Deploy None Deploy Ubuntu 22.04.2 64-bit, 4 vCPU, 16GB RAM, 10GB, 20GB, &
50GB disks

Prime Client None PrimeClient Ubuntu 22.04.2 64-bit, 4 vCPU, 32GB RAM, three 50GB disks

Client None Client Ubuntu 22.04.2 64-bit, 64 vCPU, 96GB RAM, 50GB disk

The following sections discuss each of these application and infrastructure workloads.

Scalable Web Simulation: Weathervane

Weathervane is an application-level performance benchmark implementing an interactive web site where
users bid in real-time auctions. The Weathervane application, called Auction, is a scalable web application that
interacts with multiple support services for web, messaging, coordination, and database operations.

In VMmark 4.x, Weathervane runs with two workload variants: one workload that runs the application and
service processes on virtual machines, a second workload that runs the application and services in Docker
containers on Kubernetes nodes (VMs). The elements of these two variants are performed across 13 workload
virtual machines running on the system under test (detailed in Table 1-1). Load is generated by drivers
running on a client virtual machine simulating users interacting with the Weathervane Auction application.

The Kubernetes nodes (i.e., VMs) running the Weathervane workload containers use vSphere Container
Storage to store persistent data. During provisioning, the VMmark harness creates Persistent Volume Claims
(PVCs) on the CSI datastore configured for that tile. There are limitations to managing VMs that use PVCs. The
Kubernetes Node VMs (AuctionKA, AuctionKB, AuctionKC, and AuctionKD) can’t be moved off the cluster
or datastore on which they were created. Instead, the VMmark tile should be deleted cleanly (as described
below) and reprovisioned on the new cluster or datastore. For this reason, you should create tiles on the cluster
and datastore where you’ll be performing the VMmark benchmark runs. Deletion of VMmark 4 tiles should
be done not through vCenter using the GUI or PowerShell, but should instead be done using the VMmark 4
service as described in “Delete VMmark 4 Tiles” on page 74.

QoS requirements specify 99th-percentile response-time for each operation and a required mix of operations
performed by all users.

E-Commerce Simulation: DVD Store 3.5

Databases running transactional workloads support a wide array of applications, typically as part of a
multi-tier architecture. Databases tend to be resource intensive and exercise most server and infrastructure
components. In many cases, database systems also face strict response-time demands. Transaction processing
often exhibits bursty behavior, resulting in widely varying resource demands over time. The ability of the
underlying platform to support usage spikes is critical to maintaining acceptable performance.

DVD Store Version 3.5 (DS35) is a complete online e-commerce test application with a back-end database
component, a web application layer, and driver programs (see http://github.com/dvdstore/ds35 for additional
information). The DS35 driver simulates users logging into a web server and browsing a catalog of products
using basic queries. Users may select items for purchase then proceed to check out or continue shopping. Users
are also able to read reviews, rate reviews, provide new reviews, and join as a premium member. Each web
server communicates with a database server that maintains user account and inventory data.

Though not necessary in order to use it as part of VMmark, additional information about DS35 is available at
http://github.com/dvdstore/ds35.

16 Broadcom
 Chapter 1 What is the VMmark Benchmark?

The DS35 workload used in VMmark 4.x utilizes four virtual machines in each tile—three web servers and one
database server—all running 64-bit Ubuntu 22.04. The three virtual machines in the DS35 web tier
(DS35WebA, DS35WebB, and DS35WebC) each run the Apache 2.4.52 web server and have 6 vCPUs and 0.5GB
of memory. The DS35 database tier runs the PostgreSQL database on a virtual machine with 24 vCPUs and
48GB of memory.

One of the web servers delivers a constant load to the database throughout each benchmark interval. The other
two web servers deliver periodic load to the database during the benchmark interval to create a bursty overall
load profile and varying resource demands. For VMmark 4.x, each web server is driven by 64 driver threads
when active. The performance metric for this workload is the total number of transactions per minute.
Minimum quality-of-service metrics must also be met.

Social Network Simulation: DeathStarBench

DeathStarBench is an open-source benchmark suite for cloud microservices. VMmark 4.x uses one of these
benchmarks, Social Network, which simulates users registering, logging in, creating posts, reading posts and
users' timelines, following/unfollowing users (and receiving recommendations on users to follow), and
searching for posts or users.

The Social Network application consists of a number of Docker containers, including a frontend NGINX web
server, middle-tier logic services, a backend of memcached and Redis for in-memory caching and storage, as
well as MongoDB database instances to store users, posts, timelines, etc.

In VMmark 4.x, the Social Network application runs on a virtual machine with 36 vCPUs and 16GB of memory.
The Social Network client generates an HTTP workload with a predefined number of threads, connections,
and throughput. The performance metric for this workload is number of transfers per second.

More details on DeathStarBench applications can be found at https://github.com/delimitrou/DeathStarBench.

A characterization of their behavior can be found at “An Open-Source Benchmark Suite for Microservices and
Their Hardware-Software Implications for Cloud and Edge Systems,” Y. Gan et al., ASPLOS 2019.

Database Workload: NoSQLBench

NoSQLBench (https://github.com/nosqlbench) is a performance testing tool for NoSQL environments. It was
originally developed at DataStax as an internal tool specifically for Cassandra based workloads, but was
enhanced to support many types of workloads in the NoSQL space and was open-sourced in March of 2020.

In VMmark 4.x NoSQLBench has been configured to run as a workload on a 3-node Cassandra clustered
NoSQL database. The three virtual machines (NoSQLBenchA, NoSQLBenchB, and NoSQLBenchC) each have
8 vCPUs and 8GB of memory. The NoSQLBench client uses the cql-keyvalue workload profile (one of the
profiles included with NoSQLBench). Results are determined in terms of request throughput and response
time, which are obtained from the NoSQLBench results files.

Virtual Machine Cloning and Deployment

Creating a new virtual machine and installing a guest operating system and applications can be time
consuming. Using virtual machine cloning technology, administrators can make many copies of a virtual
machine using a single installation and configuration process. Cloning, configuration, and deployment
operations create bursty loads on platform resources, particularly the storage subsystem as the virtual
machine files are copied.

This infrastructure workload clones the VMmark template virtual machine, powers-on and pings the clone,
takes a snapshot, performs a hot add of CPU and memory, takes another snapshot, creates a small MySQL
database, then reverts the snapshots, again pings the clone, and finally deletes it. The benchmark then waits
40 seconds and repeats this process, continuing for the duration of the benchmark period. The number of
concurrent clone and deploy operations increases with the number of tiles and the number of hosts in the
benchmark cluster. The performance metric used is the number of clone and deploy operations per hour.

Broadcom 17
VMmark User’s Guide

Dynamic Virtual Machine Relocation Between Servers

Live migration technology such as VMware vMotion® leverages the complete virtualization of servers,
storage, and networking to move an entire running virtual machine seamlessly from one server to another.
During a vMotion operation, the active memory and precise execution state of a virtual machine is rapidly
transmitted over a high speed network from one physical server to another and access to the virtual machine’s
disk storage is instantly switched to the new physical host. This transition can result in bursty loads on
platform resources, particularly the networking subsystem. VMmark mimics the manual relocation of a
virtual machine, which can be a common task performed by an administrator.

This infrastructure workload acts on one of the Standby virtual machines selected in a round-robin fashion
from among all the tiles. A destination host is selected at random from among all hosts in the benchmark
cluster (other than the virtual machine’s current host), the virtual machine is moved to the destination host,
left there for two minutes, then returned to its original host. VMmark then waits another two minutes and
repeats this process, continuing for the duration of the benchmark period. The number of concurrent
relocation operations increases with the number of tiles and the number of hosts in the benchmark cluster. The
performance metric used is the number of relocations per hour.

Dynamic Virtual Machine Relocation Across Storage

Live migration of virtual machine disk files across or within storage arrays enables enormous flexibility for
storage maintenance, upgrades, and load balancing. Storage relocations can create bursty loads on platform
resources, particularly the storage subsystem.

In this infrastructure workload, VMmark relocates a virtual machine's disk files to a maintenance partition,
then returns them to their original location. This round-trip approach models an administrator temporarily
evacuating a disk partition, performing maintenance on the storage system, then returning the system to its
initial state.

This infrastructure workload acts on one of the AuctionWebF virtual machines selected in a round-robin
fashion from among all the tiles. The virtual machine’s files are moved to the maintenance partition, left there
for two minutes, then moved back to their original location. VMmark then waits another two minutes and
repeats this process, continuing for the duration of the benchmark period. The number of concurrent storage
relocation operations increases with the number of tiles and the number of hosts in the benchmark cluster. The
performance metric used is the number of relocations per hour.

Simultaneous Server and Storage Virtual Machine Relocation

The live migration of virtual machines simultaneously across both servers and storage (vMotion without
shared storage) allows even more flexibility than either capability alone. This infrastructure workload
produces a combination of the infrastructure loads created by the individual operations.

In this infrastructure workload, VMmark uses vMotion to relocate a virtual machine while simultaneously
invoking the storage relocation of the same virtual machine's disk files to a maintenance partition. After 2
minutes the virtual machine is returned to its original host and the files are returned to their original location.
VMmark then waits another 5 minutes and repeats the process. This workload models an administrator
temporarily evacuating a host and disk partition, performing maintenance on the host and/or storage system,
then returning the system to its initial state.

This infrastructure workload acts on one of the DS35WebA virtual machines selected in a round-robin fashion
from among all the tiles. The number of concurrent relocation operations increases with the number of tiles
and the number of hosts in the benchmark cluster. The performance metric used is the number of relocations
per hour.

Automated Load Balancing

Automatically balancing resource demands among multiple physical servers using technology such as
VMware vSphere® Distributed Resource Scheduler™ (DRS) has become a fundamental part of modern
virtualized data centers. Intelligently allocating and balancing resources allows the underlying platform to
respond effectively to bursty load conditions even when utilizations are high.

18 Broadcom
 Chapter 1 What is the VMmark Benchmark?

VMmark requires that DRS be enabled and running at (or above) a specific level to ensure that rebalancing
occurs in a timely manner when utilizations are high. This should improve overall performance by addressing
load imbalances occurring during the benchmark interval.

Figure 1-2 illustrates the overall VMmark architecture.

Figure 1-2. Example VMmark Architecture

Broadcom 19
VMmark User’s Guide

VMmark Client Systems

Each VMmark tile requires a client system which runs software to drive most of the workloads. The prime
client also runs the VMmark Harness software that can start, stop, and measure the performance of the
workloads.

The client systems must run in a vSphere cluster that is not part of the system under test.

NOTE Running the client systems in the same cluster as the SUT might work; however, doing so is
unsupported and any results obtained on such a configuration would be non-compliant.

VMmark Harness
The VMmark Harness is a utility run on the prime client system that can start and stop the applications
running on the workload virtual machines and can report the results of a test run.

The VMmark Harness is based on the open-source Software Testing Automation Framework (STAF, see
http://staf.sourceforge.net/index.php) and its companion execution engine, STAX. These tools support the
development and running of distributed coordinated tests across heterogeneous machines and operating
systems.

The VMmark Harness consists of several STAX XML modules, the VMmark4.properties file, and several
workload-specific configuration files. The main STAX module, vmmark4_main.xml, processes the
VMmark4.properties file to configure the test to be run. Each workload has its own
<workload>_functions.xml module that contains the workload-specific code needed to initialize the test,
run the test, and collect the results.

The VMmark4.properties file defines the actual test, identifying all the clients and server virtual machines
involved in the test, the number of tiles to be run, and the workloads within each tile.

After the VMmark4.properties file has been processed, the VMmark Harness performs pre-run system and
timing validation and initiates the setup phase for the VMmark infrastructure operations and for each
workload in each tile. After the setup has completed, the VMmark Harness simultaneously initiates the
individual workloads in all the tiles. When the workload runs have completed, the harness again validates the
timing, then collects the results into a results directory.

Contents of VMmark Package

VMmark 4 is distributed as an ova file named VMmark-4.*.*-version.ova (where version is the version of
the release).

Other relevant files are available on the VMmark download page.

VMmark Run and Reporting Rules

In addition to the guidelines presented in this User’s Guide, specific run and reporting rules must be followed
for a VMmark test to be considered compliant and for the results of that test to be publishable. These VMmark
Run and Reporting Rules are available at the VMmark web page:
http://www.vmware.com/products/vmmark.html

NOTE VMware updates the VMmark Run and Reporting Rules from time to time. Before performing a
VMmark test, you should confirm that you have the latest available version.

20 Broadcom
 Chapter 1 What is the VMmark Benchmark?

VMmark 4.x Benchmark Scoring Methodology

A VMmark 4.x score aggregates the throughput metrics of all application and infrastructure workloads to
create a single overall benchmark score that can be used to quickly compare different platform configurations.
Specific application workloads must also pass minimum quality-of-service requirements for the benchmark
result to be considered compliant. The scoring methodology is conceptually straightforward despite the
underlying complexity and large scope of the benchmark.

After a VMmark benchmark test run completes, each individual application and infrastructure workload
reports its relevant performance metric. These performance metrics are shown in Table 1-2.

Table 1-2. Individual VMmark Workload Metrics

Workload Name Application(s) Metric

WVAuctionVM Nginx, Tomcat, Zookeeper, RabbitMQ, PostgreSQL, Cassandra Operations/second

WVAuctionK8S Docker, Kubernetes, Nginx, Tomcat, Zookeeper, RabbitMQ, Operations/second

PostgreSQL, Cassandra

DVDStoreA Apache, PostgreSQL Operations/minute

DVDStoreB Apache, PostgreSQL Operations/minute

DVDStoreC Apache, PostgreSQL Operations/minute

NoSQLBenchA Cassandra Calls/second

NoSQLBenchB Cassandra Calls/second

NoSQLBenchC Cassandra Calls/second

SocialNetwork Docker, Nginx, RabbitMQ, Memcached, Redis, MongoDB MB transferred/second

Standby server None None

vMotion None VM migrations/hour

Storage vMotion None VM migrations/hour

XvMotion None VM migrations/hour

Clone and deploy None VM deployments/hour

NOTE The standby server workload does not produce a metric that affects the benchmark score. However the
standby server needs to answer to a periodic heartbeat request in order for the VMmark test to be considered
compliant. Likewise, DRS must be enabled to allow the platform to automatically balance resources during
the benchmark run. DRS does not produce a metric but will affect the performance of other workloads by
managing the overall resource allocations to improve performance and provide stability.

The DVD Store implementation in VMmark 4.x is a multi-tier workload containing three web server virtual
machines with varying load patterns accessing a single database virtual machine. For scoring purposes, each
web server generates a separate results file to be processed independently from the others. This approach
provides a clearer understanding of platform behavior.

The Weathervane workload in VMmark 4.x uses two independent instances of the Weathervane Auction
application, one running in VMs, the other in containers. Because in today's data centers it's increasingly
common to have self-scaling applications that dynamically add and remove resources to meet demands, the
container instance takes advantage of Weathervane's elasticity-related capabilities to add and remove an
application server and a web server throughout the run. This elastic component (along with the cyclical
application profile generated by DVD Store 3.5) allows VMmark 4.x to more accurately represent today's
bursty environments.

These metrics are collected at frequent intervals during the course of the run. The standard VMmark 4.x
workload is designed to run for at least 3 hours with workload metrics reported every 60 seconds. This means
that rather than having a single number upon completion of a test run, the user will have a series of numbers
for each of the workloads. The series of data points for each workload is averaged to generate a single score
for that workload which is then listed in the VMmark results file (Score_N_Tile_Test.txt).

Broadcom 21
VMmark User’s Guide

Normalization allows the integration of the different component metrics into an overall score. A reference
system is commonly used for normalization when computing benchmark scores. For example, SPEC CPU
2017 (see http://www.spec.org/cpu2017/) took this approach, using a Sun Fire V490 with 2100 MHz
UltraSPARC-IV+ processors as its reference platform. VMmark 4.x measures consolidation workloads within
virtual environments and therefore requires a reference platform capable of successfully running a single tile.

The steady state for the benchmark is defined as the middle two hours of the three-hour run. The first and last
half hours are the ramp-up and ramp-down times, respectively. The steady state is further divided into three
40-minute phases. For each of the 40-minute phases we compute the overall result for the platform and select
the median score of the three as the reported score.

After a valid run, the metrics of the application workloads within each tile are computed and aggregated into
a score for that tile. This aggregation is performed by first normalizing the different performance metrics (such
as Actions/minute and operations/minute) with respect to a reference platform. Then a geometric mean of the
normalized scores is computed as the final score for the tile. The resulting per-tile scores are then summed to
create the application-workload portion of the final metric.

The metrics for the infrastructure workloads are aggregated separately using the same mathematical
technique of normalization with respect to a reference platform followed by the computation of the geometric
mean. Unlike the application workloads, the infrastructure workloads are not scaled explicitly by the user.
Consequently, the infrastructure workloads are compiled as a single group and no multi-tile sums are
required.

The final benchmark score is then computed as a weighted average of the application-workload component
and the infrastructure-workload component. VMmark 4.x gives weights of 80% to the application-workload
component and 20% to the infrastructure-workload component. These weights were chosen to reflect the
relative contribution of infrastructure and application workloads to overall resource demands.

The benchmark helps measure the virtualization overheads of the individual workloads as well as the
scalability of the entire system. Therefore results for multi-tile runs are reported as the aggregate score for all
tiles, the individual scores for each of the tiles, and the scores for the workloads within the tiles as well as the
individual scores for each infrastructure workload.

If any of the workloads within any tile fails to run, produces errors during a run, or fails its minimum
quality-of-service requirement, that entire VMmark run is considered to be invalid. This applies to programs
running on both the servers and the client systems. Also, the configuration of the workloads, the versions of
the benchmarks, operating systems, tools, and all other software used must conform to the specifications in
the VMmark documentation.

To illustrate the scoring methodology, consider the following two examples. The first example demonstrates
how to compute the score for a single-tile benchmark run, while the second example demonstrates how to
compute the scores for a multiple-tile benchmark run.

For these examples, assume the reference system had the scores shown in Table 1-3.

Table 1-3. Example Reference System Workload Scores (Artificial Data)

Workload Name Score

WVAuctionVM 1000 operations/second

WVAuctionK8S 1000 operations/second

DVDStoreA 1000 operations/minute

DVDStoreB 1000 operations/minute

DVDStoreC 1000 operations/minute

NoSQLBenchA 1000 calls/second

NoSQLBenchB 1000 calls/second

NoSQLBenchC 1000 calls/second

SocialNetwork 1000 MB transferred/second

Standby server n/a

22 Broadcom
 Chapter 1 What is the VMmark Benchmark?

Table 1-3. Example Reference System Workload Scores (Artificial Data)

Workload Name Score

vMotion 10 VM migrations/hour

Storage vMotion 10 VM migrations/hour

XvMotion 10 VM migrations/hour

Clone and deploy 10 VM deployments/hour

Example One: Single-Tile Benchmark Scoring

Suppose you perform a single-tile VMmark run and, according to the methodology described earlier, you
remove the first and last thirty minutes, calculate results for each of the three 40-minute phases, and select the
median of the three results to obtain the scores shown in Table 1-4.

Table 1-4. Single-Tile Example Test System Workload Scores (Artificial Data)
Workload Name Score

WVAuctionVM 940 operations/second

WVAuctionK8S 910 operations/second

DVDStoreA 1020 operations/minute

DVDStoreB 980 operations/minute

DVDStoreC 970 operations/minute

NoSQLBenchA 1100 calls/second

NoSQLBenchB 1150 calls/second

NoSQLBenchC 1120 calls/second

SocialNetwork 970 MB transferred/second

Standby server None

vMotion 11 VM migrations/hour

Storage vMotion 9 VM migrations/hour

XvMotion 8 VM migrations/hour

Clone and deploy 9 VM deployments/hour

To compute the score for this tile, you first compute each workload's normalized scores by dividing the score
for each workload by the reference score for that workload:

WVAuctionVM 940/1000 = 0.94

WVAuctionK8S 910/1000 = 0.91

DVDStoreA 1020/1000 = 1.02

DVDStoreB 980/1000 = 0.98

Broadcom 23
VMmark User’s Guide

DVDStoreC 970/1000 = 0.97

NoSQLBenchA 1100/1000 = 1.10

NoSQLBenchB 1150/1000 = 1.15

NoSQLBenchC 1120/1000 = 1.12

SocialNetwork 970/1000 = 0.97

Standby server n/a n/a

vMotion 11/10 = 1.10

Storage vMotion 9/10 = 0.90

XvMotion 8/10 = 0.80

Clone and deploy 9/10 = 0.90

You would then combine the normalized scores for the application workloads using a geometric mean:

(0.94 * 0.91 * 1.02 * 0.98 * 0.97 * 1.10 * 1.15 * 1.12 * 0.97)^(1/9) = 1.01

Next, you would combine the normalized scores for the infrastructure workloads using a geometric mean:

(0.90 * 1.10 * 0.90 * 0.80)^(1/4) = 0.92

Finally, you would combine these two geometric means using a weighted average (80% for application
workloads, 20% for infrastructure workloads):

(0.8 * 1.01) + (0.2 * 0.92) = 0.99

The VMmark score for this tile is 0.99. For reporting, the VMmark results file will include this score as well as
the individual scores for all the workloads (both raw and normalized).

Example Two: Multiple-Tile Benchmark Scoring

This example demonstrates the scoring for multiple tiles. The scores for each tile in a multiple-tile run are
calculated just as the score for the tile in a single-tile run is calculated. Suppose you perform a VMmark run
containing four tiles and obtain the scores shown in Table 1-5.

Table 1-5. Multiple-Tile Example Test System Workload Scores (Artificial Data)
Workload Name Tile 1 Score Tile 2 Score Tile 3 Score Tile 4 Score

WVAuctionVM 940 950 890 940

WVAuctionK8S 910 920 915 930

DVDStoreA 950 1000 990 970

DVDStoreB 940 1010 1020 970

DVDStoreC 930 1020 960 1050

NoSQLBenchA 1100 1110 1000 1200

NoSQLBenchB 1150 1130 1050 1190

NoSQLBenchC 1120 1110 1100 1180

SocialNetwork 980 1000 990 970

Standby server n/a n/a n/a n/a

vMotion 22

Storage vMotion 11

24 Broadcom
 Chapter 1 What is the VMmark Benchmark?

Table 1-5. Multiple-Tile Example Test System Workload Scores (Artificial Data)
Workload Name Tile 1 Score Tile 2 Score Tile 3 Score Tile 4 Score

XvMotion 26

Clone and deploy 21

The geometric means of the normalized scores for the application workload tiles would be:

Geometric mean of
normalized scores: Tile 1: 1.00 Tile 2: 1.03 Tile 3: 0.99 Tile 4: 1.04

The application workload portion of the VMmark score would be the sum of these geometric means:

1.00 + 1.03 + 0.99 + 1.04 = 4.05

The infrastructure workload portion of the VMmark score would be:

Geometric mean of
normalized scores: 1.91

The overall VMmark score for the system would be the weighted average of these two numbers:

(0.8 * 4.05) + (0.2 * 1.91) = 3.62

Along with the overall VMmark score of 3.62, the VMmark results file will include both the individual tile
scores and the workload scores (both raw and normalized).

Broadcom 25
VMmark User’s Guide

Reference Scores
The VMmark 4.x reference scores were obtained using the environment detailed in Table 1-6.

Table 1-6. VMmark Reference System Details

Servers

Servers Two HPE ProLiant DL385 Gen11

Processors Two AMD EPYC™ 9354P 3.25 GHz 32-core processors per server
Four processors total

Cores 32 cores per processor; 64 cores per server; 128 cores total

RAM 1.5TB DDR5 memory per server; 3TB total

Storage

Primary Storage VMware vSAN 8.0 Update 2 ESA - All Flash

Infrastructure storage One HPE SN1610E 32Gb 2-port FC HBA per server, two total (connected at 16Gb)
connectivity

vSAN storage devices Eight HPE 1.6TB NVMe MU SCN U.3 PM1735a SSDs per server, 16 total

Infrastructure storage Dell EMC Unity 650F all-flash array

Network

Workload traffic Broadcom BCM57414 NetXtreme-E NIC (connected at 10Gb)

vSAN and vMotion traffic Mellanox ConnectX-6 Dx EN 100GbE

Client

Servers Two HPE ProLiant DL385 Gen11

Processors Two AMD EPYC™ 9354P 3.25 GHz 32-core processors per server
Four processors total

Cores 32 cores per processor; 64 cores per server; 128 cores total

RAM 1.5TB DDR5 memory per server; 3TB total

Network connectivity Broadcom BCM57414 NIC (connected at 10Gb)

Storage connectivity One HPE SN1610E 32Gb 2-port FC HBA per server, two total (connected at 16Gb)

Storage array Dell EMC Unity 650F all-flash array

Software

Server virtualization VMware ESXi 8.0 Update 2, build 22380479

Client virtualization VMware ESXi 8.0 Update 2, build 22380479

Data center management VMware vCenter Server 8.0 Update 2a, build 22617221

The reference scores obtained on this system are shown in Table 1-7.

Table 1-7. Reference System Workload Scores

Workload Name Score

WVAuctionVM 14000 operations/second

WVAuctionK8S 9200 operations/second

DVDStoreA 2900 operations/minute

DVDStoreB 2150 operations/minute

DVDStoreC 1530 operations/minute

NoSQLBenchA 56500 calls/second

NoSQLBenchB 56500 calls/second

NoSQLBenchC 56500 calls/second

26 Broadcom
 Chapter 1 What is the VMmark Benchmark?

Table 1-7. Reference System Workload Scores (Continued)

Workload Name Score

SocialNetwork 72MB transferred/second

Standby server n/a

vMotion 29.5 VM migrations/hour

Storage vMotion 13 VM migrations/hour

XvMotion 13 VM migrations/hour

Clone and deploy 14.5 VM deployments/hour

Broadcom 27
VMmark User’s Guide

VMmark Version Notes

For the latest information about a particular VMmark release, including security notices and any changes that
occurred after the release of this guide, view the appropriate Release Notes, available at the VMmark
download page.

VMmark 4.0.2
VMmark 4.0.2 contains updates to improve workload stability, logging, and error handling as well as
improved result post-processing, examples, and documentation. VMmark 4.0.2 benchmark scores are
considered directly comparable to VMmark 4.0 and 4.0.1 benchmark scores.

VMmark 4.0.1
VMmark 4.0.1 corrects an issue with deploy operation automatic calculations, improves the restore process for
multiple application workloads, and includes minor updates for ancillary scripts and documentation.
VMmark 4.0.1 benchmark scores are considered directly comparable to VMmark 4.0 benchmark scores.

VMmark 4.0
VMmark 4.0 adds new NoSQLBench and Social Network workloads, an updated Weathervane workload that
now runs in both virtual machines and Docker containers on Kubernetes nodes, an updated DVD Store 3.5
workload that uses PostgreSQL, higher overall workload levels, support for partial tiles, a completely new
Quick Start mode, an automated disclosure creator, message and results delivery via Slack or Google Chat, and
a variety of other updates and improvements. Since there are new workloads and significantly higher load
levels in VMmark 4.0 as compared to VMmark 3.x, the benchmark scores are in no way considered
comparable.

VMmark 3.1.1
VMmark 3.1.1 fixes a bug in the Weathervane VMmark workload. VMmark 3.1.1 benchmark scores are
considered directly comparable to VMmark 3.0 and 3.1 benchmark scores (though see the VMmark Run and
Reporting Rules for limitations on comparisons between results with different security vulnerability
mitigations).

VMmark 3.1
VMmark 3.1 contains updates to improve workload scalability and enhance security as well as including new
requirements regarding security vulnerability mitigations. VMmark 3.1 benchmark scores are considered
directly comparable to VMmark 3.0 benchmark scores (though see the VMmark Run and Reporting Rules for
limitations on comparisons between results with different security vulnerability mitigations).

VMmark 3.0
VMmark 3.0 provided a completely new, highly-automated benchmark installation process; it used free or
open-source software throughout, eliminating the need to purchase any software; it replaced DVD Store 2
with DVD Store 3 and added Weathervane, a new application-level cloud performance benchmark; and it
added XvMotion as a new infrastructure workload. The workloads and load levels of VMmark 3.0 were
completely changed from VMmark 2.x and the benchmark scores were in no way considered comparable.

VMmark 2.5.2
VMmark 2.5.2 was a minor maintenance release. It included an updated VMmark Benchmarking Guide.
VMmark 2.5.2 benchmark scores were considered directly comparable to VMmark 2.0, 2.1, 2.1.1, 2.5, and 2.5.1
benchmark scores.

28 Broadcom
 Chapter 1 What is the VMmark Benchmark?

VMmark 2.5.1
VMmark 2.5.1 was a minor maintenance release. It included an updated VMmark Benchmarking Guide.
VMmark 2.5.1 benchmark scores were considered directly comparable to VMmark 2.0, 2.1, 2.1.1, and 2.5
benchmark scores.

VMmark 2.5
VMmark 2.5 added support for optional power monitoring, support for client systems running additional
versions of Windows Server 2008, support for the VMware vCenter Server® Appliance™, a new Turbo Mode,
message and results delivery via Growl/Prowl, and a variety of other updates and improvements. VMmark
2.5 benchmark scores were considered directly comparable to VMmark 2.0, 2.1, and 2.1.1 benchmark scores.

VMmark 2.1.1
VMmark 2.1.1 was a minor maintenance release. VMmark 2.1.1 benchmark scores were considered directly
comparable to VMmark 2.0 and 2.1 benchmark scores.

VMmark 2.1
VMmark 2.1 added support for client systems running certain versions of Windows Server 2008 (in addition
to Windows Server 2003, which was supported in VMmark 2.0) as well as support for virtualized clients
(subject to certain conditions). The benchmark harness now scales the storage relocation workload in the same
fashion as the other infrastructure workloads. The VMmark Benchmarking Guide is updated to reflect these
changes. VMmark 2.1 benchmark scores were considered directly comparable to VMmark 2.0 benchmark
scores.

VMmark 2.0
While VMmark 1.x was designed as a single-system consolidation benchmark consisting of six isolated
single-tier workloads, VMmark 2.0 was designed as a multi-host benchmark reflecting typical, modern-day
usage of virtualized infrastructure. VMmark 2.0 consisted of two single-tier application workloads, two
multi-tier application workloads, and four infrastructure-level workloads. The workloads and load levels of
VMmark 2.0 were completely changed from VMmark 1.x and the benchmark scores were in no way
considered comparable.

VMmark 1.1.1
VMmark 1.1.1 was a minor maintenance release. It included updates to the VMmark Benchmarking Guide,
updated reporting script and disclosure template, and new versions of the Run and Reporting Rules and
Review Panel Guidelines and Procedures. VMmark 1.1.1 benchmark scores were considered directly
comparable to VMmark 1.0 and 1.1 benchmark scores.

VMmark 1.1
In order to reflect the increasing use of 64-bit operating systems in data centers, VMmark 1.1 used 64-bit
operating systems and applications in three of the workload virtual machines (Java server, Web server, and
Database server). The workloads and load levels were unchanged from VMmark 1.0, however, and the
VMmark 1.0 and 1.1 benchmark scores were thus considered directly comparable.

VMmark 1.0
This was the initial release of VMmark.

Broadcom 29
VMmark User’s Guide

30 Broadcom
VMmark Benchmark Requirements 2
This chapter describes the hardware and software required in order to perform VMmark benchmark testing.
It consists of the following sections:

 “VMmark Version and Settings Requirements” on page 31

 “VMmark Hardware Requirements” on page 31

VMmark Version and Settings Requirements

In order for a VMmark benchmark run to be compliant, the following version and settings requirements must
be met:

 Both VMware vCenter® and VMware ESXi™ versions must be Publicly Available or planned to be
Publicly Available within 90 days (using the definition of “Publicly Available” in the VMmark Run and
Reporting Rules).

 The systems under test must be running ESXi version 8.0 Update 2 or later.

 The virtualization environment used must include Distributed Resource Scheduler (DRS) as a supported
feature. Such environments include VMware Cloud Foundation (VCF) and VMware vSphere Foundation
(VVF). Note that vSphere Standard does not include DRS.

 The DRS migration threshold on the SUT cluster must be set to the most aggressive or the second-most
aggressive setting.

 There must be no DRS rules set for the SUT cluster.

VMmark Hardware Requirements

This section describes the hardware required in order to run VMmark.

System Under Test Hardware Requirements

The system under test (SUT) must use server hardware supported by the ESXi version being used and must
be configured to meet or exceed the minimum CPU, minimum memory, and other basic requirements for that
version. The subsections below provide information about the required storage, memory, and other elements
of the test environment.

Broadcom 31
VMmark User’s Guide

System Under Test CPU Requirements

Each tile in the VMmark benchmark uses a total of 169 vCPUs (the vCPUs required by the prime and tile clients
are not included here because they’re not part of the SUT; instead they’re addressed in “CPU Requirements for
VMmark Clients” on page 33). While CPU overcommitment is allowed for the SUT, it does have the potential
to adversely affect the VMmark score.

NOTE The first VMmark tile also uses an additional 4 vCPUs for a deploy VM. Depending on the number of
tiles and hosts in your VMmark run, additional deploy VMs might be required, each of which uses an
additional 4 vCPUs. See “Calculate the Number of Simultaneous Infrastructure Operations” on page 54 for
more details.

System Under Test Memory Requirements

Each tile in the VMmark benchmark allocates a total of 324GB of memory (the memory required by the prime
and tile clients is not included in this total because it’s not part of the SUT; instead it’s addressed in “Memory
Requirements for VMmark Clients” on page 33). While memory overcommitment is allowed for the SUT, it
does have the potential to adversely affect the VMmark score.

NOTE The first VMmark tile also allocates an additional 16GB of memory for a deploy VM. Depending on the
number of tiles and hosts in your VMmark run, additional deploy VMs might be required, each of which
allocates an additional 16GB of memory. See “Calculate the Number of Simultaneous Infrastructure
Operations” on page 54 for more detail.

System Under Test Storage Requirements

Configure the system under test with enough shared datastore capacity to hold the disks and paging files for
all the virtual machines required for the VMmark Benchmark runs. This is approximately 2003GB per tile (not
counting the storage space required by the prime and tile clients, addressed in “Storage Requirements for
VMmark Clients” on page 33), less if thin provisioned.

NOTE The first VMmark tile also uses an additional 96GB of storage for a deploy VM. Depending on the
number of tiles and hosts in your VMmark run, additional deploy VMs might be required, each of which uses
an additional 96GB of storage. See “Calculate the Number of Simultaneous Infrastructure Operations” on
page 54 for more detail.

NOTE In order to provide source and target datastores for the storage relocation operations that are part of
the benchmark, VMmark requires a minimum of two datastore partitions.

NOTE If your VMmark runs will use VMware vSAN™ as the primary storage solution, a secondary storage
solution is needed for infrastructure operations. This secondary storage could be any one of a variety of
storage types, including a traditional SAN, an NFS share, iSCSI, or VMware vSAN HCI Mesh. For one NFS
option, see “Add an NFS Datastore for Infrastructure Operations” on page 67.

Additionally, the benchmark requires that all ESXi hosts used in a test have access to the same shared storage.

NOTE Some vSAN configurations, such as two-node vSAN clusters, require a witness host. For VMmark the
vSAN witness function must be performed by a dedicated virtual appliance running on the VMmark client
hardware. For more information about vSAN witness appliances, see the following resources: What Are vSAN
Two-Node Clusters, vSAN 2-node Cluster Guide, and Deploying a vSAN Witness Appliance.

The VMmark benchmark needs high-throughput, low-latency storage. While the exact bandwidth
requirements will vary based on other aspects of the environment, a single VMmark tile can drive about 50,000
IOPS (Input/Output Operations Per Second), with additional tiles typically each driving somewhat less. The
latency requirements will also vary based on other aspects of the environment; a review of published VMmark
results will provide a sense of the storage solutions that work with VMmark. Thus in addition to ensuring that
you have enough storage capacity, you should also make sure your storage system will have adequate
performance.

VMmark can be used with any storage hardware type that meets the above requirements.

32 Broadcom
 Chapter 2 VMmark Benchmark Requirements

Other System Under Test Requirements

In order to support the vMotion and DRS infrastructure operations included in VMmark, the benchmark
requires a minimum of two ESXi hosts and requires that all ESXi hosts used in a test be vMotion compatible.

Client Hardware Requirements

Each VMmark deployment requires a single prime client virtual machine. Each VMmark tile requires its own
tile client virtual machine. These prime and tile client virtual machines must meet or exceed the specifications
detailed in this section and in the subsections below.

 The client virtual machines must not use resources that are part of the system being tested. This means
they must not be run on the same ESXi servers as the benchmark workloads and ideally should use
separate storage resources.

 All tile clients in a VMmark deployment must have identical resource, scheduling, and tuning
configurations.

 The servers hosting the VMmark client virtual machines must be running VMware ESXi 8.0 Update 2 or
later.

 The servers hosting the VMmark client virtual machines must be enterprise-class hardware.

CPU Requirements for VMmark Clients

 The prime client uses 4 vCPUs. Each tile client uses 64 vCPUs.

 The CPUs in the servers hosting the prime and tile client virtual machines must not be overcommitted.

NOTE This means the sum of the vCPUs configured for all the virtual machines running on a host must
not exceed the number of logical CPUs that host contains.

Each hyper-threaded core can be considered two CPUs when calculating the level of CPU commitment.

 In order not to affect the VMmark score, we recommend that the hosts’ physical processors be equivalent
to or faster than AMD EPYC™ 9004 Series (“Genoa”).

NOTE As a very rough approximation, client systems will need about half as much processing power per
tile as the systems under test. In addition, compute resources must be allocated for the single prime client.

 If desired, users can increase the vCPU count for the prime client virtual machine.

 If desired, users can increase the vCPU count for the tile client virtual machines (provided all tile clients
are provisioned identically).

Memory Requirements for VMmark Clients

 The prime client requires 32GB of memory. Each tile client requires 96GB of memory.

 The memory in the servers hosting the client virtual machines must not be overcommitted.

NOTE This means the sum of the memory allocated to all the virtual machines running on a host must
not exceed the amount of physical memory that host contains.

 If desired, users can increase the memory for the prime client virtual machine.

 If desired, users can increase the memory for the tile client virtual machines (provided all tile clients are
provisioned identically).

Storage Requirements for VMmark Clients

 In typical deployments, the prime client requires 182GB of storage for its disks and paging files. In
environments with a large number of ESXi hosts, the prime client can require more storage space.

Broadcom 33
VMmark User’s Guide

 Each tile client requires 146GB of storage for its disks and paging files.

 The physical storage used for the virtual clients should be typical of a modern data center and should offer
performance sufficient to meet the client virtual machines’ resource requirements without introducing
delays that might affect the benchmark results.

vCenter Hardware Requirements

The host on which the vCenter Server runs must be supported by the vCenter version being used and must be
configured to meet or exceed the minimum CPU, minimum memory, and other basic requirements for that
product and version.

NOTE The vCenter Server virtual machine must be on a client host. See “Install vSphere vCenter Server” on
page 40.

Network Requirements
VMmark tests require a network between the server system and the client systems as well as a separate
dedicated vMotion network for infrastructure operations. Some configurations might need multiple network
links for optimal performance.
We recommend that the VMmark systems be on dedicated private networks and that the server and client
systems be connected with links of at least 25 Gb/s speed.

For compliant benchmark runs, IP addresses must be statically assigned. There must be no DHCP server
reachable from any of the workload virtual machines.

NOTE Although VMmark can be run with DHCP-assigned IP addresses for the workload and client virtual
machines, such deployments are not supported and can not be used to generate compliant benchmark runs.

Network Performance
The physical network infrastructure should be typical of a modern data center and should offer performance
sufficient to meet the virtual machines’ resource requirements without introducing delays that might affect
the benchmark results.

Network bandwidth requirements can vary significantly with different environments, but each VMmark tile
can potentially consume 11 Gb/s of network bandwidth.

Network Topology
The most accurate benchmark results can be obtained when the VMmark environment is on a private network
and, during the VMmark tests, neither the client systems nor the SUT systems are connected to any other
network, whether an internal company network or the external Internet. Use of a completely private network
ensures that extraneous network traffic does not impact the VMmark tests. However, the VMmark prime client
can be connected to a company-wide network in order to provide user access to the VMmark installation.

For compliant benchmark runs, configure the ESXi servers, each workload virtual machine, and each client
virtual machine with a static IP address. The prime client can, in addition to the private network connection,
also be configured with a DHCP-assigned IP address on a company-wide network. (VMmark can be run with
DHCP-assigned IP addresses for the workload and client virtual machines, but such deployments can not be used
to generate compliant benchmark runs.)

34 Broadcom
 Chapter 2 VMmark Benchmark Requirements

Figure 2-1. Sample VMmark Benchmark Network Configuration (Not Shown: vCenter Server, Clients,
Dedicated vMotion Network)

Broadcom 35
VMmark User’s Guide

36 Broadcom
Prepare the Infrastructure for
VMmark 4.0 Benchmark Tests 3
This document describes the steps required in order to prepare the infrastructure for VMmark 4.0 tests. It
consists of the following sections:

 “Potential Security Vulnerabilities” on page 38

 “Create the Virtualization Infrastructure” on page 39

 “Obtain the VMmark Template and Create the Prime Client” on page 46

 “Deploy and Run VMmark” on page 50

 “Configure Your VMmark Environment for Multiple Tiles” on page 53

 “Optional Configuration Changes to the VMmark Environment” on page 55

Broadcom 37
VMmark User’s Guide

Potential Security Vulnerabilities

VMmark uses a wide variety of open source software. In some cases, that software has known security
vulnerabilities. The VMmark team believes that the risks of using VMmark are minimal due to the
non-sensitive nature of the data VMmark contains and creates as well as it being run within an isolated,
non-production environment. However, each organization must make their own risk assessment. To assist in
doing so, Table 3-1 lists the known security vulnerabilities with a CVSS3 score of 9.0 or greater contained in
the latest release of VMmark 4.0. Details of each one are provided in Appendix C.

Table 3-1. Known Security Vulnerabilities

Package Name Vulnerability

zlib1g-1:1.2.13.dfsg-1 CVE-2023-45853

GNU C Library-2.32 CVE-2019-1010022, CVE-2021-33574,

CVE-2021-35942, CVE-2022-23219, CVE-2022-23218,
CVE-2023-0687

busybox-1.30.1, busybox-initramfs-1:1.30.1-7ubuntu3, CVE-2022-48174

busybox-static-1:1.30.1-7ubuntu3

apparmor-3.0.4-2ubuntu2.2, libapparmor1-3.0.4-2ubuntu2.3 CVE-2016-1585

libarchive13-3.6.0-1ubuntu1, libarchive-3.6.0 CVE-2022-36227

php8.1-8.1.2-1ubuntu2.13, php8.1-readline-8.1.2-1ubuntu2.14, CVE-2016-9138

php8.1-opcache-8.1.2-1ubuntu2.14, php8.1-common-8.1.2-1ubuntu2.14,
php8.1-cli-8.1.2-1ubuntu2.14, php8.1-8.1.2-1ubuntu2.14,
libapache2-mod-php8.1-8.1.2-1ubuntu2.14

libc6-2.31-13+deb11u5 GNU, C Library-2.36.9000, CVE-2019-1010022, CVE-2023-0687

libc6-dev-2.36-9+deb12u4, libc6-2.36-9+deb12u4,
libc-dev-bin-2.36-9+deb12u4, libc-bin-2.36-9+deb12u4

github.com/moby/buildkit-v0.12.1 CVE-2024-23652, CVE-2024-23653

Linux-Pam-v1.5.2 CVE-2022-28321

OpenJPEG-v2.4.0 CVE-2021-3575

linux-libc-dev-6.1.76-1 CVE-2024-26584

Linux Kernel-5.17.6, Linux Kernel-v5.15.15, Linux Kernel-v5.16.18 CVE-2023-5178, CVE-2022-47939, CVE-2022-43945,

CVE-2024-26584, CVE-2023-38431, CVE-2023-38432,
CVE-2023-38430, CVE-2023-40791, CVE-2023-38428,
CVE-2023-40791, ZDI-23-979

38 Broadcom
 Chapter 3 Prepare the Infrastructure for VMmark 4.0 Benchmark Tests

Create the Virtualization Infrastructure

The following subsections describe the steps needed to prepare the virtualization infrastructure for VMmark
tests. See Figure 3-1 for an example of what a complete 1-tile deployment will look like.

NOTE VMmark and the benchmark hardware and software components (vCenter, ESXi, and the VMmark
template, clients, workload virtual machines, systems under test, storage, etc.) must be used only in an isolated
test environment. None of the benchmark hardware or software components can ever be connected to a
production network or transitioned to a production environment.

NOTE Don’t include spaces or special characters in vSphere names (usernames, datacenters, clusters, etc.).

Figure 3-1. Example 1-Tile VMmark 4 Deployment (Note: actual template name might differ)

Broadcom 39
VMmark User’s Guide

Install vSphere vCenter Server

Install a version of VMware vSphere vCenter Server compatible with VMmark 4.0.

NOTE This instance of vCenter Server must be run on the VMmark client cluster, which you’ll create in
“Configure vCenter Server” on page 41. You can thus save a step by putting it on a host that will become part
of that cluster.

VMmark 4.0 is designed to run with vCenter Server 8.0 Update 2 or later. In order for a VMmark
benchmarking run to be compliant, the installed version of vCenter Server must either be Publicly Available
(using the definition of “Publicly Available” in the VMmark Run and Reporting Rules) or must meet the
pre-release requirements detailed in the VMmark Run and Reporting Rules. Make sure that all relevant
updates and patches are installed.

Make note of the vCenter Server user credentials, which need to be specified in the VMmark4.properties file.
This user doesn’t need to be [email protected], but must have administrative privileges.

NOTE All ESXi hosts must be able to reach the same shared storage in order to allow the VMmark
infrastructure operations.

Proceed to the appropriate section below, “Install VMware vCenter Server using the OVF Tool” or “Configure
an Existing VMware vCenter Server.”

Install VMware vCenter Server using the OVF Tool

If you have not yet installed vCenter Server, you can deploy it in an automated fashion using the Open
Virtualization Format Tool (ovftool). There are numerous properties you can configure, as shown in the
example command below. The key properties for VMmark are in blue below; these include enabling SSH,
setting the default shell to Bash, enabling time synchronization, and configuring an NTP server.

NOTE The vCenter Server Appliance .ova can be found under the vcsa directory of the .iso, and ovftool
is under the ovftool subdirectory.

ovftool --name="vCSA-VMmark" --datastore=[datastore_name] --acceptAllEulas --noSSLVerify \

--diskMode=thin --deploymentOption="small" \
--net:"Network 1"="VM Network" --network="VM Network" \
--powerOn --X:injectOvfEnv \
--prop:guestinfo.cis.vmdir.domain-name="vsphere.local" \
--prop:guestinfo.cis.vmdir.password='VMmark123!' \
--prop:guestinfo.cis.appliance.root.passwd='VMmark123!' \
--prop:guestinfo.cis.appliance.ssh.enabled=true \
--prop:guestinfo.cis.appliance.root.shell='/bin/bash' \
--prop:guestinfo.cis.appliance.net.addr.family="ipv4" \
--prop:guestinfo.cis.appliance.net.mode="static" \
--prop:guestinfo.cis.appliance.net.prefix="24" \
--prop:guestinfo.cis.appliance.net.addr="X.X.X.145" \
--prop:guestinfo.cis.appliance.net.dns.servers="X.X.X.1,X.X.X.2" \
--prop:guestinfo.cis.appliance.net.gateway="X.X.X.251" \
--prop:guestinfo.cis.appliance.time.tools-sync="True" \
--prop:guestinfo.cis.appliance.ntp.servers="X.X.X.99" \
/vcsa/VMware-vCenter-Server-Appliance-8.0.2.00000-22385739_OVF10.ova \
'vi://root:VMmark123!@[ESXi IP]'

NOTE You’ll need to replace the variables in the above example (IP addresses, network labels, passwords, VM
names, etc.) with values from your own environment.

40 Broadcom
 Chapter 3 Prepare the Infrastructure for VMmark 4.0 Benchmark Tests

Configure an Existing VMware vCenter Server

If you’ve already installed vCenter Server, or you choose to install vCenter Server using the traditional GUI
method instead of using the ovftool as described above, you’ll need to perform the following configuration
tasks:

1 Enable SSH login (but not Bash Shell):

a In the vCenter Server Appliance Management Interface, click Access, and click Edit.

b Enable SSH login.

NOTE When the GUI is used to enable the Bash Shell, the change doesn’t persist, so we don’t make
that change here.

c Click OK to save the settings.

2 Set Bash as the default shell by running this command on the prime client:
ssh -o PubkeyAuthentication=no root@[vCenter IP] "shell chsh -s /bin/bash root"

You should be prompted for the root password, then you should see this message:
Shell access is granted to root

For more information or troubleshooting, see KB article 2107727.

NOTE The vCenter Server UI might report the Bash Shell is disabled; this is expected.

Install and Configure VMware ESXi

Install and configure VMware ESXi version 8.0 Update 2 or later on all servers to be used in the test according
to VMware instructions. The ESXi version must either be a Publicly Available (using the definition of “Publicly
Available” in the VMmark Run and Reporting Rules) or must meet the pre-release requirements detailed in
the VMmark Run and Reporting Rules. Make sure that all relevant updates and patches are installed.

Make sure that hardware on which you are installing ESXi meets the requirements outlined in “System Under
Test Hardware Requirements” on page 33, including the datastore capacity and vMotion compatibility.

Use the vSphere Client to add all the ESXi hosts you plan to use in the test to the vCenter Server.

Configure vCenter Server

1 Using the vSphere Client, create a new datacenter on your vCenter Server (in the Hosts and Clusters pane
right-click the vCenter Server, select New Datacenter..., enter a name, then click OK).

NOTE If your vCenter Server already has a datacenter, you can optionally use that one instead.

NOTE Don’t include spaces or special characters in vSphere names (usernames, datacenters, clusters,
etc.).

2 Create a VMmark systems under test (SUT) cluster in the datacenter as follows:

a Still in the Hosts and Clusters view, right-click on the new datacenter and select New Cluster....

b At the New Cluster Basics window:

i Enter a name for the cluster (recording the name for later use; this will be
vCServerVMmark4Cluster in the VMmark4.properties file).

NOTE Each cluster name must be unique, even across datacenters.

ii Next to vSphere DRS use the slider to activate DRS.

iii Next to vSAN use the slider to activate vSAN if desired.

iv Select or unselect Enable vSAN ESA as desired.

Broadcom 41
VMmark User’s Guide

v Click on Next to leave the New Cluster Basics window.

vi Click on Next to leave the New Cluster Image window.

vii Click on Finish to close the New Cluster Review window.

3 With the newly created cluster selected, click on the Configure, then click on vSphere DRS.

i Click on EDIT... near the upper right corner.

ii In Edit Cluster Settings, under Automation, make sure the Automation Level is set to Fully
Automated.

iii Set Migration Threshold, move the slider to the second-most aggressive setting (this is the
fourth position counting from the left, as shown in Figure 3-2).

NOTE Those wishing to further increase the DRS aggressiveness can, optionally, set the
Migration threshold slider to the most aggressive setting (this is the fifth—or rightmost—
position).

iv Click on OK to close the Edit Cluster Settings window.

Figure 3-2. Setting the DRS Migration Threshold to 4

4 Add hosts to your systems under test cluster by right clicking on the cluster name and selecting Add
Host....

5 In order to allow VMmark to perform infrastructure operations, make sure that all the ESXi hosts in your
systems under test cluster can reach the same shared storage.

NOTE We recommend that the ESXi hosts not be connected to any storage resources that aren’t part of
the same VMmark installation. This is to avoid potential problems caused by identically named virtual
machines on different VMmark installations.

42 Broadcom
 Chapter 3 Prepare the Infrastructure for VMmark 4.0 Benchmark Tests

NOTE If your VMmark runs will use vSAN storage as the primary storage solution, a secondary storage
solution is needed for infrastructure operations. This secondary storage could be any one of a variety of
storage types, including a traditional SAN, an NFS share, iSCSI, or VMware vSAN HCI Mesh. For one
NFS option, see “Add an NFS Datastore for Infrastructure Operations” on page 67.

6 Make sure that all hosts can vMotion to all other hosts within the systems under test cluster.

7 Create a VMmark client cluster in the data center (recording the name for later use; this will be
vCServerClientCluster in the VMmark4.properties file) and add one or more ESXi hosts to it.

NOTE Optionally, you can activate DRS for this client cluster and set it to any aggressiveness level you
choose.

8 If the vCenter Server instance that will be used for the VMmark benchmark isn’t already on a host in the
client cluster, migrate it there now.

Configure Networking
Configure two networks each on both the SUT cluster and the client cluster.

NOTE This document is written for environments with two prime client NICs. Configuring the prime client
with only one NIC (connected to the private network) is not covered in this User's Guide but is nevertheless a
compliant configuration for benchmarking purposes.

NOTE Some VMmark 4 workloads use Docker containers, which by default are configured to use IP addresses
in the ranges 172.17.*.* or 172.18.*.*. You must therefore not use these IP address ranges for either of the two
networks described in this section.

The first network must be an external network; this is generally VM Network, the default network configured
when installing vSphere.

NOTE Though referred to as an “external network,” this network must not be connected to production
systems.

The second network must be a “private” network; all VMmark workload traffic uses this network.

You can achieve these two networks in either of the following ways:

 with two physical NICs in each ESXi host and two vSwitches, or

 with a single NIC in each ESXi host with two port groups.

If you use a distributed vSwitch, it needs to be configured with ephemeral port bindings. See Add a
Distributed Port Group for more information.

For guidance on configuring a network in vSphere 8.0, see Create a vSphere Standard Switch or Create a
vSphere Distributed Switch.

Broadcom 43
VMmark User’s Guide

Configure Time Synchronization on the vCenter Server and the ESXi Hosts
The time on the vCenter Server and the ESXi hosts (in both the systems under test and client clusters) must be
synchronized to a single NTP server or a pool of NTP servers (such as pool.ntp.org). In either case, the NTP
server (or server pool) must be external to the SUT. (Client and workload virtual machines must use VMware
Tools to set their clocks from their ESXi host.)

NOTE In addition:

 If using a pool of NTP servers, variations of more than a few seconds between the servers in that pool
could negatively impact benchmark results.

 The vCenter Server, the workload and client ESXi hosts, and the workload and client virtual machines
(described later) must all be set to the USA standard date and time format (that is, month/day/year) as
opposed to the UK standard (that is, day/month/year).

 The vCenter Server and the workload and client virtual machines (described later) must all be set to the
same time zone.

The following sections describe how to configure this time synchronization.

Configure Time Synchronization for ESXi Hosts

To use the vSphere Client to configure ESXi hosts to synchronize with an external NTP server, follow the
instructions at this link:

http://kb.vmware.com/kb/57147

To confirm that NTP is working correctly, run the following command from each ESXi host:
ntpq -p

If NTP is working correctly on the ESXi host, you’ll see something similar to the following:
remote refid st t when poll reach delay offset jitter
==============================================================================
*118.163.81.61 198.18.0.3 2 u 841 1024 377 1.967 0.414 0.362

If NTP is not working correctly on the ESXi host, you’ll see something more like this:
remote refid st t when poll reach delay offset jitter
==============================================================================
193.168.1.100 .INIT. 16 u - 1024 0 0.000 0.000 0.000

If you don’t see an asterisk (“*”) in the “remote” column, as shown above, NTP is likely not working correctly.
Other indications of trouble include a “-” in the “when” column and zeros in many of the other columns.

If needed, you can find more information here:

https://kb.vmware.com/s/article/1005092

Configure Time Synchronization for vCenter Server

To configure vCenter Server to synchronize with an external NTP server, follow the instructions at this link:

http://kb.vmware.com/kb/57146

Non-VMmark Virtual Machines in a VMmark Cluster

Some environments might require one or more non-VMmark virtual machines to run in the VMmark systems
under test cluster. When a VMmark run is initiated the presence of such non-VMmark virtual machines will
cause the benchmark to generate a warning. Such virtual machines must have been approved by VMware in
order for a benchmark run to be compliant.

NOTE vSphere Cluster Services (vCLS) agent VMs are approved for use in compliant benchmark runs.

44 Broadcom
 Chapter 3 Prepare the Infrastructure for VMmark 4.0 Benchmark Tests

For questions about non-VMmark virtual machines in the VMmark systems under test cluster, contact
VMware at [email protected].

Broadcom 45
VMmark User’s Guide

Obtain the VMmark Template and Create the Prime Client

VMmark 4.x uses a single template as the source for all client and workload virtual machines used in the
benchmark. This section describes obtaining that template and creating the first client (the “prime client”).

NOTE VMmark and the benchmark hardware and software components (vCenter, ESXi, and the VMmark
template, clients, workload virtual machines, systems under test, storage, etc.) must be used only in an isolated
test environment. None of the benchmark hardware or software components can ever be connected to a
production network or transitioned to a production environment.

Obtain the VMmark Template

The VMmark template can be obtained using the download link on the VMmark pages of the VMware
website:

http://www.vmware.com/products/vmmark.html

Deploy the VMmark Template

Once you’ve obtained the .ova format VMmark template (vmmark-4.*.*-nnn), deploy the template to your
VMmark systems under test cluster using either ovftool or the vSphere Client, as described in the following
two sections.

Use ovftool to Deploy the VMmark Template

To use the Open Virtualization Format Tool (ovftool) to deploy the VMmark template, use a command
similar to the following, but customized for your environment:
ovftool --diskMode=thin --noSSLVerify --acceptAllEulas --datastore=[datastore_name]
/path/to/VMmark-4.*.*-nnn.ova vi://[email protected]:[VC password]@[VC
IP]/[Datacenter name]/host/[SUT cluster name]/

Use the vSphere Client to Deploy the VMmark Template

To use the vSphere Client to deploy the VMmark template, follow these steps:

1 From the vSphere Client, in the inventory pane, select Hosts and Clusters.

2 If needed, expand the tree on the left hand side to show your clusters.

3 Select your VMmark systems under test cluster.

4 Select Actions on the top bar, then select Deploy OVF Template.

5 Under Select template, click Local file, browse to select the .ova template file, then click Next.

6 Under Select name and location, leave the Name: field at the default, select the folder to which you’d like
to deploy the template, then click Next.

NOTE The VMmark 4 template must be placed in the systems under test cluster.

7 Under Select a resource, select where you’d like to run the template, then click Next.

8 Under Review details, click Next.

9 Under Select storage, change the virtual disk format if desired, select the datastore where you’d like to
deploy the .ova, then click Next.

NOTE The disk type (that is, Thin, Thick Lazy, or Thick Eager) of the primary disk for provisioned VMs
will be equivalent to the selection used for the template.

The disk type (that is, Thin, Thick Lazy, or Thick Eager) of any additional disks for provisioned VMs is
controlled by the value of the extra_disk_mode parameter in the VMmark4.properties file.

10 Under Select networks, select your external network (which is VM Network by default), then click Next.

46 Broadcom
 Chapter 3 Prepare the Infrastructure for VMmark 4.0 Benchmark Tests

11 Under Ready to complete, click Finish.

The OVA takes a few minutes to deploy.

NOTE Though this virtual machine is referred to as the “VMmark template,” it should be left as a virtual
machine as opposed to converting it to a template in vCenter.

Configure the VMmark Template

Whether you used ovftool or the vSphere Client to deploy the VMmark template, perform the following
configuration steps:

1 Verify VMware Tools time synchronization:

a From the vSphere Client, right-click on the newly-deployed VMmark template and select Edit
Settings....

b Select the VM Options tab.

c Expand VMware Tools.

d Make sure time synchronization is activated by placing marks in the check boxes for both the
Synchronize at startup and resume (recommended) and Synchronize time periodically options.

e Click OK.

2 If desired, change the passwords for the VMmark template’s root and vmmark4 accounts. These changes
propagate from the VMmark template to the VMmark prime client and to the tiles provisioned by that
prime client.

NOTE The default credentials for the VMmark template are:

login: root, password: vmmark4
login: vmmark4, password: vmmark4

Create the Prime Client

Follow these steps to clone the VMmark template to a new prime client:

1 From the vSphere Client, right click on the newly-deployed VMmark-4.*.*-nnn virtual machine and select
Clone > Clone to Virtual Machine….

2 Under Select a name and folder enter PrimeClient, select the location for the prime client, then click
Next.

3 Under Select a compute resource, expand the tree and select your VMmark client cluster (and, if DRS is
not activated for the client cluster, select an ESXi host), then click Next.

4 Under Select storage, select your desired datastore to provision the prime client, then click Next.

NOTE You can also change the virtual disk format and/or the VM storage policy if desired.

5 Under Select clone options, don’t select any options (remove the check from Customize the virtual
machine’s hardware if it’s checked), then click Next.

6 Under Ready to complete, select Finish.

7 Once the clone process is complete, edit the prime client’s settings as follows:

a Right click on the prime client virtual machine and select Edit Settings....

b Add a second disk:

Click the ADD NEW DEVICE button (at the top), select Hard Disk, and set the size to 50GB.

Broadcom 47
VMmark User’s Guide

c Add a third disk:

Click the ADD NEW DEVICE button (at the top), select Hard Disk, and set the size to at least 50GB.

NOTE This disk is used to store benchmark test results; making it larger provides room to store more
test results. The size of each test result directory depends on various factors, such as the number of
tiles, the number of hosts, inclusion of esxtop data, etc. Each result might take 1GB or more.

d Add a second NIC:

Click the ADD NEW DEVICE button (at the top), select Network Adapter, and select your private
network.

e Click OK.

Configure the Prime Client

Once the prime client is created, follow these steps to configure it.

NOTE This document is written for environments with two prime client NICs (as described above).
Configuring the prime client with only one NIC (connected to the private network) is not covered in this User's
Guide but is nevertheless a compliant configuration for benchmarking purposes.

1 Right click on PrimeClient and select Power > Power On.

2 Right click on PrimeClient and select Open Remote Console.

3 Within the console, log in as root.

(Default root password: vmmark4.)

4 Configure the network settings for the first network adapter, which will connect to the external network:

a Click Applications (in the upper left corner), select Settings, then select Advanced Network
Configuration.

b Select the first network adapter, then click the Settings wheel.

c In the Connection name box enter a name.

d Select IPv4 Settings, select a method, and configure other network settings as needed (i.e., Address,
Netmask, Gateway, DNS, etc.) for a public IP address.

e Click Save.
5 Configure the network settings for the second network adapter, which will connect to a private network:
a Click Applications (in the upper left corner), select Settings, then select Advanced Network
Configuration.

b Select the second network adapter (typically VM Network or VMware Ethernet), then click the
Settings wheel.

c In the Connection name box enter a name.

d Select IPv4 Settings, select a method, and configure other network settings as needed (i.e., Address,
Netmask, Gateway, DNS, etc.) for a private IP address.

NOTE To use the default private IP address for the prime client, select Method Manual, then click
Add. Use Address 198.18.4.251 and Netmask 255.255.0.0; leave Gateway blank. Using these
values will reduce the amount of input required later for provisioning and running.

e Click Save.

f Restart the virtual machine. One way to accomplish this is by opening a terminal entering the
command reboot now.

48 Broadcom
 Chapter 3 Prepare the Infrastructure for VMmark 4.0 Benchmark Tests

6 Convert this virtual machine into the prime client by running the make-prime script:

a Click Applications (in the upper left corner) and select Terminal emulator.

b In the terminal window, enter the following command:

make-prime.sh primeclient_private_ip_address
(replace primeclient_private_ip_address with the private IP address for the prime client).

7 Confirm the prime client’s time zone:

Follow these steps to make sure the prime client is configured for the correct time zone:

NOTE The prime client, tile clients, and all workload virtual machines must be set to the same time zone.
This step describes how to set the time zone for the prime client. This does not, however, set the time zone
for the tile clients and the workload virtual machines. You can set those time zones during tile
provisioning, as described in “Use Quick Start to Deploy and Run the Benchmark” on page 50.

NOTE Though you can change the default time zone from America/Los_Angeles to suit your
environment, the prime client, tile clients, and all workload virtual machines must be left at their default
US localization, which ensures that the time and date are in the format expected by the VMmark harness.

a In a terminal window on the prime client, run the date command to determine the prime client’s
current time zone. If it outputs the default time zone of America/Los_Angeles, and that is correct for
your environment, skip ahead to “Configure Passwordless SSH”.
b List the available time zones, and locate the correct one for your environment:
timedatectl list-timezones

c Set the correct time zone. For example:

timedatectl set-timezone US/Central
(replacing US/Central with the correct time zone for your environment).

d Make a note of what time zone you set in this section; you will need to set the same time zone when
you later use Quick Start mode to provision the first tile and again when you add additional tiles.

e Restart the virtual machine. One way to accomplish this is by opening a terminal and entering the
command:
reboot now

Configure Passwordless SSH

Obtaining VMmark results requires the VMmark Reporter to run (Reporter = 1 in the VMmark4.properties
file, which is the default). The Reporter, in turn, requires passwordless SSH access from the prime client to the
ESXi hosts (in both the SUT and client clusters) and the vCenter Server. To configure passwordless SSH, follow
these steps:

1 Activate SSH for all SUT and client ESXi hosts as described in Enable ESXi Shell and SSH Access with the
Direct Console User Interface.

2 Open a Web Console or Remote Console to the prime client VM and log in as root.
(Default root password: vmmark4.)

3 Click Applications (in the upper left corner) and select Terminal emulator.

4 In the terminal window run the following script for each SUT and client ESXi host and the vCenter Server:
VMmark4-Configure-Passwordless-SSH.exp HOSTNAME ‘PASSWORD’
(replacing HOSTNAME with each hostname and PASSWORD with the password for that host, and ensuring
that the password is enclosed in single quotes.)

For example:
VMmark4-Configure-Passwordless-SSH.exp esx-sut-host1 'password!'
VMmark4-Configure-Passwordless-SSH.exp esx-sut-host2 'password!'
VMmark4-Configure-Passwordless-SSH.exp esx-client-host1 'password!'

Broadcom 49
VMmark User’s Guide

VMmark4-Configure-Passwordless-SSH.exp esx-client-host2 'password!'

VMmark4-Configure-Passwordless-SSH.exp vcsa.vmware.com 'password!'

5 Still on the prime client, confirm that passwordless SSH works by using SSH to remotely run the “date”
command on each ESXi host and the vCenter Server:
ssh ESXihost date

You should see the date displayed without being prompted for a password. If you are prompted for a
password, double-check that all of the commands above were entered correctly.

Deploy and Run VMmark

This section describes how to use the new Quick Start mode to deploy VMmark and perform a short test run.
It then describes how to perform a full-length benchmark run.

WARNING Once a VMmark tile is provisioned, it should not be deleted using vCenter, either through the GUI
or PowerShell. Doing so will leave orphaned Persistent Volume Claim entities (PVCs) in Kubernetes that will
take up space on the datastore and are difficult to identify. Instead, delete tiles by following the instructions in
“Delete VMmark 4 Tiles” on page 74.

Use Quick Start to Deploy and Run the Benchmark

VMmark 4 includes a new Quick Start mode. This mode completely provisions the benchmark, waits for the
application workload provisioning to finish, and performs a short (30 minute) benchmark run that allows you
to validate that the benchmark is running correctly and quickly troubleshoot any configuration issues. This
entire process typically takes between about three and six hours. Quick Start also generates a
VMmark4-quickstart.properties file that can be used to easily create the VMmark4.properties file, which
is in turn used to control subsequent VMmark runs.

NOTE As an alternative to Quick Start mode, users can manually provision the benchmark, as described in
“Manually Provision the Benchmark” on page 69.

NOTE In previous versions of VMmark, users might have used vMotion or Storage vMotion to move VMmark
tiles off the cluster or datastore where they were originally created.

In VMmark 4, the tiles include Weathervane Kubernetes workloads. These workload virtual machines use
vSphere Container Storage; they can’t be moved off the cluster or datastore on which they were created.
Instead, the VMmark tile should be deleted as described in “Delete VMmark 4 Tiles” on page 74, and later
reprovisioned.

For this reason, you should create tiles on the cluster and datastore where you’ll be performing the VMmark
benchmark runs.

Quick Start mode is started from vmmark4service:

vmmark4service --mode quick_start
If no additional parameters are provided, the service runs in user interactive mode, prompting users for the
minimum set of parameters required.

NOTE If you have set your prime client to a time zone other than the default of America/Los_Angeles, you’ll
also need to set the time zone using the --time_zone parameter in vmmark4service.

Parameters can also be included in a Quick Start mode command and vmmark4service will prompt for any
missing required parameters. Note that some vmmark4service parameter defaults can be overwritten by
supplying that parameter to allow Quick Start mode to operate in a wider variety of environments. Quick Start
can completely provision one or more tiles, update the /etc/hosts file, generate a
VMmark4-quickstart.properties file, and perform a benchmark run.

50 Broadcom
 Chapter 3 Prepare the Infrastructure for VMmark 4.0 Benchmark Tests

The required Quick Start parameters are:

--vcenter_ip '<ip_addr>'
--vcenter_password '<password>'
--datacenter '<datacenter>'
--cluster '<cluster>'
--client_cluster '<cluster2>'
--tile_number 1
--provisioning_source '<template-VM>'
--datastore '<datastore_name>'
--client_datastore '<client_datastore>'
--infra_datastore '<different_datastore>'
---network_label 'testbed'
(where the values in italics are replaced with actual values for your environment).

NOTE If you have set your prime client to a time zone other than the default of America/Los_Angeles, you’ll
also need to set the time zone using the --time_zone parameter in vmmark4service.

Example use of Quick Start mode (including some optional parameters described in the following section):
OUTPUT=quickstart-test1.txt
nohup vmmark4service --mode quick_start --vcenter_ip '<ip_addr>' \
--vcenter_password '<password>' --datacenter '<datacenter>' --cluster '<cluster>' \
--client_cluster '<cluster2>' --tile_number 1 --provisioning_source '<template-VM>' \
--datastore '<datastore_name>' --client_datastore '<client_datastore>' \
--infra_datastore '<different_datastore>' \
--network_label 'testbed' > $OUTPUT 2>&1 & \
--network_switch_type 'dvswitch' \
--time_zone ‘US/Central’

NOTE For more example uses of vmmark4service, including powering on VMs and deleting a tile, see:
/root/VMmark4/examples/

Optional Quick Start Parameters

In addition to the required Quick Start parameters described above, there are a number of optional
parameters, described in this section.

NOTE If you will use non-default values for any of the standard parameters in your VMmark environment
(the prime client IP address, the static IP address range for VMs, etc.), these should be passed to the
VMmark4-quickstart.properties file using the quick_start_options parameter even if they were
included as Quick Start parameters.

Alternatively, you can create your own VMmark4.properties file using the provision mode.

Parameters can be added to the VMmark4-quickstart.properties file that’s generated by vmmark4service.

This is done by including the desired options (comma separated, in single quotes) after
--quick_start_options. For example:
--quick_start_options 'Reporter = 0'
generates a Quick Start file that includes the parameter to turn off the Reporter.

By default, VMmark masks the vCenter password within the VMmark results files. To change this behavior,
the mask_password parameter can be set to 0 (default 1) to include the password in these files. For example,
you could include in the quick_start_options:
--mask_password = 0

The default provisioning_mode for quick_start is one datastore. If this is not the provisioning_mode
selected, you need to add the --csi_datastore parameter. See the vmmark4service help for details.

Broadcom 51
VMmark User’s Guide

If you are using a distributed vSwitch for your VMmark network, add to your Quick Start command:
--network_switch_type 'dvswitch'

NOTE If you will be using a distributed vSwitch, it needs to be configured with ephemeral port bindings.

Optionally, you can modify the runtime_seconds parameter to change from the default Quick Start run
duration of 30 minutes. For details see “Set the runtime_seconds Parameter” on page 69.

Interpreting Quick Start Results

1 A completed run results in an output or terminal window containing:
VMmark4 Benchmark completed

2 If you encounter errors, correct them and rerun.

For guidance, see “Troubleshooting” on page 71.

If the previous Quick Start command created VMmark VMs but did not complete successfully, use the
vmmark4service mode delete_all_vmmark4 to delete all VMmark VMs except for the prime client.
VMmark VMs should be deleted using the vmmark4service mode delete_all_vmmark4, not using the
vCenter Server Client user interface.
Example delete_all_vmmark4 command which will delete 1 tile:
/root/VMmark4/tools/vmmark4service --mode delete_all_vmmark4 \
--vcenter_ip <ip_addr> --vcenter_username <vc_username> \
--vcenter_password <password> --datacenter <datacenter> \
--cluster <sut_cluster> --datastore <datastore> --csi_datastore <datastore> \
--tile_number 1 | tee <output_file>

After using delete_all_vmmark4, run the Quick Start command again with corrected values.

3 Once the run is completed, inspect the results files:

 Results are stored in subdirectories under the /root/VMmark4/ directory:

 Provisioning results are in the /root/VMmark4/provision directory.

 Benchmark results are stored in subdirectories under the /root/VMmark4/results directory.

 Workload output files have a .wrf extension.

For example, to see the workload output for Standby0, from the relevant results directory run:
cat Standby0.wrf

4 If there was an error during execution of the benchmark that isn't shown in the workload .wrf file, the
VMmark4-STAX*.log might be helpful. From the relevant results directory run:
less VMmark4-STAX*

Perform a Full One-Tile VMmark Run

Now that you’ve completed a one-tile turbo VMmark run (that is, a 30-minute run), try out a full 3-hour run.
The results of this full run will help you determine if your system can support additional tiles. After using
Quick Start mode to deploy a VMmark environment, you’ll typically control subsequent benchmark runs
using STAX as follows:

1 Update the relevant parameters in /root/VMmark4/VMmark4.properties with any uncommented lines

from /root/VMmark4/VMmark4-quickstart.properties, except for the TurboMode and
RunTimeSeconds parameters.

2 Modify the VMmark4.properties file as needed. Because you provisioned one tile in the previous step,
use the parameter Tiles = 1 in VMmark4.properties.

NOTE Anything not specified in this file uses defaults from

/root/VMmark4/source/defaults.properties.

52 Broadcom
 Chapter 3 Prepare the Infrastructure for VMmark 4.0 Benchmark Tests

3 You can start the benchmark either using a GUI or through command line options. To use the GUI, start
STAX by double-clicking the VMmark4-STAX icon on the desktop. (You can also start STAX by entering
VMmark4-STAX.sh from a terminal window.)

NOTE Alternately, to start a benchmark run via the command line, enter the following:
VMmark4-STAX-console.py -c <path to vmmark4 properties> -j <job_name>

For more information on this topic, see “Start a VMmark Run Without a GUI” on page 58.

4 If the XML Job File is not already be pre-populated, within the STAX UI:

a Click Submit New Job....

b Select local machine and click Browse....

c Browse to /root/VMmark4/source/vmmark4_main.xml.

d Optionally, to use a properties file with a different filename than the default VMmark4.properties:

NOTE The default VMmark4.properties filename is used throughout this User’s Guide and other
VMmark documentation. If you chose to use an alternate properties filename, be sure to substitute
the alternate name as required.

i Click Job Wizard....

ii Under Functions, click Main (default).

iii Under Arguments for function Main, click Edit.

iv Specify the full path to the alternate properties filename, and click Save.

v Click Save again and Yes to confirm.

e Click Submit New Job to start a VMmark run.

5 If you encounter errors, correct them and rerun.

For guidance, see “Troubleshooting” on page 71.

6 A completed run results in an output or terminal window containing something similar to:
STAX Job Monitor Machine:local JobID:1 <Completed>

7 Once the run is completed, inspect the results files:

 Benchmark results are stored in subdirectories under the /root/VMmark4/results directory.

 Workload output files have a .wrf extension.

For example, to see the workload output for Standby0, from the relevant results directory run:
cat Standby0.wrf

8 If there was an error during execution of the benchmark that isn't shown in the workload .wrf file, the
VMmark4-STAX*.log might be helpful. From the relevant results directory run:
less VMmark4-STAX*

Configure Your VMmark Environment for Multiple Tiles

Once you have your VMmark environment running with one tile, you can use the results to evaluate how
heavily utilized your system is and how many more full tiles, or even partial tiles, it might be able to support
while still meeting the required quality of service limits. You’ll typically want to add (i.e., “provision”)
additional tiles, as described in the following subsections.

Broadcom 53
VMmark User’s Guide

Add Tiles to the VMmark Environment

Once you have your VMmark environment running with one tile, you’ll typically want to add (i.e.,
“provision”) additional tiles. To do so, follow the below steps.

1 Confirm that you have sufficient disk space and other resources for the number of tiles you plan to add.
For details, see “VMmark Benchmark Requirements” on page 31.

2 Power off your tile 0 VMs.

3 Copy the vmmark4service command you used for the Quick Start process, but with the following
changes:

 Replace --mode quick_start with --mode provision

 Change the --tile_number parameter to your desired number of tiles.

For example, to add one more tile to the single tile you already have, replace:
--tile_number 1
with:
--tile_number 2

For example:
OUTPUT=provision-second-tile.txt
nohup vmmark4service --mode provision --vcenter_ip <ip_addr> \
--vcenter_password '<password>' --datacenter <datacenter> --cluster <cluster> \
--client_cluster <cluster2> --tile_number 2 --provisioning_source <template-VM> \
--datastore <datastore_name> --client_datastore <client_datastore> \
--infra_datastore '<different_datastore>' --network_label 'testbed' > $OUTPUT 2>&1 &

4 Double check the /etc/hosts file on your prime client to ensure everything is configured correctly.

5 Copy VMmark4-quickstart.properties to VMmark4.properties, then edit VMmark4.properties

file, changing Tiles to your desired number.

6 Run the benchmark as normal.

Calculate the Number of Simultaneous Infrastructure Operations

A variety of infrastructure operations are performed during a VMmark run. These are clone and deploy,
vMotion, Storage vMotion, and XvMotion.

In addition to these baseline infrastructure operations, additional instances of these infrastructure operations
are performed in larger environments. If your VMmark system under test cluster contains four or more hosts
and you run four or more tiles, VMmark performs these additional instances of infrastructure operations. The
number of simultaneous infrastructure operations is half of the smaller of either the number of hosts or the
number of tiles, rounded down to an integer; that is:

floor(min(#hosts in VMmark cluster, #tiles)/2)

Refer to Table 3-2 for examples.

Calculate this number, then record it for use later in the preparation process. (This number will determine how
many copies of the template you’ll need to make and how many instances will be required to be entered in the
DeployVMinfo parameter in the VMmark4.properties file.)

Table 3-2 provides examples of this calculation.

Table 3-2. Example Numbers of Simultaneous Infrastructure Operations
Number of Hosts Number of Tiles Number of Simultaneous Infrastructure Operations

2 Any 1

3 Any 1

4 Up to 3 1

4 4+ 2

54 Broadcom
 Chapter 3 Prepare the Infrastructure for VMmark 4.0 Benchmark Tests

Table 3-2. Example Numbers of Simultaneous Infrastructure Operations

Number of Hosts Number of Tiles Number of Simultaneous Infrastructure Operations

5 Up to 3 1

5 4+ 2

6 Up to 3 1

6 4-5 2

6 6+ 3

Make Copies of the VMmark Template for Infrastructure Operations

The original VMmark template will be used for baseline infrastructure operations. If you’ll have more than
one simultaneous infrastructure operation, you’ll need to make additional copies of the template.

Create enough additional copies of the VMmark template for all the simultaneous infrastructure operations
your environment will need (as calculated in “Calculate the Number of Simultaneous Infrastructure
Operations” on page 54), naming the copies with the template name, a hyphen, and a sequential number. For
example, if the original template was named VMmark-4.0.0-100, you’d name these copies
VMmark-4.0.0-100-2, VMmark-4.0.0-100-3, VMmark-4.0.0-100-4, and so on.

Note the names of these copies for later use, to be entered under Deploy/Templates when you edit the
VMmark4.properties file.

Optional Configuration Changes to the VMmark Environment

In order to keep your VMmark environment functional, and to remain compliant with the VMmark Run and
Reporting Rules, make changes to the prime client (or to the tile clients or workloads) only if those changes are
explicitly allowed in this User’s Guide or in the VMmark Run and Reporting Rules. Check with VMware
before making any other changes.

Changes that could cause problems include, but are not limited to, installing, updating, or removing software.

Update the Prime Client

If desired, make the following configuration change to the prime client:

 Optionally, set the display resolution:

a Click Applications (in the upper left corner), select Settings, then select Display.

b Select a different resolution (for example, 1280x800 16:10)

NOTE To set the resolution to anything above 1280x768, the prime client virtual machine’s video card
Total video memory needs to be increased from its default of 4MB (16MB would be a reasonable value).

 Optionally, install PowerShell and PowerCLI:

a Install PowerShell:
https://learn.microsoft.com/en-us/powershell/scripting/install/install-ubuntu?view=powershell-7.4

b Install PowerCLI:
https://developer.broadcom.com/powercli/installation-guide

Update VMmark Workload Virtual Machines

If desired, you can at this time make certain changes to the workload virtual machines’ virtual hardware
version, VMware Tools version, or virtual hardware, as described below.

NOTE If any of the changes in this section are made, those changes must be made to all workload VMs of the
same type, across all tiles.

Broadcom 55
VMmark User’s Guide

NOTE VMware is aware of a potential issue in which the numeric value for resource shares shown within the
vCenter user interface and the resulting VC cluster report text files might not reflect the actual values.

We recommend reviewing the corresponding virtual machine .vmx files to confirm what share values are
actually configured.

Update Virtual Hardware Version

If desired, you can update the virtual hardware version for the workload virtual machines to a newer
supported version.

NOTE Upgrading the virtual hardware version is supported; downgrading the virtual hardware version is
not supported.

Update VMmark Tools Version

If desired, you can update VMmark Tools in the workload virtual machines to the latest version supported in
your environment.

Make One-for-One Replacements of Virtual Hardware

If desired, you can replace virtual hardware storage or network devices in the workload virtual machines with
devices of the same type on a one-for-one basis. For example:

 One virtual disk can be replaced with one virtual persistent memory (PMem) device.

 One vNIC can be replaced with one vNIC of another type.

If any such replacements are made, you can also install in the guest operating system any supporting software,
such as drivers or configuration tools, required to support those devices.

Prohibited Changes to Workload Virtual Machines

Use of the following advanced configuration options will render a VMmark benchmark run non-compliant:

 Mem.BalancePeriod

 Mem.SamplePeriod

Disk Types for Provisioned VMs

 The disk type (that is, Thin, Thick Lazy, or Thick Eager) of the primary disk for provisioned VMs will be
equivalent to the selection used for the template.

 The disk type (that is, Thin, Thick Lazy, or Thick Eager) of any additional disks for provisioned VMs is
controlled by the value of the extra disk mode parameter in the VMmark4.properties file or passed
to vmmark4service.

56 Broadcom
Run the VMmark Benchmark 4
This chapter describes the process of running the VMmark benchmark. It consists of the following sections:

 “Prepare for a VMmark Run” on page 57

 “Start a VMmark Run” on page 57

 “VMmark Results Files” on page 60

 “Create and Submit a Benchmark Results Package” on page 63

Prepare for a VMmark Run

To prepare for a VMmark run, open the VMmark4.properties file and edit the VMmark Run Configuration
section to reflect your environment, following the directions included in that file.

Start a VMmark Run

Once the VMmark4.properties file has been appropriately edited, you can start a VMmark run using either
the GUI or a command line, as described in the two sections below.

NOTE When performing a benchmarking run, it's best to have powered on only those VMs that are being used
in the run.

To power off tiles in an environment, open a terminal window on the prime client and run:
cd ~/VMmark4
vmmark4service -m power_vmmark4_tiles -c VMmark4.properties -t n --tile_state off
(where n is the number of tiles to power off; for example, if n=3, tiles 0, 1, and 2 will be powered off).

To power on tiles in an environment, open a terminal window on the prime client and run:
cd ~/VMmark4
vmmark4service -m power_vmmark4_tiles -c VMmark4.properties -t n
(where n is the number of tiles to power on; for example, if n=2, tiles 0 and 1 will be powered on).

Start a VMmark Run Through the GUI

To start a VMmark run through the GUI, follow these steps:

1 From the prime client’s Web or vSphere Client console, start STAX using one of these methods:

 To start STAX using the GUI, double click on the VMmark4-STAX icon on the desktop.

 To start STAX using the command line, open a terminal window and enter the following:
VMmark4-STAX.sh

Broadcom 57
VMmark User’s Guide

2 From within the STAX window:

a Click on Submit New Job.

b In the Job Info tab, local machine should be selected and Filename: should already be filled in. If
they’re not:
XML Job File > local machine > Filename
Browse... to /root/VMmark4/source/vmmark4_main.xml

c In the STAX 3 Job Monitor window, under the Job Info tab, under Job Options, enter a job name.

NOTE The STAX job name must not include spaces.

d Optionally, to use a properties file with a different filename than the default VMmark4.properties:

NOTE The default VMmark4.properties filename is used throughout this User’s Guide and other
VMmark documentation. If you chose to use an alternate properties filename, be sure to substitute
the alternate name as required.

NOTE Before proceeding, we strongly recommend changing the display resolution to higher than
the default 800x600 so the Job Wizard window is fully visible. This can be accomplished by following
the steps in the “Optional Configuration Changes to the VMmark Environment” on page 55.

i Click Job Wizard....

ii Under Functions, click Main (default).

iii Under Arguments for function Main, click Edit.

iv Specify the full path to the alternate properties filename, and click Save.

v Click Save again and Yes to confirm.

e Click Submit New Job to start a VMmark run.

The STAX Job Monitor window opens and STAX starts the VMmark harness. The current status of the
running workloads is shown in this window.

NOTE After the above information has been entered, you can use the Resubmit Previous Job button. The
names entered are remembered across restarts of the Monitor. STAX locates the latest VMmark4.properties
file and the XML Harness code each time.

Start a VMmark Run Without a GUI

To start a VMmark run without a GUI, follow these steps:

1 SSH to the prime client:

ssh root@primeclient
(replacing primeclient with the public IP address of your prime client).

2 At the command line, enter the following:

VMmark4-STAX-console.py -c <path to vmmark4 properties> -j <job_name>

VMmark4-STAX-console.py offers a number of command-line options, shown with -h (and reproduced

here):
root@primeclient:~# VMmark4-STAX-console.py -h
VMmark4-STAX-console.py: v0.0.6 11092023
usage: VMmark4-STAX-console.py [-h] [-c PROPERTIES] [-t TILES] [-j JOB_NAME] [-l LOG] [-ss
SLEEP_SECONDS] [-q]
options:
-h, --help show this help message and exit
-c PROPERTIES, --properties PROPERTIES
Full path to VMmark4 properties file (default:
/root/VMmark4/VMmark4.properties)

58 Broadcom
 Chapter 4 Run the VMmark Benchmark

-t TILES, --tiles TILES

Optional number of Tiles to run (overrides properties file) (default: )
-j JOB_NAME, --job_name JOB_NAME
Optional job name (default: )
-l LOG, --log LOG Output log path and filename ('' to disable) (default:
/root/VMmark4/STAX_Job_User.log)
-ss SLEEP_SECONDS, --sleep_seconds SLEEP_SECONDS
How often log is output to console/log file (default: 5)
-q, --quiet Quiet mode: Do not output log to console (default: False)
For example, to specify a properties file, job name, and log file (note: specify the full paths
of properties and log files):
VMmark4-STAX-console.py --properties /root/VMmark4/VMmark4-quickstart.properties
--job_name quickstart-run2 --log /root/VMmark4/quickstart-run2.log

Broadcom 59
VMmark User’s Guide

VMmark Results Files

After a VMmark benchmark test run has completed, the VMmark harness automatically collects the results
data from the client system (or systems) and places those results in the results directory on the prime client,
then processes the results to score the test.

The benchmark captures the 60-second throughput measurements for each workload and stores them in files
with the .wrf (Workload Results File) suffix in the Results_<datestamp> directory. At the end of a compliant
test run, the average throughput scores for each workload are recorded in the Score_N_Tile_Test.txt file
(where N is the number of tiles in the test) along with the normalized scores and the composite VMmark metric.

The Score_N_Tile_Test.txt file, a sample of which is included below, contains additional details of the test,
such as duration, start time, and end time as well as:

 TILE_N_Scores: The actual workload scores for each VM during each of the three test phases (p0, p1, and
p2).

 TILE_N_Ratios: The scores normalized against the VMmark reference platform.

 TILE_N_QoS: Quality of Service (QoS) metrics for each workload.

 For DVD Store workloads, this metric represents average application latencies in milliseconds. When
the average application latencies exceed the workload's QoS limit, the run is marked non-compliant,
as indicated by an asterisk for the test phase in question.

 For Weathervane Auction (VM) and Weathervane Auction (Kubernetes), two metrics are displayed,
separated by a bar:

 The metric to the left of the bar represents the normalized average response times for all the
various operation types Weathervane performs during a run. Each operation’s response time is
normalized against that operation’s QoS limit.

When any Weathervane operation type’s 99th percentile response time exceeds that operation’s
QoS limit, an asterisk on the relevant test phase marks the run as non-compliant.

 The metric to the right of the bar is related to the percentage of operations that failed QoS
requirements. VMmark calculates the percentage of operations within each operation type that
failed QoS requirements then displays in this position the maximum percentage across any
operation type. When this value is greater than 0.5, an asterisk (*) on the relevant test phase
marks the run as non-compliant.

 Additionally, to ensure benchmark consistency from run to run, the proportion of each operation
type in the overall mix of operations must match a proportion requirement. When this
proportion is violated, a plus sign (+) on the relevant test phase marks the run as non-compliant.
The composite VMmark score at the bottom of the Score_N_Tile_Test.txt file is the median of the sums
of each phase’s geometric mean per tile.

The VMmark4-Graph-Throughput.html file plots the application throughput of each VMmark workload over
time for each tile. The VMmark4-Graph-QoS.html file plots the quality of service (QoS) of each VMmark
workload over time for each tile. The VMmark4-Graph-Infrastructure-Ops.html file plots the time it took
for the infrastructure operations to complete over the duration of the run. These graphs help provide an
in-depth and visually intuitive look at performance during the run.

The STAX_Job_N_User.log uses a default format that can be awkward to read. An easier-to-read processed
version, VMmark4-STAX_Job_N_User-Parsed.log, is included in the results directory. The only difference
between the two is their format.

60 Broadcom
 Chapter 4 Run the VMmark Benchmark

Sample VMmark Results File

VMmark4-Tilescore.py : v0.9.4 03272024
Tiles = 1.0 : Full Tiles = 1 : Fractional Tiles = 0.0
Referencing_tile: 0.
First Sample : 1712795160 : Wed Apr 10 17:26:00 2024 PDT : Thu Apr 11 00:26:00 2024 UTC
Info: t = 1 : Time = 1712795220 : Full Tiles = 1 : Fractional Tiles = 0.0 : App Workloads = 10
Run_start : 1712795160 : Wed Apr 10 17:26:00 2024 PDT : Thu Apr 11 00:26:00 2024 UTC
Time_start : 1712795220 : Wed Apr 10 17:27:00 2024 PDT : Thu Apr 11 00:27:00 2024 UTC
Time_end : 1712805600 : Wed Apr 10 20:20:00 2024 PDT : Thu Apr 11 03:20:00 2024 UTC
Run_end : 1712805960 : Wed Apr 10 20:26:00 2024 PDT : Thu Apr 11 03:26:00 2024 UTC
Duration : 173.00 minutes
Steady_state_start : 1712797020 : Wed Apr 10 17:57:00 2024 PDT : Thu Apr 11 00:57:00 2024 UTC
Steady_state_end : 1712804220 : Wed Apr 10 19:57:00 2024 PDT : Thu Apr 11 02:57:00 2024 UTC
Phase_0_begin : 1712797020 : Wed Apr 10 17:57:00 2024 PDT : Thu Apr 11 00:57:00 2024 UTC
Phase_1_begin : 1712799420 : Wed Apr 10 18:37:00 2024 PDT : Thu Apr 11 01:37:00 2024 UTC
Phase_2_begin : 1712801820 : Wed Apr 10 19:17:00 2024 PDT : Thu Apr 11 02:17:00 2024 UTC

TILE_0_Scores: WVAuctionVM WVAuctionK8S DVDStoreA DVDStoreB DVDStoreC NoSQLBenchA

NoSQLBenchB NoSQLBenchC SocialNetwork Standby
p0 14121.56 9252.23 2617.50 1829.33 1351.60 51093.86
51160.50 51151.88 72.03 1.00
p1 14143.07 9238.03 2609.28 1867.58 1281.00 53174.13
53174.81 53170.56 72.00 1.00
p2 14108.78 9246.64 2576.88 1806.25 1309.70 53186.82
53187.40 53184.33 71.96 1.00

TILE_0_Ratios: WVAuctionVM WVAuctionK8S DVDStoreA DVDStoreB DVDStoreC NoSQLBenchA

NoSQLBenchB NoSQLBenchC SocialNetwork Standby Geo.Mean
p0 1.01 1.01 1.05 1.08 1.13 1.04
1.04 1.04 1.00 1.00 1.04
p1 1.01 1.00 1.04 1.10 1.07 1.09
1.09 1.09 1.00 1.00 1.05
p2 1.01 1.01 1.03 1.06 1.09 1.09
1.09 1.09 1.00 1.00 1.05

TILE_0_QoS: WVAuctionVM% WVAuctionK8S% DVDStoreA DVDStoreB DVDStoreC NoSQLBenchA

NoSQLBenchB NoSQLBenchC
p0 0.26 | 0.00 0.24 | 0.00 653.33 784.61 861.89 0.57
0.57 0.57
p1 0.30 | 0.00 0.24 | 0.00 656.75 754.84 887.78 0.59
0.59 0.59
p2 0.30 | 0.00 0.26 | 0.00 693.32 822.15 944.22 0.61
0.61 0.61

p0_score = 1.04
p1_score = 1.05
p2_score = 1.05

Infrastructure_Operations_Scores: vMotion SVMotion XVMotion Deploy

Completed_Ops_PerHour 29.00 13.00 15.00 14.50
Avg_Seconds_To_Complete 1.52 58.92 38.97 203.76
Failures 0.00 0.00 0.00 0.00
Ratio 0.98 1.00 1.15 1.00
Number_Of_Threads 1 1 1 1

Warnings Messages::
p0 : WVAuctionVM0 Exceptions : 59
p0 : WVAuctionK8S0 Exceptions : 14
p1 : WVAuctionVM0 Exceptions : 58
p1 : WVAuctionK8S0 Exceptions : 11
p2 : WVAuctionVM0 Exceptions : 75
p2 : WVAuctionK8S0 Exceptions : 14
rampdown : WVAuctionVM0 Exceptions : 37
rampdown : WVAuctionK8S0 Exceptions : 35

Summary ::

Broadcom 61
VMmark User’s Guide

Run_Is_Compliant
Median_Phase : p2

Unreviewed_VMmark4_Applications_Score : 1.05
Unreviewed_VMmark4_Infrastructure_Score : 1.03
Unreviewed_VMmark4_Score : 1.05 @ 1 Tiles

Results table generated : VMmark4-Results-Table.html

Graph generated : VMmark4-Graph-Throughput.html
Graph generated : VMmark4-Graph-QoS.html
Graph generated : VMmark4-Graph-Infrastructure-Ops.html
Graph generated : prme-perf-vmmark07.eng.vmware.com-powermetrics.html
Graph generated : prme-perf-vmmark10.eng.vmware.com-powermetrics.html

62 Broadcom
 Chapter 4 Run the VMmark Benchmark

Create and Submit a Benchmark Results Package

Preparing VMmark results for publication according to the VMmark Run and Reporting Rules involves the
following tasks:

 Make sure your prime client is configured to log in without a password to your vCenter Server and every
ESXi hosts in your VMmark environment, as described in “Configure Passwordless SSH” on page 49.

 Verify the Reporter is enabled and set the Disclosure Settings in the VMmark4.properties file, then run
a compliant test.

 Run the disclosure_creator tool to automatically generate a disclosure.html file.

 Edit the disclosure.html file.

 Submit the benchmark results for review.

The following sections detail these steps.

Verify the Reporter is enabled and set the Disclosure Settings in the
VMmark4.properties file, then run a compliant test
When you are satisfied with your benchmark run and are ready to submit the results, you need to run a full
test with the Reporter enabled. You can also optionally set the Disclosure Settings in the VMmark4.properties
file. To do so:

1 Edit the VMmark4.properties file and set the following parameter:

Reporter = 1
(this should be the default, unless you have previously changed it to 0).

2 (This step is optional, but recommended) Still in the VMmark4.properties file, go to the bottom,
uncomment the lines that start with # Disclosure/, and change them to suite your environment. For
example:

Before:
# String Disclosure/TestedBy : Default empty : Specifies who ran the benchmark test.
# Disclosure/TestedBy = <TesterName>
# String Disclosure/SUTAvailabilityDate : Default MM-DD-YYYY : Specifies the date
when all components of the SUT are publicly available
# Disclosure/SUTAvailabilityDate = <MM-DD-YYYY>
(etc.)

After:
# String Disclosure/TestedBy : Default empty : Specifies who ran the benchmark test.
Disclosure/TestedBy = Broadcom
# String Disclosure/SUTAvailabilityDate : Default MM-DD-YYYY : Specifies the date
when all components of the SUT are publicly available
Disclosure/SUTAvailabilityDate = 12-31-2024
(etc.)

3 Run a compliant VMmark test.

The VMmark harness automatically gathers information about the workload virtual machines, the ESXi
virtualization hosts, and the vCenter server, and places that information in the results directory.

Run the disclosure_creator Tool to Generate the disclosure.html File

Once you have a compliant VMmark test, you can run the disclosure_creator tool to automatically
generate a disclosure.html file (recommended) or copy the disclosure template into your results directory
and fill in the required fields manually.

Broadcom 63
VMmark User’s Guide

To use the disclosure_creator tool:

1 On the prime client change to the appropriate results directory:

cd ~/VMmark4/results/Results_datestamp
(where datestamp refers to the results to be submitted).

2 Issue the command:

disclosure_creator -p .

NOTE If your prime client is not connected to the Internet, you may see warning messages about not
being able to download files/updates. These are expected and can be ignored.

To fill in the disclosure manually:

1 On the prime client change to the appropriate results directory:

cd ~/VMmark4/results/Results_datestamp
(where datestamp refers to the results to be submitted).

2 Copy the template disclosure.html file from the tools directory to the results directory:
cp ~/VMmark4/tools/disclosure.html .

Edit the disclosure.html File

Edit the disclosure.html file with a text editor (visual/WYSIWYG HTML editors often change the HTML
formatting in unexpected ways).

NOTE If you used disclosure_creator, many fields will have been automatically filled in, thus the fields
below will already be populated. Nevertheless, all fields must be reviewed for accuracy.

1 Replace ### @ ### Tiles with the score and tile count from the VMmark4-Results-Table.html file.

2 In the section titled Performance, replace ### with the contents of the VMmark4-Results-Table.html
file.

3 Search for all remaining instances of ### and replace ### with the appropriate details of your system
configuration and data about the benchmark run. Keep in mind the following points:

 For vSAN results specify in the storage section whether this result used vSAN OSA (Original Storage
Architecture) or vSAN ESA (Express Storage Architecture).

 For benchmark runs with non-identical servers, make additional copies of the relevant sections of the
template disclosure.html file as needed to describe the servers.

 Other Hardware must disclose:

 Server power supply information, including:

- Quantity
- Name/part number
- Wattage
- Firmware
- Type (voltage, AC or DC)

 Server cooling information, which is one of the below:

- Air cooling: Traditional cooling, either with or without fans
- Closed loop cooling: any liquid cooling method implemented within the server chassis
- Direct liquid cooling: any liquid cooling method implemented outside the server chassis
- Other: Any other cooling method (provide additional detail)

 Networking Notes must disclose assignment of virtual machines and vmnics to virtual switches.

 In the Primary Storage section, provide details of the storage used for the application workload
virtual machines. For benchmark runs using multiple storage solutions, list all such solutions in the
Primary Storage section and number each one.

64 Broadcom
 Chapter 4 Run the VMmark Benchmark

 If the infrastructure target storage (that is, the storage used for the Storage vMotion, Cross-host
Storage vMotion, and deploy targets) or the deploy template are located somewhere other than on
the primary storage hardware, the details for that other storage must be included in the Storage Notes
section.

 All disclosures of non-default values or settings must also disclose the default value or setting:
<parameter> = <non-default value> (default <default value>)

 The Virtualization Software Notes section must state the DRS Migration threshold. To find this value,
in your results folder open the VC-ClusterReport.txt file. Under the heading VMmark4 Cluster
Information, you'll see DRS Threshold N (where N is your DRS threshold). Disclose this value in the
Virtualization Software Notes section as follows:
DRS Migration threshold set to level N
(substituting your DRS threshold for N).

NOTE There is a potential point of confusion between the DRS Migration threshold values
displayed in the vSphere Client GUI and the values shown in the above-mentioned file. The
correct values for the disclosure are those in the above-mentioned file.

 MM-DD-YYYY strings are included in the template as examples and must be replaced with the relevant
month, day, and year.

4 Save the disclosure.html file.

Submit the Benchmark Results for Review

Once all the necessary files are in the appropriate results directory on the prime client, you are ready to submit
the benchmark results for review. For guidance on the submission process, email
[email protected].

Broadcom 65
VMmark User’s Guide

66 Broadcom
Optional Configurations and Settings A
This appendix provides information about optional configurations for many of the programs and utilities used
to perform the VMmark benchmarking tests.

Add an NFS Datastore for Infrastructure Operations

If your VMmark environment doesn’t include a second traditional shared-storage datastore (such as a Fibre
Channel LUN) to use for infrastructure operations, you can create an NFS datastore for this purpose.

This section describes how to configure a copy of the VMmark 4 template as an NFS datastore. This is not
required if you already have a second datastore available.

1 Clone a VMmark 4 template into a VM to use as an NFS server.

2 Configure networking to the NFS server VM so that it can communicate with the SUT ESXi hosts.

3 Add a 300GB hard disk to the NFS server VM.

4 Configure the NFS server:

a Install the nfs-kernel server package:

apt install nfs-kernel-server

b Make a filesystem for the NFS share:

Warning: This is destructive! Make sure you update /dev/sdb to point to the correct hard disk. For
example, if you already have two disks, the commands might need to point to /dev/sdc instead of
/dev/sdb.
mkdir /mnt/nfs_share
flock /dev/sdb parted /dev/sdb -a optimal --script -- mklabel msdos mkpart primary 1MiB -1
partprobe
mkfs.ext4 -L nfs-share /dev/sdb1
printf "LABEL=nfs-share\t/mnt/nfs_share\text4\tdefaults\t\t0\t0\n" >> /etc/fstab
mount -a

c Set the directory ownership for the NFS share directory:

sudo chown -R nobody:nogroup /mnt/nfs_share/

d Set the file permissions in the nfs_share directory using:

sudo chmod 777 /mnt/nfs_share/

e Grant access to the client system from all the ESXi hosts in the SUT cluster:
i Open /etc/exports for editing.

ii Update /etc/exports to include individual entries for each host from which you’ll want to
mount the newly created NFS share (changing the IP addresses for your environment):
/mnt/nfs_share 10.159.208.6(rw,async,no_subtree_check)
/mnt/nfs_share 10.159.208.7(rw,async,no_subtree_check)
.
.

Broadcom 67
VMmark User’s Guide

f Export the NFS share directory:

sudo exportfs -a

g Restart the NFS server:

sudo systemctl restart nfs-kernel-server

h Verify the NFS share:

exportfs -v

i Make sure the NFS server starts up every time the VM is powered on:
sudo systemctl enable nfs-kernel-server

5 Once the NFS server VM is up and running, create an NFS datastore using this server:

a In your vCenter navigate to Actions --> <SUT Cluster> --> Storage --> New Datastore.

b In the New Datastore pop up, select the option to Create an NFS datastore on an NFS share over the
network.

c In NFS version, leave the version at the default of 3.

d In Name and configuration, specify the following:

 A name for the datastore (for example, NFS_datastore).

 The name of the folder you created when configuring the NFS server (for example,
/mnt/nfs_share).

 The server (the NFS server virtual machine’s IP address you configured in Step 2, above).

e Select the hosts from which you want the NFS datastore to be accessible (this is typically all the hosts
in the SUT cluster).

f Click Finish.

68 Broadcom
 Appendix A Optional Configurations and Settings

Manually Provision the Benchmark

As an alternative to Quick Start mode, users can manually provision the benchmark, as described below.
# The below example can be placed into a sh file and run, creating one output log.
#
OUTPUT=provision-test1.txt
# First we deploy:
nohup vmmark4service --mode provision --vcenter_ip '<ip_addr>' \
--vcenter_username '<VC-username>’ --vcenter_password '<password>’ \
--datacenter '<datacenter>' --cluster '<cluster>' --tile_number 1 \
--provisioning_source '<template-VM>' \
--provisioning_ip_mode static --static_ip_startaddress 198.18.4.1 \
--static_ip_endaddress 245 --provisioning_ip_gateway 198.18.4.251 \
--provisioning_ip_subnetmask 255.255.0.0 --client_datastore <client_datastore> \
--provisioning_ip_dnsserver_list 8.8.8.8, 8.8.8.4 \
--provisioning_ip_dnssuffix_list eng.vmware.com, labs.vmware.com \
--datastore <datastore_name> --primeclient_ip=198.18.4.251 \
--network_label 'VM Network' > $OUTPUT 2>&1 &
#
# Next we do a small sleep (for now):
nohup echo "sleeping 5" > $OUTPUT 2>&1 &
sleep 5
#
# Now we run the monitor command with our workload list to ensure the workloads are done:
nohup vmmark4service -m vmmark4_monitor_provision > $OUTPUT 2>&1 &
#
# At this point all VMmark4 VMs have been deployed & set up. We are ready to run it.

NOTE The above example uses a non-default IP address scheme to show the relevant options. If the default
IP address scheme (198.18.4.*) is used, the options above with 192.168.1.* IP addresses could be omitted.

Load Parameters from a VMmark4.properties file

As another alternative to Quick Start mode, the -c parameter (or --properties) loads variables from a
VMmark4.properties file. For example:
OUTPUT=provision-tile4.log
nohup vmmark4service -m provision -c /root/VMmark4/VMmark4.properties -t 4 -ps VMmark-4.*.*-nnn
-ds LUN1-20TB > $OUTPUT 2>&1 &

Set the runtime_seconds Parameter

By default, Quick Start mode automatically sets the runtime_seconds parameter to 1800 seconds and enables
a Turbo Mode run of the benchmark. To disable this default behavior, modify the runtime_seconds
parameter to something other than its default of 120 seconds. For example, to run the benchmark for 300
seconds, use:
--runtime_seconds 300

The default value for the runtime_seconds parameter is 120. When Quick Start encounters this default value,
it takes the following actions:

If quick_start mode is used and the runtime_seconds parameter is left at the default (120 seconds):

 runtime_seconds is automatically set to 1800.

 TurboMode = 1 (in addition to any other parameters the user adds) is pushed into the automatically
created VMmark4-quickstart.properties file.

If quick_start mode is used and the runtime_seconds parameter is set by the user to some value other than
120 (to 10800, for example):

 runtime_seconds is left as set by the user.

 TurboMode = 1 is not added to the VMmark4-quickstart.properties file.

Broadcom 69
VMmark User’s Guide

Perform a Partial Tile Run

To perform a partial tile run, set the Tiles parameter in the VMmark4.properties file to a fractional value.
Valid values are the addition of 0.2, 0.4, 0.6 or 0.8; for example, Tiles = 1.4. See Table A-1 for a list of the
workloads associated with each partial tile value.

NOTE For each full tile and for any partial tile all application workloads must be powered on, regardless of
whether or not all of those workloads will be used. Thus, for example, running 2.2 tiles requires three full tiles
to be provisioned and powered on, even though from the third tile only the SocialNetwork workload will be
run.

Table A-1. Partial Tile Workloads

Fractional Value Adds This Workload Composition of Partial Tile

0.0 None None

0.2 SocialNetwork SocialNetwork

0.4 DS35 SocialNetwork and DS35

0.6 NoSQLBench SocialNetwork, DS35, and NoSQLBench

0.8 AuctionK8 SocialNetwork, DS35, NoSQLBench, and AuctionK8

Perform a Benchmark Run Without Infrastructure Workloads

You can run the VMmark benchmark without the infrastructure workloads, but any such runs would not be
compliant. You might wish to do so when testing a new environment, for example.

To do this, edit your VMmark4.properties file adding the line:

InfrastructureList =
(leaving the line blank after the = sign).

You will get a notice in the Score file that the run is non-compliant and it will have only an application score
rather than a full VMmark 4 score, but you’ll otherwise have a complete test.

Upgrade from Previous Versions of VMmark

There is no upgrade path to VMmark 4.0 from previous versions. To use VMmark 4.0, follow the instructions
in Chapter 3, “Prepare the Infrastructure for VMmark 4.0 Benchmark Tests,” on page 37.

70 Broadcom
Troubleshooting B
This appendix provides assistance in troubleshooting the VMmark Benchmarking tests.

Benchmark Provisioning Issues

CNS Naming in vCenter 8.0 Update 3

In vCenter 8.0 Update 3, some Cloud Native Storage (CNS) roles are created by default with slightly different
names than those used by the VMmark 4 scripts.

The VMmark 4 script ~/VMmark4/tools/postprovision-auction.sh uses CNS-DATASTORE in two places,

whereas vCenter might have a role created with variation of this name, such as CNS-Datastore or CNS
Datastore. This discrepancy might result in name conflicts when permissions are being set during tile
provisioning, which could in turn lead to tile provision failure.

We expect this issue to be fixed in a future patch for vCenter 8.0 Update 3.

Until a patch is released, a workaround is, prior to the first VMmark 4 tile provisioning, delete the conflicting CNS
role(s) from vCenter (assuming those roles are not being used by an existing wcpsvc or vSphere Cloud Native
Storage configuration, including an existing VMmark 4 tile).

The initial provision following role deletion should create the roles with compatible names such that
subsequent provisions do not encounter issues.

Error: “Requested number of processors 4 is more than 1 processors this

virtual machine is configured for”
If you see vCenter Server errors that state "Requested number of processors 4 is more than 1 processors this
virtual machine is configured for" and Deploy infrastructure operations are failing, make sure the VMmark
template VM "Cores per Socket" value is set to 1 (which is the default).

Warning: “Line -1: Unsupported value 'pciBridgeN.present'”

If, when deploying OVAs , you see warnings that begin with something like “Line -1: Unsupported value
'pciBridgeN.present',” they are safe to ignore.

Broadcom 71
VMmark User’s Guide

Benchmark Execution Issues

Keyboard and mouse become unresponsive in a VM

If the keyboard and mouse become unresponsive in a Linux virtual machine (such as the VMmark prime
client), use this temporary workaround:

1 Power off the affected virtual machine.

2 Add the following line to that virtual machine’s .vmx file:

keyboard.allowBothIRQs = “FALSE”

3 Power on the virtual machine.

Error “All required properties are not set in /root/VMmark4/VMmark4.properties,

provisioning failed. Exiting...”
The error:
“All required properties are not set in /root/VMmark4/VMmark4.properties, provisioning failed. Exiting...”
is typically seen during --mode "provision" when the user has not added the required csi entries to the
VMmark4.properties file. See the VMmark4.properties file for details on the required csi entries.

Error Similar to “Could not properly start the data services for run 1-0. Exiting.”
If you encounter STAX output such as:
“Could not complete Setup for the following 1 Wklds: Auction Tile0 failed setup”
and an error in restorefiles/ConfigAuction_0.txt similar to:
“Could not properly start the data services for run 1-0. Exiting.”
you might have inadvertently deleted the files backing the Persistent Volume Claim entities (PVCs) in
Kubernetes from your vSphere datastore. If this happens, you can delete the PVCs in Kubernetes (as described
in “Delete Weathervane PVCs Before Removing Kubernetes Cluster” on page 72), thus allowing Weathervane
to create new PVCs, or you can reprovision the VMmark tile.

Delete Weathervane PVCs Before Removing Kubernetes Cluster

The Weathervane workload VMs use vSphere Container Storage. This creates Persistent Volume Claim entities
(PVCs) for persistent data on the CSI datastore configured for that tile during provisioning. For more details,
see “Scalable Web Simulation: Weathervane” on page 16. Before a Kubernetes cluster is removed, the PVCs
should first be deleted. This can be accomplished by deleting the VMmark tiles (but only as described in
“Delete VMmark 4 Tiles” on page 74) or the PVCs for a tile can be deleted on the tile client VM using the
~/VMmark4/tools/misc/pod_pvc_cleanup.sh script (also described in “Delete VMmark 4 Tiles” on
page 74). If tiles are deleted without first deleting the PVCs, the PVCs will remain and will continue to
consume space, but will not be visible to kubectl commands from new Kubernetes clusters are annoying to
clean up.

The following PowerShell script can be used to list and remove such PVCs.

CAUTION This script will remove all PVCs accessible by the user, not just the leftover ones and not just those
created by VMmark. For this reason the removal code is commented out in the script below.

<#
#>
Write-host "Connecting to vCenter server.."
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false
-DisplayDeprecationWarnings:$false -Scope User
Connect-VIServer -Server <ip_address> -User [email protected] -Password '<password>'

Write-host "Listing pvc-*..."

Get-VDisk "pvc-*" | Format-Table -AutoSize -Wrap

#Write-host "Removing pvc-*..."

72 Broadcom
 Appendix B Troubleshooting

#foreach ($vdisk in Get-VDisk "pvc-*") {

# Remove-VDisk $vdisk -confirm:$false
#}

Empty “Please wait” Popup Box When Opening STAX

When opening STAX if you see an empty Please wait popup box that doesn't close when clicking on the X,
close the parent STAX 3 Job Monitor window and reopen it to continue.

No Score in the Score_n_Tile_Test.txt file / “Error: Turbo mode disabled but

Duration (Runtime) = 180 seconds. Rerun with -T/--turbo”
If you find no score in the Score_n_Tile_Test.txt file and see “Error: Turbo mode disabled but Duration
(Runtime) = 180 seconds. Rerun with -T/--turbo,” check for NoSQLBench errors as follows:

1 On the prime client change to the appropriate results directory:

cd ~/VMmark4/results/Results_datestamp
(where datestamp refers to the results to be submitted).

2 Issue the command:

grep ERROR NoSQLBench*.out

 If you see Timed out waiting for server response errors, continue with the steps below.

 If you don't see any errors, this is not the issue. For questions, contact VMware at
[email protected].

3 Verify that your VMmark environment meets the requirements listed in “Client Hardware Requirements”
on page 33.

4 NoSQLBench errors on the clients can be resolved by increasing the number of vCPUs on your client VMs,
for example from the default of 64 vCPUs to 72 vCPUs. If you make this change, be sure to apply it to all
client VMs.

5 Reboot your client VMs and rerun the benchmark to see if the issue is resolved.

Warning “Failed to find principal: VSPHERE.LOCAL\WorkloadStorage”

When the reporter is run — either by the benchmark harness or manually — you might see a warning (which
could be mistakenly reported as an error) similar to:
WARNING authzData.py:116 Failed to find principal: VSPHERE.LOCAL\WorkloadStorage

You can safely disregard this message.

Error: “Failed - Feature ‘cpuid.mwait’ was 0, but must be 1.”

If you see the error:
Feature ‘cpuid.mwait’ was 0, but must be 1
this might mean that MWAIT is disabled in your system BIOS. This can conflict with the vCLS VMs (even
when other VMs don't have any issues).

Sometimes the Maximum Performance setting in BIOS can disable MWAIT by default. It should nevertheless
be possible to enable MWAIT in BIOS (where it might be called MONITOR/MWAIT).

Cancel a Benchmark Run

There are two ways a VMmark run can end before its allotted run time. A run can be canceled manually or it
can cancel automatically if an error occurs. Both scenarios are described below.

Broadcom 73
VMmark User’s Guide

Manually Cancel a Benchmark Run

If you decide to cancel an in-progress benchmark run, it’s possible to do so while still preserving many of the
workload results files. To cancel a running test:

1 Go to the STAX Monitor window and right-click on the active job. Select Terminate Job.

2 Click Yes to confirm.

3 In a terminal window on the prime client, enter:

bash VMmark4-CancelTest.sh <full path to canceled run results directory> <full path
to VMmark4.properties file>

For example:
bash VMmark4-CancelTest.sh /root/VMmark4/results/Results_testrun-2tiles-R1
/root/VMmark4/VMmark4.properties

This script stops the test’s running processes on all client and workload VMs and attempts to copy each
tile’s workload output files (.wrf) into the specified results directory.

4 To prevent issues with subsequent runs:

 Check that the Storage vMotioned and XvMotioned VMs (AuctionWebF and DS35WebA) are on their
original datastore, rather than on the SVMotionLUN or XVMotionLUN specified in the
VMmark4.properties file.

 Delete any DeployVMs from the SUT cluster.

 Reboot all workload virtual machines and all clients except the prime client.

Automatically Cancel on Error

If the Error Immediate function is enabled in the VMmark4.properties file (ErrorImmediate = 1), a
benchmark run will cancel automatically if an error occurs during the run.

After a benchmark run cancels automatically, do the following to prevent issues with subsequent runs:

 Check that the Storage vMotioned and XvMotioned VMs (AuctionWebF and DS35WebA) are on their
original datastore, rather than on the SVMotionLUN or XVMotionLUN specified in the
VMmark4.properties file.

 Delete any DeployVMs from the SUT cluster.

 Reboot all workload virtual machines and all clients except the prime client.

Delete VMmark 4 Tiles

Once a VMmark tile is provisioned, it should not be deleted using vCenter, either through the GUI or through
PowerShell. Doing so will leave orphaned Persistent Volume Claim entities (PVCs) in Kubernetes that will take
up space on the datastore and are difficult to identify. Instead, delete VMmark tiles using one of the following
methods:

 To delete specific tiles, use the vmmark4service "delete_tile" mode:

vmmark4service --mode delete_tile --tile_number X
replacing X with the tile you want to delete.

NOTE Specifying 1 here indicates the first tile, which is tile 0; 2 indicates the second tile, which is tile 1,
and so on.

 To delete all tiles, use the vmmark4service "delete_all_vmmark4" mode:

vmmark4service --mode delete_all_vmmark4
(Also see the VMmark4-Delete-All-Workload-VMs-example.sh script in /root/VMmark4/examples.)

What is not supported is deleting tiles using vCenter through the GUI or PowerShell. This is discussed in more
detail in “Delete Weathervane PVCs Before Removing Kubernetes Cluster” on page 72.

74 Broadcom
 Appendix B Troubleshooting

Using the VMmark4-ConfigChecker Script

The VMmark4-ConfigChecker Perl script (VMmark4-ConfigChecker.pl) can be used by VMmark submitters
to review multiple configuration details of a result for potential compliance and configuration issues. It is not
intended to be an exhaustive check, but is rather a framework that can be built upon. The script’s output also
does not indicate the presence or absence of errors or noncompliant items in the benchmark configuration. The
script simply highlights areas to evaluate for such issues.

NOTE The VMmark4-ConfigChecker script supports only results run on vSphere 8.0 Update 2 or later.

The script is supplied as part of the VMmark benchmark (in the ~/VMmark4/tools directory).

To run the VMmark4-ConfigChecker script, follow these steps:

1 On the prime client change to the appropriate results directory:

cd ~/VMmark4/results/Results_datestamp
(where datestamp refers to the results to be checked).

2 Run the script with the -v (verbose) parameter. For example:

VMmark4-ConfigChecker.pl -v

3 Review the output of the script. The detailed results of each check are written to the
vmmarkconfigchecker-results subdirectory.

NOTE The script decompresses reporter files (with the .tar.xz extension) in the results directory, which
creates subdirectories for the vCenter Server and all ESXi hosts. Once you’re ready to submit your results, run
VMmark4ConfigChecker.pl --cleanup to delete the reporter and vmmarkconfigchecker-results
subdirectories.

Additional points:

 For help, a list of configuration details that are checked, and command line options, run:
VMmark4-ConfigChecker.pl -h

 You are encouraged to extend the mechanisms utilized in the script to add additional functionality. If you
need assistance extending the script, please contact VMware at [email protected].

Broadcom 75
VMmark User’s Guide

Manually Running the VMmark Reporter Scripts

VMmark results for publication must include detailed information about the ESXi and vCenter Server
instances in which the test was run. This information is usually obtained automatically by reporter scripts.
However, if the scripts do not complete correctly, you can run them manually.

Run only the reporter script that is needed to replace the one that did not complete correctly. Before running
the VMmark reporter script manually, ensure that:

 the testbed configuration is the same as it was during the test run,

 all the virtual machines that were powered on during the test was run are powered on, and

 no additional virtual machines (that were not on during the test run) are powered on.

The following section describes running the VMmark reporter script.

Manually Running the Reporter Script

From the prime client, run the VMmark4-Reporter.sh script once for each host for which the reporter bundle
is needed. The script takes several minutes to complete; you can run the script in multiple terminal windows
on the prime client at the same time if you need to run it for more than one host.
Make a new directory for the reporter bundles, and change to it by running these commands (adjust the
directory name if needed):
mkdir /root/VMmark4/results/Reporter_Manual
cd /root/VMmark4/results/Reporter_Manual

For each ESXi host and/or vCenter Server for which the reporter bundle is needed, run this command,
replacing <hostname> with the hostname or IP address of the ESXi host and/or vCenter Server:
VMmark4-Reporter.sh <hostname> > Reporter-<hostname>-manual.out 2>&1
For example: VMmark4-Reporter.sh 10.211.197.12 > Reporter-10.211.197.12-manual.out 2>&1

The script will take a few minutes to complete and the new .tar.xz bundle,
Reporter-<hostname>manual.out and ReporterResult<hostname>.txt will be saved into the
/root/VMmark4/results/Reporter_Manual directory.

You can display the directions to run the reporter manually by running the following on the prime client:
VMmark4-Reporter.sh

76 Broadcom
 Appendix B Troubleshooting

How to Enable and Analyze esxtop Performance Data

To help diagnose problems or simply to obtain the best performance from a virtualized platform it can be
useful to get and analyze esxtop performance data from your VMmark run. This section describes how to do
so.

Obtain a Tool to View the esxtop Performance Data

There are various ways to view esxtop performance data. In this section we’ll describe using the
NMONVisualizer tool, though you can also use perfmon or other tools.

NMONVisualizer can be downloaded from here:

https://nmonvisualizer.github.io/nmonvisualizer/

The tool can be run on macOS, Windows, and Linux.

Enable esxtop Performance Data Collection in VMmark

Follow these steps to enable esxtop performance data collection in VMmark:

1 To enable esxtop collection on all SUT hosts, set the EsxtopCollection parameter to 1 in the
VMmark4.properties file:
EsxtopCollection = 1

2 Specify a LUN on which the esxtop file should be saved (it must be shared storage for the systems under
test):
EsxtopLUN = LUN-NAME
(where LUN-NAME is your datastore name).

3 To enable esxtop collection on all client hosts, set the ClientEsxtopCollection parameter to 1 in the
VMmark4.properties file:
ClientEsxtopCollection = 1

4 Specify a LUN on which the esxtop file should be saved (it must be shared storage for the systems under
test):
ClientEsxtopLUN = LUN-NAME
(where LUN-NAME is your datastore name).

Capture the esxtop Performance Data

Follow these steps to capture the esxtop performance data:

1 Run VMmark.

2 Transfer the VMmark results folder to wherever you put the NMONVisualizer .jar file.

3 The VMmark results folder will include files like HOSTNAME-Esxtop-SUT.csv.gz and
HOSTNAME-Esxtop-Client.csv.gz. Unzip these .gz files.

NOTE When unzipped, these files can be very large (200MB or more).

Open the esxtop Performance Data for Analysis

Follow these steps to open the esxtop performance data for analysis:

1 Open the NMONVisualizer program .jar.

2 In NMONVisualizer, select File > Load.

3 Browse to the VMmark results folder where you unzipped the .gz file. In that folder, locate a .csv file. It
will be something like HOSTNAME-Esxtop-SUT.csv or HOSTNAME-Esxtop-Client.csv.

Broadcom 77
VMmark User’s Guide

4 Click Parse.

NOTE This operation can be slow. If loading freezes, you might need to increase the amount of system
memory allowed for NMONVisualizer. For example, to set the maximum size of the memory allocation
pool to 10GB, open NMONVisualizer use the command:
java -Xmx10g -jar NMONVisualizer_.jar*

Example esxtop Performance Data Analysis: CPU Utilization

To view CPU utilization levels from the VMmark run, in the NMONVisualizer left sidebar select <your
hostname> > Physical Cpu > Total > % Core Util Time. This will show the core utilization time during the run.

% Core Util Time is the metric typically used to represent CPU utilization, though there are several other
metrics available that measure utilization slightly differently.

The bottom pane in NMONVisualizer shows the average, minimum, maximum, and so on over the interval.
Note that this interval includes the benchmark ramp-up, not just the steady state. If you want to see just the
utilization over the steady state, the start and end time of the steady state period can be found in
Score_N_Tile_Test.txt on the lines Steady_state_start and Steady_state_end. ESXi hosts use the
UTC timezone, so use the UTC timezone timestamps on that line in reference to the esxtop files.

In addition to CPU utilization, all the other performance metrics are in the same file and can be analyzed in a
similar manner.

78 Broadcom
Potential Security Vulnerabilities C
VMmark uses a wide variety of open source software. In some cases, that software has known security
vulnerabilities. The VMmark team believes that the risks of using VMmark are minimal due to the
non-sensitive nature of the data VMmark contains and creates as well as it being run within an isolated,
non-production environment. However, each organization must make their own risk assessment. To assist in
doing so, this Appendix provides details of the known security vulnerabilities with a CVSS3 score of 9.0 or
greater contained in the latest release of VMmark 4.0.

Broadcom 79
VMmark User’s Guide

Package Name:
zlib1g-1:1.2.13.dfsg-1

Vulnerability: CVE-2023-45853
CVSS3 Base Score: 9.8

Description:
MiniZip in zlib through 1.3 has an integer overflow and resultant heap-based buffer overflow in
zipOpenNewFileInZip4_64 via a long filename, comment, or extra field. NOTE: MiniZip is not a supported
part of the zlib product.

80 Broadcom
 Appendix C Potential Security Vulnerabilities

Package Name:
GNU C Library-2.32

Vulnerability: CVE-2019-1010022
CVSS3 Base Score: 9.8

Description:
DISPUTED ** GNU Libc current is affected by: Mitigation bypass. The impact is: Attacker may bypass stack
guard protection. The component is: nptl. The attack vector is: Exploit stack buffer overflow vulnerability and
use this bypass vulnerability to bypass stack guard. NOTE: Upstream comments indicate "this is being treated
as a non-security bug and no real threat."

Vulnerability: CVE-2021-33574
CVSS3 Base Score: 9.8

Description:
The mq_notify function in the GNU C Library (aka glibc) has a use-after-free. It may use the notification thread
attributes object (passed through its struct sigevent parameter) after it has been freed by the caller, leading to
a denial of service (application crash) or possibly unspecified other impact.

Vulnerability: CVE-2021-35942
CVSS3 Base Score: 9.1

Description:
The wordexp function in the GNU C Library (aka glibc) through 2.33 may crash or read arbitrary memory in
parse_param (in posix/wordexp.c) when called with an untrusted, crafted pattern, potentially resulting in a
denial of service or disclosure of information. This occurs because atoi was used but strtoul should have been
used to ensure correct calculations.

Vulnerability: CVE-2022-23218
CVSS3 Base Score: 9.8

Description:
The deprecated compatibility function svcunix_create in the sunrpc module of the GNU C Library (aka glibc)
through 2.34 copies its path argument on the stack without validating its length, which may result in a buffer
overflow, potentially resulting in a denial of service or (if an application is not built with a stack protector
enabled) arbitrary code execution.

Vulnerability: CVE-2022-23219
CVSS3 Base Score: 9.8

Description:
The deprecated compatibility function clnt_create in the sunrpc module of the GNU C Library (aka glibc)
through 2.34 copies its hostname argument on the stack without validating its length, which may result in a
buffer overflow, potentially resulting in a denial of service or (if an application is not built with a stack
protector enabled) arbitrary code execution.

Vulnerability: CVE-2023-0687
CVSS3 Base Score: 9.8

Broadcom 81
VMmark User’s Guide

Description:
A vulnerability was found in GNU C Library 2.38. It has been declared as critical. This vulnerability affects the
function __monstartup of the file gmon.c of the component Call Graph Monitor. The manipulation leads to
buffer overflow. It is recommended to apply a patch to fix this issue. VDB-220246 is the identifier assigned to
this vulnerability. NOTE: The real existence of this vulnerability is still doubted at the moment. The inputs that
induce this vulnerability are basically addresses of the running application that is built with gmon enabled.
It's basically trusted input or input that needs an actual security flaw to be compromised or controlled.

82 Broadcom
 Appendix C Potential Security Vulnerabilities

Package Names:
busybox-1.30.1, busybox-initramfs-1:1.30.1-7ubuntu3, busybox-static-1:1.30.1-7ubuntu3

Vulnerability: CVE-2022-48174
CVSS3 Base Score: 9.8

Description:
There is a stack overflow vulnerability in ash.c:6030 in busybox before 1.35. In the environment of Internet of
Vehicles, this vulnerability can be executed from command to arbitrary code execution.

Broadcom 83
VMmark User’s Guide

Package Names:
php8.1-8.1.2-1ubuntu2.13, php8.1-readline-8.1.2-1ubuntu2.14, php8.1-opcache-8.1.2-1ubuntu2.14,
php8.1-common-8.1.2-1ubuntu2.14, php8.1-cli-8.1.2-1ubuntu2.14, php8.1-8.1.2-1ubuntu2.14,
libapache2-mod-php8.1-8.1.2-1ubuntu2.14

Vulnerability: CVE-2016-9138
CVSS3 Base Score: 9.8

Description:
PHP through 5.6.27 and 7.x through 7.0.12 mishandles property modification during _wakeup processing,
which allows remote attackers to cause a denial of service or possibly have unspecified other impact via
crafted serialized data, as demonstrated by Exception::toString with DateInterval::_wakeup.

84 Broadcom
 Appendix C Potential Security Vulnerabilities

Package Names:
apparmor-3.0.4-2ubuntu2.2, libapparmor1-3.0.4-2ubuntu2.3

Vulnerability: CVE-2016-1585
CVSS3 Base Score: 9.8

Description:
In all versions of AppArmor mount rules are accidentally widened when compiled.

Broadcom 85
VMmark User’s Guide

Package Name:
libarchive13-3.6.0-1ubuntu1, libarchive-3.6.0

Vulnerability: CVE-2022-36227
CVSS3 Base Score: 9.8

Description:
In libarchive before 3.6.2, the software does not check for an error after calling calloc function that can return
with a NULL pointer if the function fails, which leads to a resultant NULL pointer dereference. NOTE: the
discoverer cites this CWE-476 remark but third parties dispute the code-execution impact: "In rare
circumstances, when NULL is equivalent to the 0x0 memory address and privileged code can access it, then
writing or reading memory is possible, which may lead to code execution."

86 Broadcom
 Appendix C Potential Security Vulnerabilities

Package Names:
libc6-2.31-13+deb11u5, GNU C Library-2.36.9000 libc6-dev-2.36-9+deb12u4 libc6-2.36-9+deb12u4,
libc-dev-bin-2.36-9+deb12u4 libc-bin-2.36-9+deb12u4

Vulnerability: CVE-2019-1010022
CVSS3 Base Score: 9.8

Description:
DISPUTED ** GNU Libc current is affected by: Mitigation bypass. The impact is: Attacker may bypass stack
guard protection. The component is: nptl. The attack vector is: Exploit stack buffer overflow vulnerability and
use this bypass vulnerability to bypass stack guard. NOTE: Upstream comments indicate "this is being treated
as a non-security bug and no real threat."

Vulnerability: CVE-2023-0687
CVSS3 Base Score: 9.8

Description:
A vulnerability was found in GNU C Library 2.38. It has been declared as critical. This vulnerability affects the
function __monstartup of the file gmon.c of the component Call Graph Monitor. The manipulation leads to
buffer overflow. It is recommended to apply a patch to fix this issue. VDB-220246 is the identifier assigned to
this vulnerability. NOTE: The real existence of this vulnerability is still doubted at the moment. The inputs that
induce this vulnerability are basically addresses of the running application that is built with gmon enabled.
It's basically trusted input or input that needs an actual security flaw to be compromised or controlled.

Broadcom 87
VMmark User’s Guide

Package Name:
github.com/moby/buildkit-v0.12.1

Vulnerability: CVE-2024-23652
CVSS3 Base Score: 9.1

Description:
BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable
manner. A malicious BuildKit frontend or Dockerfile using RUN --mount could trick the feature that removes
empty files created for the mountpoints into removing a file outside the container, from the host system. The
issue has been fixed in v0.12.5. Workarounds include avoiding using BuildKit frontends from an untrusted
source or building an untrusted Dockerfile containing RUN --mount feature.

Vulnerability: CVE-2024-23653
CVSS3 Base Score: 9.8

Description:
BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable
manner. In addition to running containers as build steps, BuildKit also provides APIs for running interactive
containers based on built images. It was possible to use these APIs to ask BuildKit to run a container with
elevated privileges. Normally, running such containers is only allowed if special `security.insecure`
entitlement is enabled both by buildkitd configuration and allowed by the user initializing the build request.
The issue has been fixed in v0.12.5. Avoid using BuildKit frontends from untrusted sources.

88 Broadcom
 Appendix C Potential Security Vulnerabilities

Package Name:
OpenJPEG-v2.4.0

Vulnerability: CVE-2021-3575
CVSS3 Base Score: 7.8

Description:
A heap-based buffer overflow was found in openjpeg in color.c:379:42 in sycc420_to_rgb when decompressing
a crafted .j2k file. An attacker could use this to execute arbitrary code with the permissions of the application
compiled against openjpeg.

Broadcom 89
VMmark User’s Guide

Package Name:
Linux-Pam-v1.5.2

Vulnerability: CVE-2022-28321
CVSS3 Base Score: 9.8

Description:
The Linux-PAM package before 1.5.2-6.1 for openSUSE Tumbleweed allows authentication bypass for SSH
logins. The pam_access.so module doesn't correctly restrict login if a user tries to connect from an IP address
that is not resolvable via DNS. In such conditions, a user with denied access to a machine can still get access.
NOTE: the relevance of this issue is largely limited to openSUSE Tumbleweed and openSUSE Factory; it does
not affect Linux-PAM upstream.

90 Broadcom
 Appendix C Potential Security Vulnerabilities

Package Name:
linux-libc-dev-6.1.76-1

Vulnerability: CVE-2024-26584
CVSS3 Base Score: N/A

Description:
In the Linux kernel, the following vulnerability has been resolved: net: tls: handle backlogging of crypto
requests Since we're setting the CRYPTO_TFM_REQ_MAY_BACKLOG flag on our requests to the crypto API,
crypto_aead_{encrypt,decrypt} can return -EBUSY instead of -EINPROGRESS in valid situations. For
example, when the cryptd queue for AESNI is full (easy to trigger with an artificially low
cryptd.cryptd_max_cpu_qlen), requests will be enqueued to the backlog but still processed. In that case, the
async callback will also be called twice: first with err == -EINPROGRESS, which it seems we can just ignore,
then with err == 0. Compared to Sabrina's original patch this version uses the new tls_*crypt_async_wait()
helpers and converts the EBUSY to EINPROGRESS to avoid having to modify all the error handling paths. The
handling is identical.

Broadcom 91
VMmark User’s Guide

Package Names:
Linux Kernel-5.17.6, Linux Kernel-v5.15.15, Linux Kernel-v5.16.18

Vulnerability: ZDI-23-979
CVSS Score: 5.9

Description:
This vulnerability allows remote attackers to create a denial-of-service condition on affected installations of
Linux Kernel. Authentication is not required to exploit this vulnerability, but only systems with ksmbd
enabled are vulnerable.

The specific flaw exists within the handling of chained requests. The issue results from dereferencing a NULL
pointer. An attacker can leverage this vulnerability to create a denial-of-service condition on the system.

Vulnerability: CVE-2022-43945
CVSS3 Base Score: 7.5

Description:
The Linux kernel NFSD implementation prior to versions 5.19.17 and 6.0.2 are vulnerable to buffer overflow.
NFSD tracks the number of pages held by each NFSD thread by combining the receive and send buffers of a
remote procedure call (RPC) into a single array of pages. A client can force the send buffer to shrink by sending
an RPC message over TCP with garbage data added at the end of the message. The RPC message with garbage
data is still correctly formed according to the specification and is passed forward to handlers. Vulnerable code
in NFSD is not expecting the oversized request and writes beyond the allocated buffer space.

Vulnerability: CVE-2022-47939
CVSS3 Base Score: 9.8

Description:
An issue was discovered in ksmbd in the Linux kernel 5.15 through 5.19 before 5.19.2. fs/ksmbd/smb2pdu.c
has a use-after-free and OOPS for SMB2_TREE_DISCONNECT.

Vulnerability: CVE-2023-5178
CVSS3 Base Score: 9.8

Description:
A use-after-free vulnerability was found in drivers/nvme/target/tcp.c` in `nvmet_tcp_free_crypto` due to a
logical bug in the NVMe/TCP subsystem in the Linux kernel. This issue may allow a malicious user to cause
a use-after-free and double-free problem, which may permit remote code execution or lead to local privilege
escalation.

Vulnerability: CVE-2023-38428
CVSS3 Base Score: 9.1

Description:
An issue was discovered in the Linux kernel before 6.3.4. fs/ksmbd/smb2pdu.c in ksmbd does not properly
check the UserName value because it does not consider the address of security buffer, leading to an
out-of-bounds read.

Vulnerability: CVE-2023-38430
CVSS3 Base Score: 9.1

92 Broadcom
 Appendix C Potential Security Vulnerabilities

Description:
An issue was discovered in the Linux kernel before 6.3.9. ksmbd does not validate the SMB request protocol
ID, leading to an out-of-bounds read.

Vulnerability: CVE-2023-38431
CVSS3 Base Score: 9.1

Description:
An issue was discovered in the Linux kernel before 6.3.8. fs/smb/server/connection.c in ksmbd does not
validate the relationship between the NetBIOS header's length field and the SMB header sizes, via pdu_size in
ksmbd_conn_handler_loop, leading to an out-of-bounds read.

Vulnerability: CVE-2023-38432
CVSS3 Base Score: 9.1

Description:
An issue was discovered in the Linux kernel before 6.3.10. fs/smb/server/smb2misc.c in ksmbd does not
validate the relationship between the command payload size and the RFC1002 length specification, leading to
an out-of-bounds read.

Vulnerability: CVE-2023-40791
CVSS3 Base Score: 6.3

Description:
extract_user_to_sg in lib/scatterlist.c in the Linux kernel before 6.4.12 fails to unpin pages in a certain situation,
as demonstrated by a WARNING for try_grab_page.

Vulnerability: CVE-2024-26584
CVSS3 Base Score: N/A

Description:
In the Linux kernel, the following vulnerability has been resolved: net: tls: handle backlogging of crypto
requests Since we're setting the CRYPTO_TFM_REQ_MAY_BACKLOG flag on our requests to the crypto API,
crypto_aead_{encrypt,decrypt} can return -EBUSY instead of -EINPROGRESS in valid situations. For
example, when the cryptd queue for AESNI is full (easy to trigger with an artificially low
cryptd.cryptd_max_cpu_qlen), requests will be enqueued to the backlog but still processed. In that case, the
async callback will also be called twice: first with err == -EINPROGRESS, which it seems we can just ignore,
then with err == 0. Compared to Sabrina's original patch this version uses the new tls_*crypt_async_wait()
helpers and converts the EBUSY to EINPROGRESS to avoid having to modify all the error handling paths. The
handling is identical.

Broadcom 93
VMmark User’s Guide

94 Broadcom