ObjectScale Next Evolution in Object Storage
ObjectScale Next Evolution in Object Storage
IN OBJECT STORAGE
Nithin Shankar
Inside Product Specialist 2
Dell Technologies
[email protected]
Kshitij Yadav
Inside Product Specialist 2
Dell Technologies
[email protected]
Courses are offered to meet different learning styles and schedules, including self-paced On
Demand, remote-based Virtual Instructor-Led and in-person Classrooms.
Whether you are an experienced IT professional or just getting started, Dell Technologies
Proven Professional certifications are designed to clearly signal proficiency to colleagues and
employers.
Disclaimer: The views, processes or methodologies published in this article are those of the
authors. They do not necessarily reflect Dell Technologies’ views, processes or methodologies.
Dell.com/certification 3
1. Introduction
1.1 IDC
According to IDC, by 2026, large companies forecast that unstructured data stored as files or
object storage in on-premises, edge, or public clouds will triple compared to 2021. By 2025,
60% of the global storage capacity for unstructured data will be deployed as software-defined
storage (SDS), up from less than 25% in 2021.
1.2 Background
The unstructured data storage market is evolving rapidly. New capabilities are needed to meet
the growing challenges of exponential data growth, rapid digitization, and business
globalization. Storage infrastructure products based on distributed file systems and object
storage are growing faster than ever, both in scale and capacity, as a platform for meeting the
growth of unstructured data in global data centers.
Gartner end users report that unstructured data has increased by more than 30% year-on-year.
As a result, Infrastructure and Operations (I&O) leaders are looking for scalable on-premises
storage products that can meet growing digital business use cases while reducing acquisition,
operational, and management costs. I&O leaders want scalability, flexibility, lifecycle
management, manageability, and analytical insights into the data.
Rapid growth of unstructured data for new and established workloads requires new types of
products and cost efficiency. Most products in this market are driven by SDS that can provide
petabytes of storage. SDS can also leverage hybrid cloud workflows with public cloud
infrastructure as a service (IaaS) to reduce total cost of ownership (TCO) and improve data
mobility. Newly established storage vendors continue to develop scalable storage scalable file
systems and object storage products to address the cost, agility, and scalability limitations of
traditional scale-up storage environments.
Gartner defines file systems and object storage as a distributed design-based software and
hardware platform that supports object and/or scale-out file system technology to meet the
growing demands of unstructured data. This market is based on a distributed computing
architecture with no single points of failure or contention throughout the system. More
specifically, the product should have a fully distributed architecture. In this architecture, data and
metadata are distributed, replicated, or erasure-coded across multiple nodes in the cluster.
When managing a multi-petabyte scale system, it is important to be able to scale out the
capacity and throughput of the cluster by adding independent nodes to the global namespace /
file system.
Object storage, according to Gartner, also refers to systems and software that store data in
"objects" and offer it to clients using RESTful HTTP (application programming interface) APIs,
such as Amazon Simple Storage Service (S3), which has become the de facto standard for
SDS is a storage architecture that separates storage and software from the underlying hardware
infrastructure. Unlike traditional network attached storage (NAS) or storage area network) (SAN)
systems, SDS is typically designed to run on industry standard systems, and the software does
not rely on proprietary hardware.
Decoupling storage software from its hardware allows you to expand your storage capacity as
needed, rather than adding another piece of proprietary hardware. You can also upgrade or
downgrade hardware at any time. Basically, SDS offers a great deal of flexibility.
• Flexibility in hardware selection: The SDS you choose does not have to be from the
same company that sold you the hardware. You can use any commodity server to build
your SDS-based storage infrastructure. This means that you can maximize the capacity
of your existing hardware as your storage needs grow.
• Cost efficiency: SDS is distributed and scales out instead of scaling up, allowing you to
adjust capacity and performance independently.
• You can join many data sources to build your storage infrastructure: You can
network object platforms, external disk systems, disk or flash resources, virtual servers,
and cloud-based resources (even data dedicated to workloads) to create a unified
storage volume.
• SDS can adjust automatically based on your capacity needs: SDS doesn't depend
upon hardware; automation in SDS is, well, automatic in the sense that it can pull from
any storage volume it’s connected to. Without administrator involvement, new
connections, or new hardware, the storage system can adapt to data needs and
performance.
• Limitless Scalability: Traditional SAN are limited to the number of nodes (devices with
assigned IP addresses) they can use. SDS, by its very definition, is not similarly
constrained. That means – theoretically – it is infinitely scalable.
Dell.com/certification 5
2. ObjectScale
2.1 Introduction
IT must provide developers with agile infrastructure that gives them more efficient ways to
provision storage for their apps while ensuring the entire environment is secure and governed
according to policy as containers and Kubernetes continue to drive cloud-native modernization
initiatives. This necessitates S3-compatible, Kubernetes-compliant object storage that works in
tandem with CI/CD (continuous integration and continuous deployment) pipelines to speed up
application development and reduce time-to-market.
2.2 ECS
Dell Technologies has a long history in the field of object storage. In 2001, Dell developed Dell
EMC Centera, the first CAS-based commercial object storage solution, and since then
continually redesigned the portfolio to match customers' needs.
Elastic Cloud Storage (ECS), Dell EMC's flagship enterprise-class object storage platform, is
designed to handle both traditional and next-generation workloads. ECS has unrivalled
scalability, performance, robustness, and economics to meet the demands of a modern
organization, and it can be deployed as a software-only architecture or as a turnkey appliance.
With ECS, you can create a cost-effective object storage private cloud with TCO as low as 59
percent. As an S3-compatible, globally scalable object-store, ECS combines the capabilities of
the public cloud with the command and control of a private cloud infrastructure.
ECS is a software-defined cloud storage platform that allows you to store, manipulate, and
analyze unstructured data on commodity hardware at a large scale. It blends commodity
infrastructure's low cost with traditional arrays' enterprise-level reliability, availability, and
serviceability.
ECS has a scalable design with several nodes and storage devices attached. The nodes and
storage devices are commodity components that are housed in one or more racks and are
similar to devices that are widely accessible.
• Data Services: The data services layer supports various object protocols; S3 (Simple
Storage Service), Atmos, Swift and CAS (Content Addressable Storage) as well as file
system protocols like HDFS and NFS (Network File System) for accessing the ECS
object store. In general, ECS supports multiprotocol access, which means that data
ingested via one protocol can be accessed via another. Data ingested using S3 can, for
example, be modified using Swift, NFS, or HDFS.
• Provisioning Portal / Mgmt: The ECS portal is the centralized access point where you
go to manage the ECS appliance. It provides options for both monitoring and operating
(managing, licensing, and provisioning) ECS nodes.
The following detailed reporting options are available through the portal:
• Storage engine: The storage engine layer provides an unstructured storage engine for
data storage and retrieval, transaction management, and data protection and replication.
Objects ingested via numerous object storage protocols, as well as the NFS and HDFS
file protocols, are accessible through the storage engine.
Dell.com/certification 7
• Fabric: Cluster health management, software management, configuration management,
upgrade capabilities, and alerting are all provided by the fabric component layer. The
fabric layer is in charge of keeping services up and running as well as managing
resources like discs, containers, firewalls, and networks. It monitors and reacts to
changes in the environment, such as failure detection, and sends out notifications about
system health. Fabric Firewall Manager protects the 9069 and 9099 ports, which are
public IP ports. Outside of the cluster, the port is unavailable.
• Infrastructure: ECS runs on top of SUSE Linux Enterprise Server 12 (SLES 12)
operating system as a Java application; hence, Java Virtual Machine (JVM) is installed
as well. Docker is installed on the infrastructure to deploy the encapsulated ECS layers.
The containers are isolated only sharing the underlying operating system resources and
hardware.
• Hardware: The Gen3 or EX-series appliance models of ECS are based on standard Dell
servers and switches.
As shown in Figure 2, every single node runs every single service. This will simplify the
deployment model in an ECS appliance. However, it is limited in terms of flexibility.
Virtual Data Center - A VDC is made up of a set of ECS infrastructure governed by a single
fabric instance, generally referred to as a site or geographically distinct region.
Storage Pool - A storage pool (SP) is a subset of nodes and their associated storage that are
part of a VDC. A storage pool can have any number of nodes; however, the minimum
recommended is five. An ECS node can only belong to one storage pool. A storage pool is a
tool that can be used to physically separate data from distinct applications. At the storage pool-
level, erasure coding is done using either a 12+4 or a 10+2 scheme.
Replication Group - Replication groups (RG) specify where content from storage pools is
safeguarded and where data can be read or written. Objects inside the same VDC are protected
by local replication groups from disc, node, and rack failures. Global replication groups
safeguard objects from disc, node, rack, and site failures across many VDCs. A VDC can be
associated with numerous RGs.
Namespace - A namespace's most distinguishing feature is that users from one namespace are
unable to access objects from another namespace. A namespace can represent a department
or a group inside a department.
Buckets - Buckets are object data containers that are created in a namespace to allow apps to
access data stored in ECS. These containers are referred to as "buckets" in S3, and ECS has
embraced this term. In ECS, buckets are global resources. Each bucket has its own
namespace, and each namespace has its own RG. A bucket is replicated across sites when the
replication group spans multiple sites.
Dell.com/certification 9
It is at the high-level design of the ECS appliance where we come in to dedicate a physical node
and all its sub-resources. We allocate that into a storage pool. That storage pool is allocated to
a VDC. Replication groups are sliced and behaviors for replication will be on top of the VDC and
storage pools and so on. Lastly, we do our tenancy model across those buckets using
Namespaces.
ObjectScale uses the same Dell EMC ECS codebase. ObjectScale inherits the worldwide
Global scalability, robust S3 compatibility, data protection, and security benefits that have
helped make ECS the market leader in on-premises object storage. What makes it game-
changing is that we do not have to map and use all of the underlying resources at any given
time. ObjectScale enables distribution of the different storage services that interact with the
physical disks, adding to the amount needed when it is needed. For example, choose the
number of instances of the management services actually required. Another example would be
having just two API gateways (Figure 4) versus having API gateways on every single node.
In short, ECS is basically monolithic containers, and with ObjectScale as architecture, ECS rips
all that apart, micro-servicing it. So, we use what we need when we need it plus the added
capabilities that come with using Kubernetes for orchestration such as load balancing,
scheduling, self-healing, resource optimization, etc.
Dell.com/certification 11
2.3.1 Object Store
The next question is how does ObjectScale then differ in terms of architecture? Principles that drive
ObjectScale from an infrastructure perspective is Kubernetes. With the ECS appliance, we use VDC, storage
pools and such to drive object storage. With ObjectScale, we are moving into a construct called Object Store,
a completely resource-isolated deployment of an object platform. ECS’s processes are broken up into
individually scaled Kubernetes “pods” to efficiently use only the resources needed. ObjectScale allows
multiple object stores in one Kubernetes cluster. The object stores will interface with a global ObjectScale
manager that orchestrates them: create, upgrade, delete etc. (lifecycle of multiple object stores).
• Identity and Access Management: IAM is framework of policies and technologies that ensure the
right users have the appropriate access to technology resources. ObjectScale supports a wide set of
granular permissions, conditions, and policies. It enables bucket, object and user tagging and tags to
be referenced in IAM policies.
• Federation: managing a geographically distributed environment as a single logical resource.
• A web-based GUI to manage, license and provision ObjectScale clusters.
• GraphQL: an open-source data query and manipulation language for APIs, and a runtime for fulfilling
queries with existing data.
Disks will be provided into an object store and the object store will then be the instance of what we are doing
with the deployment. Looking at a high-level design of objectscale, we have the Kubernetes platform, which
has either virtual or physical nodes. We will also have our ObjectScale manager. Kubernetes acts as a
connector between physical infrastructure, such as disc and network, and containerized application services.
ObjectScale relies on Kubernetes' resource-management features to manage operating-system and
hardware interactions.
Te global infrastructure model for ObjectScale provides a regional construct where namespaces can be
shown in one region or multiple regions. This enables replication of a bucket using Cross Region Replication
(CRR) from the object store in one region to another bucket object store in another region (called as global
replication) or even locally replicate the buckets. Replicating data across environments helps organizations
protect workloads from outages and share data with teams across the globe, while native multi-tenancy
provides resource isolation and secure global access through multi-site failure tolerance. ObjectScale
Replication can be used to fuel everything from dev/test sandboxes to globally distributed data lakes.
Dell.com/certification 13
3. Deployment Options with ObjectScale
ObjectScale runs on Kubernetes distributions including Red Hat OpenShift and VMware vSphere with Tanzu,
allowing developers to focus on code while IT professionals manage ObjectScale Storage to policy. With
deep VMware integration, ObjectScale can be deployed on Dell EMC VxRail, a hyper-converged
infrastructure (HCI) system purpose-built for VMware environments. It’s flexible IT at your fingertips. Let’s
discuss further.
The Red Hat OpenShift Container Platform is an on-premises private platform-as-a-service product. It helps
enterprises establish a DevOps model for developing cloud-native apps by providing developers and
operations teams with the tools and services they need to address today's needs while also planning for the
future. It is built around application containers powered by Docker, with orchestration and management
provided by Kubernetes, and runs on Red Hat Enterprise Linux and Container Linux.
Dell Technologies in association with Red Hat enables enterprises to deploy and run S3-compliant object
storage on the OpenShift Container Platform to support cloud-native apps from an on-premises data center.
3.1.2 Architecture
With ObjectScale's integration with Red Hat OpenShift, developers can use Kubernetes APIs to provide and
scale high-performance object storage while providing IT with unrivalled simplicity, manageability, and
reliability.
Deploying ObjectScale with OpenShift alongside traditional applications in a full-stack infrastructure that
comprises servers, primary storage, and supporting software is one approach to maximize business value.
Customers can choose from a variety of options for ObjectScale support and deployment.
• Dell EMC ProDeploy and ProDeploy Plus for Enterprise offer remote and in-person deployment
options along with site readiness review, implementation planning, etc.
• For qualifying 3rd party software, Dell EMC ProSupport and ProSupport Plus for Enterprise offers
collaborative or single source support options.
• Dell EMC Data Migration services include a project manager and design specialists who use tools,
techniques, and worldwide best practices to migrate data from Dell EMC and third-party storage to
new infrastructure securely and efficiently.
VMware is dominating applications and infrastructure modernization. VMware created the ideal environment
for running modern apps within the enterprise with the release of VMware Cloud Foundation with Tanzu in
April of 2021. The latest developments in VMware vSphere and VMware vSAN, which foster collaboration
between development and IT operations, are at the heart of VMware Cloud Foundation with Tanzu.
With Kubernetes integrated into vSphere, data center managers can manage both virtual machines and
Kubernetes clusters from a single interface, on the same platform they already know and love. Developers
may quickly deploy infrastructure services on demand thanks to an open ecosystem of Kubernetes-based
APIs. Customers can now provision persistent cloud-scale object storage for modern stateful applications
using VMware vSAN DPp (Data Persistence platform), which is integrated with VMware Cloud Foundation
with Tanzu.
The vSAN DPp provides a platform for VMware partners who deliver modern stateful services to interact with
the underlying virtual infrastructure, enabling execution of stateful services with high velocity scalability,
simpler IT management, and lower TCO. VMware vSphere with Tanzu (VMware vSAN, VMware NSX, and
VMware Tanzu) licenses are also required for the vSAN Data Persistence platform.
Dell.com/certification 15
3.2.1 Architecture
Dell EMC ObjectScale’s integration with VMware vSphere with Tanzu provides developers freedom to provision
and scale high performance object storage via Kubernetes APIs while ensuring unmatched simplicity,
manageability, and reliability for IT.
Organizations can use VMware vSAN Direct Configuration to deploy ObjectScale alongside traditional
applications on a regular vSAN cluster using the vSAN-SNA (vSAN Support for Shared Nothing Architecture)
policy or use VMware vSAN Direct Configuration (a technology enabling direct access to the underlying
direct-attached hardware which can be optimized for the application needs) to deploy it on a dedicated vSAN
cluster. Both choices benefit from excellent storage efficiency for stateful services by employing service-level
replication and unified management of services in VMware vCenter.
3.3 Dell EMC ObjectScale on VMware vSphere with Tanzu on Dell EMC VxRail
Containers and Kubernetes are driving enterprise-wide modern application initiatives. While cloud-native
applications promise to speed up development and innovation, many of these next-generation apps require
modern S3-compatible storage to function properly. To fully embrace a genuine DevOps model, IT must also
give developers with self-service infrastructure resources, such as storage, to enable smooth integration into
CI/CD pipelines. To add to the complexity, IT departments are frequently required to support new apps and
cloud-native efforts using current data center resources and skills.
Organizations will be able to support Kubernetes workloads and containers with ObjectScale software-
defined object storage by running VMware vSphere with Tanzu and the vSAN Data Persistence platform on
Dell EMC VxRail, the only fully integrated, pre-configured, and proven HCI system designed for VMware
vSAN. It provides a turnkey solution for quicker, better, and simpler workload management in private and
hybrid cloud settings.
ObjectScale can be implemented on VxRail thanks to its integration with the vSAN Data Persistence
technology, allowing organizations to take advantage of new modern application opportunities on familiar
hyperconverged infrastructure.
Consistent infrastructure and operations managed from VMware: With ObjectScale's integration with
VMware's vSAN Data Persistence platform, IT teams do not have to learn new skills or manage an
external storage footprint. They can help developers with cloud-native storage using the VMware tools
and capabilities they've developed in-house. They can now control shadow IT while freeing developers to
use Kubernetes APIs to support CI/CD procedures and agile techniques in a self-service manner by
running ObjectScale on VxRail.
Storage Policy-Based Management (SPBM) can be extended from VM volumes to container volumes,
providing for a more uniform management strategy. Furthermore, VxRail Manager and vCenter are 100%
natively integrated, and all VxRail management is done through the familiar vServer interface. Intelligent
lifecycle management (LCM) automates non-disruptive updates, patches, node additions, and node
retirements while keeping the VxRail infrastructure in a continuously validated state to guarantee
workloads operate and clusters are optimized. ObjectScale and VxRail combine to provide persistent
cloud-scale object storage for modern stateful applications on a uniform architecture with a consistent
management experience, minimizing TCO.
Dell.com/certification 17
Storage infrastructure supported by a single vendor: Storage supported by a single vendor - Dell
Technologies: Companies can take advantage of the integrated storage stack offered by Dell with
ObjectScale and VxRail, providing a consistent, unified experience for all storage support and
maintenance requirements. Dell Technologies provides a comprehensive set of deployment, support, and
advisory services to help you get the most out of your technology. Dell EMC ProDeploy, ProSupport, and
Data Migration are some of the services available. To comply with the various financial strategies that
businesses follow, flexible consumption models and payment solutions are also available.
Dev / Test: ObjectScale is the appropriate sandbox for S3-enabled modern application
development. Object stores and buckets are simple to install thanks to automatic provisioning and
an integrated workload sizer. Identity and access management (IAM) controls offer safe data
access, while multi-tenancy enables resource separation. You can set up scratch object stores in
minutes, decommission them when they're no longer needed, and then push them to production
with a few simple clicks using ObjectScale. Data can be easily shared between development
teams, allowing for more collaboration and productivity.
Consolidated data lake: Sensor telemetry, machine-generated logs, and application data may all
be stored in an indefinitely scalable, centralized data lake using ObjectScale. You can choose the
performance profile that best fits your workloads by deploying it on all-flash or capacity-optimized
media. Connect several sites through federation to avoid data silos and offer data access from the
edge to the core. Tag objects to improve analysis and make data more discoverable. Cloudera,
Dremio, and Vertica provide powerful data lake engines and analytics systems.
AI & ML: ObjectScale supports next-generation AI and ML workloads, the next frontier for object
storage, by delivering performance at scale. ObjectScale offers large datasets at high transfer
rates to the most demanding CPU and GPU servers, allowing AI training algorithms to access
more data without the need for HPC storage. Clusters may be easily scaled out to increase
performance and capacity in a linear fashion. Object tagging also gives inference models more
data to work with, allowing them to generate better predictions.
Analytics: Generate operational insights at the pace your organization requires by running
lightning-fast queries on a performant ObjectScale data lake. Storage performance is no longer a
limitation thanks to the option to deploy on NVMe-based all-flash devices. Custom tags give data
more context, allowing for easier discovery and faster results. S3a allows Hadoop applications to
read and write data to ObjectScale without having to manage a sophisticated HDFS cluster.
Enhancing Dell EMC VxRail: ObjectScale adds S3-compatible, cloud-native object storage to
Dell EMC VxRail, allowing it to serve workloads ranging from high-performance analytics to long-
term archiving. ObjectScale can handle stateful sets thanks to the VMware vSAN Data
Persistence platform, which can be configured directly through the vCenter interface. You can
now support modern application development projects without having to acquire new
infrastructure because VxRail is integrated, optimized, tested, and verified to run VMware's stack.
Dell.com/certification 19
5. ObjectScale and ECS Positioning
To position ObjectScale and ECS, it is important that we understand their differences.
ECS is Dell EMC’s current enterprise-grade object storage system that features a microservices framework.
ECS is designed for organizations that require a fully integrated turnkey appliance for diverse object storage
requirements, though it can also be deployed as a software-defined solution that sits on top of certified
hardware. ECS has a long legacy of enterprise-proven functionality and performance.
ObjectScale uses functionality native to the Kubernetes platform, with architecture that can be easily
deployed and scaled. ObjectScale builds upon Dell’s success in the object storage market by implementing a
software-defined stack that enables object storage capabilities for VMware and Red Hat today, and additional
use cases and platforms in the future.
6. Conclusion
ObjectScale clusters, which are built on a scale-out architecture, may extend from a few terabytes to
petabytes and beyond, with no restriction on the amount of object stores, buckets, or objects stored. Clusters
can begin with just three nodes and expand from there. ObjectScale Replication allows you to replicate object
data from the edge to the core data center, from any location where an ObjectScale footprint exists. By
replicating data throughout the environment, users may protect workloads against failures and share data
with colleagues worldwide. D@RE, erasure coding, versioning, resource isolation, IAM, WORM (Write Once
Read Many) features, and more are included in ObjectScale's enterprise security and data protection
capabilities. It's storage technology you can rely on from the industry leader.
Dell.com/certification 21
Having two product offerings in ECS and ObjectScale enable Dell Technologies to better meet the
increased demand for object storage while serving two unique use cases and deployment models.
Both products are an important part of the Dell Technologies product portfolio and both are
designed to provide enterprise-grade object storage. However, each is optimized for different usage
and deployment scenarios.
ECS will continue to serve organizations that require large-scale industrial archive and enterprise
solutions using some of the densest appliance form factors in the market.
ObjectScale is ideal for edge use cases or when organizations want to deploy flexible, lightweight
object storage close to the applications they support. Due to its software-defined nature and S3
compatibility, ObjectScale can also be deployed on existing virtualized infrastructure, making it an
excellent platform for cloud-native applications and dev/test scenarios.
Dell Technologies believes the information in this publication is accurate as of its publication date. The
information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” DELL TECHNOLOGIES MAKES NO
RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN
THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY
OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying and distribution of any Dell Technologies software described in this publication requires an
applicable software license.
Copyright © 2022 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell EMC
and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of
their respective owners.
Dell.com/certification 23