TH E

OPTIM IZ ATION
B E NCH MARK

Q2 / 2015

+23%

The Race to Find the

Ideal Customer Experience

- 9%

+11%

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

Contents
50%
40%

Introduction
30%

Why Optimizers Test

20%

How Optimizers Build Their Teams

10%

How Optimizers Manage Their Process

0%
September

October

November

Whats Next for Optimization

Methodology and Audience Composition

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

Introduction
Optimization is a fast-growing capability,
and is being adopted and resourced as
a competitive advantage for marketing,
growth, product, and engineering-focused
organizations around the world.

The practice of optimization has evolved significantly

needed to accelerate the development of their

from its origins of A/B testing small tweaks to add

optimization programs. The result? Faster business

incremental conversion improvements. Optimization

benefits and in greater orders of magnitude.

is now a holistic strategy that takes into account

how to optimize every customer touchpoint and

For the companies that are already practicing

continuously drives improvements. Businesses

optimization, it is imperative to align around

looking to create better digital experiences for

emerging best practices. They should challenge

their customers and to increase the value of those

their organizations to optimize more holistically,

interactions are turning to experience optimization as

and make sure that they are supported by ample

a program that can drive proven business benefits.

resources so they can continue to earn dividends

on their technology investment in optimization.

In Optimizelys independent survey of digital channel

owners with Dimensional Research, we observed an

For companies who arent yet investing in

emerging set of behaviors that indicate sophisticated

optimizaton, the opportunity cost of waiting on

optimization being done by teams actively running

the sidelines will only grow, as other companies

A/B and multivariate experiments. These teams

continue to develop the organizational skill of

are doubling down on the people and processes

optimization into a competitive advantage.

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

Why Optimizers Test

In the context of this report,
Optimization is defined as: The process
of running A/B/n and multivariate
experiments on desktop websites, mobile
websites, mobile apps, or other digital
channels. This process lets individuals test
ideas for improvements to their customers
digital experiences in a controlled manner,
collect data on the outcome, and apply
statistical inference to decide on the
optimal experience moving forward.

Within the normal course of doing business, the

We were impressed by just how rapidly optimization

following statement is inevitably made: This headline

has been integrated across a wide range of teams

doesnt capture my attentionwe should change

and companies, and to see how sophisticated

it. Or, This page isnt ____ enough. Maybe it

the early adopters in this area have become.

lacks visual contrast, clear language, or creates

confusion and uncertainty for the visitor or user. In

Companies are turning to optimization as a natural

some cases, maybe the conversation is based on

fit for some of their largest areas of uncertainty

analytics data, where a low-performing webpage,

demonstrating ROI, understanding and taking action

app flow, or other digital experience is called out

on company data, and making quick decisions.

and suggestions for improvements are made.

According to our survey results, companies are unsure

These discussions can only make clear progress

of their ability to understand and take action on their

towards a better, more valuable customer experience if

company data, with only 66% and 61% of respondents

they are tested. To learn more about how optimization

stating they do so well or very well, respectively.

impacts businesses, we asked nearly 400 individuals

how they incorporate optimization into their workflows.

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

CHALLENGES FACING DIGITAL BUSINESSES

The following is a list of characteristics that many companies struggle with. How well
does your company do each of the following?

20%

37%

31%

to-use experiences across web, mobile, email,

12%

social, and myriad other channels are driving a

Deliver customized experiences to customer segments

26%

26%

15%

complex path of interactions between businesses

9%

and their customers. These experiences are

Make quick decisions

24%

35%

23%

constantly evolving, and need to be optimized.

15%

Take action on company data

Optimization empirically demonstrates the impact

27%

34%

31% 6%

Embrace the creative talents of my team

30%

35%

24%

10%

Very good

of changes to the customer experience, promotes a

Good

culture of learning and experimentation, and drives a

OK

Make data-driven decisions

31%

39%

21%

8%

Very bad

31%
10

20

30

35%
40

50

60

27% 6%
70

80

90

faster pace of decision-making within organizations as

part of an ongoing process of testing and learning.

Not good

Understand company data

ongoing changes in technology that steadily raise

the bar for digital experiences. Engaging, easy-

Maximize the ROI of my resources

24%

At the same time, businesses today are faced with

100

According to our survey results, companies

are uncertain of their ability to understand
and take action on company data, with
only 66% and 61% of respondents stating
they do so well or very well, respectively.

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

THE BENEFITS OF OPTIMIZATION

When asked how optimization impacts their business,

Do you agree or disagree with each of the following statements about optimization?

survey respondents overwhelmingly agreed that

Is valuable to the success of my marketing campaigns and/or product strategy

optimization has been a boon for them. 98% of

98%

respondents agreed that optimization is valuable to

Improves my understanding of my customers needs and desires

94% 6%

the success of their marketing and product strategy.

Helps me to deliver relevant content to my customers

93% 7%

Optimization also has qualitative and quantitative

Increases my conversion and engagement rates

benefits91% of those surveyed agree that optimization

91%

increases their conversion and engagement metrics, and

87% agreed that it improves the ROI of their marketing
improves their understanding of their customers.

to increase their budgets for optimization.

13%

Disagree

13%

80%

20%

Improves my teams speed to execute on ideas

76%

process, optimization also improves the speed to execute

the majority of those optimizing are planning

87%
87%

that 87% of survey respondents agree with. In terms of

both tangible and intangible, it makes sense that

Agree

Improves the creative output of my team

cultures of experimentation and learning, a statement

Since optimization drives clear business benefits,

12%

Has created a culture of learning and experimentation at my company

Optimization is seen as a catalyst for building company

creative output of the team for 80% of respondents.

88%
Allows me to take action on data quickly

campaigns. Furthermore, 94% agree that optimization

on ideas for 76% of those surveyed, and improves the

9%

Improves the ROI of my marketing campaigns

10

20

30

40

50

60

70

24%
80

90

100

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

BUDGET TRENDS FOR OPTIMIZATION

More than half (53%) of optimizers surveyed

How do you plan to change the amount of budget allocated to optimization in the next 12 months?

indicated that they plan to increase their budget

for optimization in the next 12 months. Managers

Percentage

and executives were also more likely to report a

planned increase, at 57% and 61%, respectively.
Additional budget dedicated to optimization will

53%

4%

43%

help businesses double down in an area where

they are already seeing benefits. When asked why,
respondents cited reasons like, As revenue and
budget allowance increases, we plan to put this money
back into better optimization, and The more we put

10

20

30

40

50

into optimization, the more benefits we will receive

60

70

80
Increase

post-optimization. They even cited optimization as a

90
Decrease

100
No change

competitive advantage: Optimization has become

even more crucial in todays market, so more budget
is required to make a position in the market.

that have ventured into the world of optimization have

Optimization dollars are being used to fund a broad

seen numerous concrete benefits. They are choosing to

array of resources, from people to software, in order

reinvest in optimization in order to achieve future gains.

to help these programs flourish. Optimizers are

actively hiring, purchasing testing software as well
complementary technologies, and preparing to run more
experiments in anticipation of higher traffic volumes.
Based on the findings from this research, companies

For companies that have yet to begin optimizing,

the proficiency gap will only continue to widen, as
teams who conduct optimization become more wellresourced and well-practiced in 2015 and beyond.

More than half (53%) of optimizers

surveyed indicate that they plan to increase
their budget for optimization in the next
12 months. Managers and executives
were also more likely to report a planned
increase, at 57% and 61%, respectively.

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

How Optimizers Build Their Teams

Human creativity, organization, and effort are
essential for the success of optimization. Optimization

How many members of your team are focused on optimization?

technologies do not automate this process, nor should

they. People are needed to generate ideas for what to
optimize, when, and to manage and communicate the

29%

29%

30%

process of experimentation. But just how many, and in

what teams? We asked our survey panelists to find out.

NUMBER OF TEAM MEMBERS

DEVOTED TO OPTIMIZATION

Successful optimization programs require human

23%

25%

20%

17%

15%

resources to develop a testing roadmap, maintain

a queue of ideas, set up and execute experiments,

10%

and conduct analysis on the results.

More than half of companies that conduct optimization

5%

2%

2%

(52% of survey respondents) have one or more

dedicated employees assigned to the process. These
program managers are often the central resource for
organizing and conducting optimization, but they also
draw on inputs from across their organization. Since

0%
One, part-time

One, full-time
Multiple, part-time

An agency runs our optimization

Multiple, full-time

None

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

good ideas for what to optimize, as

Which teams are INVOLVED in optimization at
your company?

well as knowledge and understanding

What team LEADS optimization at

your company?

of the customer, can come from many

departments and individuals, we wanted

Marketing

Marketing

32%

62%

to know which teams are involved

in optimization across the board.

E-commerce

Design

14%

53%
UI or Front-End Development

12%

UI or Front-End Development

40%

Product

12%
Product

36%

Engineering

9%
Engineering

Design

35%

8%
Executive Staff

E-commerce

5%

31%

3%
30%
Customer Service or Support

Customer Service or Support

23%

When asked which teams participate

in optimization, respondents indicated
that a wide array of technical and nontechnical teams are involved. 62% of
respondents stated that marketing is
involved with optimization, followed by
a cascading portion of programs that
involve design (53%), UI design and

Other
Sales

TEAMS INVOLVED WITH AND

LEADING OPTIMIZATION

frontend development (40%), product

(36%), engineering (35%), and an

3%

e-commerce team, if they have one (31%).

3%

However, when asked to indicate which

Sales

team is likely to lead optimization efforts

Executive Staff

20%

across the company, 32% incidated it is

their marketing team, making them the

Other

3%

clear leader for housing the capability.

10

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

How Optimizers Manage Their Process

An optimization strategy is iterative and continuous,
and requires a disciplined process and visionoriented goals to drive meaningful business results.

Approximately how often do you run A/B or multivariate tests to optimize your website, mobile app,
or other digital channels?

We asked optimizers about pacing, prioritization, and

inputs for their experiments to better understand

37%

how they think about and execute their strategy.

PAC E O F E X P E R I M E N TAT I O N

Digital channel owners that do optimize are

21%

running experiments frequently. The faster

the pace of experimentation, the more

15%

opportunities there are for learning, conversion

12%

and revenue gains within an organization.

9%
7%

More than half (58%) of those surveyed run

experiments on at least a weekly cadence.
Another quarter (27%) of respondents are
working on a monthly or semi-monthly pace,
with the remainder running experiments a
handful of times a year, or not at all.

Multiple Times per Week

Weekly

Multiple Times per Month

Monthly

Multiple Times per Year

We Dont Run Tests

11

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

INPUTS FOR OPTIMIZATION

Optimization does not occur in a vacuum. The

Which of the following inform your optimization strategy?

more inputs are taken into account when designing

Analytics

experiments and structuring a broader program, the

72%

stronger and more comprehensive the program will

Results from Previous Experiments

become, with greater opportunity for influence and

57%

collaboration with other teams within a company.

Customer Surveys

51%

When looking at the potential inputs for optimization,

User Research Studies

we found that 72% of those surveyed use analytics* to

50%

inform optimization, above all other tools and tactics.

A/B Testing Case Studies

This confirms our longstanding belief that optimization

44%

enables companies to turn data into action.

Support Tickets or Live Chat Feedback

38%

Many of the other inputs for optimization bring qualitative

Heat Maps

data into the optimization process alongside quantitative

28%

data, in the form of surveys (51%), research studies

Other

3%

(50%), support tickets (38%), or heat map tools (28%).

The use of inputs like previous experiment results or

10

20

30

40

50

60

70

industry case studies speaks to the level of procedural

discipline needed to optimize and grow organizational
knowledge. Optimization organizations must learn
from their own prior successes and failures, as well

to include a wide array of tools, from Google Analytics

as the learnings of their industry peers, if they are

or other dedicated analytics software, to BI data, to the

going to succeed in building a long-term program.

analytics reporting found in many optimization tools.

Regardless, quantitative information from any source

*The analytics umbrella could have been interpreted

is data that can be acted upon via optimization.

72% of those surveyed use

analytics* to inform optimization,
above all other tools and tactics.

12

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

PRIORITIZING WHAT TO OPTIMIZE

Prioritizing what to optimize is an essential process

How do you prioritize which digital interactions with your customers require optimization?

that organizations must develop to ensure they are

tracking towards their goals and making efficient

Potential to increase the value of each customer interaction, or increase overall revenue

65%

use of the resources they have for optimization.

The impact on ease of use and satisfaction for my customer

The top two prioritization techniques are very

58%

telling of why companies choose to optimize in the

Improvement to website or app stability or performance

first place: an equivalent focus on improving the

48%

value of each interaction (65% of those surveyed),

The inefficiencies that currently exist in my marketing and product funnels

as well as improving ease of use and satisfaction

33%

for the customer (58% of those surveyed).

The cost to acquire customers on various platforms

31%

Even better? Only 2% of optimizers do not

prioritize what they should optimize, which

We dont do this kind of prioritization

2%

shows a focus on discipline and process for the

vast majority (98%) of companies running A/B

Other

and multivariate tests and experiments.

Interestingly, the impact on ease of use and

2%
0

10

15

20

25

30

35

40

45

50

55

60

65

satisfaction for the customer is a prioritization

method that rang true more for individual contributor
level optimizers (68%) than for executives (51%).

thinking about or designing customer experiences.

This could be an indication that individuals closest

Optimization creates an experimental culture that

to the customer experience are more sensitive to

allows for ideas of individual contributors, inspired by

how customers feel about their engagement with

direct customer interaction, to surface and get tested.

the business. An executive might be more removed,

This ultimately drives a focus on customer needs and

focused on other metrics or stakeholders when

desires, and brings more creative ideas to the table.

Only 2% of optimizers do not

prioritize what they should optimize,
demonstrating a focus on discipline
and process amongst the vast majority
(98%) of companies testing.

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

13

WHAT OPTIMIZERS TEST

What do you test when you are optimizing?

When it comes down to what optimizers are actually

conducting experiments on, the responses from

User Experience Components (Onboarding Patterns, Search Functionality, Modals, etc.)

63%
Landing Pages

61%
Messaging and Copy

57%
Forms

our survey varied widely. Everything from user

experience components to landing pages, payment,
and social sharing functionality ranked for at least
a one-third of participants. The top three choices
were user experience components (63%), landing
pages (61%), and messaging and copy (57%),

54%

representing a wide array of experiment complexity.

Offers and Promotions

42%
After segmenting these responses out by conversion

Recommended Content Functionality

38%

goal, it became clear that different industries

Social Sharing Functionality

websites and mobile apps lend themselves to

37%

different combinations of elements to test.

Payment Functionality

37%
Engagement sites and apps were more focused on

Other

2%
0

testing social sharing (50% of respondents), B2B sites

10

15

20

25

30

35

40

45

50

55

60

with a lead goal were more likely to focus on optimizing

landing pages (74%) and forms (66%), and transactional
e-commerce websites were more likely to focus on
testing offers and promotions (51%) and payments (48%).

14

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

What do you test when you are optimizing? (Displayed by vertical)

User Experience Components (Onboarding Patterns, Search Functionality, Modals, etc.)

62%

67%
67%

Landing Pages

65%

74%

65%
Messaging and Copy

61%

Engagement sites and apps were

more focused on testing social sharing
(50% of respondents), B2B sites with
a lead goal were more likely to focus on
optimizing landing pages (74%) and forms
(66%), and transactional e-commerce
websites were more likely to focus on
testing their offers and promotions (51%)
and payments (48%).

60%

66%

Forms

61%

69%

53%
Offers and Promotions

46%
44%

51%

Recommended Content Functionality

39%
42%
42%
Social Sharing Functionality

32%

36%

50%

Payment Functionality

39%
38%
0

10

20

30

40
E-Commerce

48%

50
B2B

60
Media

70

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

MEASURING SUCCESS

How do you measure the success of your optimization program?

After the research, prioritization, and execution of

experiments has been completed, how do optimizers
sum the value of their optimization efforts?

Conversion Improvements from Experiments

58%
Although many teams tout the overall number of
Revenue Generated from Experiments

52%

experiments they execute in a given timeframe, it

actually was not the leading success metric that
optimizers use, at only 37% of respondents.

ROI (Combination of Conversion Improvement and Revenue)

49%
The leading metrics that teams use to measure their
Output or Velocity of Experiments

success are cumulative conversion improvements

37%

(58% of those surveyed), and revenue generated from

experiment improvements (52% of those surveyed).

Number of Team Members or Departments Involved

25%
Other

1%
0

10

15

20

25

30

35

40

45

50

55

15

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

Whats Next for Optimization

While progress in the field of
optimization has been substantial,
there is still room for growth. We believe
that optimization programs of the
future will continue to evolve, and that
they will each be as unique and diverse
as the companies they represent.

We are also optimistic about how optimization

We look forward to asking these questions of

will continue to impact the challenges that many

digital channel owners again in the future, to gauge

marketers, product owners, and engineers face,

developments in the field of optimization over time. In

and expect that optimization will give them greater

the meantime, share your requests and feedback on

confidence to face these challenges over time.

industry research you would like to see in the future.

With your help, we are looking forward to uncovering

A focus on hiring or partnering with the right talent

for optimization will be essential. At the same time,
organizations should commit to learning best-in-class
processes to support optimization while establishing
a culture that relies on data to make decisions in
pursuit of their ideal customer experience.

and understanding the changes in our industry.

16

17

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

Methodology and Audience Composition

The goal of the research underlying the
Optimization Benchmark report was
to determine the current practices and
attitudes towards optimization as a part
of modern marketing, growth, product, and
engineering organizations.

Survey respondent company size

8%

Third-party market research firm Dimensional

21%

Research invited individuals with responsibility for

digital channels to participate in an online survey.
A total of 417 qualified individuals participated. The

22%

respondents represent a diverse group of companies,

25 or Fewer

departments, levels, and digital channels.

25 - 250

27%

251 - 1,000
1,001 - 5,000

22%

5,000 or More

18

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

Survey respondent level

Survey respondent job function

49%

Optimization

10%

40%

Marketing

35%

Business Intelligence or Analytics

21%

34%

Product Management

43%
Manager

31%

Engineering

Individual Contributor
Executive
Consultant

26%

19%

Sales or Business Development

16%

People Manager

4%

Other
0

Weve seen the tremendous rise in growth as both a

function and a title within organizations that invest in
optimization. 49% of respondents chose Optimization
as a job function, which included overlap with disciplines
including business, marketing, and web titles.

10

15

20

25

30

35

40

45

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

The participants represent a wide array of digital

marketing channels and customer touchpoints. When

Survey respondent digital channels

asked to choose their primary channel for engaging

88%

Desktop website

79%

Mobile website

with their customer, desktop web emerged as the

primary focus (54%), with mobile web (15%) and mobile
apps (14%) and rounding out the top three channels.
*This distribution of primary digital channels

75%

Email

is meant to reflect the qualifications of the

respondents, not to reflect the market at large, since

70%

Social media

participants were invited to take the survey based

on their expertise with the above channels.

58%

Online advertising

51%

Mobile app

8%

Wearable devices
0

10

20

30

40

50

60

70

80

19

T H E O P T I M I Z A T I O N B E N C H M A R K | Q 2 / 2 0 1 5 | T H E R A C E T O F I N D T H E I D E A L C U S T O M E R E X P E R I E N C E

ABOUT OPTIMIZELY

Optimizely is the worlds leading optimization platform, providing A/B testing,

multivariate testing, and personalization for websites and iOS applications.
The platforms ease of use empowers organizations to conceive of and run
experiments that help them make better data-driven decisions. With targeting
and segmentation using powerful real-time data, Optimizely meets the diverse
needs of any business looking to deliver unique experiences to their visitors.

To learn more about Optimizely,

schedule a live demo today at

OPTIMIZELY.COM/DEMO

ABOUT THIS GUIDE

The Optimization Benchmark

WRITTEN BY

Shana Rusonis, Content Marketing Specialist

@srusonis
DESIGNED BY

Jules Forrest, Communication Designer

@jules_forrest
THANKS TO

Diane Hagglund of Dimensional Research, Andrew Gori, Jessie Becker, Jason

Gatoff, Heather Watkins, Ross Camp, Olga Antonenko Young, and Jessie Ren

20