WHITE PAPER

The Ultimate Guide to Office

365 Reporting:

Native Reports

www.cogmotive.com
www.cogmotive.com
 WHITE PAPER

CONTENTS
INTRODUCTION 3
WHY REPORTS ARE CRUCIAL TO OFFICE 365 5
NATIVE REPORTING SOLUTIONS 6
POWERSHELL 6
APIs 7
NATIVE REPORTS 7
THE ADMIN CENTER 8
AVAILABLE ADMIN ROLES 8
EXAMPLE OF THE OFFICE 365 PORTAL 10
THE ADMIN CENTER - OLD AND NEW 11
THE OFFICE 365 ADMIN CENTER MENU 12
GRANTING ACCESS TO REPORTING 13
GETTING WHAT YOU NEED IN THE ADMIN CENTER 15
NATIVE REPORTING CAPABILTIES BY ADMIN ROLE 16
REPORTS AVAILABLE WITHIN THE OFFICE 365 ADMIN CENTER 18
THE EVOLUTION OF NATIVE REPORTS 19
EXPLORING THE APPLICATION OF NATIVE REPORTING 23
EXAMPLE ONE: SKYPE FOR BUSINESS 23
USE CASE: ASSESSING USER ACTIVITY FOR MANAGERS 24
EXAMPLE TWO: LICENSING VS ACTIVE USAGE 28
USE CASE: LICENCE RECONCILIATION 29
CONCLUSION 31
REFERENCES 33

All images, screenshots, and the commentary surrounding them, are related to the
Office 365 portal as it appeared in February 2016, and are representative of the
portal at that point in time.

www.cogmotive.com 2
 WHITE PAPER

INTRODUCTION
Whether your organisation is newly migrated to Office 365, or made the transition
some time ago, you’re in good company. With benefits like speed, collaboration and
scalability, it is clear why cloud adoption is ever-increasing amongst small and large
organisations alike. In fact, IDC research predicts that by 2020, there will be no
‘clouds’ as we now know them, whether ‘public’ or ‘private’. It will not be a service to
choose or move to; it will be simply the new way that business is done, and how IT is
provisioned.

While IDC’s prediction is bold, it is not unreasonable. We all know that good IT can
mobilise and transform at a rapid pace - and four years is a long time in technology
terms. But perhaps more importantly, their claim also highlights an attitude shift
which is being propelled through cloud technology, and the data it can generate. In
the past IT has often been considered a marginalised function, conducting facilities
management in a way that is somewhat detached from the organisation itself. This is
no longer the case, IT is being pulled into the forefront of operations; it is becoming
central to ‘the new way business is done’. The reason for this? The potential for
business intelligence and data insight has enabled IT to become a key influencer for
critical decisions, spanning the entire organisation.

But what does this mean in real terms? Your cloud environment, including Office
365, is absolutely teeming with data. A truly unnerving amount. The term ‘Big Data’
is frequently bandied around in the abstract. This is partly because it can be difficult
to comprehend, but also because it is evasive, and often spewed out in an unusable
or unmanageable form, which is why you might be told to ‘harness’ or ‘capture’ its
power – like a wild, mysterious beast. Cut out the jargon, and it is simply an ever-
increasing volume of available information – visible or not, accessible or not…useful or
not, it’s there.

www.cogmotive.com 3
 WHITE PAPER

There is little value in data that cannot be understood, and equally, you can’t get
the most out of your Office 365 environment if you cannot interpret the data that
it provides. It is promoted as an efficient, agile, scalable platform, but if you can’t
measure these elements in specific, granular detail then how can you guarantee
that your investment has been returned, either financially or through the increased
performance that the platform promises. In order to achieve complete management,
you have to be able to see into your Office 365 environment, and understand as much
of its inner workings as possible. For that you need comprehensive, visible data - or
more specifically, reports.

www.cogmotive.com 4
 WHITE PAPER

WHY REPORTS ARE CRUCIAL TO OFFICE 365

No matter the context, the ultimate purpose of data is to enable understanding. In
order to achieve this, information needs to be processed from its raw, unintelligible
form and represented in a way that can be widely consumed and actioned. Of
everything that you need to know about Office 365 reporting that’s covered in this
white paper series, the first is just how integral reports are to understanding the
platform.

Once adopted, Office 365 effectively becomes your IT infrastructure. Your activity,
growth and operational metrics sit within it, so it is essential that these can be
accessed and reviewed, particularly as some of the traditional data sources, such as
event logs or desktop tools, are no longer available once you’re in the cloud. In Office
365 for Exchange Professionals, Tony Redmond, Paul Cunningham and Michael Van
Horenbeeck explain some of these challenges, ‘Microsoft does not expose IIS tracking
logs generated inside Office 365’, so you cannot get client connection information
from IIS (internet information services) logs, unlike on-premises solutions. He
explains:

“The same need to understand what you use exists in Office 365, but there’s a big
difference. Instead of having full control over all moving parts and thus being able
to retrieve any piece of information you might think is interesting, you consume a
service with limited external interfaces.”

For the average user, this reduction in external interfaces goes relatively unnoticed,
but as the individual tasked with explaining issues, fixing problems and managing the
administration of Office 365, it can feel like a loss of some control: that you can only
get what you’re given.

To provide a better understanding of the reporting solutions available for Office 365
overall, this white paper will explore native reporting within the platform, including its
limitations. The series will then continue by examining planned future development
of native reporting capabilities, and comparing this new functionality against
independent third-party solutions that allow comprehensive, business-critical data to
be extracted from the Office 365 platform.

www.cogmotive.com 5
 WHITE PAPER

NATIVE REPORTING SOLUTIONS

In-depth, intelligent reporting is one of best ways to achieve extensive visibility,
review operational efficiency, and gain value from your IT environment. Microsoft
offers two primary ways to gain this kind of information from Office 365:
PowerShell and native reports. There are also a number of APIs that you can use to
programmatically access this data, through tools such as PowerBI. An example of this
is the Reporting Web Service, however, this is being wound down and replaced with a
range of new APIs that were announced at Ignite 2015, and are not available at the
time of writing. We will consider current APIs briefly, and will look at the planned new
reporting functionality Microsoft has announced in the next paper in this series.

Before diving into native reports, we will briefly consider PowerShell and APIs as
reporting tools.

POWERSHELL
PowerShell is a task automation and configuration management framework, which
allows you to run scripts from the command line shell. PowerShell is extremely useful
for both on-premises and Office 365, and can retrieve extensive information about
many aspects of its environment, which often leads to the question: Why not just use
PowerShell for your reporting needs?

As a tool for regular, hassle-free reporting, PowerShell has some limitations. For
example, one of the most useful ways to analyse data is by looking at performance
historically, to identify changes or trends over time. This is something that PowerShell
is less able to do, as the cmdlets (a lightweight command that is used in the Windows
PowerShell environment) which retrieve the data from Office 365 often don’t provide
the granular level of information that is needed to achieve this.

PowerShell also requires the time and resources of a skilled user, or someone willing
to learn. As Tony Redmond highlights, it is ‘fantastic for querying data but is less
good at formatting it’. While it is possible to export data in CSV format, it takes
time to manipulate the data in Excel, especially if there is a large volume, which,
as Redmond points out, is just ‘not the same as creating a nicely crafted report.’
Finally, PowerShell’s performance can be compromised by throttling – a process by
which resource-intensive activities are regulated, slowed, or de-prioritised in order to
maintain overall system performance.

www.cogmotive.com 6
 WHITE PAPER

Redmond explains that PowerShell reports are ‘likely to be throttled by Office 365 if a
script attempts to process details of hundreds or thousands of mailboxes’. Due to its
all-round resource-demanding nature, PowerShell is not always the best solution for
reports. As Redmond observes, PowerShell is not a ‘substantial’ reporting tool, as the
user input is high, with the need to ‘format the recovered data’ and manipulate it to
be ‘more accessible and easily understood.’

APIS
There are a number of APIs available across the Microsoft Office 365 ecosystem that,
while not designed for reporting, can be used to infer usage information across Users,
Groups, OneDrive, SharePoint and Email. Recently, some of these APIs have been
brought together into the Microsoft Graph, a unified API endpoint, but others such as
the Office 365 Management APIs and the Reporting Web Services remain separate.
The Reporting Web Service allows developers to incorporate information on various
Office 365 activities: ‘email and spam, antivirus activity, compliance status and Lync
Online conferences’ into their own custom reporting applications and web portals.
These APIs are incredibly fast and powerful and can be used to really enrich your
reports with other pieces of relevant data. Unfortunately, the way you interact with
them and the format of the data returned is different with each API. A company
seeking to leverage this extra information will need to have developers on staff to
build customised solutions to consume and display this data. Often these APIs are
deprecated and replaced with newer versions, which means that any customised
solutions that have been written for the old API will need to be re-designed. This
would mean that, with any new changes or updates, a home-grown solution is likely
to break, and will need re-coding. As previously mentioned, an impending example
of this is Microsoft’s recent announcement that the Reporting Web Service will be
replaced by a new, yet un-disclosed native reporting API at some point in the future.

NATIVE REPORTS
If you don’t have the time or resources to use PowerShell, an alternative option is the
native reports that Office 365 offers. These consist of a selection of predefined reports
on popular areas and features, providing insight into activities within the platform. We
will conduct an in-depth analysis on the reporting feature’s past, present and future
functionality as we go on, but in order to use the native reports - first you have to
find them.

www.cogmotive.com 7
 WHITE PAPER

THE ADMIN CENTER

Native reports are located within the Admin Center. As many Office 365 business
users will be aware, the Admin Center is only visible and accessible to designated
IT administrators for your company. In smaller businesses, with Office 365 Small
Business and Office 365 Small Business Premium plans, one person is tasked with
managing all functions as a global admin. Whereas larger organisations, with Office
365 Enterprise, Office 365 Midsize Business, Office 365 Business Essentials or
Premium plans, have a role-based access control structure (RBAC). This means that
each admin is assigned a specific role, and has certain permissions pertaining to that
role.

AVAILABLE ADMIN ROLES

The following list explains the eight possible admin roles within Office 365 for these
plans:

Global admin: This permits access to all administrative features. Only global
admins can assign other admin roles. It is possible to have more
than one global admin, and the individual who signs up to buy
Office 365 becomes the global admin by default.

Billing admin: The role is able to make purchases, manage subscriptions and
support tickets, and monitor service health.

Password admin: This allows you to reset passwords and monitor service health.

Service admin: This role monitors health and manages any service requests.

User Management This admin manages user accounts (excluding some admin roles),
admin: user groups, and service requests. They can reset passwords, and
monitor service health.

Exchange admin: This permits administrative access to Exchange Online through the
Exchange Admin Center.

SharePoint admin: This role has full administrative access to SharePoint through the
SharePoint Online Admin Center.

Skype for Business This enables administrative access to Skype for Business through
admin: the Skype for Business admin center, and can perform almost any
task in Skype for Business Online.

www.cogmotive.com 8
 WHITE PAPER

Regardless of whether an organisation has one admin, or several, the center acts as a
space where all areas of the platform can be coordinated and controlled, such as the
management of users, passwords, licenses, domains, and much more.

All admins have access to role-specific data that could be considered sensitive or
confidential, such as employee or company details, which could include passwords,
or extremely sensitive documents. As such, limiting access to users with admin
rights ensures that Office 365 can be maintained and supported centrally to increase
security.

The issue with native reports in Office 365, is that only admins can access, view and
interact with them, therefore most members of the organisation do not have the
permissions to view them. This can be frustrating, as there is a wide range of reasons
why individuals (operating outside of the IT department) might have a reason to
generate a report from the platform, and this is not a task that they can complete
themselves.

Outside of IT administration, some examples of role-based needs are:

• Finance – Many organisations have a process whereby each department is charged

for their IT costs to provide a more accurate picture for the business overall of
where costs are incurred. To do this, finance staff need information on the quantity
and type of Office 365 licenses. A report that provides this quickly and easily would
save time and effort. If it could be filtered by department or other attribute, this
would streamline the process even further.

www.cogmotive.com 9
 WHITE PAPER

• HR – When large volumes of employees leave or join a company it can be hard

to verify whether the proper processes have been completed. Failure to complete
these processes has consequences. For example, the organisation may be paying
for licenses for users no longer employed, or may be exposing the company
to security vulnerabilities by not shutting down user access to the company’s
systems. A report showing new and deleted mailboxes or the types of devices that
have connected to Office 365 could be used to verify that an ex-employee hasn’t
got access to anything that they shouldn’t have.

• Sales – If team members are often in the field, a quick report on user activities like
the number of emails sent, or Skype for Business calls made, can work as a quick
provisional check that weekly targets are being met.

There are numerous use cases for reporting that extend beyond general IT monitoring
and maintenance, and can save time by providing immediate insight and act as
crucial business intelligence.

It is likely that you will know whether you are an admin for your organisation, but if
you are unsure, the process of verifying your level of access is simple. On entering
your Office 365 portal, take a look at the application tiles (an example of the Office
365 portal is shown below. Depending on your subscription, you may see some or all
of these tiles in your environment).

EXAMPLE OF THE OFFICE 365 PORTAL

www.cogmotive.com 10
 WHITE PAPER

THE ADMIN CENTER - OLD AND NEW

Microsoft are in the process of updating the Admin Center - the current version is
included above. This is being replaced at the moment, and some tenants have already
been upgraded to the new Office 365 Admin Portal (featured below). As you can see, it
has a new look, with a fresh interface and dashboards showing key information.
To access the reports in the new portal click the Reports icon in the Hamburger menu
on the left hand side

www.cogmotive.com 11
 WHITE PAPER

THE OFFICE 365 ADMIN CENTER MENU

If you are an admin, you will see the grey ‘Admin’ tile included amongst the applications.
When you click into it, you will see the drop-down menu, featured below. The reports
section is highlighted in pale blue. This is where the reports can be accessed in the
current version of the Admin Center.

www.cogmotive.com 12
 WHITE PAPER

GRANTING ACCESS TO REPORTING:

IMPLICATIONS AND CONSIDERATIONS

If you want to explore native reporting, and do not currently have access to the
Admin Center, then you will need to be granted admin rights, or request the data
from an existing admin. A concern with increasing the number of designated admins
is that with greater visibility comes great responsibility, and the additional capabilities
that come with admin access have to be taken into consideration.

Increasing the number of admins has the potential to make your environment more
vulnerable to ungoverned changes, users errors or mistakes, particularly if the
individual is not skilled or specialised in IT administration or management.

Here’s an example of what is best avoided: An organisation assigns global

admin rights to a member of the Finance team for reporting on licenses and

costs. The individual goes into the Office 365 portal and, after making a

few wrong clicks, accidentally changes a number of user subscriptions for

Dynamics CRM. The entire Sales team is now unable to get access to their

CRM system, leading to significant downtime.

These circumstances can cause user disruption affecting productivity, a potential loss
of earnings, and it also demands time and effort to rectify. For this reason, granting
more users admin rights, so they can get the information they need from Office 365
directly is not always the right answer.

www.cogmotive.com 13
 WHITE PAPER

Ultimately when assessing whether it’s appropriate to grant admin rights more widely,
there are several factors to be weighed:

• Whether the person has an operational need to access certain information, and
admin rights can give them that access;
• However, granting them admin rights also exposes the organisation to greater risk
because aside from accessing the information they need operationally, they can
also:
◦◦ View sensitive company or user information that they may not need
access to; AND
◦◦ Make changes to your IT environment that could affect other users, the
company’s security or system configuration.

Therefore, it’s important to consider whether the operational need for information is
proportionate to the risk of granting admin rights – or whether there is an alternative
solution that could meet the operational need without the same risks. Getting the
information from an admin is certainly an option, but this then has an operational
impact on the IT team’s time, which comes at a cost. The other alternative is a third
party tool that offers fine-grained access control to this information without allowing
changes to your environment. We will explore this option in more detail at a later
point in this white paper series.

www.cogmotive.com 14
 WHITE PAPER

GETTING WHAT YOU NEED IN THE ADMIN CENTER

Once inside the center, there are some restrictions that could prohibit your access to
reports. On their support page for service usage reporting, Microsoft explains that:

“The reports are separated into categories such as Mail, Devices, Lync, SharePoint,
OneDrive for Business, Auditing, Protection, Rules, and DLP (data loss prevention).
You may see different reports depending on what services your organization is
subscribed to.”

In addition to this, some reports are also disabled depending on the Office 365 plan
your organisation has selected. It is also possible to cross reference the available
reports against your plan, which can be found here. If you do not have access to a
certain report, you are likely to see something resembling the screenshot below.

‘You do not have permissions to see this report’: If you can access the admin center,
but don’t have permissions for a certain report, you are likely to see this message. The
report on ‘Team sites deployed’ is related to SharePoint, and therefore, only SharePoint
admins, or global admins are able to see the data.

www.cogmotive.com 15
 WHITE PAPER

Another more important aspect that limits the available reports is also explained in
these support pages:

”depending on your admin role, you will have different reporting capabilities”.

Reports are filtered and displayed depending on the admin role type, so for
example, a SharePoint admin cannot see what a global admin can, as they have a
tailored subset of reports visible to them, which are specific to their position. The
information on report access can be quite confusing, and the details on permissions
and capabilities are spread over a number of support pages, with no comprehensive
overview of exactly who can do or see what.

The available breakdown of permissions and capabilities is featured below for

reference, but it is not particularly informative as the different levels of permission
(view only, export, manage) are not provided. The support pages use the term
‘manage reporting’, but as we have seen – it is not that simple, as there are a number
of different access levels - which constitute as management?

NATIVE REPORTING CAPABILTIES BY ADMIN ROLE

The image below lists the admin roles and their corresponding reporting capabilities.
This information has been condensed from an article in the support page listing
‘Office 365 admin permissions by role’ for all areas of the platform.

Office 365 Admin Role Manage Reporting

Billing Admin No
Global Admin Yes
Password Admin No
Service Admin No
User Management Admin No
Exchange Admin Yes
SharePoint Admin Yes
Skype for Business Admin No

www.cogmotive.com 16
 WHITE PAPER

As you can see above, various restrictions are placed upon different roles. There
are admins who cannot conduct any reporting, some who have access to some
reports, as dictated by their role, and some which have ‘view-only’ restrictions
meaning that they cannot export data. Only the global admin has full access to all
reporting features.

In essence, if you are an admin that is able to use reports, and have a view to
generate role-specific information, you should be able to find what you need within
native reports. Role-based access control means that the scope for reporting
beyond your designated role is relatively restrictive.

In a scenario where an admin (or non-admin) needs information for a specific

business use case that they deem relevant, yet lies outside of their permission
level, their request is likely to reach a ‘bottleneck’ as it has to be passed through to
a couple of senior IT staff with global admin access - whose time is simultaneously
needed elsewhere.

For example: A member of the operations department wants to review the

project mailboxes for ongoing and completed projects to identify where

mailboxes may no longer be required and remove them. They can’t get to

this information themselves, and they can’t ask one of eight admins for a

quick report on new and deleted groups in Exchange Online because those

admins don’t have access to provide this information. So instead they have

to speak to the Global Admin, or Exchange Admin. And as anyone in IT

knows, there is never a quiet moment, and senior IT staff will have a long

list of critical tasks to complete before they can get to this request.

www.cogmotive.com 17
 WHITE PAPER

REPORTS AVAILABLE WITHIN THE OFFICE 365

ADMIN CENTER
This screenshot captures the older version of the eports section (as seen by a global
admin).

In the image above you will notice that some services seem to have numerous
reports, such as Skype for Business. In contrast, the section on Mail has far fewer
reports, with a particularly sparse collection.

Typically, the Office 365 features that are newer tend to have more extensive
reporting surrounding them, suggesting an acknowledgement of the importance
and growing need for more comprehensive native reporting. In the next section, we
explore how the native reports offering you see above came about, and how it we
might expect to see it evolve in the future.

www.cogmotive.com 18
 WHITE PAPER

The screenshot above shows the reports section in the new Admin Center. Rather
than listing all available reports like the previous version, the new home page has
a dashboard which enables the user to see snippets of information quickly. The
page displays high-level information, alongside clear graphs, both of which are an
immediate improvement to the simplicity of the interface.

www.cogmotive.com 19
 WHITE PAPER

THE EVOLUTION OF NATIVE REPORTS

In the past, the native reports section of the Admin Center has been extremely
sparse, providing static information on a limited number of areas. As a result, it is an
area of Office 365 that has often come under scrutiny from its users. Microsoft have
been very responsive to this feedback and have recently launched an updated Office
365 Reporting portal. Whilst these reports are a great improvement, there remain
limitations, which we will explore.

In an ‘Ask the Admin’ article for Petri IT Knowledge Base, Russell Smith, lists the
common complaints directed at Office 365 reporting:

“the pre-defined reports in the web portal is the quickest and easiest way to get
started with Office 365 reporting, but there are some significant limitations:

• Not all available information can be returned in the report results.

• Some pre-defined reports are only available using PowerShell.

• You can’t save the results.

• There are only two options for viewing: either as a graph or table.”

This article was published in 2015, and many of the points above are gradually being
resolved by Microsoft with each iteration it releases. For example, an ever-increasing
quantity and quality of reports enable more coverage of the available data, and more
of the reports now offer an export feature, so while you cannot save the report itself,
you can extract and format some data for your own reference.

www.cogmotive.com 20
 WHITE PAPER

The lack of detail is also regularly cited as an issue with native reporting. Tony
Redmond summarises this clearly:

“One thing that’s obvious from an examination of the data returned from Office 365 is
that the reporting service focuses on high-level aggregate data and doesn’t dive down
into the detail that is often necessary to make good management decisions.”

This link made between decision making and report detail is an interesting one. As
data can be represented and interpreted in a variety of ways, the reports that display
this information need to have the capability to manipulate data (filtering, sorting, drill
down, etc) to provide the necessary insights. For organisations looking to use this
data to inform making decisions, this could present a problem.

Achieving a high level understanding is useful, but if you cannot drill down into
subsets of data, or access the specific details you need, then it becomes difficult to
extract business intelligence that is useable.

Redmond’s observation above coincides with the trend observed at the beginning of
this paper about the transforming role of IT. Increasingly, IT is in a position to inform
and influence decisions, but only if it is equipped with the right information and ability
to generate insights that can inform those decisions.

User feedback on native reports is not something that has gone unnoticed by
Microsoft. In fact they have openly said they are working towards addressing these
functional and performance limitations.

www.cogmotive.com 21
 WHITE PAPER

As Microsoft rolls out improvements to the Admin Center to all tenants, the following
information has been displayed on the ‘Reports’ page.

According to the recent announcement from Microsoft, the new reports provide:

• Usage metrics across the entire organization, and user-level details.

• Consistent reporting periods for all user activity reports: last 7 days, last 30 days,
last 90 days, and 180 days. Although, as the reports are new, most will start out
with only 1 or 2 months of data available.

• No more reporting limits on the number of users you have in your organization.
You can page through all of the users or you can export the all of the user data to
a CSV file.

Progress has certainly been made towards improving native reporting, but there
remain limitations.

The changes to native reports will be considered more extensively in the next white
paper in this series. However it is important to highlight these additions here, in order
to clearly identify where the existing gaps are, and the direction that the improved
reports will take. It is also worth considering that when these reports are first
released, they will have a limited date span (at least initially) and will not offer the
ability to customise the date period or filter the data. The updates also suggest that
they will still prioritise high-level, organisation-wide data as their focus.

www.cogmotive.com 22
 WHITE PAPER

The impending changes to native reports will be considered more extensively in the
next white paper in this series. However it is important to highlight these additions
here, in order to clearly identify where the existing gaps are, and the direction that
the improved reports will take. It is also worth considering that (at least initially)
these reports will have a limited date span and the updates suggest that they will still
prioritise high-level, organisation-wide data as their focus.

As well as continuously implementing their own improvements. Reporting is also an

area where Microsoft have been sharply focused on building a community of ISVs
(independent software vendors), encouraging them to build upon the data available,
and develop products to fill the gaps in functionality - enabling increased performance
within Office 365 and providing their customer base with a range of more specialised
solutions.

Over the past year or so Microsoft has developed and announced a range of APIs
that make it easier for ISVs to develop solutions to meet the needs of their diverse
customer base. These are widely available, but as they require significant time and
effort, individual admins and organisations don’t usually have the time to make use of
them. ISVs form a crucial part of Microsoft’s business model as they continue to refine
and develop the Office 365 platform. As Tony Redmond observes,

“Microsoft is encouraging ISVs to use the management activity data to develop

solutions that generate activity reports, including suspicious or out-of-norm actions,
data visualization and analysis to aid planning and oversight for tenants.”

The importance and value of comprehensive reporting is communicated throughout

Microsoft’s activities, both through their own ongoing work to develop native
functionality, and their initiatives to encourage ISVs to help them fill the gaps. There
are a number of vendors offering enhanced Office 365 reporting, with features such
as advanced filtering, powerful custom report builders, scheduling and other export
capabilities - and importantly, non-admin access – these will be discussed later in this
series.

www.cogmotive.com 23
 WHITE PAPER

EXPLORING THE APPLICATION OF NATIVE REPORTING

Putting aside the limitations and planned improvements to native reports, if you need
to get information on a certain aspect of your Office 365 environment, native reports
are a good place to start your search.

In this next section we will look at a couple of popular native reports, and what
you can get out of them in their current format. We will then consider how these
reports can be used to meet a typical use case, where the limits are to their current
functionality.

EXAMPLE ONE: SKYPE FOR BUSINESS

As native reporting functionality goes, Skype for Business reporting is more
comprehensive than most. The screenshot on the following page is an example of the
Active Users report providing information about user activity (total number of users
per day using IM, audio, video, application sharing, file transfer, etc.)

As you can see on the left panel, there are options to group data according to these
variables. This report is useful if you need to find information on a specific element,
such as the number of active users for file transfer (perhaps you want to know why
your team aren’t using SharePoint to collaborate instead), you can quickly and easily
see that there were two instances on 2/1/2016. It is also possible to view trends
within the data as it now ranges back as far as 90 days.

www.cogmotive.com 24
 WHITE PAPER

One feature of native reports which is worth noting is the current limit on historical
data: it can only retrieve and store information for up to 90 days. For data that
extends beyond that time, Tony Redmond suggests,

“if you want to use the information retrieved from Office 365 for long-term analysis,
some arrangement usually has to be made to store the data in another repository”

This should be a consideration when assessing your organisation’s reporting needs

and how you will meet them: it will require significant, regular effort to manually
update your own records if you need to monitor historical trends beyond the 90 day
limit.

To explore this further, let’s look at a use case for the Skype for Business native
reports.

USE CASE: ASSESSING USER ACTIVITY FOR MANAGERS

The Global Head of Sales would like to see various usage reports for Skype for
Business so that he can gain better insight into why a remote team’s sales figures
decreased throughout the last quarter, while their performance statistics (number
of calls made, emails sent) seemed to stay the same. He has spoken to the team’s
management, but this provided little insight, and generated some disagreements as
to the specific cause. His concern is that the team is not following up with high quality
leads and prospects regularly.

In order to move forward, the Global Head thinks that information on the team’s
activities might give him the insight he needs. Using a combination of the Skype
for Business reports on ‘active users’ and ‘audio and video minutes’, he wants to
see a breakdown of the team’s activity, and check his sales team are active on the
application. Using the audio and video minutes report, he would like to review the
frequency of calls and ensure that the team are not making frequent, ‘dummy calls’
in order to hit their outreach targets. He wants to monitor the overall length of audio
sessions, to calculate an average call time. As he has a meeting scheduled with that
entire team this afternoon, he needs the report urgently. If the information proves to
be helpful, he plans to use the reports regularly to inform monthly performance reviews,
and biweekly pipeline meetings.

www.cogmotive.com 25
 WHITE PAPER

Above is an example of the report that the Global Head of Sales would like to access.

In order to get this data using native reports in Office 365:

1. He will have to submit the request to an admin. This step inevitably slows the pro-
cess down, and the IT admin has to take time out of their busy day to locate and
generate these reports.

2. As the reports can’t be shared or exported to send to a non-admin directly, they

are left with two options:

a. Take a static screenshot, which is quick, but removes any interactive

functionality; or
b. Export the data to a CSV file – which can only be done if the specific
report allows for this (not all of them do).

In this instance, the Active Users report can be exported in CSV, but the Audio and
Video Minutes report cannot.

www.cogmotive.com 26
 WHITE PAPER

3. It’s also important to isolate the data relating just to the remote sales team so
that the report can show the Global Head of Sales what he needs to see. In
general, native reports allow for reporting across the whole tenant (organisation),
or in only some limited cases, you can view data user by user – but this user level
data is shown for every user in the organisation. There is no filtering functionality
to isolate a sub-set of users (such as a team or department), even where a report
drills down to this level.

a. The Audio and Video Minutes report only provides aggregate data for
the whole organisation, so cannot be used to understand the remote team’s
activities.
b. However the Active Users report does provide user detail. Unfortunately,
there is no department field that can be used to quickly identify a group
of users.

So in order to get the information on average call times for the team of 30 staff, data
on User Activities will needs to be exported in CSV for the whole organisation, and
then data for each member of the team will need to be located in the CSV file one by
one, and extracted to put in a separate report to show data only for the remote sales
team. In all likelihood, IT won’t have time to do this and instead will provide the raw
file with data for all users in the organisation.

4. The raw data will then require formatting, or the admin can send it on in a form
that may be difficult to read making it challenging to gain the necessary insights.

www.cogmotive.com 27
 WHITE PAPER

These solutions are less than ideal, and there is no option to schedule or subscribe
to this report so that he can receive the data regularly. If he does need these reports
regularly, the Global Head of Sales will have to make ongoing requests, increasing the
workload for IT due to the limitations in native reporting functionality and the inability
to automate it.

As a result, the Global Head of Sales is sent a screenshot, providing a snapshot view
of the whole organisation’s activity, and a CSV file that, after a few attempts, he could
not interpret. Despite the best intentions the plan is abandoned, and the potential
business insight that the report could deliver is lost.

In order to be useful, and to be used within an organisation, reports must be

easy to generate and share: they should not be an unnecessary drain on time and
resources. Administrators with access to all of the reporting data for an organisation
are extremely valuable staff members, both in terms of company cost, and expertise.
How much of their time should be spent generating and interpreting these sorts of
reports to support a handful of business needs, which (arguably) fall outside of their
jurisdiction?

While it is clear that reporting has the potential to provide business intelligence, and
value, if it not easy and accessible it is likely to fall by the wayside - let’s not forget,
that IT Admins do have an actual job to do, and reporting is – at most - only one
small part of it.

www.cogmotive.com 28
 WHITE PAPER

EXAMPLE TWO: LICENSING VS ACTIVE USAGE

The Licensing Vs Active Usage report (shown below) allows you to see the active
usage of your organisation’s Office 365 services, over the past 30 days, and compares
this against the licenses you own. On the face of it, this is an extremely useful report
as it allows you to quickly see your license types, and ensure that you are getting
the most out of your Office 365 investment. It is clearly laid out, enabling you to
breakdown the information on licence type, and view the full details in the table below
the graph.

However, it is worth noting that this report has some quirks that may affect its
usefulness. In the screenshot above, you can see 35 SharePoint licences – a
considerably higher number than Exchange or Skype for Business. So high, in
fact, that you may want to look into why there is such a large discrepancy. After
investigation, it turns out that “Office 365 Extra File Storage” – the additional storage
you can purchase (by GB) to ensure your users can store all necessary files on
SharePoint - is counted as an additional user in this licensing report.

www.cogmotive.com 29
 WHITE PAPER

In the example above, there are 14 paid licences for Exchange, Skype and Yammer,
and 35 paid licences for SharePoint – a difference of 21 licences. In fact, this account
shows 20 GB of extra file storage was purchased, which is being counted as licences
- accounting for all but one of the 21 licence disparity between paid licences for other
services, and SharePoint.

USE CASE: LICENCE RECONCILIATION

A common use case for this report is within the Finance department, where it’s
common practice to validate what the company is being billed for by Microsoft against
what it’s actually using. A study by IDC (IDC FutureScape: Worldwide Software
Pricing and Licensing 2016 Predictions) predicts that ‘software license complexity will
indirectly cost organizations an average of 25% of their software license budgets in
2016’. This is a huge avoidable cost, simply due to ‘license complexity’, and even in
large organisations there should be a way to simplify software licensing and reconcile
charges.

If the Finance Manager is able to run a regular Licensing Vs Active Usage report, she
would be able to identify how many paid Office 365 licenses are available, how many
are being used, if there is an unused surplus, or whether more need to be purchased.
This would provide immediate visibility into licensing cost and ROI.

As with the previous use case, the barrier to entry is access. The Finance Manager
goes to IT and asks for Office 365 licensing information, and finds that she cannot
access this information herself. As she knows she needs to get this information
regularly, she asks if she is able to get access set up. Unfortunately because giving
access to reports also means she would be able to access information and make
changes that she shouldn’t, there were concerns that granting access to reports might
put the environment at risk of user error. Her request for admin rights was denied. As
a compromise, the global admin offered to send the data through to her each month.

The Licensing Vs Usage report cannot be exported, so the global admin has to send
screenshots to share this data, and it would then need to be manually copied into
another document such as Excel if it needs to be stored, compared and tracked over
time.

www.cogmotive.com 30
 WHITE PAPER

As this data is only stored for up to 30 days, it is important that the data is collected
and stored somewhere else before it’s lost – meaning the reports need to be sent
through regularly and consistently. Unfortunately, the IT admin who was sending
on report information became extremely busy and forgot the months of April and
September and sent through for the wrong date range in November. This meant that
the ability to confirm actual usage for particular months, and for the year overall, was
compromised.

Clearly this process could be streamlined by scheduling reports, meaning that each
month could be accounted for, with minimal effort. If it was automated in a way that
enabled it to land in the Finance Manager’s inbox every month, fully formatted, it
would allow her to validate the Microsoft charges and track this information over the
course of the year without involving IT. This would allow the report to provide sizeable
business value, with minimal effort from either the admin, or end user.

In many organisations, the finance department calculate the IT costs for each
department and the total is charged to the specific team’s budget. Currently, native
reports have some very limited and piecemeal filtering capabilities, allowing you to
add or remove limited defined variables.

Unfortunately, there is no functionality for filtering by department, which undermines

a lot of the potential value that this data could generate. More powerful filtering
functionality would allow for more extensive accounting and ROI analysis within Office
365, and would allow the user to drill down into each department to understand
usage and costs.

www.cogmotive.com 31
 WHITE PAPER

CONCLUSION
In their current state, native reports are a starting point for organisations looking
to begin or increase their reporting. It provides a solid range of popular reports,
retrieving a base set of data, and as they continue to improve, there is an increasing
level of functionality being built in. All of these elements provide reports that can act
as a foundation to understanding your infrastructure. That said, there are a handful of
functional and performance limitations which inhibit native reports from being a fully-
fledged solution for comprehensive reporting, particularly when this data could be
used for broader business intelligence purposes that go beyond IT.

The current limitations include:

• Restricted access to the data - which diminishes its value from an organisation-
wide perspective, obstructing its ability to inform critical business decisions.

• Reports cannot be scheduled, shared, or automated – and export capabilities are

more limited than you would expect.

• Date ranges are very limited (currently up to 180 days).

• The filtering capabilities are limited, do not drill into the data sufficiently, and you
can only filter by a small set of predefined fields that fall short of most practical
applications.

www.cogmotive.com 32
 WHITE PAPER

While notable progress has been made since their initial release, native reports are
likely to be too restrictive for some of the larger enterprise customer base. Microsoft
serves a vast and complex customer base, and meeting every need is always going to
be time-consuming and challenging. At present, there are probably many customers
- particularly in the small business sector - who get what they need from the current
native reports, and don’t expect to use Office 365 data as the rich source of business
intelligence that it has the potential to be.

On the other side, Office 365 customers with more complex data needs that require
more powerful and responsive reporting are not left out in the cold – there is a strong
and growing pool of ISVs offering third party solutions that fill the gaps in native
reporting to provide comprehensive reporting and analytics tools.

In the next white paper in this series, we will continue to explore reporting in Office
365 with an in-depth exploration of the future of native reporting, and alternative
methods for gaining comprehensive insight into your environment.

www.cogmotive.com 33
 WHITE PAPER

REFERENCES
https://www.capgemini.com/thought-leadership/big-fast-data-the-rise-of-insight-
driven-business

http://searchexchange.techtarget.com/feature/Office-365-reporting-strategies

Microsoft Support Articles:

View and download reports about service usage
Office 365 Reporting capabilities
Assigning Admin Roles

https://www.petri.com/3-ways-to-connect-to-the-office-365-reporting-service-from-
excel

http://windowsitpro.com/blog/reporting-office-365

http://www.idc.com/getdoc.jsp?containerId=259833

Office 365 for Exchange Professionals, September 2015 edition, written by Tony
Redmond with Paul Cunningham and Michael Van Horenbeeck

http://www.infoworld.com/article/2981909/cloud-computing/5-office-365-admin-
settings-you-must-get-right.html

www.cogmotive.com 34