Paper Title: Sustainable investing: The black box of environmental, social and governance (ESG)

ed
ratings

Subhash Abhayawansa, Associate Professor in Accounting and Finance, Swinburne University of

iew
Technology, Hawthorn, Australia (email: sabhayawansa@[Link])

Shailesh Tyagi, Founding & Managing Partner - Asia Pacific, Viridi Global.
(shailesh@[Link]).

ev
Accepted for publication in Journal of Wealth Management

r
er
pe
ot
tn
rin
ep
Pr

This preprint research paper has not been peer reviewed. Electronic copy available at: [Link]
 Paper Title: Sustainable investing: The black box of environmental, social and governance (ESG)

ed
ratings

Abstract

iew
Environmental, social and governance (ESG) investing is becoming mainstream, and the COVID-
19 pandemic has amplified the momentum. The interest in ESG investing has created greater
demand for ESG data, ratings and rankings together with a proliferation of agencies offering these
products which are unquestioningly relied on by investors, academics and regulators. Research
highlights that different ESG ratings and rankings produce significantly different assessments of

ev
the ESG performance of companies. In this paper, we examine the causes of the differences in the
ratings and ranking produced by different agencies. It is found that the divergences between raters
can be attributed to differences in defining ESG constructs (i.e., theorisation problem) and
methodological differences (i.e., commensurability problem). While users of ESG ratings and

r
rankings are advised to study the definitions and methodologies prior to their use, lack of
transparency about the data sources, weightings and methodologies makes it difficult to ensure that

er
companies’ true ESG performance is accounted for when making portfolio selection and
investment decisions. As a solution, we suggest that instead of attempting to compare and contrast
ratings and rankings of different agencies, investors should determine ESG constructs material to
pe
their investment strategy and match them with an ESG ratings/rankings product that closely
resemble those constructs.
Highlights:
1. There are significant divergences between the ratings and rankings provided by different
ot

ESG rating agencies.

2. Differences between various ESG ratings and rankings are caused by the differences in
defining ESG constructs (i.e., theorisation problem) and differences in methods applied for
tn

measuring ESG performance of companies (i.e., commensurability problem).

3. Agencies providing ESG rating and ranking are not transparent about what constitute ESG
performance and how ESG performance is measured, including information sources used.
rin

4. The theorisation, commensurability and transparency problems contribute to masking the

true ESG risks and performance of companies
ep

Keywords: Environmental, social and governance (ESG) ratings, socially responsible investing,
commensurability, sustainability reporting
JEL Code: P45
Pr

This preprint research paper has not been peer reviewed. Electronic copy available at: [Link]
 SUSTAINABLE INVESTING: THE BLACK BOX OF ENVIRONMENTAL, SOCIAL, AND

ed
GOVERNANCE (ESG) RATINGS

Overview

iew
Sustainable investing is an approach to investing that considers environmental, social and
governance (ESG) factors in portfolio construction and management. Once the domain of socially
responsible investors trying to exclude sustainability laggards from their portfolios, sustainability
investing is no longer a niche approach. The assets under management using sustainable investing
strategies1 surpassed US$ 30.7 trillion (or around 9% of global debt and equity) in the five major
markets of Europe, the United States, Canada, Japan, and Australia and New Zealand at the

ev
beginning of 2018 (Global Sustainable Investment Alliance, 2018). In the two years to 2020,
professionally managed funds adopting sustainable investing strategies in the U.S alone increased
by 42% to US$17.1 trillion (US SIF, 2020). The inflows to ESG investing funds have continued to

r
grow in 2020 with the full-year amount for 2019 being achieved by the middle of the year2 (Hale,
2020). A 2019 survey of more than 550 members of the CFA Institute spread out globally shows

er
that more than half of the quantitative analysts or portfolio managers incorporate environmental
and social factors and two-thirds incorporate governance factors into their investment analysis
(Singh and Peters, 2019).
pe
The rise in sustainable investing has given birth to a new industry that trades in ESG data
and ratings on companies, funds dedicated to rated companies and ESG index providers. Hawley
(2017) notes that there are more than 600 products from over 150 organisations providing ESG
data, ratings and rankings, with MSCI Inc and Sustainalytics being the leading players. The
industry is dynamic with new ratings appearing and disappearing, and rating organisations merging
ot

and realigning at a rapid pace (Sadowski et al., 2010a). ESG ratings are used by academics, for
instance, to understand how the financial and market performance of sustainable companies differs
from others and what drives those differences. ESG ratings as well as academic research based on
tn

them can potentially influence regulators, impacting policies and regulations relating to the
disclosure and use of ESG data3. The unquestioning reliance by various parties, including
investors, academics and regulators, on ESG ratings makes it important to understand whether ESG
rin

1
Sustainable assets under management include assets that are managed using one of the following seven strategies: (1)
negative/exclusionary screening; (2) positive/best-in-class screening; (3) norm-based screening; (4) ESG integration;
(5) Sustainability themed investing; (6) impact/community investing; and (7) corporate engagement and shareholder
action (see, for more details, Global Sustainable Investment Alliance, 2018).
ep

2
Flows are estimated for 315 sustainable open-end and exchange-traded funds available to U.S. investors which
include equity, fixed-income, allocation, and alternatives funds that have an ESG, impact, or sustainable sector focus
(see Hale, 2020).
3
For example, the U.S. Department of Labor (2020) new rule relating to the consideration of Financial Factors in
Selecting Plan Investments prohibits Employee Retirement Income Security Act (ERISA) plan fiduciaries from
Pr

choosing investments to promote environmental, social, and other public policy goals unrelated to the interests of plan
participants and beneficiaries. Also, recommendations of the SEC Investor Advisory Committee (2020) relating to
ESG disclosure show that the growth in ESG data, ratings and ranking providers has been taken into consideration in
the policy discussion relating regulating ESG reporting.
3

This preprint research paper has not been peer reviewed. Electronic copy available at: [Link]
 ratings of companies reflect true ESG performance of those companies and whether different ESG

ed
ratings converge. The purpose of this paper is to identify the causes of the differences in the ratings
and ranking produced by different agencies.

All ESG ratings are not made equal

iew
Several studies and media articles point out that ESG ratings produced by different agencies
show significant divergence (e.g., Chatterji et al., 2016; Yont et al., 2018). Allen (2018) shows that
in September 2018, FTSE placed Tesla last among the global automotive companies on ESG
performance while MSCI placed it at the top and Sustainalytics' placed it somewhere in the middle.
Semenova and Hassel (2015) explored the convergent validity of the environmental ratings of

ev
MSCI (formerly known as Kinder, Lydenberg, and Domini Research & Analytics; KLD),
Thomson Reuters’ ASSET4 (Now Refinitiv) and Global Engagement Services and found that
while the ratings have common dimensions, on aggregate, they do not converge. Dorfleitner et al.
(2015) compared individual environmental, social, governance and economic scores as well as the

r
aggregate ESG scores of three rating products: Thomson Reuters’ ASSET4, MSCI/KLD ratings
and ESG data set of Bloomberg. Using a subsample containing companies covered by all three
er
rating providers between 2002 and 2012 they show that correlations between ASSET4 and
Bloomberg were as high as 0.62 for the aggregate score and varied between 0.47 and 0.60 for
individual dimensions, whereas MSCI/KLD ratings showed little resemblance to the other two
pe
ratings (correlation varying between 0.05 and 0.39). Similarly, Chatterji et al. (2016) conclude that
the six ratings they compared (i.e., MSCI/KLD, ASSET4, Calvert, FTSE4Good, DJSI, and
Innovest) exhibit low convergence in their assessments of ESG factors. Futher, Hawley (2017)
showed that the correlation between the rankings based on the MSCI/KLD and Fortune Magazine’s
‘Best 100 Firms’ is merely 14%.
ot

Why do various ESG ratings differ?

tn

Our review of the literature shows that definitional and methodological differences
contribute to the low convergence between different ESG ratings. Disagreements about the
definitions of ESG factors, their composition and the weightings for each of the E, S ad G factors
are known as the theorisation problem, and having a variety of approaches or methodologies to
rin

measure the ESG factors (even if definitions are agreed upon) is called the commensurability
problem (Hawley, 2017). The theorisation problem originates in the different views held by rating
agencies on the types of ESG factors considered financially material and the degree to which they
are considered material (i.e., the differences in their materiality maps). Dorfleitner et al. (2015,
ep

p.256) identify the theorisation problem in their comparison of the compositions of ASSET4,
MSCI/KLD and Bloomberg ESG ratings. As an example of the theorisation problem, they
highlight that “animal testing is solely covered by ASSET4, whereas the question of whether a
company’s practice complies with environmental regulations is only considered by Bloomberg and
Pr

KLD”. Chatterji et al. (2016), upon comparing social ratings from six ESG raters find differences
between ratings even when adjusted for the explicit differences in the definitions of ESG factors.

This preprint research paper has not been peer reviewed. Electronic copy available at: [Link]
 Thus, Chatterji et al. (2016) demonstrate that the commensurability problem is independent of the

ed
theorisation problem.
The commensurability problem relates to the differences in how the individual ESG
dimensions are measured, especially relating to the level of detail. One aspect of the
commensurability problem relates to the differences in the way financially material ESG factors

iew
are measured using indicators. Dorfleitner et al. (2015) explain that the three ratings they examined
broadly cover the same aspects within the social dimension, and indicators within it account for
approximately half of the total data points/binary indicators. However, they note that the number of
indicators dedicated to particular aspects within the social dimension, such as health and safety,
differed significantly between the three ratings. The extent to which different raters conflate ESG

ev
impacts (which indicate ESG risks) and ESG performance contributes to low commensurability
(Mattingly and Berman, 2006), and constitutes the other aspect of the commensurability problem.
Commenting on the environmental dimension of ESG, which was the subject of their study,
Mattingly and Berman (2006, p.250) claim that “industry-specific risk drives EP [environmental

r
performance] and that performance and risk are different constructs to be clearly separated”.
Notwithstanding, Dorfleitner et al. (2015) found that ESG risk measures have the lowest level of
correlation between different raters.
er
Finally, the commensurability problem also relates to the differences in the information
sources used in developing the ratings. The information sources vary from companies’
pe
sustainability reports and other publicly disclosed information to surveys and interviews with the
company and independent information channels. Sadowski et al. (2010b) state that about one-half
of the 120 raters they examined rely only on public information sources while the rest use either
corporate self-disclosure alone or self-disclosure combined with public information sources. The
ot

use of information disclosed publicly by companies can bias the ratings in favour of larger
companies (as larger companies provide more disclosures than smaller companies after controlling
for their level of corporate social performance) and companies domiciled in geographical regions
tn

such as Europe where there are greater regulations and relatively mature traditions of sustainability
reporting. Similarly, Sadowski et al. (2010b) identify a bias towards companies responding to
information requests.
Moreover, the ratings of those agencies relying on publicly disclosed information, which is
rin

often based on annual reporting cycles, may not reflect the same period as that of the agencies
using more timely information sources. Then, there is the problem of reliability and comparability
of information disclosed in sustainability reports, as there are neither commonly accepted standards
for sustainability reporting nor a requirement to have the reports audited. Also, depending on the
ep

sustainability reporting guidelines adopted, companies might either disclose ESG information with
the investors in mind (e.g., if Sustainability Accounting Standards Board guidelines are adopted) or
take in to account a multiplicity of stakeholders (e.g., if Global Reporting Initiative Standards are
applied). Hence, the same data source can provide different perspectives about different
Pr

companies, making comparisons of ESG performance and risks across companies difficult.

This preprint research paper has not been peer reviewed. Electronic copy available at: [Link]
 Differences between ESG ratings and actual ESG performance

ed
One can identify several limitations in the generally adopted rating methodologies that
contribute to a gap between the perceived (based on ESG ratings) and true ESG performance and
risks of companies. As the scores on various indicators are added together to generate an overall

iew
ESG rating and ratings on each of the three factors (i.e., E, S and G), trade-offs between indicators
are inevitable (Hawley, 2017). For instance, a company that scores exceptionally on the diversity
aspect might, nonetheless, perform poorly on board independence for having directors with longer
tenures, smaller boards and a smaller share of independent directors (Yont et al., 2018). Most
rating methodologies are likely to assign these two indicators similar weightings and combine
them, although low board independence might constitute a greater risk than the absence of a

ev
gender-diverse board. Combining metrics masks the important differences between companies in
relation to performance and risk characteristics by aggregating empirically and conceptually
distinct ESG constructs (Mattingly and Berman, 2006).

r
Similarly, most raters fail to distinguish between disclosure (e.g., carbon emissions) and
performance (e.g., actions to lower emissions) and disregard the need to weigh them differently

er
(Hawley, 2017). These problems are aggravated when multiple aspects relating to a particular ESG
dimension are combined in one indicator (in contrast to one dimension) by one rater while they are
kept separate by another (Dorfleitner et al., 2015). Moreover, raters are criticised for not
pe
adequately focusing on material issues and trying to compare companies across industries which
differ on material ESG issues (Sadowski et al., 2010a). Related to determining materiality is the
time period over which given aspects of E, S or G factors are considered material. For example,
environmental factors such as loss of biodiversity or stranded assets resulting from climate change
regulation might only be material in the long-term, whereas other transitional risks of climate
ot

change or release of toxic waste might be material in the short term, despite the time horizons over
which the materiality of a given E, S, or G aspect differing between industries. Rating agencies do
not often disclose the time-horizons used for determining materiality, thus, denying users of critical
tn

information to understand companies’ true ESG risks and ESG performance. The result of these
limitations is ESG ratings that are incapable of representing organisational realities. Allen (2018)
argues that “Just because you can measure this stuff doesn't mean that you necessarily should.”
rin

Exercise caution

While the quality and comparability of ESG data and ratings remain hotly contested,
investors are cautioned against over-relying on, even, top-level ESG scores (Chatterji et al., 2016;
ep

Yont et al., 2018). At the very least, the users of ESG data and ratings should understand what the
methodology of the raters they choose actually measures. Unfortunately, this too is problematic as
most providers of ESG ratings and formulators of ESG rankings are not transparent, or only
transparent to a degree about their data sources, weighting and methodologies (Hawley, 2017). For
Pr

instance, some raters disclose categories and weightings, but not many explain how these
categories and weightings were determined or the specific criteria applied within the categories

This preprint research paper has not been peer reviewed. Electronic copy available at: [Link]
 (Sadowski et al., 2010a). The transparency problem is unlikely to go away as the rating

ed
methodologies (e.g., ESG indicators and weightings), data sources (e.g., interview participants) and
measurement instruments (e.g., questionnaires) are proprietary and considered intellectual property
of the raters. The usefulness of ESG ratings is further complicated by the potential conflicts of
interest of some rating and ranking agencies due to having connections with the companies they

iew
rate or produce indices based on their ratings.

Concluding remarks

The use of ESG ratings is tipped to increase with the market starting to price company
specific ESG risks and integrate ESG considerations alongside traditional financial analysis.

ev
Together with this growth, regulations on ESG disclosure, ESG ratings and rankings and use of
ESG factors in the investment processes are likely to follow (see, for example, Commission, 2020;
U.S. Department of Labor, 2020). The theorisation, commensurability and transparency problems
associated with ESG ratings are unlikely to disappear anytime soon. Thus, the only safeguard for

r
investors is educating themselves on the types and amount of ESG data, ratings and rankings
available, and the strengths and limitations of various methodologies and nuanced processes used
er
by the raters (LaBella et al., 2019). We do not claim that ESG ratings are unusable: to the contrary,
they can be valuable if used with caution and as one of the inputs. The best way to embed ESG
considerations within the investment process is by first determining what ESG constructs are
pe
material for the investor and then identifying an ESG rating that includes measures that closely
resemble those constructs.
ot
tn
rin
ep
Pr

This preprint research paper has not been peer reviewed. Electronic copy available at: [Link]
 References

ed
Allen, K. (2018), "Lies, damned lies and ESG rating methodologies", Financial Times.
Chatterji, A. K., Durand, R., Levine, D. I. and Touboul, S. (2016), "Do ratings of firms converge?
Implications for managers, investors and strategy researchers", Strategic Management
Journal, Vol. 37 No. 8, pp. 1597-1614.

iew
Commission, S. a. E. (2020), Modernization of Regulation S-K Items 101, 103, and 105 (U.S.
Securities and Exchange Commission: Washington D.C).
Dorfleitner, G., Halbritter, G. and Nguyen, M. (2015), "Measuring the level and risk of corporate
responsibility – An empirical comparison of different ESG rating approaches", Journal of
Asset Management, Vol. 16 No. 7, pp. 450-466.
Global Sustainable Investment Alliance (2018), 2018 Global Sustainable Investment Review

ev
(Global Sustainable Investment Alliance).
Hale, J. (2020), "Sustainable Funds Continue to Rake in Assets During the Second Quarter",
available at: [Link]
rake-in-assets-during-the-second-quarter (accessed 2020 December 12).

r
Hawley, J. (2017), ESG Ratings and Rankings: All over the Map. What Does it Mean? (Truevalue
Labs).
LaBella, M. J., Sullivan, L., Russell, J. and Novikov, D. (2019), "The Devil is in the Details: The
er
Divergence in ESG Data and Implications for Responsible Investing", QS Investors.
Mattingly, J. E. and Berman, S. L. (2006), "Measurement of Corporate Social Action: Discovering
Taxonomy in the Kinder Lydenburg Domini Ratings Data", Business & Society, Vol. 45
pe
No. 1, pp. 20-46.
Sadowski, M., Whitaker, K. and Buckingham, F. (2010a), Rate the Raters: Phase One - Look Back
and Current State (SustainAbility: New York).
Sadowski, M., Whitaker, K. and Buckingham, F. (2010b), Rate the Raters: Phase Two - Taking
Inventory of the Ratings Universe (SustainAbility: New York).
SEC Investor Advisory Committee (2020), Recommendation of the SEC Investor Advisory
ot

Committee Relating to ESG Disclosure (U.S. Securities and Exchange Commission:

Washington D.C).
Semenova, N. and Hassel, L. G. (2015), "On the Validity of Environmental Performance Metrics",
tn

Journal of Business Ethics, Vol. 132 No. 2, pp. 249-258.

Singh, m. and Peters, S. (2019), The Case for Quarterly and Environmental, Social, and
Governance Reporting (CFA Institute: New York).
U.S. Department of Labor (2020), Financial Factors in Selecting Plan Investments (US
Department of Labour: Washington D.C).
rin

US SIF (2020), 2020 Report On US Sustainable, Responsible And Impact Investing Trends (US
SIF The Forum for Sustainable and Responsible Investment: Washington D.C).
Yont, C., Jamie, A. and Zhou, M. (2018), Hard decisions: Asia faces tough choices in CG reform
(CLSA and Asian Corporate Governance Association Hong Kong).
ep
Pr

This preprint research paper has not been peer reviewed. Electronic copy available at: [Link]