Chapter 1
jects) and individual preferences (on the side of subjects) of which people
are not even aware, but that are manifested in their appreciative behavior.
Patterns of explanation, or what do we do when we talk
about aesthetics
Computational analysis may enable us to extract patterns and formal struc-
tures, but it does not provide an understanding of how such patterns affect
human perception, emotion, and cognition. Patterns considered in isolation
from human meaning are ultimately empty. Art historian Michael Baxandall
(in his 1985 book Patterns of Intention) has persuasively described the es-
sence of the critical language that we use when talking about any artwork or
cultural product. For Baxandall, any discourse that we create is neither a
merely factual description of features, nor a subjective report of a person's
reactions, but consists in highlighting the relationship between the object
and human responses (the meaning they give and the aesthetic reaction they
manifest). This relationship is further mediated by an understanding of the
object’s symbolic and cultural meanings. A critic, so to speak, tells the reader
what kind of reaction is expected (or would have been expected for people
in the past) in front of a specific object. Expressed in the terms of the map
from the previous section, this would mean drawing a connection between
the description of the object and the description of the corresponding sub-
jective reactions.
15
ARTIFICIAL AESTHETICS
Following Baxandall, cultural and critical explanations in art are not mere
descriptions or classifications: they are “primarily a representation of our
thoughts about it” (Patterns of intention, p. 10). What we describe is a “partially
interpretative description”: “one does not describe pictures, but our thoughts
of having seen pictures” or at least hypotheses on those thoughts. The effica-
cy of a critic’s argumentation lies in his or her ability to compellingly per-
suade the reader that the artifact elicits the kind of reactions and thoughts
that the critic is claiming to make explicit. Moreover, the critic’s use of words
and concepts, while sharpening the perception of an object, at the same time
deepens the meaning of the concept itself: “concepts and object reciprocally
sharpen each other” (Patterns of intention, p 34). For example, if we describe
The Scream by Edvard Munch (1893) as inspiring a sense of dread, then the
very concept of dread as an aesthetic notion will be made richer by using
Munch’s famous painting as a case in point.
Artifacts in synthetic media (images, songs, texts) are generated by networks
that have already been trained on large databases of similar, preexisting arti-
facts, as in following diagram:
However, if the generated content is expected to have aesthetic value, the
generative networks must take into account not only the formal dimension
(how artifacts are made), but also their corresponding subjective interpreta-
tion and reaction, including people’s aesthetic preferences. Otherwise, we
would be able to generate infinite variations of patterns, but not have a clue
on how they relate to our appreciation. If description of patterns without
meaning is empty, as we said, generation of patterns without human inter-
pretation is blind.
16
Chapter 1
In AI-media generation today, humans operate generative networks by se-
lecting, adjusting, and tweaking the process to obtain a desired result. This
result also depends on humans following their own aesthetic sensibility: for
example, a music expert had to evaluate and filter the different generations
of Schubert's "Unfinished Symphony”. Only algorithmic analysis of subjective
responses (“Studying subjects”) would allow a progressive automation of this
evaluative step.
Moreover, “Studying subjects” would involve both individual and collective
reactions. The latter involves analyzing historically sedimented responses
towards cultural objects. Ideally, an AI capable of creating meaningful art and
design would take into account the history of what exists, not only to extrap-
olate patterns from the artifacts, but also to interpret their collective recep-
tion, that is, how people over time have reacted to these artifacts. Thus, using
an AI to generate new cultural artifacts (and assist human creators) will re-
quire using an AI for cultural analysis. This would necessitate bringing artifi-
cial aesthetics into contact with the various fields that deal with this issue:
philosophical aesthetics, art history, psychology of art, anthropology and so-
ciology of culture, and so on. Granted, new technical developments can gen-
erate entirely new kinds of artifacts that need not resemble the cultural pro-
duction of the past. However, if we want to better grasp how these artifacts
could affect people, an understanding of how we typically react and give
meaning to aesthetic objects could save us from wandering in the dark. The
near future may hold entirely new aesthetic artifacts, but it is unlikely to hold
an entirely new human nature.
17
ARTIFICIAL AESTHETICS
Computation and psychology
Aesthetic phenomena involve a complex relationship between all human
faculties, from low-level perceptual mechanisms to higher-level affective and
cognitive processes. It is no coincidence that by the end of the 19th century
Gustav Fechner, the father of experimental psychology, had already identified
aesthetics as the most critical challenge for his new methods in scientific
psychology.10 In fact, researching how people react and behave during an
aesthetic experience (the domain we defined as “Studying people”) has long
been a tradition in so-called psychological experimental or empirical aes-
thetics. Fechner investigated, for example, whether people prefer shapes that
follow the golden ratio rule. While Fechner’s findings seem to confirm the
rule, later studies failed to replicate the same results. This line of research
continued steadily for almost a century: for instance, Birkhoff’s Aesthetic Mea-
sure11 tried to capture in a quantifiable formula the optimal aesthetic rela-
tionship between a shape’s complexity and order: high order with high com-
plexity would correlate, according to him, to a higher aesthetic pleasure. In
the 1970s, Daniel Berlyne’s new experimental aesthetics12 introduced moti-
vational factors as a key component in aesthetic pleasure and appreciation:
aesthetic value is not only a function of an object’s features, but also of the
hedonic tone of a subject, namely his or her level of interest and stimulation.
His inverted-U relationship between complexity and enjoyment suggests an
optimal middle point between too little and too much complexity in a stimu-
lus. This has been empirically investigated as well, albeit with divergent re-
sults. At the turn of the new century, researchers felt that it was necessary to
move from aseptic psychophysical experiments based on simple abstract
patterns to observing how people react in front of real artworks, artifacts, or
natural entities.13 Neuropsychological approaches have recently become
popular in this field, extending their focus to issues such as creativity and the
mechanisms of reception and interpretation in specific art forms (visual art-
works, music, movies, literature).
18
Chapter 1
A researcher in this field typically conducts experiments with small groups of
people under carefully controlled conditions, using statistical techniques to
analyze the collected data. For example, in many experiments in visual aes-
thetics, a group is shown a particular set of images (the dataset can be pre-
existing or created specially for the experiment), and people are asked to ex-
press their preferences in some way, such as rating all images on a numerical
scale. Decades of investigation in experimental aesthetics led to many find-
ings. For example, psychologists showed that more prolonged exposure to a
stimulus leads to a growing familiarity with the object, inducing a preference
for it as well as for prototypes in the object’s category. That is, we like what is
more typical, and that overall fluency, the ease in processing an experience,
correlates with aesthetic preference. Furthermore, research findings showed
a preference for symmetry in facial features, a preference for smooth and
curved shapes over angular ones; specific preferences for natural landscapes
over man-made scenes, and for architectural scenes with naturalistic aes-
thetics.14 Numerous studies have tested the classical rules of harmony, bal-
ance, and “good composition,” such as the “rule of thirds” or the principles de-
scribed by Gestalt-theory (which were first applied to art by Rudolf Arnheim
in his 1954 classic work, Art and Visual Perception).
We should note that these experiments often use college students as their
test subjects. Their aesthetic judgment could mirror a specific taste, without
being representative of the judgments of artists, designers, or critics. Differ-
ent studies have repeatedly confirmed a significant difference between ex-
perts and non-experts in aesthetic evaluation. It should be noted, moreover,
that most of the research does not point to conclusive findings, showing in-
stead that aesthetic preference depends on numerous underlying variables,
like context and subjective attitudes. One example of a contextual factor
would be the verbal description of an artwork: titles change our appreciation
of paintings and how we look at them.15 The order of presentation (which
object do we see first? Which next?), spatial disposition (which object is on
the left? Which on the right?) and juxtaposition (do we compare similar or
19
ARTIFICIAL AESTHETICS
_____________________________________
Big data does not require us to assume a
universal human aesthetic subject
very different objects?) also affects how people judge objects.16 The envi-
ronment also influences how we evaluate and appreciate art.
For instance, our reception of an artwork may differ depending on whether
we look at it in a typical “white cube” space or in a more informal context.
Variations exist depending on the observer's characteristics: factors such as
one's emotional state and level of arousal, expertise, personality traits and
culture all contribute to the aesthetic experience and judgment. Instead of
looking for generic universal rules – like the golden ratio, “unity in multiplici-
ty,” and Berlyne’s inverted-U model – experimental research investigates very
subtle mechanisms while considering contextual, personal, and culturally
specific factors. In summary, the field has generated and tested many inter-
esting theories to account for human aesthetic experiences, demonstrating at
the same time that none of them seem to hold universally.17
There are two crucial differences between today’s computational methods
and traditional experimental aesthetics. First, experimental aesthetics mostly
focuses on subjects, while artificial aesthetics focuses on objects. Further-
more, experimental aesthetics uses specially selected and highly controlled
stimuli, while artificial aesthetics uses “big data” from real life human behav-
ior, which is often collected through digital platforms.
While experimental aesthetics usually produces stimuli in controlled settings
and looks at people’s responses, computational methods make use of large,
available datasets of expressed preferences, like Photo.net or Dpchallenge.-
com (used for computation studies in the late 2000s), allowing researchers to
explore how people give their “likes” on social platforms. In other cases, they
capture and measure people’s actual consumer behavior on online platforms,
20
Chapter 1
like streaming services for music and film, with the aim of inferring features
from the most popular artifacts.
In experimental aesthetics, a subject-focused approach emphasizes the
analysis of so-called “dependent variables”. These include the controlled re-
sponses of subjects, measured through judgments on well-calibrated scales,
as well as physiological reactions (heart rate, skin conductance, pupil dilation
etc.) and brain activity, measured with EEG or fMRI, which theoretically obvi-
ate the problems associated with verbal evaluation. Computational analysis
of aesthetic behavior, on the other side, is an object-focused approach and is
particularly strong at describing “independent variables”, i.e., the aesthetic
contents that are consumed and judged by people every day. This strength
stems from its capacity to gather and analyze large numbers of features from
images, music and other cultural artifacts. As previously mentioned, the key
advantage of computational approaches to aesthetics is the fact that they
are not bound to seek aesthetic universals or to take the common responses
of (relatively small) groups of subjects to be representative of general atti-
tudes. Instead, algorithms can track individual preferences and behavior
without needing to model aesthetic responses based on aggregated aver-
ages. Big data does not require us to assume a universal human aesthetic
subject.
Despite these advantages, an artificial aesthetics that focuses on aesthetic
preferences still has to deal with the methodological challenges that charac-
terize all experimental approaches. We shall briefly mention two of them,
concerning 1) the difficulty of isolating the features linked to our aesthetic
evaluation, and 2) the difficulty of determining what kind of response we are
trying to describe.
Concerning the first point, features of aesthetic objects are hard to isolate.
For example, to study how variations in the shape of a design item influence
aesthetic appreciation, an experiment should use a controlled setting that
analyzes the effect of minimal variations in the shape and avoids confound-
ing multiple variations at once (e.g., changing shape and color, or shape and
texture etc.). However, aesthetic variables can also interact with each other.
21
ARTIFICIAL AESTHETICS
Consequentially, this set-up would not allow us to draw a one-to-one corre-
spondence between the feature and the aesthetic responses to the feature
on this particular object. It is certainly possible to determine general trends
in people's preferences: e.g., we could observe that a certain musical style is
more popular than another one with a particular demographic in a given
country. However, it is not always easy to reach greater granularity and com-
prehend the precise role of each factor in the final aesthetic effect: what ex-
actly makes the one musical style more appealing than the other? In order to
achieve this level of understanding, we would need a large number of similar
aesthetic artifacts that present only small variations from each other.
In some cases, digital platforms allow us to study a vast number of different
but not too heterogeneous stimuli which are available on the web. For ex-
ample, in a study from 201418, the authors used hundreds of features from
micro-videos (up to six seconds’ duration) on Vine, a former media sharing
platform, to predict whether people would judge them as “creative” or “non-
creative”. The study used a crowdsourcing platform to have 284 people judge
3800 videos. Each video received evaluations from multiple people, the aver-
age agreement of which was calculated to be 84%. The features covered
scene content, filmmaking techniques, photographic techniques, composition,
visual affect, audio affect, and novelty. All these features were defined math-
ematically and calculated automatically from the videos through an analysis
of their frames and soundtrack. The authors report the classification accuracy
for each group of features, concluding: “The best results are achieved when
we combine novelty features with aesthetic value features, showing the use-
fulness of this twofold definition of creativity.” Used separately, composition
and photographic techniques outperform scene content (classification accu-
racy is 77% vs 73%), while novelty video features outperform novelty audio
features (74% vs 63%). To get these kind of results, it is necessary to have a
sufficiently wide data set whose features are manageable (like a short six-
second film), which is not always the case with human cultural production.
22
Chapter 1
_____________________________________
What are we actually measuring when we
ask a subject about her aesthetic
experience?
Concerning the second point, human aesthetic responses (i.e., dependent
variables in a psychological experiment) also pose their own challenges.
What are we actually measuring when we ask a subject about her aesthetic
experience? Our relationships with aesthetic objects have many layers and
dimensions. They can range from sub-personal physiological reactions to
complex critical formulations, from a “like” given to an image in a social net-
work, to actual consumption behavior, up to sophisticated critical judgment.
We get a different answer depending on whether we ask someone if she
“likes” a movie or if she considers it a masterpiece, or if we simply observe
her physiological reactions while watching that movie. Moreover, we should
distinguish between value judgement and mere subjective preference/desire:
in general, we can say that value judgments are more stable than momentary
preferences or desire for a certain object. I can consider song X to be a mas-
terpiece (and superior to song Y), but lack the desire to listen to X at present,
instead experiencing a greater desire to listen to Y, maybe because of my
emotional state or because I listened to X too many times. This means that
my consumption behavior can reveal preferences that do not necessarily ex-
press my general idea of aesthetic value: I may be an avid consumer of ac-
tion movies and yet consider arthouse films aesthetically superior, even
though I watch them more rarely. Artificial systems that gather data about
human aesthetic consumption should take these issues into consideration if
we want to avoid overly simplistic models of human aesthetic experience
and judgment, both of which are used in artificial evaluative and generative
algorithms.
23
ARTIFICIAL AESTHETICS
1Datta et al. propose 56 different rules and features. Datta, Ritendra, et
al. “Studying aesthetics in photographic images using a computational
approach.” European Conference on Computer Vision. Springer, 2006, pp.
288-301.
2 Kao, Yueying, et al. “Deep aesthetic quality assessment with seman-
tic information.” IEEE Transactions on Image Processing, vol. 26, no. 3,
2017, 1482-1495. IEEE Xplore, https://doi.org/10.1109/
TIP.2017.2651399.
3 Hadjeres, Gaëtan, et al. “DeepBach: a Steerable Model for Bach
Chorales Generation.” Proceedings of Machine Learning Research, vol. 70,
2017. arXiv, https://arxiv.org/abs/1612.01010; Emerging Technology
from the arXiv. “Deep-Learning Machine Listens to Bach, Then Writes
Its Own Music in the Same Style.” MIT Technology Review, 14 December
2016, technologyreview.com/2016/12/14/155416.
4 “Can You Tell the Difference between AI and Human Composers?”
YouTube, uploaded by TwoSetViolin, 23 September 2020, youtu.be/Pm-
L31mVx0XA; “Bach vs AI: spot the difference.” YouTube, uploaded by
Orchestra of the Age of Enlightenment, 22 November 2019, youtu.be/
lv9W7qrYhbk.
5 Davis, Elizabeth. “Schubert’s ‘Unfinished’ Symphony completed by ar-
tificial intelligence.” Classic FM, 6 February 2019, classicfm.com/com-
posers/schubert/unfinished-symphony-completed-by-ai.
6Xue, Alice. "End-to-End Chinese Landscape Painting Creation Using
Generative Adversarial Networks." Proceedings of IEEE WACV, 2021, pp.
3863-3871. arXiv, https://arxiv.org/abs/2011.05552.
7 By using well-defined and ill-defined problems, we are referring to the
crucial distinction made by Herbert Simon in 1973 regarding artificial
intelligence. See Simon, Herbert. “The structure of ill structured prob-
lems.” Artificial Intelligence, vol. 4, 1973, pp. 181-201. ScienceDirect,
https://doi.org/10.1016/0004-3702(73)90011-8.
24
Chapter 1
8 The practical applications of such analysis include designing interac-
tive online interfaces for museum collections. For example, when a
person chooses a particular artwork, the system shows other artworks
in the collection that are most similar. Further applications can be
found in digital art history: changes in any extracted feature or a com-
bination of features can be plotted over time to analyze the evolution
of a single artist or entire historical periods.
9 See Djudjic, Dunja. “The Rise Of The Machines: Google’s AI Will De-
cide If Your Photos Are Aesthetically Pleasing.” DIY Photography, 26 De-
cember 2017, diyphotography.net/rise-machines-googles-ai-will-de-
cide-photos-aesthetically-pleasing; and Mikhailiuk, Aliaksei. “Deep Im-
age Quality Assessment.” towards data science, 15 March 2021, to-
wardsdatascience.com/deep-image-quality-assessment-30ad71641-
fac.
10 Fechner, Gustav. Vorschule der Aesthetik. Breitkopf und Härtel, 1876.
11 Birkhoff, George D. Aesthetic Measure. Harvard University Press, 1933.
12Berlyne, Daniel E. Studies in the New Experimental Aesthetics. Wiley,
1974.
13 Leder, Helmut, et al. “A Model of Aesthetic Appreciation and Aesthet-
ic Judgements.” British Journal of Psychology, vol. 95, 2004, pp. 489-508.
14For an overview, see Locher, Paul L. “Contemporary Experimental
Aesthetics: Procedures and Findings.” Handbook of the Economics of Art
and Culture, edited by Victor A. Ginsburgh and David Throsby, vol. 2,
North Holland, 2013; The Cambridge Handbook of the Psychology of Aes-
thetics and the Arts, edited by Pablo P. L. Tinio and Jeffrey K. Smith,
Cambridge University Press, 2014.
15Leder, Helmut, et al. “Entitling Art: Influence of Title Information on
Understanding and Appreciation of Paintings.” Acta Psychologica, vol.
121, 2006, pp. 176-198.
16Khaw, Mel W., and David Freeberg. “Continuous Aesthetic Judgment
of Image Sequences.” Acta Psychologica, vol. 188, 2018, pp. 213-219.
25
ARTIFICIAL AESTHETICS
17 Even the assumption that beholders would universally associate par-
ticular forms with specific qualities or “aesthetic effects” has been put
into question. See Specker, Eva, et al. “Warm, Lively, Rough? Assessing
Agreement on Aesthetic Effects of Artworks.” PLOS One, 13 May 2020,
https://doi.org/10.1371/journal.pone.0232083.
18Redi, Miriam, et al. "6 Seconds of Sound and Vision: Creativity in Mi-
cro-videos." IEEE Conference on Computer Vision and Pattern Recognition,
2014, pp. 4272-4279, https://doi.org/10.1109/CVPR.2014.544.
26
