Journal of Engineering Design

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/cjen20

Product innovation concept generation based on

deep learning and Kansei engineering

Xiong Li, Jianning Su, Zhipeng Zhang & Ruisheng Bai

To cite this article: Xiong Li, Jianning Su, Zhipeng Zhang & Ruisheng Bai (2021): Product
innovation concept generation based on deep learning and Kansei engineering, Journal of
Engineering Design, DOI: 10.1080/09544828.2021.1928023

To link to this article: https://doi.org/10.1080/09544828.2021.1928023

Published online: 03 Jun 2021.

Submit your article to this journal

Article views: 80

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=cjen20
JOURNAL OF ENGINEERING DESIGN
https://doi.org/10.1080/09544828.2021.1928023

Product innovation concept generation based on deep

learning and Kansei engineering
Xiong Lia,b , Jianning Suc , Zhipeng Zhangc and Ruisheng Baia
a School of Mechanical & Electronical Engineering, Lanzhou University of Technology, Lanzhou, People’s

Republic of China; b School of Bailie Mechanical Engineering, Lanzhou City University, Lanzhou, People’s
Republic of China; c School of Design Art, Lanzhou University of Technology, Lanzhou, People’s Republic of
China

ABSTRACT ARTICLE HISTORY

Industrial designers often present their initial concepts as design Received 25 May 2020
sketches. Rapid creation of new product conceptual images that Accepted 6 May 2021
meet users’ affective preferences remains challenging in real design KEYWORDS
environments. However, few published works in affective design deep learning; PCGA-DLKE;
directly assist industrial designers in creating product conceptual Kansei engineering; PD-GAN;
images. Thus, we propose a product concept generation approach product concept generation
framework based on deep learning and Kansei engineering (PCGA-
DLKE) to assist industrial designers. Our work focuses on dataset col-
lection, pre-processing, affective preferences recognition, concep-
tual image generation model and product style transfer networks. To
mark users’ affective preferences, we established an affective recog-
nition model by Kansei engineering and deep convolutional neu-
ral networks. To address the product conceptual image generation
problem, we proposed a product design GAN model (PD-GAN), gen-
erating product conceptual images with affective preferences. An
improved fast neural style transfer network was successfully trained
to meet users’ style preferences. This study aims to assist industrial
designers in finding innovative concepts with affective preference.
The Kansei evaluation shows that the innovation of the new product
concept has been enhanced, indicating that the approach can better
assist industrial designers in creating designs that meet users’ emo-
tional needs. Hand drill design and bicycle helmet design are taken
as a case study.

1. Introduction
Nowadays, consumers are not only concerned with the functionality and reliability of a
product, but they are also concerned with product emotions related to the feelings and
impressions of the product, such as form, texture, colour and style (Yanagisawa and Fukuda
2005). People’s demands are have become diversified, especially with the rapid progress
of technology, emotional needs have become more prominent. Simultaneously, for enter-
prises, the era of gaining a competitive advantage by focusing only on product functions
has passed (Khalid 2006). Enterprises need to consider the consumption preferences of

CONTACT Jianning Su [email protected]

© 2021 Informa UK Limited, trading as Taylor & Francis Group

2 X. LI ET AL.

users and consumers, especially emotional needs (Kwong et al. 2013). Products need to
appeal to users and customers on an affective level to succeed in the highly competitive
market (Chan et al. 2020). Therefore, affective design becomes the key point in the product
upgrade process.
However, most of the initial concepts of product design are designed by industrial
designers to explore the design direction through the hand-painted conceptual image pat-
tern, which is a design pattern of personal introspection (Self 2019). This design process
operates entirely in a black box by the designer’s brain (Sutera, Yang, and Elsen 2014). This
traditional design pattern is difficult to capture the emotional preference needs of users and
consumers quickly. Therefore, how to quickly and effectively create conceptual images of
products that meet consumer’s emotional needs is a challenge for industrial designers in a
competitive design environment. In this study, we propose a product concept generation
approach framework based on deep learning and Kansei engineering (PCGA-DLKE).
Kansei engineering is widely used as a quantitative analysis method of new product
affective design and development. It has three core tasks: affective modelling, determi-
nation of design elements settings, and design execution, that is, conceptual generation
(Nagamachi 1995). Many previous researches mainly focused on the first two. Reviewing
related literatures, we find that there are three types of concept generation methods. The
first type concerns algorithm or data-driven design generation. A small number of investi-
gations developed a specialized design tool in the case study, but they cannot be used as
general-purpose tools. For example, Chen et al. developed a 3D knife design tool based
on Kansei engineering and Visual Basic software (Chen and Chang 2014). Big-data min-
ing brings new challenges and opportunities for designer’s idea generation. For example,
Liu et al. proposed data-driven concept network-assisted design concept generation (Liu
et al. 2020). Second, generate a new design scheme using 3D software. Guo et al. (2014), for
example, used Rhino to create a digital camera scheme after the Kansei engineering mole
was determined. Third, design sketches show the results of Kansei engineering. For exam-
ple, the case in (Chen, Yeh, and Lin 2010) shows chair design sketches drawn by a product
designer base on design elements determined by Kansei engineering. These three meth-
ods lack continuity with the first two core tasks in Kansei engineering. Because the design
schemes are not directly generated by Kansei engineering. This shows that concept gen-
eration is a shortcoming of Kansei engineering. Such a short board can easily lead to the
separation of design research and design practice in the implementation process.
Over the past few years, deep learning technology, such as deep convolutional neural
networks (DCNN), have made breakthroughs in many fields, for instance, computer vision,
autopilot, Medical Science and games (Hadji and Wildes 2018). These efforts have resulted
in new state-of-the-art performance on a wide range of classification (Krizhevsky, Sutskever,
and Hinton 2017) and regression tasks (Eigen and Fergus 2015). In industrial design, Pedro
et al. (2018) proposed using CNN to evaluate the usability of a product, and a case study
using a thermostat as an example for rapid product evaluation and development. Pan et al.
(2017) used a scalable deep learning approach to predict and interpret customer aesthetic
perceptions of heterogeneous market design attributes, and used automotive aesthetic
perceptions as a case study. However, the model constructed lacks explanations for the
aesthetic perception of the heterogeneous market. Wang, Mo, and Tseng (2018) proposed
an affective modelling approach based on deep learning techniques that automatically
correlate customer needs with product design parameters. Besides, the invention of the
 JOURNAL OF ENGINEERING DESIGN 3

Generative Adversarial Networks (GAN) (Goodfellow et al. 2014) has breathed new life into
the innovative design of the product. Kim et al. (Kim et al. 2017) used GAN to implement
the cross-domain generative design, which improves the associative innovation capability
of an aided design system. Chai et al. (2018) proposed a GAN-based automatic colour-
ing model for footwear design sketches, but their model lacks consideration of consumer
affective preferences. Chen et al. (2019) proposed a visual concepts combination model
of the GAN model consisting of a double discriminator for product conceptual innova-
tion, but their model almost ignores the functionality of the product. Quan, Li, and Hu
(2018) proposed a combined deep learning and Kansei engineering method for product
style transfer that can automatically generate a new scheme with a specific style, but this
approach simply changes the style of the product and does not create a new product form.
In addition, the recognition and prediction of product affective preferences is a key issue in
Kansei engineering research (Nagamachi 1995; Quan, Li, and Hu 2018; Wang, Mo, and Tseng
2018). Traditional machine learning methods, such as SVM and Multi-Layer Perceptron, are
unable to recognise the affective preferences of the products at the level of overall visual
perception as humans do.
To overcome the above problem, we proposed a product concept generation approach
framework based on deep learning and Kansei engineering (PCGA-DLKE). This framework
benefits for applying computer-aided design and creation and realizes the integration of
creativity and inspiration.
The success of deep convolutional neural networks in the field of image recognition such
as Alex-Net (Krizhevsky, Sutskever, and Hinton 2017), VGG-Net (Simonyan and Zisserman
2015) reminds us of the recognition of product affective preferences in affective design.
While the invention of GAN (Goodfellow et al. 2014) shocked us, we saw its great potential in
design. The style transfer network (Gatys, Ecker, and Bethge 2015) reborn Van Gogh’s style,
which aroused widespread concern in academia and industry. This also prompted us to
imagine the space of product style. In short, these works inspired our passion for developing
research into product design applications.
This paper proposes a framework model that produces product concept design for
an industrial designer to inspire innovation. We propose a relatively complete product
concept image generation framework based on deep learning technology and Kansei
engineering.
Our contribution can be described as three points of innovation: (1) An integrated
approach (PCGA-DLKE), which combines Kansei engineering and deep learning techniques,
is proposed for innovation concept generation, to improve the product conception design
process efficiently. It has three core modules. A product affective preferences recognition
module, based on the modified deep residual networks and Kansei engineering. A valid PD-
GAN algorithm framework for product conceptual image generation and a suitable PD-GAN
model based on DCGAN and Residual Networks. We propose using the fast-neural style
transfer technique for a new generative product by reconstructing and merging the style
image’s pattern features. We also improved the structure of the original fast-neural style
transfer networks with more robust feature transfer capability. (2) Previous studies generate
a limited space for conceptual designs. In this paper, our approach framework gener-
ates many innovative conceptual designs with affective preferences and rapidly changes
the style features of the generated solutions. (3) We applied the proposed PCGA-DLKE
framework to the hand drill and bicycle helmet creative concept generation problem to
4 X. LI ET AL.

demonstrate the value of our method. Experiments demonstrate the innovative design
potential of the methodological framework to improve affective design efficiency through
an end-to-end learning approach.
The rest of this paper is organised as follows. In Section 2, we firstly present the overall
research framework. Next, we describe the product image data preparation, affective pref-
erences recognition and label estimation, product design generative adversarial networks
(PD-GAN), and product style fast-neural style transfer networks (PS-FNSTN). In Section 3, an
empirical case is given to verify the proposed framework for an industrial designer, and the
related experimental results are shown. Finally, discussion, conclusions, and future work are
given in Sections 4 and 5.

2. Methods
2.1. Research framework
In order to exploit artificial intelligence (AI) for innovative conceptual design, we propose
a generative approach to the product concept image, the PCGA-DLKE based on deep con-
volutional neural networks, deep convolutional generative adversarial networks and fast
neural style transfer networks model. As shown in Figure 1, the PCGA-DLKE framework
contains four parts. In part 1, product image data crawling and a series of image data pre-
processing. Step A of data preparation aims to obtain the original data of the target product.
In this step, crawler technology crawls the product pictures from the target websites, and
the design team draws the concept image. Step B involves the pre-processing of the orig-
inal data, aiming to obtain a clean image data set. Based on this module, we can obtain a
clean and uniform product image dataset. In part 2, Kansei engineering (KE) and deep resid-
ual networks are used for label estimation of product images. In part 3, we use a variant of
vanilla generative adversarial networks (named PD-GAN) to create a newer product design
that meets user and customer’s affective needs. In the last part, the style transfer technique
is applied to extend the affective domain of the product. Because we applied the fast-neural
style networks method to product images, we named it PS-FNSTN. By trained the VGG-19
(Simonyan and Zisserman 2015) model, we can turn the content image and style image into
a stylised image, which is generated entirely new product.

2.2. Product image data preparation

Data is one of the three critical components (big data, computing power and algorithms)
of artificial intelligence. High-quality image dataset is indispensable for successful image
generation. Currently, there are few dedicated open-source product images dataset for
training image generation models. Previous works (Jin et al. 2017; Mattya 2015) have shown
that images crawled directly from the web have high inter-image variance and noise. To
obtain high quality, clean product images dataset, we proposed an efficient operation that
includes two parts. In the first part, we obtain the original product images data through
the web crawler and the work of the design team. The design team can only draw a lim-
ited number of product images. Therefore, the original images data are mainly captured
from the object web pages through web crawlers. We found the Bing search engine to be a
practical tool for collecting product images. To collect more object images, it is important to
 JOURNAL OF ENGINEERING DESIGN 5

Figure 1. The product concept generation approaches framework based on deep learning and Kansei
engineering (PCGA-DLKE).

Figure 2. Left: Easy Image Modifier. Middle: Vector Magic. Right: Easy Photo Denoise.

constantly change search keywords for web crawlers and continuously update the landing
page. In the second part, we need to perform some series of pre-processing on the origi-
nal images, including image filter for selecting good images, pixel pre-treatment, uniform
in size, data augmentation, and image noise reduction. During image data pre-processing,
we found that three tools can provide effective auxiliary functions. They are Easy Image
Modifier, Vector Magic, Photoshop and Easy Photo Denoise, as shown in Figure 2. Vector
Magic and Photoshop can be used together to enhance image quality. The combination
of Easy Image Modifier and Photoshop can be used to achieve data augmentation. We did
not use any data augmentation techniques provided by deep learning frameworks (e.g.
6 X. LI ET AL.

Figure 3. Overall flow of labels estimation approach based on Kansei engineering (KE) and deep residual
networks (ResNet).

Figure 4. 5-point semantic differential scale for the online questionnaire.

PyTorch and TensorFlow), as we found that they often lead to incomplete product images.
This is not good for subsequent affective preference recognition and product image gener-
ation. To obtain a clean image, image noise reduction is necessary. In this study, Easy Photo
Denoise is used for image noise reduction.

2.3. Affective preference recognition and label estimation

According to Figure 3, the proposed approach to label estimation consists of two mod-
ules: separately affective evaluation data collection and product affective preference
 JOURNAL OF ENGINEERING DESIGN 7

recognition. this section gives a detailed introduction to the method of affective preference
labels estimation.
Affective evaluation data collection of the product includes collecting product images
from various websites, which is taken from the first part of the approach framework and
Kansei words collection and web questionnaires. This affective evaluation data collection
process aims to build the data onto users and customers’ Kansei evaluations. The product
affective preference recognition includes data splitting and a training process of the deep
convolutional neural networks. It uses the dataset collected from the last step to train the
deep learning models of the product features and affective preference labels (that is Kansei
labels). The network is continuously updated with the growth of data. Lastly, the affective
preference recognition application involves applying the PAR-ResNet to help designers
estimate the types of users’ affective preference and label automatically for the product
image.

2.3.1. Affective evaluation data collection

The data used for affective evaluation includes product image sample data (image
database) and corresponding affective preference labels (Kansei database). Product image
sample data can be randomly extracted from the image data collected in the previous step.
The affective preference labels in the second part can be obtained from the Kansei words.
Because Kansei words are usually adjectives that describe people’s emotions and feelings,
and are widely used in design research and practice.
Affective preferences collection. Generally, the acquisition of affective preferences in
design research and practice requires two steps. First, the extensive collection of Kan-
sei vocabulary. Kansei words usually use adjectives, an expression channel that intuitively
reflects people’s emotions and feelings about the product. They can be collected through
various channels, such as magazines, academic paper, product test reports, product manu-
als, expert reviews, user opinions, web reviews and customer interview (Chou 2016). In this
study, we collect Kansei words from related academic literatures and the internet. Second,
clustering all collected Kansei words result in affective preference attributes. This process
is called affective clustering. Currently, there are three Kansei clustering methods that are
frequently used in an affective design, which are the clustering method based on fuzzy
equivalence relation (Chou 2016), the Kansei clustering method based on a design struc-
ture matrix (DSM) (Huang, Chen, and Khoo 2012), the rough set-based clustering method
(Zhai, Khoo, and Zhong 2009). In this study, we used the methods and steps proposed by
Li et al. (2018), and hand drills are selected to conduct the case study. There was a similar
study about battery drill used 25 adjectives as Kansei words, and cluster analysis (Grimsaeth
et al. 2010).
Questionnaire construction. Because the internet and smartphones have become an
integral part of people’s lives, such as online office, online learning and online shopping, it
is natural to save time and labour by performing surveys online traditional surveys. Besides,
what excites us most is that the speed of data update online surveys is unmatched by
traditional surveys. Consumers’ and user’s affective responses obtained through online
questionnaires are more authentic than conventional surveys. We used the determined
affective preference tags and product images to construct an online questionnaire based
on the semantic differential. Semantic differential, proposed by Osgood, Suci, and Tan-
nenbaum (1957), an American psychologist, is a user-centric design technique commonly
8 X. LI ET AL.

used to obtain consumer affective scores for products. This study uses the 5-point seman-
tic differential scale for quantifying consumers and user’s affective preferences. As shown in
Figure 4, from 1 to 5, each point represents a preference level of the customers and users.
For instance, 1 and 5 represent a pair of bipolar adjectives, while 3 represents a medium
level. Finally, the user Kansei evaluation data set was obtained through online surveys.

2.3.2. Affective preferences recognition

Essentially, the product affective preferences recognition is a classification problem.
Obtaining the user evaluation Kansei evaluation data set, we use a deep learning algorithm
to model the relation between product (images) and consumers’ and users’ affective
preferences tags.
Deep learning, especially deep convolutional neural networks (CNNs), has been widely
used in image recognition (Krizhevsky, Sutskever, and Hinton 2017; Simonyan and Zis-
serman 2015). Because of their ability to hierarchically abstract representations with local
operations (Hadji and Wildes 2018). Many experiments have proved that CNNs can extract
the identifiable overall features of the images through local receptive fields, weight
sharing and local down sampling (Dong et al. 2014). More and more network architec-
tures with high representation capabilities have been proposed and refreshed the image
recognition records again and again, for example, LeNet (Lecun et al. 1998), Alex-Net
(Krizhevsky, Sutskever, and Hinton 2017), VGG-Net (Simonyan and Zisserman 2015), Goole-
LeNet (Szegedy et al. 2015) and ResNet (He et al. 2016). In this study, we used a modified
ResNet18 as the affective preferences recognizer, namely PAR-ResNet. Therefore, more new
product images can be automatically recognised by the trained model and labelled with the
affective preference tags. Since it is a classification problem, we use the cross-entropy loss
function to evaluate PAR-ResNet.
  m
1
L(θ)PAR-ResNet = − [log hθ (x (j) ) + (1 − y(j) ) log(1 − hθ (x (j) ))] (1)
m
j=1

The architecture of par-resnet will be presented in Section 3.2, the recognition accuracy
is measured by the confusion matrix, which is calculated by Equation (2):

TP + TN
accuracyDataset = (2)
TP + TN + FP + FN
where, TP is True Positive, TN is True Negative, FP is False Positive and FN is False Negative.

2.4. Product design generative adversarial network

Generative Adversarial Networks (GANs) is a hybrid generative model whose core idea
comes from the two-person zero-sum game in game theory (Goodfellow et al. 2014). The
basic GAN model consists of two networks: A Generator and a Discriminator. Sampling ran-
dom variable z from a probability distribution Pz (for example, Gaussian) as input to the
Generator G, through a nonlinear mapping of G, output signal G(z). The discriminator D
takes G(z) or x as input, and determines whether the input data comes from real data or
generated data by calculating the probability that it belongs to real data. Specifically, the
original GAN adopts the adversarial learning strategy to train G and D, so that the training
 JOURNAL OF ENGINEERING DESIGN 9

objectives of the two are opposite. Mathematically, the goal of GAN can be expressed as

min max V(D, G) = Ex∼Pdata (x) [log D(x)] + Ez∼Pz (z) [log(1 − D(G(z)))] (3)
G D

In Equation (3), the discriminator D tries to maximise the log likelihood function from real
data. Meanwhile, it also minimises the log likelihood function from fake data generated by
generator G. In contrast, the goal of G is to minimize the log-likelihood function and make
the distribution PG of G(z) approach the distribution Pdata of real data. The discriminator D
is the object deceived by generator G.
Although Goodfellow et al. (2014) have theoretically proved the convergence of the
GAN model. However, in practice, GANs still has problems such as training instability,
mode collapse, convergence difficulties, difficulty inaccurate control of generated content
(Arjovsky and Bottou 2017). Therefore, different architectures, loss functions, conditional
techniques and constrain methods were introduced, easing the convergence of GAN mod-
els. In improving the architecture of networks, such as Deep Convolutional GAN (2016)
(Radford, Metz, and Chintala 2016), Deep Regret Analytic GAN (2017) (Kodali et al. 2017),
PGAN (2018) (Karras et al. 2018) and Big-GAN (2019) (Brock, Donahue, and Simonyan 2019).
In terms of improving the loss function, such as Least Squares GAN (2016) (Mao et al. 2017),
Wasserstein GAN (2016) (Arjovsky, Chintala, and Bottou 2017) and Boundary-Seeking GAN
(2018) (Hjelm et al. 2017). In terms of improving the conditional techniques, such as Condi-
tional GAN (2014) (Mirza and Osindero 2014), Auxiliary Classifier GAN (2016) (Odena, Olah,
and Shlens 2017), Triple GAN (2017) (Li et al. 2017) and Style GAN (2019) (Karras, Laine,
and Aila 2019). Unfortunately, there is currently no comprehensive comparative study of
GAN models. However, we can always make the right choice since no free lunch theorem
in machine learning and considering the reality of industrial product design.
In this paper, we choose DCGAN (Radford, Metz, and Chintala 2016) as the basic GAN
model. We have three reasons: First, the number of images of the same product in real world
is limited, for example, the number of hair dryers is far less than the number of faces. Sec-
ond, the network architecture should not be too complex because of the limited amount of
data. The parameters that need to be calculated for a complex network structure will also
increase dramatically. Third, the model must be easy to train. Besides, the work of Mattya
(2017) and Elgammal et al. (2017) also shows that DCGAN’s generation ability is excellent.
In our experiments, we successfully train the PD-GAN model based on ResNet (He et al.
2016) and DCGAN (Radford, Metz, and Chintala 2016). Figure 5 shows an overview of our
PD-GAN algorithm framework. The architecture of the generative network will be presented
in Section 3.3.
The discriminator model needs to determine whether each sample is true or false, and
it also needs to complete to a categorisation task to predict affective preference type c by
adding an auxiliary classifier. Thus, the loss function of the discriminator is described as
following:

Ladv (D) = −Ex∼Pdata (x) [log D(x)] − Ez∼Pz (z),nc∼Pnc (nc) [log(1 − D(G(z, nc)))] (4)
Lcls (D) = Ex∼Pdata (x) [logPD (cx |x)] + Ez∼Pz (z),nc∼Pnc (nc) [logPD (nc|G(z, nc))] (5)
2
Lgp (D) = Ex̃∼Pdata (x̃) [(∇x̃ D(x̃)2 − 1) ] (6)

where the term Lgp from DRAGAN (Kodali et al. 2017).

10 X. LI ET AL.

Figure 5. Product Design-GAN algorithm framework.

For the generator model, the loss function is as follows:

Ladv (G) = Ez∼Pz (z),nc∼Pnc (nc) [log(D(G(z, nc)))] (7)

Lcls (G) = Ez∼Pz (z),nc∼Pnc (nc) [PD (nc|G(z, nc))] (8)

Summarily, the loss function of PD-GAN can be expressed as the following two simple
expressions:

L(D) = Ladv (D) + β Lcls (D) + λLgp (D) (9)

L(G) = Ladv (G) + β Lcls (G) (10)

where β and λ are balance factors for the adversarial loss and gradient penalty, respectively.

2.5. Fast neural style transfer network

Style transfer is also called style conversion. The intuitive analogy adds a filter to the input
image, but it is different from traditional filters. Generally, style transfer uses a convolutional
neural network to automatically apply styles from one image to another. A content image
and a style image are used to create an output image whose ‘content’ mirrors the content
image and whose style resembles that of the style image (Gatys, Ecker, and Bethge 2015). In
contrast to Neural Style, Fast Neural Style has designed a network specifically for style trans-
fer. When inputting a picture, the network automatically generates the target picture in real
time (Johnson, Alahi, and Fei-Fei 2016). This network needs to train a corresponding style
network for each style image, but once the training is completed, it only takes 20 s less to
complete a style transfer. This is well suited to early product concept design needs and helps
designers quickly experiment with different styles in order to make the right judgment.
The network structure of a fast-neural style consists of two parts: an image transformation
network fw (that is generative network) and a loss network φ. Figure 6 shows a structural
overview of our fast-neural style transfer network, in other words, generative network on
the left and loss network on the right. The former is the network we need to train, the latter
is a trained network. We utilised the ‘Encoder-IN-Decoder’ architecture. The architecture of
the generative network is presented in section 3.4.
The transform network fw takes a content image c and a style image s as inputs and
synthesises an output image that recombines the content of the former and style of the
 JOURNAL OF ENGINEERING DESIGN 11

Figure 6. Fast style transfer model for product design. Left: style transfer network. Right: loss network.

later. In the task of style transform, x is the input image, y c = x, y s is the style image. For the
input image x, fw can return a new image ŷ, so fw is naturally the style transform network
we want to design. ŷ is similar to yc in content, but is similar to y s in style. Generally, the
loss network can be used to calculate visual features and style features without training.
In this study, we used VGG-19 (Simonyan and Zisserman 2015), which has been trained on
the ImageNet dataset. The choice of content presentation layer and style presentation layer
is derived from the literature (Luan et al. 2017). Therefore, all three (ys , ŷ and y c ) are input
into the loss network φ, and corresponding losses (including content loss and style loss) are
generated. In this way, we can get ideal style images by end-to-end training.
The content loss is the (squared and normalised) Euclidean distance between the stylised
image and the original image:

φ,j 1
Lcontent (ŷ, y c ) = ||φ (ŷ) − φ j (yc )||22 (11)
C j Hj W j j

where C j Hj W j is the shape of the feature map. According to our definition of style percep-
tion, the style loss can be simply understood as removing the spatial information and fusing
the feature response of each channel. The style loss is the squared Frobenius norm of the
difference between the Gram matrices of the output and target images:

φ,j 1 φ φ
Lstyle (ŷ, ys ) = ||G (ŷ) − Gj (ys )||2F (12)
C j Hj W j j

The loss function that is ultimately used for training is weighted sum of content loss and
style loss.
φ,j φ,J
Ltotal = α Lcontent (ŷ, y c ) + β Lstyle (ŷ, y s ) + γ Ltv (ŷ) (13)
where Ltotal is the loss function, α is the weight of the content loss, β is the weight of the
style loss, Ltv (ŷ) total variation regularizer, and γ is total variation weight.

3. Empirical study
In this paper, two case studies of generating hand drills and bicycle helmets were con-
ducted to verify the proposed framework’s practicality and effectiveness. It had the fol-
lowing steps: (1) Data collection and data pre-processing. To obtain a clean dataset, we
pre-processed the messy raw data by web crawler using a series of the combined steps
and methods. (2) A modified ResNet18 (PAR-ResNet) was trained to identify the type of
12 X. LI ET AL.

Figure 7. Example of a Bing Search Web page search hand drills.

affective preference of the target product. (3) The PD-GAN model based on DCGAN and
residual blocks was trained to generate new product images and evaluate them. (4) A mod-
ified fast neural style transfer networks (PS-FNSTN) was trained to transfer the style of the
new product concept image.
Our PCGA-DLKE framework was developed using Python. Comprehensively, PAR-
ResNet, PD-GAN, PS-FNSTN were built using the Python language along with Python
modules such as PyTorch, TorchVision, TorchNet, Torchsummary, Visdom and NumPy. All
experiments were run on a YunXuan workstation (YunXuan Inc., Shanghai, China) with Intel
i7-9700K and RTX2080 8G (double) and Ubuntu 18.04. operating system.

3.1. Image data preparation

It is well known that an image dataset in high quality is essential, if not most impor-
tant, to the success of product concept image generation. However, it is not easy to get
enough product pictures. Our collection of images is based on the approach mentioned
in Section 2.2. First, some of the product images are crawled from a web page provided
by the Bing search engine using a web crawler tool, and others are crawled from Ama-
zon.com, JD.com and Taobao.com. For example, the results of the Bing search are shown
in Figure 7. Table 1 lists the keywords and samples we used in the process of searching
for hand drill images. Second, we manually check all product images and remove unde-
sired images. It should be noted that we do not use a hammer drill as a search keyword.
Thus, there are no hammer drill images in our dataset. Third, we used Easy Image Mod-
ifier for image filtering, format conversion (∗.jpg), standard size (resolution: 256 × 256,
128 × 128). The combination of Vector Magic and Photoshop was used to improve pixel
quality. Easy Photo Denoise was used for image noise reduction. Easy Image Modifier and
Photoshop were combined for data augmentation technology (random colour and image
flipping). Finally, we harvested 18,285 images of hand drills (i.e. hand drill dataset, named
HD dataset) and 15,456 images of bicycle helmets (i.e. bicycle helmet dataset, named BH
dataset).
 JOURNAL OF ENGINEERING DESIGN 13

Table 1. The list of search keyword for hand-held electric tools.

Sample picture Search keyword Sample picture Search keyword Sample picture Search keyword

Hand Drill Power Drill Electric Drill

Cordless Drill Battery Drill Electrical Drill

Cord Drill Automatic Screwdriver Electric Screwdriver

3.2. Product affective recognition and labelling

3.2.1. Affective evaluation data
The procedure for obtaining affective evaluation data uses the approach proposed in
Section 2.3. The data consists of two parts: image database, that is hand drill images and
Kansei database, that is affective preference labels data. In this study, 840 hand drills are
randomly selected from the previous hand drill image dataset.
Table 2 shows the collected Kansei words and Kansei labels (i.e. affective preference
labels). We collected 168 Kansei words from the existing literatures and online shop reviews.
We referred to the method mentioned in (Li et al. 2018) to cluster and optimise the Kan-
sei words. First, the collected Kansei words with opposite meanings are clustered into a
cluster by semantic cluster analysis. Sixteen clusters were obtained in total. Second, all the
clusters were ranked by the value of the sum of the amount and total frequency of the
Kansei words. Third, the first six high-scoring clusters were naturally selected, and the affec-
tive attribute of each cluster is determined based on the semantic matching degree and
common degree. Finally, six representative affective attributes were identified. They are
Ergonomic–Uncomfortable, concise–complex, powerful–powerless, Practical–Unpractical,
Handy–Bulky and Appealing–Unaesthetic. This means that we have 6 Kansei preference
dimensions, that is, 12 Kansei labels. These Kansei words relate to four essential aspects of
the hand drill: function, material, operation control and appearance, which are of interest
to users (Grimsaeth et al. 2010).
The online questionnaire consists of three parts: a hand drill image, 6 Kansei dimensions
(i.e. 12 Kansei labels) and a 5-point semantic difference scale. The first example is shown in
Figure 8. About 840 different hand drill samples we selected to be investigated in total.
There are two types of questionnaires, mobile and computer versions, distributed
through social networking sites. To ensure the accuracy of the evaluation and consider-
ing the respondent’s visual tolerance, we randomly divided 840 hand drill samples into 20
parts, each part containing 42 samples. Each participant was given an incentive during the
implementation. The minimum number of participants for each online questionnaire was
set at 32, and each participant answered the question up to 2 times. In the end, a total of
655 people participated in the online survey on affective preferences for the hand drill. Their
age range: 28–45 years. They have diverse careers, including engineers, construction work-
ers, designers, college students, renovators, etc. Therefore, only 42 sample images were
scored for each participant, and 640 (32 × 20) complete questionnaires were obtained. Each
of the six Kansei dimensions is evaluated independently of each other. Finally, the number
14 X. LI ET AL.

Table 2. Kansei labels and Kansei words.

Kansei labels Kansei words (examples) References (examples)
K1 Uncomfortable– comfortable, handling comfort, (Kim et al. 2019), (Grimsaeth
Ergonomic cozy, restrained, Ergonomic, et al. 2010), (Chang and
round Chen 2016), Online shop
reviews
K2 Complex– minimal, simple, plain, (Grimsaeth et al. 2010), (Chang
Concise complex, complicated, and Chen 2016), (Razza and
dazzling, simplistic, compact Paschoarelli 2015), Online
shop reviews
K3 Powerless– powerful, strong, forceful, (Vieira et al. 2017), (Wang et al.
Powerful vigorous, energetic, 2016), online shop reviews
energetical, weak, flaccid,
powerless
K4 Unpractical– quality, reliable, high-quality, (Kim et al. 2019), (Guo et al.
Practical sturdy, safe, accurate, robust, 2016), (Hsiao, Chen, and Liao
solid, durable, unreliable, 2017), Online shop reviews
K5 Bulky– handy, portable, heavy, (Kim et al. 2019), (Chou 2016),
Handy ingenious, bulky, flexible, Online shop reviews
lightweight
K6 Unaesthetic– artistic, aesthetic, appealing, (Chou 2016), (Fung et al. 2014),
Appealing cute, elegant, good-looking, (Jiao, Zhang, and Helander
exquisite, eye-catching, 2006), Online shop reviews
unaesthetic, artless,
attractive

Table 3. The Kansei evaluation data of hand drill.

Label Ergonomic Uncomfortable Concise Complex
Number 447 393 452 388
Label Powerful Powerless Practical Unpractical
Number 445 395 446 394
Label Handy Bulky Appealing Unaesthetic
Number 464 376 416 424

of samples corresponding to each Kansei label is shown in Table 3. Figure A1 shows the
distribution of hand drill images in our dataset.

3.2.2. PAR-ResNet architecture

Figure 9 shows PAR-ResNet’s architecture, a minor modification from ResNet18 (He et al.
2016). We only added a data deformation layer before the last Linear layer, which is named
Flatter layer. Of course, the number of output features of the Linear layer must modify to
what we need. The Model contains four modules made up of ResBlocks, each of which uses
two ResBlocks with the same number of output channels. There are four convolutional lay-
ers in each module except for 1 × 1 convolutional layer. The number of all channels in the
first module is 64. Because the previous layer is a maximum pooling layer with a stride of
2, there is no need to reduce the height and width. A residual block with a 1 × 1 convolu-
tion layer can double the number of channels of the previous residual block, and the height
and width are also halved. Furthermore, Gaussian error linear units (GELUs) (Hendrycks and
Gimpel 2016) is introduced as activation function in our model.
 JOURNAL OF ENGINEERING DESIGN 15

Figure 8. The question corresponding to the first-hand drill.

Figure 9. Product affective recognition resnet architecture.

3.2.3. Training and results

Obviously, 840 data samples are not enough. During training, therefore, we used data
augmentation techniques to augment the dataset. All 840 images were flipped, including
mirror flipping (left and right) and five random rotations (−25 degrees to 25 degrees), but
the labels were not changed. To ensure the consistent visual perception of humans and
machines, we did not change the colour and cropping on all sample images.
Since there are 6-dimensioanl emotion categories, the affective recognition task in this
paper is a 6-dimensional binary classification problem, so the model needs to be trained six
times individually.
16 X. LI ET AL.

Table 4. Evaluation results of hand drill affective preferences recognition.

Recognition accuracy (%)
K1 K2 K3 K4 K5 K6
Uncomfortable- Complex– Powerless– Unpractical– Bulky– Unaesthetic– Mean
Method Ergonomic Concise Powerful Practical Handy Appealing (%)
Alex-Net 56.34 75.59 74.20 71.82 69.98 70.83 64.83
VGG-11 58.88 73.21 81.52 77.97 72.26 80.95 54.13
VGG-16 63.96 78.42 75.99 83.53 78.51 82.93 77.22
PAR-ResNet 78.57 82.93 92.31 88.49 88.86 91.46 87.10

Table 5. Number of hand drill images for each label.

Tag Ergonomic Uncomfortable Concise Complex
Number 9457 8828 7881 10,404
Tag Powerful Powerless Practical Unpractical
Number 8652 9633 8835 9450
Tag Handy Bulky Appealing Unaesthetic
Number 11,880 6405 8401 9884

PAR-ResNet model is optimized using AdamW optimizer (Loshchilov and Hutter 2017)
with β1 = 0.9, β2 = 0.999. We use a batch size of 32 in the training procedure. The learning
rate is initialised to 5e − 2, and the learning rate decay is initialised to 5e − 1. The number
of max training epoch is set to 50. Figure A2 shows the loss of PARA-ResNet in our dataset.
Given a hand drill image, PAR-ResNet can predict probabilities of belonging to 12
kinds of Kansei labels such as ‘concise’, ‘appealing’ and ‘powerful’. The average accu-
racy rate of six Kansei dimensions is 87.10% in validation set (Table 4). Besides, we used
PyTorch to construct the Alex-Net, VGG-11 and VGG-16 models for comparison with PAR-
ResNet. The recognition accuracy is calculated by the formula (2). The evaluation results are
shown in Table 4. Table 5 shows the Kansei labels and the number of HD dataset images
corresponding to each estimated label.

3.3. Product design generative adversarial network

3.3.1. PD-GAN architecture
The generator’s architecture is shown in Figure 10, which is a modification from ResNet
(He et al. 2016) and DCGAN (Radford, Metz, and Chintala 2016). The network contains five
Residual Blocks and six Transpose convolutional layer for feature map extraction. We use
Gaussian error linear units (GELUs) (Hendrycks and Gimpel 2016) as the activation function
layer. Figure 11 shows the discriminator architecture, which includes five Residual Blocks in
all. The GELUs (Hendrycks and Gimpel 2016) is also used as the activation function layer in
the discriminator. The batch norm layer is replaced by the Instance norm layer in the dis-
criminator. The batch normalisation layer is not used in all Residual Blocks, since it would
bring correlations with the mini-batch, which is undesired for the computation of the gradi-
ent norm. Additionally, the output includes two branches that perform discriminatory and
classification tasks, respectively.
 JOURNAL OF ENGINEERING DESIGN 17

Figure 10. Generator architecture.

Figure 11. Discriminator architecture.

3.3.2. Network training and results

Noise data is designed to 128 dimensions. The learning rate is initialised to 2e − 4. Two
models (PD-G and PD-D) are optimised using Adam optimiser (Kingma and Ba 2015) with
β1 = 0.5, β2 = 0.999 and learning rate η = 2e − 4. We use a batch size of 64 in the training
procedure. Regarding the super parameters β and λ of the loss function, we refer to the
literature (Zhou et al. 2018). Here we set β = 3 and λ = 0.05.
Two types of PD-GAN models were successfully trained during our actual experiments.
The first type is a hand drill generator with Kansei labels trained based on the hand drill
data set with a resolution of 256 × 256 pixels. The second type is a bicycle helmet generator
without labels, it is trained based on the bike helmet dataset with a resolution of 128 × 128
pixels.
Figure 12 shows an example with Kansei labels. By fixing the random noise part and
Kansei labels, the model can generate hand drill images that have similar morphological
features. This is a valid proof of the learning ability of the PD-GAN under labelled conditions,
showing that our generator can avoid memory training samples.
Figure 13 shows hand drill images generated from the PD-GAN model. It is worth noting
that these hand drill images were generated without any labelling conditions. From the
perspective of visual layer perception, we found that some of the newly generated hand
drill images result from PD-GAN innovative synthesis after learning real product samples.
Figure 14 shows an example of bicycle helmet designed by PD-GAN without any Kansei
labels. However, there are also new samples that memorise training samples. The main rea-
son for this result is the small number of some samples in the training sample. It is worth
emphasising that when training without labels, β is set to 0 and λ is set to 10.
We constructed a questionnaire (Figure 15) to verify whether the hand drills (Figure 12)
produced by PD-GAN matches the relevant Kansei label and the level of form innovation.
Eight professional designers, eight experts and eight graduate students from industrial
18 X. LI ET AL.

Figure 12. Generated hand drills with random noise and Kansei labels. First column: Unaesthetic.
Second column: Practical. Third column: Powerful. Fourth column: Ergonomic.

design were invited to finish the questionnaires. By sorting the data from questionnaires,
and calculating the averages, we obtained the results shown in Figure 16.
For result 1, the first column in Figure 12 had the lowest average score (3.7) on the
‘Unaesthetic-Appealing’ dimension, which is consistent with the Kansei label (Unaesthetic)
of the first column in Figure 12. The second column has the highest average score (5.5) on
‘Unpractical-Practical’ dimension, which is consistent with the Kansei label (Practical) of the
second column in Figure 12. The same consistency also happened in the third and fourth
columns of Figure 12.
For result 2, the total average of form innovation is 4.4. The average values of the four
Kansei categories are 4.2, 4.1, 4.6 and 4.5, which shows that the overall form innovation
difference was not obvious. Although the differences between the means for each category
were small, this fluctuation indicates that the PD-GAN has a different form of innovation
capabilities in different Kansei preference categories.
 JOURNAL OF ENGINEERING DESIGN 19

Figure 13. Generated hand drills with random noise. Left: 256 × 256 pixels. Right: 128 × 128 pixels.

Figure 14. Generated Bicycle Helmets without affective preference labels.

Comparing Result 1 and Result 2, we found that the second conceptual design in Figure
12 had the lowest score for the six Kansei preferences. However, the form innovation
score of 4.1 was close to the middle. This apparent difference indicates that the second
20 X. LI ET AL.

Figure 15. The question corresponding to the concept image of the fifteenth hand drill.

Figure 16. New concept image evaluation results. (a) Result 1: Score results of Kansei preferences (b)
Result 2: Score results of form innovation.
 JOURNAL OF ENGINEERING DESIGN 21

conceptual design is the result of the PD-GAN model’s imbalance in the game between
Kansei preferences and innovation. However, this is not evident in our model.
We also did a simple test just to check whether the product images generated by PD-
GAN are instructive for industrial designers. First, two industrial designers were invited to
view the product images generated by PD-GAN. Then, they were asked to draw some quick
sketches, as shown in Figure 17. Finally, we learned their feelings through interviews. The
core points as follows.
Although these images still look a little fuzzy, and the details are not clear, the forms created
by artificial intelligence have created more imagination for designers. Because design usually
starts with fuzzy concepts, it is wonderful that these concepts suddenly appear before you.

Therefore, we believe that the combination of deep learning and Kansei engineering can
generate product designs and stimulate the innovation of industrial designers.

3.4. Product style fast transfer

3.4.1. PS-FNSTN architecture
The fast-neural style model’s architecture is shown in Figure 18, which is a modification from
(Johnson, Alahi, and Fei-Fei 2016). The model contains three parts: encoder (composed of
down sampling convolutional layers), deep residual module (12 ResBlocks and a skip con-
nection ResBlock) and decoder (composed of up sampling convolutional layers). We use
GELUs (Hendrycks and Gimpel 2016) as the activation function layer.

3.4.2. Network training and results

We used the HD dataset and the MS COCO (2014) (Lin et al. 2014) dataset for content images
to train our transform network fw. The former was used to train a style transfer model
for hand drills, and the latter was used to train a style transfer model for bicycle helmets.
There are 80,000 samples in the training data of MS COCO. Although MS COCO has fewer
categories than ImageNet and SUN, it has more instances of each category, more like life
photos. We think this is more suitable for transfer learning. We used the AdamW optimiser
(Loshchilov and Hutter 2017) with default parameters setting (β1 = 0.9 and β2 = 0.999).
We used a batch size of eight content-style image pairs. The learning rate is initialised to
1e − 3. Content weight and style weight are 1e5 and 1e10, respectively. The total variation
weight γ = 0. In the pre-processing, we used TorchVision’s Scale and CenterCrop tools to
standardise the image to 256 × 256 or 128 × 128. However, our transform network fw does
not have any restrictions on the size of the image during the test because it is a fully convo-
lutional neural network (Figure 18). For all product style transfer experiments, we compute
content loss at layer relu2_2 and style loss at layers relu1_2, relu2_2, relu3_4, relu4_4 and
relu5_4 of the VGG-19 loss network φ (Figure 6).
Partial results are shown in Figure 19. We inputted the content images (Figure 19(a)) and
style images (Figure 19(b)) into our style transform network model to obtain new prod-
uct style images (Figure 19(c)). For the bike helmet results, we removed the background
colour, while the hand drills have no background colour. We found no style image suitable
for product style transfer design in our style transfer experiments. We needed to choose the
right style image according to the shape and function of the product. Otherwise, the result
is not satisfactory, as shown in Figure 20. Because the product image and style image have
22 X. LI ET AL.

Figure 17. Quick sketches drawn by two industrial designers. (a): hands drill sketches were drawn by
designer A. (b): bicycle helmet sketches were drawn by designer B.

Figure 18. Fast neural style transfer architecture.

 JOURNAL OF ENGINEERING DESIGN 23

Figure 19. The results of product style transfer based on Fast Neural Style Architecture. (a) The content
images. (b) The style images. (c) The generated results.

too much loss in content. Besides, fabric and leather products are more suitable for style
transfer, which explains why literature (Quan, Li, and Hu 2018) chose women’s coats as a
research case.
In addition, the training time, and the number of parameters for the above three deep
neural network models (i.e. PAR-ResNet, PD-GAN and PS-FNSTN) are shown in Table A1.

4. Discussion
According to the literature (Nagamachi 1995), (Quan, Li, and Hu 2018) and (Wang et al.
2016), there is always a fatal flaw (cannot directly generate products) in product concept
24 X. LI ET AL.

Figure 20. The results of product style transfer based on Fast Neural Style Architecture. (a) The content
images. (b) The style images. (c) The generated results.

design based on Kansei engineering. Although the genetic algorithm has been used for
product design generation (Hsiao and Tsai 2005; Lo, Ko, and Hsiao 2015), its design space
is limited. It is challenging to deal with complicated products like cars and hand drills.
Industrial designers still need to deal with specific design implementations based on expe-
rience in early design development. Therefore, it is necessary to develop an effective design
method to directly generate product conceptual images based on the users’ and customers’
affective preferences. A product concept generation approach framework based on deep
learning and Kansei engineering (PCGA-DLKE) is proposed in this article to automatically
recognise affective preferences, to generate designs and to quickly transfer product styles
directly.
Comparing the generated results in Figures 12 and 13 from the visual perception level,
the results with Kansei preferences in Figure 12 are more beneficial to inspire designers to
manoeuver the affective design direction of the product and help improve design efficiency
compared to the randomly generated results in Figure 13. The evaluation results in Figure
16(a) show that there is still Kansei error in the hand drill generated under the same Kan-
sei label, but the average of the six Kansei dimensions indicates that PD-GAN has a strong
design generation capability. The results in Figure 19 show that style transfer can modify
the results generated by PD-GAN and give new style preferences to the results randomly
generated by PD-GAN to meet the emotional preferences of consumers.
The advantages of the PCGA-DLKE are described below. Most of the previous works use
traditional approaches for product innovation design. Still, it can only provide theoretical
guidance for industrial designers and cannot directly generate visible product forms, which
 JOURNAL OF ENGINEERING DESIGN 25

is a fatal flaw in industrial product form design. Although the parametric design based on
script technology can directly create product forms, this is only the result of the creation of
algorithms, and it has no ‘learning’ process. For the proposed PCGA-DLKE, we build a prod-
uct form generation system based on deep learning and Kansei engineering techniques,
including product images data acquisition methods, product affective preference recog-
nition, automatically generate product images, and product style transfer. As a result, this
method constructs a relatively complete generation design system, which makes up for the
lack of Kansei engineering and cannot directly generate product conceptual images, while
providing a more effective design tool for industrial designers. Therefore, comparing to tra-
ditional approaches and digital approaches, the proposed approach not only can maximise
user satisfaction, but more importantly, the ability to create product concepts is far superior
to previous approaches, greatly improving design efficiency. The case study result shows
that PCGA-DLKE is feasible and reasonable for product design generation. However, there is
one thing to be reminded of in product style transfer: the results shown in Figure 20 prove
that no style image can be used for product style transfer design. At percent, we do not
think this is directly related to the architecture and performance of the transform network,
mainly due to the functions and attributes of the product itself.
Besides, comparing Figures 13 and 14 at the level of visual perception, we found that
most of the hand drill concept images generated by PD-GAN are better than the bicycle
helmet images it generates. This is because the morphology of bicycle helmets is inher-
ently more complex. It is more difficult for PD-GAN to learn the data distribution patterns
of bicycle helmet images than the hand drill dataset. Furthermore, we found that when
training large pixel datasets in our experiments, the residual blocks in the PD-GAN can be
removed, which helps to save memory and shorten the training time. Style transfer is the
deconstruction and reconstruction of two image features by a deep convolutional neural
network. Still, for product design sketches, it requires the industrial designer to give more
consideration to the gains and losses of the product’s morphological details.

5. Conclusions
Kansei appeal is of critical importance to customer-centric product designs in a competi-
tive market environment. This creates an ongoing challenge for industrial designers as they
need to understand the affective preferences that influence users and consumers and comp
up with innovative design concepts as quickly as possible.
In this research, we propose PCGA-DLKE, a deep learning-based product concept gen-
eration design framework for an industrial designer. Firstly, we use a web crawler to obtain
product image data and propose a simple and effective image preprocessing method. Sec-
ondly, we construct a tag estimation model to recognise the Kansei label of the product
image based on Kansei engineering and deep residual networks. Thirdly, we train a prod-
uct design GAN model to generate new designs. Finally, we construct a fast-neural style
transfer model to create a new style for the previous step. Taking the hand drill and bicycle
helmet as examples, we demonstrate the effectiveness and feasibility of PCGA-DLKE. While
deep learning-based design method has been used to create anime characters before (Jin
et al. 2017; Mattya 2015), our study is the first to combine deep learning with Kansei engi-
neering to assist industrial designers in creating product concept images, which lays a solid
foundation for deep learning-based product design.
26 X. LI ET AL.

Deep learning provides a new interface for human–machine co-design, and our pro-
posed approach is an exploration of the human–machine co-design approach. This
approach makes up for the shortcomings that Kansei engineering cannot directly gener-
ate designs and provides a new innovative design paradigm for industrial designers. The
essence of PD-GAN is to use high-dimensional random noise data and Kansei labels to
approximate and simulate distribution patterns that approximate a large amount of real
product image pixel data, which is why PD-GAN creates new concepts as well as new colour
schemes. Our results show that this research framework is indeed able to predict users’
affective preferences, and generate innovative product conceptual images and new style,
which can effectively help industrial designers break for design fixation. Besides, we show
a visual interpretation of the Kansei attributes of PD-GAN-generated conceptual images.
This paper integrates emotion recognition, concept generation and style transformation
into an effective intelligent design methodology, which is of high value in a real design envi-
ronment, not only to improve affective design efficiency, but also to stimulate the potential
of designers. The combination of deep learning and Kansei engineering provides an end-to-
end intelligent design approach that meets the emotional preferences of the user. This also
indicates that designers will have to play the design game with intelligent design systems
in the future. That is human–machine design game. This game is reflected in the fact that AI
is driven by data and algorithms, while human designers are driven by design knowledge
and experience. Artificial intelligence, such as GAN, will help us discover the underlying
patterns in human design outcomes. Conversely, the knowledge and experience of profes-
sional designers can compensate for the random creation of artificial intelligence. Soon,
AI will change the way industrial designers work, their habits of mind, and their design
processes.
Although our framework can automatically generate brand-new products with new
styles, there are still some drawbacks. For example, colour labels are overlooked in affec-
tive recognition, our data preprocessing capabilities are not fully automated, Kansei labels
are obtained in a single way, and there is room for improvement in the generative adver-
sarial networks for product design. Besides, the size of the dataset also affects the quality of
the generated image. Therefore, in the future, we will focus on improving the data prepro-
cessing capabilities, the accuracy in capturing the user’s affective preference, the structure
and loss function of generator to generate conceptual images with high affective prefer-
ences, and the scale of the extended dataset to improve our framework. This is some very
challenging and interesting work for us.
In the research of this project, we also found that there is still much more to explore
in industrial design assisted by artificial intelligence. In the cyber-physical network com-
puting environment of the Internet of Everything, Big Data, Artificial Intelligence, Cloud
Computing, and Cloud Services, AI will open another window for designers. In the future,
therefore, artificial intelligence will be applied to various fields as a general technology, and
it is naturally no exception in industrial design.

Acknowledgements
The author first thank own long-term interest in the design innovation and design drawing. Secondly,
the author thank to the open source of GAN, CGAN and DCGAN authors on GitHub, our ideas can be
implemented efficiently.
 JOURNAL OF ENGINEERING DESIGN 27

Disclosure statement
No potential conflict of interest was reported by the author(s).

Funding
This work was supported by National Natural Science Foundation of China [grant number 51465037].

References
Arjovsky, M., and L. Bottou. 2017, April 24–26. Towards Principled Methods for Training Generative
Adversarial Networks. Paper presented at the International Conference on Learning Representa-
tions (ICLR).
Arjovsky, M., S. Chintala, and L. Bottou. 2017. “Wasserstein Generative Adversarial Networks.” In
Proceedings of the 34th International Conference on Machine Learning, 214-223 PMLR.
Brock, A., J. Donahue, and K. Simonyan. 2019. “Large Scale GAN Training for High Fidelity Natural
Image Synthesis.” In. https://arxiv.org/abs/1809.11096v1: ArXiv e-Print.
Chai, C., J. Liao, N. Zou, and L. Sun. 2018. “A One-to-Many Conditional Generative Adversarial Network
Framework for Multiple Image-to-Image Translations.” Multimedia Tools and Applications 77 (17):
22339–22366. doi:10.1007/s11042-018-5968-7.
Chan, K. Y., C. K. Kwong, P. Wongthongtham, H. Jiang, Cky Fung, B. Abu-Salih, Z. Liu, T. C. Wong, and P.
Jain. 2020. “Affective Design Using Machine Learning: A Survey and its Prospect of Conjoining Big
Data.” International Journal of Computer Integrated Manufacturing, 33 (7) 645–669.
Chang, Y. M., and C. W. Chen. 2016. “Kansei Assessment of the Constituent Elements and the Over-
all Interrelations in Car Steering Wheel Design.” International Journal of Industrial Ergonomics 56:
97–105. doi:10.1016/j.ergon.2016.09.010.
Chen, H. Y., and Y. M. Chang. 2014. “Development of A Computer Aided Product-Form Design Tool
Based on Numerical Definition Scheme and Neural Network.” Journal of Advanced Mechanical
Design, Systems, and Manufacturing 8 (3): JAMDSM0033–JJAMDSM033..
Chen, L., P. Wang, H. Dong, F. Shi, J. Han, Y. Guo, P. R. N. Childs, J. Xiao, and C. Wu. 2019. “An Artificial
Intelligence Based Data-Driven Approach for Design Ideation.” Journal of Visual Communication
and Image Representation 61: 10–22. doi:10.1016/j.jvcir.2019.02.009.
Chen, C. F., C. H. Yeh, and Y. C. Lin. 2010. A Neural Network Approach to Eco-Product Form Design.
Paper presented at the 2010 5th IEEE Conference on Industrial Electronics and Applications, 15-17
June 2010.
Chou, J. R. 2016. “A Kansei Evaluation Approach Based on the Technique of Computing with Words.”
Advanced Engineering Informatics 30 (1): 1–15. doi:10.1016/j.aei.2015.11.001.
Dong, C., C. C. Loy, K. M. He, and X. Tang. 2014. Learning a Deep Convolutional Network for Image Super-
Resolution. Paper presented at the European Conference on Computer Vision (ECCV).
Eigen, D., and R. Fergus. 2015. Predicting Depth, Surface Normals and Semantic Labels with a Common
Multi-scale Convolutional Architecture. Paper presented at the 2015 IEEE International Conference
on Computer Vision (ICCV), 7-13 Dec. 2015.
Elgammal, A., B. Liu, M. Elhoseiny, and M. Mazzone. 2017. “CAN: Creative Adversarial Net-
works, Generating “Art” by Learning About Styles and Deviating from Style Norms.” In.
https://arxiv.org/abs/1706.07068v1: ArXiv e-Print.
Fung, C. K. Y., C. K. Kwong, K. Y. Chan, and H. Jiang. 2014. “A Guided Search Genetic Algorithm Using
Mined Rules for Optimal Affective Product Design.” Engineering Optimization 46 (8): 1094–1108.
doi:10.1080/0305215X.2013.823196.
Gatys, L. A., A. S. Ecker, and M. Bethge. 2015. “A Neural Algorithm of Artistic Style.” In.
https://arxiv.org/abs/1508.06576: ArXiv e-Print.
Goodfellow, I., J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio.
2014. “Generative Adversarial Networks.” In. https://arxiv.org/abs/1406.2661: ArXiv e-Print.
28 X. LI ET AL.

Grimsaeth, K., M. Gangwar, B. Baggerud, and R. Narain. 2010. “Kansei Engineering Based Emotional
Product Design: An Illustration by a Battery Drill Case-Study.” Kansei Engineering International
Journal 9 (2): 67–79. doi:10.5057/kei.9.67.
Guo, F., W. L. Liu, Y. Cao, F. T. Liu, and M. L. Li. 2016. “Optimization Design of a Webpage Based on Kansei
Engineering.” Human Factors and Ergonomics in Manufacturing & Service Industries 26 (1): 110–126.
doi:10.1002/hfm.20617.
Guo, F., W. L. Liu, F. T. Liu, H. Wang, and T. B. Wang. 2014. “Emotional Design Method of Product Pre-
sented in Multi-Dimensional Variables Based on Kansei Engineering.” Journal of Engineering Design
25 (4-6): 194–212. doi:10.1080/09544828.2014.944488.
Hadji, I., and R. P. Wildes. 2018. “What Do We Understand About Convolutional Networks?” In.
https://arxiv.org/abs/1803.08834: ArXiv e-Print.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. Deep Residual Learning for Image Recognition. Paper pre-
sented at the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 27-30
June 2016.
Hendrycks, D., and K. Gimpel. 2016. “Gaussian Error Linear Units (GELUs).” In. https://arxiv.org/abs/
1606.08415v4: ArXiv e-Print.
Hjelm, R. D., A. P. Jacob, T. Che, A. Trischler, K. Cho, and Y. Bengio. 2017. “Boundary-Seeking Generative
Adversarial Networks.” In. https://arxiv.org/abs/1702.08431: ArXiv e-Print.
Hsiao, Y. H., M. C. Chen, and W. C. Liao. 2017. “Logistics Service Design for Cross-Border E-Commerce
Using Kansei Engineering with Text-Mining-Based Online Content Analysis.” Telematics and Infor-
matics 34 (4): 284–302. doi:10.1016/j.tele.2016.08.002.
Hsiao, S. W., and H. C. Tsai. 2005. “Applying A Hybrid Approach Based on Fuzzy Neural Network and
Genetic Algorithm to Product Form Design.” International Journal of Industrial Ergonomics 35 (5):
411–428. doi:10.1016/j.ergon.2004.10.007.
Huang, Y., C. H. Chen, and L. P. Khoo. 2012. “Kansei Clustering for Emotional Design Using A Com-
bined Design Structure Matrix.” International Journal of Industrial Ergonomics 42 (5): 416–427.
doi:10.1016/j.ergon.2012.05.003.
Jiao, J., Y. Zhang, and M. Helander. 2006. “A Kansei Mining System for Affective Design.” Expert Systems
with Applications 30 (4): 658–673. doi:10.1016/j.eswa.2005.07.020.
Jin, Y., J. Zhang, M. Li, Y. Tian, H. Zhu, and Z. Fang. 2017. “Towards the Automatic Anime Characters
Creation with Generative Adversarial Networks.” In. https://arxiv.org/abs/1708.05509: ArXiv e-Print.
Johnson, J., A. Alahi, and L. Fei-Fei. 2016. “Perceptual Losses for Real-Time Style Transfer and Super-
Resolution.” European Conference on Computer Vision (ECCV), 694-711.
Karras, T., T. Aila, S. Laine, and J. Lehtinen. 2018. “Progressive Growing of GANs for Improved Quality,
Stability, and Variation.” In. https://arxiv.org/abs/1710.10196: ArXiv e-Print.
Karras, T., S. Laine, and T. Aila. 2019. A Style-Based Generator Architecture for Generative Adversar-
ial Networks. Paper presented at the 2019 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), 15-20 June 2019.
Khalid, H. M. 2006. “Embracing Diversity in User Needs for Affective Design.” Applied Ergonomics 37
(4): 409–418. doi:10.1016/j.apergo.2006.04.005.
Kim, T., M. Cha, H. Kim, J. Lee, and J. Kim. 2017. Learning to Discover Cross-Domain Relations with
Generative Adversarial Networks. Paper presented at the Proceedings of the 34th International
Conference on Machine Learning (ICML).
Kim, W., T. Ko, I. Rhiu, and M. H. Yun. 2019. “Mining Affective Experience for A Kansei Design Study on
A Recliner.” Applied Ergonomics 74: 145–153. doi:10.1016/j.apergo.2018.08.014.
Kingma, D. P., and J. L. Ba. 2015. “Adam: A Method for Stochastic Optimization.” In. https://arxiv.org/
abs/1412.6980: ArXiv Preprint.
Kodali, N., J. Abernethy, J. Hays, and Z. Kira. 2017. “On Convergence and Stability of GANs.” In.
https://arxiv.org/abs/1705.07215v5: ArXiv e-Print.
Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2017. “ImageNet Classification with Deep Convolutional
Neural Networks.” Communications of The ACM 60 (6): 84–90. doi:10.1145/3065386.
Kwong, C. K., K. Y. Fung, H. Jiang, K. Y. Chan, and K. W. M. Siu. 2013. “A Modified Dynamic Evolv-
ing Neural-Fuzzy Approach to Modeling Customer Satisfaction for Affective Design.” The Scientific
World Journal 2013: 1–11. doi:10.1155/2013/636948.
 JOURNAL OF ENGINEERING DESIGN 29

Lecun, Y., L. Bottou, Y. Bengio, and P. Haffner. 1998. “Gradient-Based Learning Applied to Document
Recognition.” Proceedings of the IEEE 86 (11): 2278–2324. doi:10.1109/5.726791.
Li, Z., Z. G. Tian, J. W. Wang, W. M. Wang, and G. Q. Huang. 2018. “Dynamic Mapping of Design Ele-
ments and Affective Responses: A Machine Learning Based Method for Affective Design.” Journal
of Engineering Design 29 (7): 358–380. doi:10.1080/09544828.2018.1471671.
Li, C., K. Xu, J. Zhu, and B. Zhang. 2017. Triple Generative Adversarial Nets.” In. https://arxiv.org/abs/
1703.02291: ArXiv e-Print.
Lin, T. Y., M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and
P. Dollár. 2014. Microsoft COCO: Common Objects in Context. Paper presented at the European
Conference on Computer Vision (ECCV), September 2014.
Liu, Q., K. Wang, Y. Li, and Y. Liu. 2020. “Data-Driven Concept Network for Inspiring Design-
ers’ Idea Generation.” Journal of Computing and Information Science in Engineering 20 (3): 1–12.
doi:10.1115/1.4046207.
Lo, C. H., Y. C. Ko, and S. W. Hsiao. 2015. “A Study That Applies Aesthetic Theory and Genetic
Algorithms to Product Form Optimization.” Advanced Engineering Informatics 29 (3): 662–679.
doi:10.1016/j.aei.2015.06.004.
Loshchilov, I., and F. Hutter. 2017. “Fixing Weight Decay Regularization in Adam.” In. https://arxiv.org/
abs/1711.05101v3: ArXiv e-Print.
Luan, F., S. Paris, E. Shechtman, and K. Bala. 2017. Deep Photo Style Transfer. Paper presented at the
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 21-26 July 2017.
Mao, X., Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley. 2017. Least Squares Generative Adversarial
Networks. Paper presented at the 2017 IEEE International Conference on Computer Vision (ICCV),
22-29 Oct. 2017.
Mattya. 2015. “Chainer-DCGAN.” In. https://github.com/mattya/chainer-DCGAN.
Mattya. 2017. “Chainer-Gan-Lib.” In. https://github.com/pfnet-research/chainer-gan-lib.
Mirza, M., and S. Osindero. 2014. “Conditional Generative Adversarial Nets.” In. https://arxiv.org/abs/
1411.1784: ArXiv e-Print.
Nagamachi, M. 1995. “Kansei Engineering: A New Ergonomic Consumer-Oriented Technol-
ogy for Product Development.” International Journal of Industrial Ergonomics 15 (1): 3–11.
doi:10.1016/0169-8141(94)00052-5.
Odena, A., C. Olah, and J. Shlens. 2017. Conditional Image Synthesis with Auxiliary Classifier
GANs. Paper presented at the International Conference on Machine Learning (ICML), August
2017.
Osgood, E. C., G. J. Suci, and P. H. Tannenbaum. 1957. The Measurement of Meaning. Urbana, IL:
University of Illinois Press.
Pan, Y., A. Burnap, J. Hartley, R. Gonzalez, and P. Y. Papalambros. 2017. “Deep Design: Product Aesthet-
ics for Heterogeneous Markets.” In Proceedings of the 23rd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, 1961–1970. Halifax, NS, Canada: Association for Computing
Machinery.
Ponce, P., D. Balderas, T. Peffer, and A. Molina. 2018. “Deep Learning for Automatic Usability Eval-
uations Based on Images: A Case Study of the Usability Heuristics of Thermostats.” Energy and
Buildings 163: 111–120. doi:10.1016/j.enbuild.2017.12.043.
Quan, H., S. Li, and J. Hu. 2018. “Product Innovation Design Based on Deep Learning and Kansei
Engineering.” Applied Sciences 8 (12): 2397–2317. doi:10.3390/app8122397.
Radford, A., L. Metz, and S. Chintala. 2016. “Unsupervised Representation Learning with Deep Convo-
lutional Generative Adversarial Networks.” In. https://arxiv.org/abs/1511.06434: ArXiv e-Print.
Razza, B., and L. C. Paschoarelli. 2015. “Affective Perception of Disposable Razors: A Kansei Engineering
Approach.” Procedia Manufacturing 3: 6228–6236. doi:10.1016/j.promfg.2015.07.750.
Self, J. A. 2019. “Communication Through Design Sketches: Implications for Stakeholder Interpreta-
tion During Concept Design.” Design Studies 63: 1–36. doi:10.1016/j.destud.2019.02.003.
Simonyan, K., and A. Zisserman. 2015. “Very Deep Convolutional Networks for Large-Scale Image
Recognition.” In. https://arxiv.org/abs/1409.1556: ArXiv e-Print.
30 X. LI ET AL.

Sutera, J., M. C. Yang, and C. Elsen. 2014. “The Impact of Expertise on the Capture of Sketched Inten-
tions: Perspectives for Remote Cooperative Design.” In International Conference on Cooperative
Design, Visualization and Engineering, 245-252.
Szegedy, C., W. Liu, Y. Q. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabi-
novich. 2015. Going Deeper with Convolutions. Paper presented at the 2015 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 7-12 June 2015.
Vieira, J., J. M. A. Osório, S. Mouta, P. Delgado, A. Portinha, J. F. Meireles, and J. A. Santos. 2017. “Kansei
Engineering as A Tool for the Design of In-Vehicle Rubber Keypads.” Applied Ergonomics 61: 1–11.
doi:10.1016/j.apergo.2016.12.019.
Wang, Y., D. Y. Mo, and M. M. Tseng. 2018. “Mapping Customer Needs to Design Parameters in
the Front End of Product Design by Applying Deep Learning.” CIRP Annals 67 (1): 145–148.
doi:10.1016/j.cirp.2018.04.018.
Wang, C. C., C. H. Yang, C. S. Wang, T. R. Chang, and K. J. Yang. 2016. “Feature Recog-
nition and Shape Design in Sneakers.” Computers & Industrial Engineering 102: 408–422.
doi:10.1016/j.cie.2016.05.003.
Yanagisawa, H., and S. Fukuda. 2005. “Interactive Reduct Evolutional Computation for Aesthetic
Design.” Journal of Computing and Information Science in Engineering 5 (1): 1–7. doi:10.1115/1.184-
6055.
Zhai, L. Y., L. P. Khoo, and Z. W. Zhong. 2009. “A Rough Set Based Decision Support Approach to
Improving Consumer Affective Satisfaction in Product Design.” International Journal of Industrial
Ergonomics 39 (2): 295–302. doi:10.1016/j.ergon.2008.11.003.
Zhou, Z., H. Cai, S. L. Rong, Y. Song, K. Ren, W. Zhang, J. Wang, and Y. Yu. 2018. “Activation Maximization
Generative Adversarial Nets.” In. https://arxiv.org/abs/1703.02000v9: ArXiv e-Print.

Appendices
Appendix 1. Data distribution of survey sample images.

Figure A1. Data distribution of affective preferences (840 samples)

 JOURNAL OF ENGINEERING DESIGN 31

Appendix 2. PAR-ResNet losses on six Kansei preference dimensions.

Figure A2. The losses of PAR-ResNet for six Kansei dimensions.

Appendix 3. The training time and the number of parameters of three

models (PAR-ResNet, PD-GAN and PS-FNSTN).

Table A1. The training time and the number of parameters of three models.
Max Number Training Trainable Total
Model Dataset epoch of GPU time (s) params params
1 PAR-ResNet HD dataset 50 Single avg. 11,180,546 11,180,546
245.46
2 PD-GAN Generator HD dataset 500 Double 96,511.18 52,969,344 52,969,344
(No ResBlocks) (256 × 256
Discriminator pixels) 11,019,744 11,019,744
(No ResBlocks)
Generator HD dataset 500 Double 47232.16 38,394,496 38,394,496
Discriminator (128 × 128 1,26,60,128 12,660,128
pixels)
Generator BH dataset 500 Double 41278.95 38,394,496 38,394,496
Discriminator (128 × 128 12,660,128 12,660,128
pixels)
3 PS-FNSTN MS COO 3 Double avg. 2,714,115 2,714,115
2014 14546.37
HD dataset 6 avg.
3948.42