Recognizing and Classifying Unknown Object

in BIM Using 2D CNN

Jinsung Kim, Jaeyeol Song, and Jin-Kook Lee(&)

Department of Interior Architecture and Built Environment,

Yonsei University, Seoul, Republic of Korea
[email protected], [email protected],
[email protected]

Abstract. This paper aims to propose an approach to automated classifying

building element instance in BIM using deep learning-based 3D object classi-
fication algorithm. Recently, studies related to checking or validating engine of
BIM object for ensuring data integrity of BIM instances are getting attention. As
a part of this research, this paper train recognition models that are targeted at
basic building element and interior element using 3D object recognition tech-
nique that uses images of objects as inputs. Object recognition is executed in
two stages; (1) class of object (e.g. wall, window, seating furniture, toilet fixture
and etc.), (2) sub-type of specific classes (e.g. Toilet or Urinal). Using the
trained models, BIM plug-in prototype is developed and the performance of this
AI-based approach with test BIM model is checked. We expect this recognition
approach to help ensure the integrity of BIM data and contribute to the practical
use of BIM.

Keywords: 3D object classification
Building element

Building information modeling
Data integrity
Interior element

1 Introduction

1.1 Research Objective

This paper aims to automatically recognize the information of an unknown object in
building information model (BIM) instances using a deep learning-based technique.
BIM and its applications have shown the various benefits in the architecture, engi-
neering, construction, and facility management (AEC/FM) industry. Practically exe-
cution of such BIM applications presupposes that the BIM model has the appropriate
data for each of specific domains [1]. However, it is difficult to ensure the BIM data
integrity due to error, mistake or omission of object information for various reasons
including; (1) technical problems resulting from the process of data conversion
between BIM tools, (2) Human error such as input from the writing process, and
(3) Utilization of dummy geometry objects created from non-BIM design tools or
obtained from the cloud points. Although a neutral format, the industry foundation
class (IFC), and various translation frameworks, the model view definition (MVD),
have been proposed to ensure interoperability of BIM data, tasks to input and validate

© Springer Nature Singapore Pte Ltd. 2019

J.-H. Lee (Ed.): CAAD Futures 2019, CCIS 1028, pp. 47–57, 2019.
https://doi.org/10.1007/978-981-13-8410-3_4
48 J. Kim et al.

the data should have been done by workers. For solving this problem, researches about
automatically recognizing and validation BIM data have been introduced. Bloch and
Sacks (2018) showed that a machine learning-based approach is more successful in
automatically classifying the name of spaces then rule-based approaches. Koo et al.
(2019) proposed automated recognizing category of basic building objects such as a
wall, column, slab, door and etc. using a machine learning-based classification tech-
nique. These researches utilize geometry data such as (width, height, volume and etc.)
and relationships between the object to the training machine learning model. However,
it is difficult to recognize the category of detailed and complex objects (e.g. Interior
elements and complex building fixtures) and various types of each category with only
low-level geometry data. This research aims to look for ways to recognize category and
detailed sub-types of various building objects using 2D-CNN-based approaches that
have demonstrated excellence in complex and diverse 3D object classification. The
visual information including a visual image (shape) and geometry data such as mass,
area, height, and width is less likely to be missing or an error and the domain experts
also rely on such information in validation task. The proposed mechanism includes the
deep learning-based CNN model and machine learning-based classification algorithm
using multi-view rendering image and geometry data of building object (Fig. 1).

Fig. 1. The objective of this research

1.2 Research Scope and Method

2D CNN model is the proven approach to effectively making computer train and
classify 3D CAD object using 2D captured images at various viewpoints. The target
classes of the object include not only general building objects such as walls, columns,
and stairs, but also interior object that may have more variations such as furniture, and
toilet fixture. There are two different scopes of classification. The first is a high-level
 Recognizing and Classifying Unknown Object in BIM Using 2D CNN 49

classification of the category. The second is a low-level classification of sub-type on

specific categories. The level of detail of the object covered in this paper is LOD 200
that basically could be visually classified. The flow of research is as follows:
(1) Preparing object data and multi-view images using image generator API we
developed, (2) Training the CNN models according to classification scopes,
(3) Demonstrating the prototyping plug-in of auto-classification on the BIM tool.
Through the application, the user could check undetermined objects in IFC instances
and override the class data automatically. It is expected to help practical operations for
further BIM applications.

2 Background

2.1 Researches on Ensuring Data Integrity of BIM Object

Researches on improving the accuracy of building information models and ensuring the
interoperability of BIM data has been studied. Eastman (2009) described that syntactic
pre-checking is required to determine that the BIM instance has the objects and their
properties required to execute a function in the application. It has been explained that the
inaccuracy in the information model derives from various reason causes. The most
critical cause is occurred in exchanging the BIM data between different BIM applica-
tions. The Industry Foundation Class (IFC) is the standard data format for BIM data
exchange. Most BIM software generally implements heterogeneous translators that bind
their native data format to the IFC format. This results in the syntactic errors of BIM data
exchanged [2]. The other cause is related to a human error in the BIM authoring and
exchanging process. The probability of human mistakes and omissions become stronger
according to increasing the size and complexity of the BIM models [3].
Researches on the mapping framework for BIM and IFC have been conducted.
Model view definition (MVD) is the most popular approach and the generic definition
of MVD is “the standard methodology and format documenting the software imple-
mentation requirements for standard IFC based data exchange” [4]. MVD is IFC sub-
schemas that represents interoperable requirements data for applications of specific
domains, so helps software vendors develop translator of IFC import and export [5].
Solibri Model Checker (SMC) [6] which is known as the most popular rule-checking
software has pre-checking module such as name and properties conventions, object
existence and others.
However, these tools and approaches can’t validate directly the correctness of
properties in building information models such as name, class or type of object.
Recently, studies on validating BIM data and inferring missing information using a
third “engine” have been introduced. The logic rule-based approach is common and
well known. Sacks, et al. (2017) described a logic rule-based approach to object
classification using single-object features and topological relationships between pairs of
objects [7]. However, the logic rule-based approach has a limitation on generalization
because every building object has different features and different rules are needed to set
respectively. Unlike this deductive approach, Machine learning-based approach to
classification of building element was presented as an alternative method that can
50 J. Kim et al.

effectively classify complex object with no specific rules [8]. Koo, et al. (2019) pro-
posed the machine learning-based approach to classifying 8 different IFC class of
object (wall, slab, door, window, railing, covering, column and beam) in building
information model instance [9]. Geometry information (e.g. width, height, length,
volume and etc.) and some representative relationship information (e.g. relation with
the wall, door and window) is used as an input value, and support vector machine
algorithm was utilized to train “ifc class classifier”. The approach showed 94.39%
accuracy of classifying the basic building element in the test phase. These results show
that the machine learning approach can successfully inductively infer the class infor-
mation without specific logic rule. Thus, it is shown that there are differences between
geometric information and relationship information for each BIM object, and it is
possible to create a model that can learn the differences and recognize them. However,
this approach can be applied to common building objects, but different approaches may
be needed for interior objects that may be similar in size and relationship. For example,
how can you distinguish between a sofa and a toilet using the geometry data? People
classify these by visual differences rather than geometric information. As a part of these
inductive approaches to recognizing building element, this paper explores a different
approach using the deep learning based image recognition techniques, which has
shown human-level performance.

2.2 3D Object Classification Approaches

Classification of a 3D object is one of the main researches subject in the computer
vision field. The Princeton ModelNet [10] is the representative project with well-
structured collections of generic 3D CAD files labeled with 10–40 different classes.
This dataset is commonly used for developing a 3D object classification model and
validating the accuracy of the model. The study of 3D object recognition can be
broadly divided into what features to target and what algorithm or methods to target to
create a classification model. Recently, machine learning or deep learning approaches
have shown more successful performance in terms of accuracy and efficiency than rule-
based approaches. Especially, Deep Learning-based approaches that utilize voxel data
or object images data are drawing attention.
The voxel-based approach generally utilizes the n  n  n grid of voxels that is
usually given by the intersection of 3D CAD models as input. On the other hand, the
image-based approach utilizes the rendering images of 3D CAD models as input. Like
human generally recognizes a 3D object with the composition of 2D images, the image-
based approach models have shown as better performance then models trained with the
voxel data [11]. It may result from the relative efficiency of 2D and 3D representations
[12]. Convolutional neural network (CNN) has been known as the most powerful deep
learning algorithm to train the computer model to recognize and classify 2D image
[13]. A CNN model [12] with single view images showed higher accuracy (85.1%)
than one (77.3%) of a model that trained voxel data [14]. Su, et al. (2015) compared the
single-view and multi-view models and showed that the multi-view models are aver-
agely more accurate by 5%. In this paper, Multi-view based CNN approach is applied
to a training model to classify building object.
 Recognizing and Classifying Unknown Object in BIM Using 2D CNN 51

We do not train the model from scratch, but fine-tuning Inception-v3 [15] that
trained ImageNet [16] data on TensorFlow framework [17]. The main feature of the
Inception-v3 model is the inception modules that includes multiple convolution layers
and pooling layer. It is confirmed that the inception-v3 shown a better performance
than the VGG model. It is beyond the scope of this study to describe the structure and
detailed operating principles of the inception-v3 model, so this paper only explains how
the model is utilized (Fig. 2). Generally, the architecture of the CNN model can be
divided into two parts: the image feature extraction part and classifier part. We reuse
the image feature extraction part of the inception-v3 and then retrain the 4 different
classifiers regarding tasks. As a result, 4 different models that recognize building
element categories, sub-type of the door, seating and toilet are trained.

Fig. 2. The pipeline of utilization of pre-trained Inception-v3 model [15] to retrain the building
elements and sub-types

3 Auto-Recognition of Category of Building Element Using

2D-CNN

3.1 Preparation of Data of Building Element

The objects covered in this study include the basic building elements and specific
interior elements that make up the building. Information about architectural objects is
basic and essential, and information about more specific objects, such as interior fur-
nishing, is increasingly required. For example, information of more detailed building
element is required for a detailed code compliance checking (such as the Installation
Standards for Public Toilets, etc.) Specific information such as sub-type of the door
(e.g. single swing, double swings, rotation and etc.) is required. Therefore, this paper
aims to recognize the categories as well as sub-types of some building elements.
Building object data for training consist of the BIM files as well as the simple CAD
files such as .dwg, .skp and .3 dm files. The object data being studied in the paper were
52 J. Kim et al.

collected from a variety of BIM libraries [18, 19] as well as from a simple CAD library
[20] without object information. 80% of the data collected is used for training and 20%
for validation.

Fig. 3. A module to generate multi-view images of BIM object on BIM tool

Data of the building objects used in this paper consists of 2D rendering bitmap
image data and geometric text data (width, depth, height, and volume). First of all, the
rendering image of an architectural object means the multi-view images that are gen-
erated from various angles of each object. For automatically creating the rendering
images of selected objects, the multi-view images generator module was developed as a
plug-in tool for using the API [21] of BIM authoring software. To obtain images of
different angles, the sets of horizontal angle are preset in 45° from 0 to 315°, the sets of
vertical angle in 30° from 0 to 90°, and 32 images can be created per object (Fig. 3).
This is used to build an image DB for learning and to create an object cache image to
be used in the inference process. Collecting the geometry data of building objects is
also conducted by using API of BIM tool. The volume and minimum bounding box of
the object are calculated. The bounding box is utilized to calculate the width, depth,
and height.

3.2 2D-CNN-Based Training a Model of 3D Object Recognition

In this paper, two types of recognition models are addressed. The first model aims to
recognize the category of building element, while the second model aims to recognize
the sub-type of some building elements. For example, if an element is recognized as a
door, it can be re-recognized whether the door is a single wing, a double swing, or a
revolving type. This approach can give the possibility to validate the more detail
information of BIM data. The target building objects and sub-types to be covered are
shown in Fig. 4. The configuration and name of categories are referred to IFC entity
and Omniclas [22]. As the target of first-stage, the category of building element
includes slab, wall, ceiling, columns, a staircase that can be described basic building
 Recognizing and Classifying Unknown Object in BIM Using 2D CNN 53

elements and only toilet fixtures that are considered as the interior element. As the
target of second-stage, sub-types of door and toilet fixture are covered. The training of
these recognition models is processed by fine-tuning the Inception-v3model that train
the ImageNet dataset and show the successful performance. The training process is
conducted in the TensorFlow environment.

Fig. 4. An abstraction of the building element

4 Evaluation of Recognition Module and Prototype

Application for Classification

4.1 Evaluation of Recognition Module with Test BIM Model

The training result can be examined with quantitative values such as accuracy of
classification and cross entropy value. Accuracy shows the ratio of the overall classi-
fication data to the label information that matches the classification results, i.e. to the
correct classification. Cross-entropy, on the other hand, shows the value of the lost
function in the classifier learning of this model, the closer the function is to zero, the
total number of objects for which the function is intended, the result is for training and
validation, respectively. Tensorboard which is the visualization tool of training on
TensorFlow allows drawing a graph about changing the value of accuracy and cross
entropy of training and validating session respectively. In training the deep learning
model, it is important to avoid seeing models over-fitting and under-fitting. In the case
of under-fitting, it means that the model is more possible to train. The under-fitting
situation can be confirmed by a graph that the price of cross entropy continues to fall.
This means that the more training steps are needed. This can be checked with the value
54 J. Kim et al.

of Cross entropy is continuously falling. On the other hand, overfitting refers to cases in
which training is classified accurately, but it is classified incorrectly in validating. This
situation can be also confirmed by a graph that the cross-entropy value of training is
falling, but the cross-entropy value of validation appears to be stagnant or even rising.
In this paper, two stages of training model are executed. The first stage is related to
classification of category types of building element (e.g. wall, column, slab, window
and etc.). The other stage is related to classification of sub-type of a specific element
(e.g. double, single or revolving door). The first stage training set and the result are
shown in Table 1. The training was executed with a different number of data by
categories, considering the visual variance and redundancy of objects. The accuracy of
training is 96.0%. The second stage training set and results are shown in Table 2.

Table 1. Building element categories training and validating set and the training accuracy
Categories Training set Validation set Total Accuracy of training
Column 48 12 60 96.0%
Wall 200 48 248
Slab 32 8 40
Stair 32 8 40
Door 84 21 105
Window 90 25 115
Seating 160 40 200
Toilet fixture 64 16 80

Table 2. Sub-type training and validating set and the training accuracy
Categories Sub-type Training set Validation set Total Accuracy of training
Door Single 32 8 105 97.4%
Double 32 8
Revolving 20 5
Seating Chair 48 12 200 94.8%
Office chair 48 12
Sofa 32 8
Stool 32 8
Toilet fixture Toilet 32 8 80 97.9%
Urinal 32 8

4.2 Demonstration of Prototype Plug-in Application for Auto-

Classification of Unknown Object in BIM
We developed a prototype plug-in on the Revit tool that automatically recognizes and
input the result values of category or sub-type information of unknown objects using
the trained models. The recognizing engine was developed in Python language, and is
 Recognizing and Classifying Unknown Object in BIM Using 2D CNN 55

called on Revit interface that was developed using Revit’s C # API. Through this plug-
in, it can be conducted to infer category of unknown dummy geometry objects. These
kinds of objects are generally recognized as generic models in Revit. It is possible to
change the entity of BIM object (e.g. stairs, lighting fixture, walls and etc.) additionally
Therefore, this plug-in inputs the result value of recognition into the general field such
as comments. The results of recognition are expressed in the general name of each
category and sub-type, but lately, it can be replaced with the IFC entity or Omniclass
title or number.
Figure 5 shows examples of application execution for stair and toilet objects
modeled in the 3D CAD tool. They are the dummy geometry object so there are not
category data. The recognition engine recognized these objects. The object looking like
a stair was recognized as a stair with 99.3% of the probability and the result value was
input into the comment field of the object. Another object looking like a toilet was
recognized as a toilet with 97.8% of probability and the result value was input into the
comment field.

Fig. 5. Demonstration of Prototype Plug-in application with test dummy geometry objects

5 Conclusion

This paper proposed an approach to automated classifying building element instance

according to category and sub-type using 2D CNN-based 3D object classification
technique Training of object recognition was executed in two stages; The first stage
training results showed the 96.0% accuracy of classification of building elements
56 J. Kim et al.

(column, stair, slab, wall, door and window) and interior elements categories (seating
furniture and toilet fixture). The second stage training result showed the average 96.7%
accuracy of classification of sub-type of door, seating and toilet fixture. Unlike previous
literature, this paper showed the possibility of recognizing interior element that has
more complex geometry then the basic building element with 2D convolutional neural
networks model. The number of target elements and volume of data is very limited, but
this approach shows the possibility of generalization and extension to other building
element that is needed to be recognized. The prototyping plug-in of auto-classification
on the BIM tool using independent python-based recognition engine is demonstrated.
Through the application, the user could quickly check undetermined objects in BIM
instances and override the category and sub-type data automatically. Since the
recognition engine is independent of the BIM tool, it has the advantage that it can be
easily attached to a variety of tools that have a 3D object rendering engine. In future
research, this approach and result will be used to pre-check IFC object data for IFC-
based design review application such as a code compliance.

Acknowledgement. This research was supported by a grant (19AUDP-B127891-03) from the

Architecture & Urban Development Research Program funded by the Ministry of Land,
Infrastructure and Transport of the Korean government.

References
1. Eastman, C., et al.: Automatic rule-based checking of building designs. Autom. Constr. 18
(8), 1011–1033 (2009)
2. Lee, Y.-C., et al.: Validations for ensuring the interoperability of data exchange of a building
information model. Autom. Constr. 58, 176–195 (2015)
3. Koo, B., Shin, B.: Applying novelty detection to identify model element to IFC class
misclassifications on architectural and infrastructure building information models. J. Comput.
Design Eng. 5(4), 391–400 (2018)
4. IFC Solutions Factory: Model View Definitions Site. http://www.blis-project.org/IAI-MVD.
Accessed 29 Mar 2019
5. Weise, M., et al.: Integrating use case definitions for IFC developments, eWork and
eBusiness in architecture and construction, pp. 637–645 (2009)
6. SMC: Solibri Model Checker. http://www.solibri.com. Accessed 29 Mar 2019
7. Sacks, R., et al.: Semantic enrichment for building information modeling: procedure for
compiling inference rules and operators for complex geometry. J. Comput. Civ. Eng. 31(6),
256–272 (2017)
8. Bloch, T., Sacks, R.: Comparing machine learning and rule-based inferencing for semantic
enrichment of BIM models. Autom. Constr. 91, 256–272 (2018)
9. Koo, B., et al.: Using support vector machines to classify building elements for checking the
semantic integrity of building information models. Autom. Constr. 98, 183–194 (2019)
10. The Princeton ModelNet. http://modelnet.cs.princeton.edu/. Accessed 29 Mar 2019
11. Maturana, D., Scherer, S.: VoxNet: a 3D convolutional neural network for real-time object
recognition. In: 2015 IEEE/RSJ International Conference Proceedings of Intelligent Robots
and Systems (IROS), pp. 922–928 (2015)
12. Su, H., et al.: Multi-view convolutional neural networks for 3D shape recognition. In:
Proceedings of the IEEE International Conference on Computer Vision, pp. 945–953 (2015)
 Recognizing and Classifying Unknown Object in BIM Using 2D CNN 57

13. Krizhevsky A., et al.: ImageNet classification with deep convolutional neural networks. In:
Proceedings of Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
14. Wu, Z., et al.: 3D ShapeNets: a deep representation for volumetric shapes. In: Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1912–1920 (2015)
15. Szegedy, C., et al.: Rethinking the inception architecture for computer vision. In:
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
pp. 2818–2826 (2016)
16. Deng, J., et al.: Imagenet: a large-scale hierarchical image database. In: Proceedings of
Computer Vision and Pattern Recognition, pp. 248–255 (2009)
17. Abadi, M., et al.: Tensorflow: large-scale machine learning on heterogeneous distributed
systems, arXiv preprint arXiv:1603.04467 (2016)
18. KBIMS Library. http://www.kbims.or.kr/lms. Accessed 29 Mar 2019
19. BIMObject: BIM Libarary. https://www.bimobject.com/. Accessed 29 Mar 2019
20. 3Dwarehouse. https://3dwarehouse.sketchup.com/. Accessed 29 Mar 2019
21. Revit and Revit API. https://www.autodesk.co.kr/products/revit/overview. Accessed 29 Mar
2019
22. Omniclass: a classification system for the construction industry. http://www.omniclass.org
Accessed 29 Mar 2019