SPECIAL SECTION ON URBAN COMPUTING AND INTELLIGENCE

Received October 1, 2019, accepted October 26, 2019, date of publication November 4, 2019, date of current version November 19, 2019.
Digital Object Identifier 10.1109/ACCESS.2019.2951294

Urban Commerce Distribution Analysis Based

on Street View and Deep Learning
NANQI YE 1, BOWEN WANG2 , MICHIHIRO KITA1 , MING XIE3 , AND WENYUE CAI2
1 Division of Global Architecture, Graduate School of Engineering, Osaka University, Suita 565-0871, Japan
2 Division of Medical information, Graduate School of Medicine, Osaka University, Suita 06-6105-6074, Japan
3 Navigation College, Dalian Maritime University, Dalian 116026, China

Corresponding author: Bowen Wang (wangbowenjisuanji@[Link])

ABSTRACT Urban commerce and its distribution have always been an important part of urban
research. However, most previous studies were based on statistical data and did not reflect real street
experience. Thanks to the Street View image and deep learning technology, researchers are able to carry
out large scale studies from real human visual experience. In this article, we aim at sensing the commercial
spaces in cities. In order to achieve this ultimate goal, deep learning is applied to process the raw data of
Street View image. We disassemble the goal into three tasks: firstly, obtaining all the Street View images
in a specific area; then classifying the Street View images according to the commercial facilities in it; and
finally creating a visualization of the detected data into a map. For the first task, we get the road network
coordinate information from the openstreetmap (OSM) website, set the sampling point on the road, and then
download the Street View images of the sampling points’ coordinate through the API provided by Baidumap.
For the second task, we adopt a two-level learning strategy rather than directly using Deep Convolutional
Neural Network for classification. For the final task, we choose the heat map as the expression of the results
and draw the map by the existing GIS software. Furthermore, the results from this study can be conveniently
combined with other data because of the use of street-network-based data structure. An application of this
method combines with street-network data, the calculation of a city’s 15-minute commercial service circle
coverage is also shown in this study.

INDEX TERMS Smart cities, urban commerce, deep learning, urban planning, street view image, object
detection.

I. INTRODUCTION Currently, the most common used data and tool for com-
Commerce is one of the most important feature of cities. mercial distribution analysis is Point-of-interest (POI) data
It can reflect many aspects of a city, such as economic and and Geographic Information System (GIS) software [6]–[9].
living conditions. For instance, Jacobs pointed out that small POI means a specific point location that someone may
business is an important node of street vitality [1]; different find useful or interesting in online maps, which often
researchers have proved the positive role of commerce for stand for stores, museums, hotels and so on [10]. However,
the community by different perspectives [2], [3]. Moreover, we face some imperfections in practices while using POI
the distribution of urban commercial space has always been data: (1) some small shops are not labeled as POI points;
an important part of commerce studies. Such researches have (2) many POI points suffer from wrong geographical coordi-
provided basic references for urban researchers, planners, nates; (3) the difference in POI data from different sources
policy makers and business sectors, especially retail compa- is significant; (4) the POI date are points on a Point on a
nies, to understand the condition of cities. As a kind of raw two-dimensional surface and do not contain enough informa-
data, it can also be combined with other data to conduct vari- tion to reflect the real world; (5) at the microscopic scale, POI
ous urban studies, such as wild birds distribution research [4], data is difficult to match with specific roads, and therefore
mechanism of urban development [5], etc. causing confusion for planners and policy makers to use it
accurately.
The associate editor coordinating the review of this manuscript and Recent years, the online map services such as Google Map
approving it for publication was Shiwen Mao . and Baidu Map [11], [12] provide an alternative data of POI

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
VOLUME 7, 2019 162841
 N. Ye et al.: Urban Commerce Distribution Analysis Based on Street View and Deep Learning

in their public GIS databases. They are called the Street the large-scale research which is difficult to achieve with
View image. They provide a service that virtually displays traditional methods.
the surrounding environment, with millions of panoramas
on the streets all across the city. In major cities, the Street II. RELATIVE WORKS
View service almost covers every single street. Thanks to the In the early stage, most of the urban business distribution
development of deep learning and public GIS databases, now researches are discussing the patterns of urban commer-
we have the raw data (Street View image API) and tools (deep cial distribution based on modernist urban planning ideas
learning) to carry out studies based on large scale Street View [20], [21]. Later, with the standardization and dataization
image. There have already been researches based on this data of urban management, there are studies based on regional
and method [13], [14]. However, there has not been any study statistical data with maps [22], of which the methodology is
on commercial space that is carried out by this method. advanced at that time. However, such research work is usually
In this paper, we aim at analyzing urban commerce distri- time-consuming and at low-precision, and not sufficient as
bution by Street View images. We collect the street network in a basis for specific construction and policy [Link] is
the range of central urban areas and download the street view because that most of the data is counted in unit of single
images via an open web API. For three cities in China (Hefei, administrative districts, and the granularity is too large for
Nanjing and Shanghai), we download around 400,000 Street practical reference. In recent years, with the growth of online
View images. All these images are used for analyzing. All map services, POI data has become a new raw data for
the images are classified by the commercial facility types. urban spatial distribution research [7]–[9]. These studies use
However, it is difficult to discriminate their categories by sim- POI data to perform detailed analysis of the distribution of
ply using the DCNN models (like in Liu et al.’s paper [13]). commerce in different cities. Moreover, Yang, et al. (2019)
To solve this problem, we choose signages as our initial iden- also try to fix the problem that POI data could not match with
tification target and analytical reference. The signages, which the street network. In our paper, we use Street View image
often stand for shops [15], play significant roles in modern based on street network to solve the problem of mismatch,
commerce. The work by Tsai et al. [16] shows that finding and realize the sensing of commerce distribution base on the
the signages of shops using Street View images is possible. feelings of human vision.
Therefore, we design a two-level learning method for image Urban researches based on deep learning and Street View
classification. In the first step, we choose an algorithm which image have gained more and more attentions in recent years.
is based on a regional convolutional neural network (faster- Researchers used Street View images to study about the
RCNN [17]) for object detection of signages. Faster-RCNN cities and communities as early as 2011 [23]. In this study,
performs well on some common data set (PASCAL VOC the artificial evaluation method is used to compare the sim-
[18], COCO [19]). For each image, detected signages will ilarity between street view photos and field surveys, and the
constitute a distribution. The characteristic of signages distri- feasibility of street view images for urban studies is studied.
bution varies from one commercial facility types to another. Generally, urban research has the characteristics of big data,
Therefore, it is possible to classify images by this method. high repetition, high similarity rate, and wide spatial range.
In the second step, for each image, signage distribution Traditionally, a lot of field research need to be done as data
information classified by the first step is converted into a source, and as a result, it is time-consuming and laborious to
fixed dimension. We name this process as data reconstruction complete such work. Online big data and machine learning
which will generate the new format of data for each image. provide a tool for researchers to get rid of the massive repet-
Then, the images are classified by a Random Forest classifier itive works. With the improvement and maturity of machine
trained by new format of data. In addition, the heat maps of learning technology, urban research scholars have begun to
cities are drawn by the total classification results. We also use deep learning for ultra-large-scale researches and other
verify our heat map with the results of other researchers. tasks that were difficult to achieve in the past. For example,
By all these steps with the more advanced algorithms, we aim Liu et al. [13] use the Street View images to research the
at a higher accuracy of the image classification and higher building condition all across Beijing. They use the images
efficiency of the total workflow than existing researches in that are perpendicular to the direction of the road as the
this field. research object, and then make their own data set and analyze
Our innovation is summarized as below: it with the advanced DCNN model at the time. Moreover,
- By using and modifying deep learning technology in a Kang et al. [14] develope a building type discriminating
traditional research field, which is urban (commercial) spatial technique, which is based on Street View images and remote
distribution, a cross-boundary research is achieved. sensing data, for urban analysis. Wang et al. [24] comprehen-
- By using the two-level learning mode of object sively analyze a variety of data including Street View image,
detection and classification, the recognition accuracy is to study the impact of greening on residents’ stress and health
improved. from a physiological-psychological-social multivariate per-
- A new commercial sensing method based on the sim- spective. In addition, the Long Ying team in Tsinghua Univer-
ulation of human visual perception is realized. The highly sity has been doing long-term urban researches based on big
efficient feature of deep learning technology helps to achieve data. They use the Street View images to study Shanghai’s

162842 VOLUME 7, 2019

N. Ye et al.: Urban Commerce Distribution Analysis Based on Street View and Deep Learning

FIGURE 2. The structure of data recording.

FIGURE 1. Work flow of this paper.

green looking ratio by image color analysis based on HVS

color mode [25], and also study the changes in street quality
from 2003 to 2013 [26] by image semantic segmentation and
change detection. In this paper, we adopt a new method to
realize the classification task that cannot be realized by previ-
ous research’s method, and expand the application scenarios
of big data and deep learning in the urban research field.

III. METHOD AND MATERIALS

The structrue of entire research method can be seen in Fig.1.
The detail process in each section will be explained in the
FIGURE 3. Example of SMCPY and SMCPY-OD (With boxes contain
following section. signages. Blue boxes mean signboard and red mean shop sign).

A. STREET VIEW IMAGE ACQUISITION

The Street View image in this study came from the Baidu
Map open platform ([Link] We apply the activities. (2) Class 1: The image of one storey small business.
following steps below to download the Street View image and (3) Class 2: The image of medium scale multi-storey shops,
create a dataset: which often indicates community centre. (4) Class 3: The
(1) The urban road network information are selected and image of shopping malls and other big shops, which often
obtained by using the rectangular frame within its scope indicates city commercial centers (see Fig.3).
through open street map [27]; It should be noted that class 0 occupies about 70% among
(2) The road network obtained in the previous step is sam- the total 400,000 images. So in the data selection, for the
pled by the CreatePointsLines plugin of the ArcGIS software, class 0, we selected some representative images (For exam-
and the density of sampling point is set to 20 meters. The ple, street view without signages, underground passages,
coordinate of all sample points are saved; buildings without signage, large areas of greenery and sky
(3) In order to get images perpendicular to the road, in the field of vision, etc.). For the three categories with
the angles of the road are calculated and the vertical directions signage, the selected images cover the various districts of the
(both left and right) of each sampling point are obtained. city (not only in business districts). We selected 1638 Street
(4) Downloaded 2 images (both left and right) through the View images captured by Baidu Map, manually marked
Baidu Map API (viewing angle 90 degrees, picture size 800× the data through the standard above and named the dataset
500 pixels) for each sampling points; ‘Streetscape datasets of Major Cities in the Pan-Yangtze
(5) The points in the data retain the following information: River Delta region (SMCPY)’. SMCPY included 500 images
the sequence number of the point; the sequence number of the from class 0, 460 images form class 1, 400 images from class
street to which the point belongs; the geographic information 2 and 278 images from class 3.
coordinates of the point. Fig.2 demonstrate the structural of For object detection, we aim at searching for signages.
the data recording. The signages are classified into two categories, shop sign
(a small signage directly above the entrance to the store) and
B. DATASET signboard (a signage with a certain propaganda function that
All the pictures are divided into four categories, and labeled is usually higher than shop sign). (see Fig.3). We build a
as Class 0, 1, 2, 3 respectively. The following is the definition new data set named SMCPY-OD according to the format of
of these four types of images. (1) Class 0: The image with PASCAL VOC. Then 1000 images which are not in SMCPY
no commercial signage, which often indicates no commercial are selected and all the signages are manually marked (the

VOLUME 7, 2019 162843

 N. Ye et al.: Urban Commerce Distribution Analysis Based on Street View and Deep Learning

the bounding box regression for further fine-tuning and the

confidence scores for each signage proposal. Nonmaximum
suppression (NMS) [17] is applied to the bounding boxes
to decide the final predictions. The intersection-over-union
(IoU) thresholds for NMS are 0.7 and 0.3 for training and
testing, respectively.

2) IMPLEMENTATION DETAILS
We use the SMVPY-OD for training. 70% of the images are
used for training, 10% of images are used for validation, and
FIGURE 4. Flowchart of the signage detection algorithm. 20% of images are used for testing. The proposed algorithm is
implemented using Keras [31] with Tensorflow backend [32]
in Python 3.6. The weights in Conv1 to Conv5 are initialized
with the ImageNet [33] pretrained VGG-16 model, and all
total number of shop sign and signboard are 2917 and
the other layers are randomly initialized. Compared with the
1385 respectively).
four-step strategy in the original faster-RCNN implementa-
tion, this end-to-end joint training is more efficient. Each
C. OBJECT DETECTION mini-batch has two images. The number of region proposals
The purpose of this step is to find the signages in each per image for training is 128. We adopt the SGD and set the
image. We observed that almost 100% of commercial estab- base learning rate to 0.001, and then reduced it a factor of 10 at
lishments (shops, shopping centers, restaurants, hotels, etc.) 10000 iterations. The network executed total of 20000 itera-
have signages in their vicinity, and large commercial estab- tions. During training, we also use data augment operations
lishments often set up multiple signages. Although some such as flip, pan, contrast adjustment, etc..
non-commercial organizations also set signages, the number
of these organizations is rather small. Therefore, our recog- D. IMAGE CLASSIFICATION
nition target is the signages of commercial establishments. Based on the object detection module, we got two types of
In this paper, ‘‘signage’’ ‘‘signboard’’ and ‘‘shop sign’’ only boxes which contained the signages for each image. Due to
refer to the signage of commercial establishments. the fact that Street View images of different roads are taken
For the depth model of object detection, yolo [28], ssd [29] from different distances to the street facade. The images did
and faster-RCNN [17] are commonly used. In this study, not share the same field of vision. Some have small vision
the detection object signages are of small size so the field because they are taken near the facade while some have
faster-RCNN is applied. large vision field when they are taken far away. Therefore the
commercial condition of the street could not be shown simply
1) NETWORK ARCHITECTURE by counting the number of boxes in the images. Additional
The VGG-16 [30] model is adopted as the pre-training model information are necessary to classify the images.
for the network. As shown in Fig.4 an input image was pro-
cessed by the convolution blocks (Conv1-Conv5) to produce 1) DATA RECONSTRUCTION
feature maps. We remove the last pooling layers (pool5). In order to train the classifier, we need to convert the infor-
The next part is a region proposal network [17] analyzing mation of the boxes in each image into a fixed dimension.
the feature maps and proposing candidate signages regions. For one image, we record the information of shop sign and
It estimates the probability of signage/non-signage on a signboard respectively. We record the number and coordinate
fixed set of anchors on each position of the feature maps. information of boxes, and calculate the mean, variance, and
Meanwhile, the position and size of each anchor obtained median of the boxes’ width and height. For the image that
by bounding box regression are fine-tuned. After investi- no box is detected, all data will be recorded as 0. After this
gating the sizes of the bounding-boxes for signage, we use process, one image will be converted into a 23-dimension
three anchor scales (64,128,and 256) and three anchor ratios data.
(1:2, 1:1, and 2:1) in this study.
The feature maps and the signage proposals are sent to a 2) CLASSIFICATION
region of interest (Roi) pooling layer afterward, which will SMCPY is used to train the classifier. Similarly, 70% of the
turn all feature maps in proposal into a fixed size (7×7 in this images are used for training, 10% of images are used for
study). These feature maps are fed into two convolution lay- validation, and 20% of images are used for testing. All the
ers, Conv6 and Conv7. We replace the 4096D fully-connected images are firstly processed with the faster-RCNN to produce
layers in original VGG-16. Conv6 consists of 1024 7 × 7 the reconstracted data. And then, we try Random Forest (RF)
filters with zero padding and stride 1. Conv7 consists of 1024 [34] and Xgboost [35] as classifier. We use sklearn to realize
1 × 1 filters with zero padding and stride 1. The 1024D the program. For RF, we set 300 subtree, using information
feature vector in Conv7 are sent to two FC layers to predict gain as critic and the max depth is 8. For Xgboost, the max

162844 VOLUME 7, 2019

N. Ye et al.: Urban Commerce Distribution Analysis Based on Street View and Deep Learning

depth is 5 and the objective function is softprob. Other values

which are not indicated are default. Logistic Regression(LR)
[36] and SVM [37] are also tried but they do not get good
results. We also set up a comparative experiment that directly
uses DCNN (DenseNet and ResNet) for classification.

E. HEAT MAP DRAWING

Kernel density analysis can effectively reflect the distribu-
tion and aggregation of points in space, and heat map is a
commonly-used visual representation of kernel density anal-
ysis. All sampling point images are processed by the trained
faster-RCNN and the classifier, in order to get its predict
label. Then we use the coordinate information of the points
saved in A section in Chapter III, and use the ArcGIS soft-
FIGURE 5. PR curve of test set for object detection.
ware to draw the urban commerce distribution heat map. The
results of the analysis are presented in C section in Chapter IV.
A. OBJECT DETECTION ANALYSIS
n
1X
n
1 X x − xi The results of the test set were represented by the Precision-
fh (x) = Kh (x − xi ) = K( ) (1) Recall(PR) curve (see Fig.5). Through the PR curve,
n nh h
i=1 i=1 we found that the model achieved good results on the shop
sign, with an Average Precision(AP) value of 0.888. But
For regular Kernel density analysis in function (1), Kh is the compared to shop sign, the result of the signboard class is
kernel function based on the quartic kernel function described at a lower recall rate. This may be because that the number of
by Silverman [38]. h is the bandwidth which has a big impact signboard is less than shop sign, and some samples are really
on the estimates obtained and mean intergrated squared error difficult to discriminate. Despite some losses, the trained
[39] is usually used. n is the number of sample points. model achieves the purpose of finding two kinds of signages.

2 !2
 Through the PR curve, we also find that the detection of the
n 
1 X
 3 · popi 1 − disti two targets would have high precision and recall values at the
Density = 
(Radius)2 π Radius same time with the confidence over 0.9. Therefore, in object
i=1
detection section, we only reserve the bounding box with a
(2) confidence over 0.9 for each image.
s
1
Radius = 0.9 ∗ min(SD, ∗ Dm ) ∗ n−0.2 (3)
ln(2) B. CLASSIFICATION ANALYSIS
We list the results of Xgboost [34] and RF [35]. For com-
In this study, we used the Kernel density analysis method parison, we also add the classification results directly using
that is integrated in the ArcGIS software, the coefficients and the DCNN, which is the method used by Liu et al. [13]. The
details are slightly different. popi is the population field value results of PR curve are shown in Fig.6.
of point I, which is an optional parameter. disti is the distance From the results, we see that it is difficult to distinguish
between point i and the (x, y) location. Dm is the (weighted) four types of images simply by using DCNN. Through our
median distance from (weighted) mean center. SD is the two-level learning method, average precision values for all
standard distance [40]. four kinds have been improved. On mAP, the highest score
was obtained by using RF as the final classifier of 0.850.
IV. RESULTS AND FINDINGS Therefore, RF is chosen for the classification section. From
In this paper, we selected 3 provincial capital cities which Fig.6 we also find that the AP of class 2 is significantly lower
locate in the middle and lower reaches of the Yangtze River than other three categories. This means that in our model,
Basin (Hefei, Nanjing and Shanghai) as the research objects. the images of class two do not show obvious characters. For
Due to the financial and computing power limitation, it is class 0, the ideal result for object detection should be no
difficult to conduct research on the whole city region. So the box. But there are some objects that are not signage being
research area is limited to the urban center. misdetected. This leads to errors in the classification after
The research area of Hefei City is about 106.3 square data reconstruction. The confusion matrix of our method
kilometers, with 49,140 sampling points. The research area which tell specific results is also shown in Fig.7.
of Nanjing is about 151.4 square kilometers (including some By remarking the classification result back onto the map,
Yangtze River water-body), with 59,420 sampling points. we can get the distribution map of commercial facilities on
The research area of Shanghai is about 120.6 square kilo- both sides of the street network of a city. Here is an example of
meters (including some Huangpu River water-body), with the city of Hefei (Fig.8). The granularity of this data depends
55,998 sampling points. on (1) the density of the street sampling points; and (2) the

VOLUME 7, 2019 162845

 N. Ye et al.: Urban Commerce Distribution Analysis Based on Street View and Deep Learning

distribution, these heat maps directly show where the

shops (signages) in the city are concentrated. By observing
and analyzing these heat maps, we could observe the follow-
ing characteristics of shop distribution in each urban central
area. Hefei: (1) Commercial distribution are highly related to
road network system. Most commercial districts locate along
the main roads separately; (2) The center points are scattered,
showing obvious Voronoi diagram distribution; (3) Business
is concentrated in old urban areas, and commercial facilities
are scarce in urban fringe areas (south and northwest corner);
(4) The distribution of small businesses and large commercial
centers is more uniform; (5) Community business centers
have obvious multi-core cluster characteristics.
Nanjing: (1) All levels of commerce area are centered and
highly concentrated around Xinjiekou; (2) Old urban areas
are generally high-continuity and high-density commercial
areas as a whole; (3) The distribution of small businesses
in the central area of the old city is uniform and has high-
density, which forms a large number of community business
centers, but these centers are neither large-scale nor continu-
ous (a large number of community business centers have been
optimized and deleted by the streets); (4) Business is highly
FIGURE 6. Precison and Recall curve of test set for classification.
concentrated in the old urban areas, and there is little com-
merce distribution in the new southern urban areas, including
those along the Yangtze River; (5) There are relatively fewer
shopping malls.
Shanghai: (1) Business is widely and evenly distributed,
with high commercial coverage and density in Puxi District
but relatively low in Pudong; (2) The concentration of Com-
merce on the same street is low, and it distributes evenly in the
whole city (after street optimization, the overall commercial
coverage is significantly reduced); (3) There is a very clear
continuous central business district; (4) Small businesses in
FIGURE 7. Confusion Matrix of our method in test set. the general community are very developed.
As indicated from the analysis above, this method could
comprehensively reflect the following characteristics of com-
mercial distribution in cities: (1) Overall commercial distri-
bution; (2) The distribution of small businesses, community
center businesses, and city center businesses; (3) Continuity
between commercial areas; (4) Shop intensity.

D. VERIFICATION OF ACCURACY
We find one commerce distribution analysis papers for
each of these three cities as the benchmark for verification.
Although these studies differ from the POI data in terms of
accuracy and actual visual perception, the rough structure in
FIGURE 8. Commerce distribution map of Hefei. city scale should be consistent.
Hefei: Two core business districts and three secondary
sampling density of the Street View images provided by the business districts can be observed. This is consistent with the
online map provider. commerce distribution characteristics of Hefei City described
in the research results by CHENG, et al (2019) [7].
C. HEAT MAPS AND ANALYSIS Nanjing: It is obvious that the Xinjiekou is an absolute
According to the results of classification, we draw the heat single-core commerce area and radiates to other directions
maps of the three cities as shown in Fig.9. Since ker- except the northeast, which is blocked by Yangtze river. This
nel density analysis actually counts the number of points result is consistent with Yang’s, et al. (2019) [9] research
within a certain range and clusters according to their spatial results on the overall commerce structure of Nanjing.

162846 VOLUME 7, 2019

N. Ye et al.: Urban Commerce Distribution Analysis Based on Street View and Deep Learning

FIGURE 9. Urban commerce distribution heat map.

TABLE 1. Coverage of different commerce types in Hefei.

Shanghai: The Puxi area is almost entirely covered by produces a result of commerce distribution of a city, which
high-level commercial areas except for the central east por- can be used as input data in other urban researches, we could
tion, while the Pudong area has no core business area except find a large research field after this.
Lujiazui. This is consistent with the results of the Shanghai Here is a list of some major following usage scenarios
Commerce Distribution Study by Huang and Du (2018) [8]. in our vision: (1) Calculation of accessibility of commercial
It should be pointed out that the existing urban commerce facilities; (2) Researches on urban spatial structure character-
distribution results based on POI data have larger granularity istics based on commerce distribution. If combined with other
and low fineness. In the three papers used as benchmarks, data, it can provide a wider range of application scenarios, for
the results of the other two papers were rough in contrast. exmaple: (3) Combining residential area location, population
In these papers some location information of the business distribution and other data to study urban commercial cov-
district described in their heat maps does not accurately erage; (4) Combining mobile phone signaling data to study
march with the streets or even the blocks. This reinforces that the interaction between commercial facilities and people;
our two-level learning method has high precision and res- (5) Combining POI data to determine the hidden commerce
olution in term of classifying commerce features/analyzing distribution of the city; (6) Based on the frequency of street
commerce distribution patterns. view map update, dynamic monitoring of urban business
changes may be achieved.
E. POSSIBLE APPLICATION OF THIS METHOD IN URBAN For example, a business accessibility analysis is conducted
PLANNING PRACTICE for Hefei City. The commerce accessibility of various parts
As a large-scale, programmable city-scale analysis tool, of the city is analyzed based on road network data and our
the application potential of the research methods applied in business point level data, with the Network Analyst feature
this study can be foreseen in the fields of smart cities, urban in ArcGIS. Here we refer to the relevant requirements of the
planning or construction management. Since this method 15-minute living circle, in which the residents can reach the

VOLUME 7, 2019 162847

 N. Ye et al.: Urban Commerce Distribution Analysis Based on Street View and Deep Learning

REFERENCES
[1] J. Jacobs, The Death and Life of Great American Cities. New York, NY,
USA: Vintage, 1961.
[2] K. Pothukuchi, ‘‘Attracting supermarkets to inner-city neighborhoods:
Economic development outside the box,’’ Econ. Develop. Quart., vol. 19,
no. 3, pp. 232–244, Aug. 2005.
[3] M. Bonaiuto, F. Fornara, and M. Bonnes, ‘‘Indexes of perceived res-
idential environment quality and neighbourhood attachment in urban
environments: A confirmation study on the city of Rome,’’ Landscape
Urban Planning, vol. 65, nos. 1–2, pp. 41–52, 2003. [Online]. Available:
[Link]
[4] R. Ortega-Álvarez and I. MacGregor-Fors, ‘‘Living in the big city:
Effects of urban land-use on bird community structure, diversity,
and composition,’’ Landscape Urban Planning, vol. 90, nos. 3–4,
pp. 189–195, 2009. [Online]. Available: [Link]
science/article/pii/S0169204608002004
[5] L. Yuming, ‘‘Research on spatial distribution and influencing factors that
affect the block-scale commercial site layout,’’ (in Chinese), M.S. thesis,
Chongqing Univ., Chongqing, China, 2018.
[6] J. Yuan, Y. Zheng, and X. Xie, ‘‘Discovering regions of different functions
in a city using human mobility and POIs,’’ in Proc. 18th ACM SIGKDD
Int. Conf. Knowl. Discovery Data Mining, 2012, pp. 186–194.
[7] C. Chezhi, L. Zhanting, and Z. Qiong, ‘‘Research on spatial distribution
and evaluation of urban leisure function in HEFEI city based on POI data,’’
(in Chinese), J. Hum. Settlements West China, vol. 34, no. 2, pp. 74–79,
2019.
FIGURE 10. Walking range coverage of different commerce types in Hefei. [8] H. Pujiang, ‘‘Research on commercial vitality spatial distribution in
central city of Shanghai,’’ (in Chinese), Geospatial Inf., vol. 16,
no. 8, pp. 8–11 and 7, 2018. [Online]. Available: [Link]
living service facility within 15 minutes of walking. It could Article_en/[Link]
be converted into a distance of about 1200 meters. The results [9] J. Yang, J. Zhu, Y. Sun, and J. Zhao, ‘‘Delimitating urban commercial cen-
tral districts by combining kernel density estimation and road intersections:
of the analysis are shown in Fig.10. A case study in Nanjing city, China,’’ ISPRS Int. J. Geo-Inf., vol. 8, no. 2,
The result shows that the commerce accessibility of the p. 93, 2019. [Online]. Available: [Link]
downtown area is generally good, and the coverage of [10] X. Ren, M. Song, E. Haihong, and J. Song, ‘‘Context-aware probabilis-
tic matrix factorization modeling for point-of-interest recommendation,’’
medium commercial centers is large. Although the city-level Neurocomputing, vol. 241, pp. 38–55, Jun. 2017. [Online]. Available:
central business districts are widely distributed, the range of [Link]
services outside the old town is very limited. This is presum- [11] Google Streetview. Accessed: Jul. 24, 2018. [Online]. Available:
[Link]
ably due to the lack of a pedestrian transportation system or [12] Baidu Streetview. Accessed: Jul. 24, 2018. [Online]. Available:
a low density street network. [Link]
[13] L. Liu, E. A. Silva, C. Y. Wu, and H. Wang, ‘‘A machine learning-
based method for the large-scale evaluation of the qualities of
V. CONCLUSION the urban environment,’’ Comput., Environ. Urban Syst., vol. 65,
In this study, we have designed a city commerce distribution pp. 113–125, Sep. 2017. [Online]. Available: [Link]
sensing method based on Street View image and deep learn- com/science/article/pii/S0198971516301831
[14] J. Kang, M. Körner, Y. Wang, H. Taubenböck, and X. X. Zhu, ‘‘Build-
ing. Through this method, the public data provided by the ing instance classification using street view images,’’ ISPRS J. Pho-
online map services can be used to quickly and accurately togram. Remote Sens., vol. 145, pp. 44–59, Nov. 2018. [Online]. Available:
analyze the commerce distribution of any city from the human [Link]
[15] What’s Your Signage? How On-Premise Signs Help Small Businesses Tap
visual experience. This method takes Street View images Into a Hidden Profit Center. Small Bus. Admin., Washington, DC, USA,
as the analysis object, uses object detection model to find 2004.
commercial signages, then classifies the sampling points, and [16] T.-H. Tsai, W.-H. Cheng, C.-W. You, M.-C. Hu, A. W. Tsui, and
H.-Y. Chi, ‘‘Learning and recognition of on-premise signs from weakly
finally uses the kernel density analysis to visualize the results labeled street view images,’’ IEEE Trans. Image Process., vol. 23, no. 3,
of the urban commerce distribution. pp. 1047–1059, Mar. 2014.
Compared with the existing researches, we use differ- [17] S. Ren, K. He, R. Girshick, and J. Sun, ‘‘Faster R-CNN: Towards real-time
object detection with region proposal networks,’’ in Proc. Adv. Neural Inf.
ent models and achieve a better result. By verification, Process. Syst., 2015, pp. 91–99.
we find that this system has achieved its theoretical advan- [18] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and
tages (more realistic reflection of pedestrian visuals on the A. Zisserman, ‘‘The Pascal visual object classes (VOC) challenge,’’ Int.
J. Comput. Vis., vol. 88, no. 2, pp. 303–338, Sep. 2009.
street) than traditional POI-based urban commerce distribu- [19] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,
tion research. We believe that with more training data, it is P. Dollár, and C. L. Zitnick, ‘‘Microsoft COCO: Common objects in
possible to further improve the scores. context,’’ in Proc. Eur. Conf. Comput. Vision. Cham, Switzerland: Springer,
2014, pp. 740–755.
Since the results of this method can be used as input data [20] R. L. Davies, ‘‘Structural models of retail distribution: Analogies
for other urban researches, we hope it could provide a foun- with settlement and urban land-use theories,’’ Trans. Inst. Brit.
dation for the future studies. Especially when combined with Geographers, vol. 57, pp. 59–82, Nov. 1972. [Online]. Available:
[Link]
other urban data, various issues related to urban commerce [21] W. E. Cox, Jr., ‘‘A commercial structure model for depressed neighbor-
could be studied. hoods,’’ J. Marketing, vol. 33, no. 3, pp. 1–9, 1969.

162848 VOLUME 7, 2019

N. Ye et al.: Urban Commerce Distribution Analysis Based on Street View and Deep Learning

[22] O. ITOH, ‘‘Distribution and spatial structure of retail trand in BOWEN WANG was born in China. He received
metropolises,’’ (in Japanese), Geographical Rev. Jpn., vol. 55, no. 9, the B.C.S. degree in computer science from Anhui
pp. 614–633, 1982. University, China. He is currently pursuing the
[23] A. G. Rundle, M. D. M. Bader, C. A. Richards, K. M. Neckerman, and M.M. degree in medical information research with
J. O. Teitler, ‘‘Using Google street view to audit neighborhood environ- Osaka University. He also serves in Osaka Univer-
ments,’’ Amer. J. Preventive Med., vol. 40, no. 1, pp. 94–100, 2011. sity Institute for Datability Science. His research
[24] R. Wang, M. Helbich, Y. Yao, J. Zhang, P. Liu, Y. Yuan, and Y. Liu, ‘‘Urban interests include computer vision and medical AI
greenery and mental wellbeing in adults: Cross-sectional mediation anal-
research. Mainly studies chest x-ray image analy-
yses on multiple pathways across different greenery measures,’’ Environ.
sis and cell slice analysis through deep learning.
Res., vol. 176, Sep. 2019, Art. no. 108535.
[25] X. Hao and Y. Long, ‘‘Street greenery: A new indicator for evaluat- He is also committed to smart city research.
ing walkability,’’ (in Chinese), Shanghai Urban Planning Rev., no. 1,
pp. 32–36 and 49, 2017. [Online]. Available: [Link]
Article_en/[Link]
[26] J. Tang, Y. Long, and W. Zhuo, ‘‘Measuring quality of street space,its
temporal variation and impact factors: An analysis based on massive
street view pictures,’’ (in Chinese), New Archit., no. 5, pp. 110–115, 2016.
[Online]. Available: [Link] MICHIHIRO KITA is currently a Professor of
[Link] urban design with Osaka University. He has
[27] Open Street MAP. Accessed: Jul. 24, 2018. [Online]. Available: extensively researched the contextual design and
[Link] improvement of urban communities in Japan,
[28] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, ‘‘You only look once: Asia, Central Europe, and Africa. He has also
Unified, real-time object detection,’’ in Proc. IEEE Conf. Comput. Vis. consulted on a wide range of urban design
Pattern Recognit., Jun. 2016, pp. 779–788. and planning issues for local municipalities,
[29] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and nonprofit organizations, and consultant com-
A. C. Berg, ‘‘SSD: Single shot multibox detector,’’ in Proc. Eur. Conf. panies. Recently, he has been working on
Comput. Vis. Cham, Switzerland: Springer, 2016, pp. 21–37. research projects concerning sustainable design
[30] K. Simonyan and A. Zisserman, ‘‘Very deep convolutional networks for and improvement of informal settlements in Africa and Asia. His research
large-scale image recognition,’’ 2014, arXiv:1409.1556. [Online]. Avail- interests include architectural and urban design for the continuation of area’s
able: [Link]
context, urban formation and reorganization in central and Eastern Europe,
[31] F. Chollet. (2015). Keras. [Online]. Available: [Link]
planning theory and reorganization of community in Japanese New-towns,
fchollet/keras
[32] M. Abadi et al., ‘‘TensorFlow: A system for large-scale machine learning,’’ and campus planning and facility management. He has and had been a
in Proc. 12th USENIX Symp. Operating Syst. Design Implement. (OSDI), Council Member or a member of academic institutions, such as Architectural
2016, pp. 265–283. Institute of Japan, Japan Association for Planning Administration, Japan
[33] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, ‘‘Imagenet: Association of Urban Housing Sciences, The City Planning Institute of
A large-scale hierarchical image database,’’ in Proc. IEEE Conf. Comput. Japan, and MERA.
Vis. Pattern Recognit., Jun. 2009, pp. 248–255.
[34] A. Liaw and M. Wiener, ‘‘Classification and regression by randomforest,’’
R News, vol. 2, no. 3, pp. 18–22, 2002.
[35] T. Chen and C. Guestrin, ‘‘Xgboost: A scalable tree boosting system,’’
in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining,
Aug. 2016, pp. 785–794. MING XIE was born in China. He received the
[36] D. W. Hosmer, Jr., S. Lemeshow, and R. X. Sturdivant, Applied Logistic [Link]. degree from Nanjing University, China, and
Regression, vol. 398. Hoboken, NJ, USA: Wiley, 2013. the [Link]. and Ph.D. degrees from the University of
[37] J. A. K. Suykens and J. Vandewalle, ‘‘Least squares support vector machine South Florida, Tampa, USA. He is currently a Post-
classifiers,’’ Neural Process. Lett., vol. 9, no. 3, pp. 293–300, Jun. 1999.
doc Researcher with Dalian Maritime University,
[38] B. W. Silverman, Density Estimation for Statistics and Data Analysis.
China. His current research interests include the
Evanston, IL, USA: Routledge, 2018.
[39] J. W. H. Swanepoel, ‘‘Mean intergrated squared error properties and opti- applications of GIS and RS technology in coastal
mal kernels when estimating a diatribution function,’’ Commun. Statist.- environment and transportation, and the cognitive
Theory Methods, vol. 17, no. 11, pp. 3785–3799, 1988. process of map and marine chart.
[40] How Kernel Density Works. Accessed: Jul. 24, 2018. [Online]. Available:
[Link]
[Link]

NANQI YE was born in China. He received

the [Link]. and [Link]. degrees from the Hefei WENYUE CAI was born in China. He received
University of technology, China, and the [Link]. the [Link]. degree in biomedical engineering from
degree from Cardiff University, Wales, U.K. in the Capital Medical University, China. He is currently
course of Eco-cities. He is currently pursuing pursuing the M.M. degree in medical informa-
the Ph.D. degree with the Architecture Depart- tion research with Osaka University. His research
ment, Osaka University, Japan. He is aiming at interest includes X-ray image research under deep
cross-boundary researches related to architecture learning methods.
and cities. His main research interests include
architectural design, architectural genealogy, com-
munity development, and smart cities.

VOLUME 7, 2019 162849