US011400925B2

( 12 ) Frazzoli
Unitedet States Patent ( 10 ) Patent No.: US 11,400,925 B2
al . (45 ) Date of Patent : Aug. 2 , 2022
( 54 ) PLANNING FOR UNKNOWN OBJECTS BY B60W 2420/42 ( 2013.01 ) ; B60W 2420/52
AN AUTONOMOUS VEHICLE (2013.01 ) ; B60W 2552/00 ( 2020.02 ) ; B60W
2554/00 (2020.02 ) ; B60W 2555/20 ( 2020.02 ) ;
( 71 ) Applicant: Motional AD LLC , Boston , MA (US ) B60W 2555/60 ( 2020.02 ) ; B60W 2556/00
( 72 ) Inventors: Emilio Frazzoli , Zurich (CH ) ; Baoxing (2020.02 ) ; B60W 2556/10 ( 2020.02 ) ; G05D
Qin , Singapore ( SG) 2201/0213 (2013.01 )
( 58 ) Field of Classification Search
(73 ) Assignee : Motional AD LLC , Boston, MA (US) None
See application file for complete search history .
( * ) Notice: Subject to any disclaimer, the term of this
patent is extended or adjusted under 35 (56 ) References Cited
U.S.C. 154 ( b ) by 128 days . U.S. PATENT DOCUMENTS
( 21 ) Appl. No .: 16 /380,002 6,768,813 B1 7/2004 Nakayama
8,605,946 B2 12/2013 Iwasaki et al .
(22) Filed : Apr. 10 , 2019 8,611,591 B2 12/2013 Coffman et al .
9,434,382 B1 9/2016 Prokhorov et al .
( 65 ) Prior Publication Data 9,476,970 B1 10/2016 Fairfield et al .
US 2019/0232957 A1 Aug. 1 , 2019 (Continued )
FOREIGN PATENT DOCUMENTS
Related U.S. Application Data
( 63 ) Continuation of application No. 15 / 451,734 , filed on EP 3091370 11/2016
Mar. 7 , 2017 , now Pat. No. 10,281,920 .
OTHER PUBLICATIONS
( 51 ) Int. Ci. Shackelford et al ., " Urban Road Network Extraction from High
B60W 30/09 ( 2012.01 )
GO8G 1/16 ( 2006.01 ) Resolution Multispectral Data ” , 2nd GRSS/ISPRS Joint Workshop
GO6V 10/26 ( 2022.01 ) on Remote Sensing and Data Fusion over Urban Areas, May 2003 ,
GO6V 20/56 ( 2022.01 ) pp . 142-146 . ( Year: 2003 ) . *
G06V 20/58 (2022.01 ) (Continued )
B60W 10/20 ( 2006.01 )
B60W 10/184 ( 2012.01 ) Primary Examiner Todd Melton
G05D 1/00 ( 2006.01 ) (74 ) Attorney , Agent, or Firm — Fish & Richardson P.C.
G05D 1/02 ( 2020.01 ) ( 57 ) ABSTRACT
( 52) U.S. CI.
CPC B60W 30/09 ( 2013.01 ) ; B60W 10/184 Among other things, a model is maintained of an environ
(2013.01 ) ; B60W 10/20 ( 2013.01 ) ; G05D ment of a vehicle. A hypothetical object in the environment
1/0088 (2013.01 ) ; G05D 1/0219 ( 2013.01 ) ; that cannot be perceived by sensors of the vehicle is included
G05D 1/0223 ( 2013.01 ) ; G06V 10/26 in the model .
( 2022.01 ) ; GO6V 20/56 ( 2022.01 ) ; G06V
20/58 ( 2022.01 ) ; G08G 1/166 ( 2013.01 ) ; 30 Claims , 18 Drawing Sheets

1605 1606 1608

1603
 US 11,400,925 B2
Page 2

( 56 ) References Cited 2017/0345311 A1 * 11/2017 Sugiura GO8G 1/167

2017/0345321 A1 11/2017 Cross et al .
U.S. PATENT DOCUMENTS 2017/0369051 A1 12/2017 Sakai et al .
2018/0113459 Al 4/2018 Bennie et al .
9,501,932 B2 11/2016 Sakai et al . 2018/0114442 Al 4/2018 Minemura et al.
9,600,768 B1 3/2017 Ferguson 2018/0120859 Al 5/2018 Eagelberg et al .
9,645,577 B1 5/2017 Frazzoli et al . 2018/0217233 A1 8/2018 Lee
9,836,895 B1 * 12/2017 Nygaard GO5D 1/0088 2018/0259967 Al 9/2018 Frazzoli et al .
10,037,036 B2 * 7/2018 Nilsson B60W 50/0097 2018/0259969 Al 9/2018 Frazzoli et al .
10,095,234 B2 10/2018 Frazzoli et al . 2019/0018421 Al 1/2019 Frazzoli et al .
10,198,951 B2 * 2/2019 Vanholme GOSG 1/16 2021/0078562 Al 3/2021 Frazzoli et al .
10,234,864 B2 3/2019 Frazzoli et al .
10,281,920 B2 5/2019 Frazzoli et al . OTHER PUBLICATIONS
10,850,722 B2 12/2020 Frazzoli et al .
2006/0155464 A1 7/2006 Smartt Coenen et al . , " A Representation Method Based on the Probability
2007/0061074 Al 3/2007 Safoutin
2008/0266168 A1 10/2008 Aso et al . of Collision for Safe Robot Navigation in Domestic Environments ” ,
2009/0024357 Al 1/2009 Aso et al . IEEE/RSJ International Conference on Intelligent Robots and Sys
2009/0058677 A1 3/2009 Tseng et al . tems ( IROS 2014 ) , Sep. 2014 , pp . 4177-4183 . ( Year: 2014 ) . *
2009/0237263 A1 9/2009 Sawyer Bertozzi et al . , “ Stereo inverse perspective mapping: theory and
2011/0231095 Al 9/2011 Nakada et al . applications” Image and Vision Computing, 1999 , 16 : 585-590 .
2012/0140039 A1 6/2012 Ota Florentine et al., " Pedestrian notification methods in autonomous
2012/0275524 A1 11/2012 Lien et al . vehicles for multi -class mobility -on -demand service .” Proceedings
2013/0223686 A1 8/2013 Shimizu of the Fourth International Conference on Human Agent Interaction ,
2013/0304349 Al 11/2013 Davidson Oct. 4 , 2016 , pp . 387-392 .
2013/0328916 Al 12/2013 Arikan et al . Himmelsback et al . , “ Fast Segmentation 3D Point Clouds for
2014/0088855 A1 3/2014 Ferguson Ground Vehicles , ” IEEE Intelligent Vehicles Symposium , Jul. 21-24 ,
2015/0066329 A1 3/2015 Mielenz 2010, 6 pages .
2015/0120125 A1 4/2015 Thomson et al . Pendleton et al . , “ Autonomous golf cars for public trial of mobility
2016/0047657 Al 2/2016 Caylor et al . on -demand service.” Intelligent Robots and Systems ( IROS ), 2015
2016/0109245 A1 4/2016 Denaro
2016/0137206 Al 5/2016 Chandraker et al . IEEE
1171 .
/RSJ International Conference on Sep. 28 , 2018 , pp . 1164
2016/0147921 Al 5/2016 VanHolme
2016/0266581 A1 9/2016 Dolgov et al . Shackelford et al ., “ Urban Road Network Extraction from High
2016/0282874 Al 9/2016 Kurata et al . Resolution Multispectral Data , ” 2nd GRSS/ISPRS Joint Workshop
2016/0327953 A1 11/2016 Nilsson et al . on Remote Sensing and Data Fusion over Urban Areas, May 23 ,
2017/0008521 A1 1/2017 Braunstein et al . 2003 , pp . 142-146 .
2017/0262750 Al 9/2017 Kozuka et al . U.S. Appl. No. 16 / 125,993 , filed Sep. 10 , 2018 , Frazzoli et al .
2017/0291608 A1 10/2017 Engel et al . U.S. Appl. No. 17 / 102,409 , filed Nov. 23 , 2020 , Frazzoli et al.
2017/0327128 A1 11/2017 Denaro
2017/0341652 A1 11/2017 Sugawara et al . * cited by examiner
U.S. Patent Aug. 2 , 2022 Sheet 1 of 18 US 11,400,925 B2

Autonomous Vehicle System 22

Autonomous Vehicle10
Sensors 24 data Database 34
Sensors 26 data
A
Communication devices 28
Computer 40 data Data Sources 30
-Processor 42
Interface devices 44
Communication devices 46 Technology 18, 20
• Memory 32 Data sources 36
Functional devices 48 Data sources 38
Reasoning algorithm 101

Environment 12

Obstades 16

Goal 14 FIG . 1
U.S. Patent Aug. 2 , 2022 Sheet 2 of 18 US 11,400,925 B2

202

.
.

2 ? 3

+
.
. . +
.
.. ?

..
*
.

. .
.
- .

19

.
.
.

*+
FIG
2
.

205

3
3

??
P.
U.S. Patent Aug. 2, 2022 Sheet 3 of 18 US 11,400,925 B2

302

5
.. .
:
.
*

.
3
.
FIG
+ .

ww

ReasonigAlorithm 321
U.S. Patent Aug. 2, 2022 Sheet 4 of 18 US 11,400,925 B2

***

906
***
...
***** .
***
.

11.
406 fu
...
*****

FIG
4
.

405
24

1 Re
3
as oni g A l o rith m 421
U.S. Patent Aug. 2 , 2022 Sheet 5 of 18 US 11,400,925 B2

..

.. + . 2

. . * 2
.
?

.
2
.

. 3

5
.
FIG

? ? ? ? ? ? ? ? ? ? ? ?:?'z
?-xi

*****
? ? ? ja.:::::f•
?:?
******
*i*ster**** *** ** * * ** * * * * * * *

***** *********
1972
!.
**********#
twit uil e * * * * * *
um
*** *****...
U.S. Patent Aug. 2 , 2022 Sheet 6 of 18 US 11,400,925 B2

19

&
FIG
6
.

OS

??
U.S. Patent Aug. 2 , 2022 Sheet 7 of 18 US 11,400,925 B2

Perception
process

World Model

Planning and Decision

Making process

Control
process

FIG . 7
U.S. Patent Aug. 2 , 2022 Sheet 8 of 18 US 11,400,925 B2

811

DATA FROM ONBOARD

SENSORS PERCEPTION
PROCESS

DATA FROM SENSORS ON WORLD

OTHER OBJECTS MODEL

820
DARK
DATA FROM MAPS ,
DATABASES AND OTHER OBJECTS
PROCESS
DATA SOURCES
812

FIG . 8
U.S. Patent Aug. 2 , 2022 Sheet 9 of 18 US 11,400,925 B2

DATA FROM ONBOARD SENSORS , SENSORS

OUTSIDE THE VEHICLE , MAPS AND OTHER
DATA SOURCES

brott STEP 1 : BOUNDARY

DETERMINATION PROCESS

902
STEP 2 : DARK OBJECTS
GENERATION PROCESS

FIG . 9
U.S. Patent Aug. 2 , 2022 Sheet 10 of 18 US 11,400,925 B2

1012

1014
1010
10 9
? 10 8 10 4
FIG
10
.

10 6
10 5 O 10 1
10 3 10 2 *****
16
O
U.S. Patent Aug. 2, 2022 Sheet 11 of 18 US 11,400,925 B2

11
.
FIG
1 06 1 01

11 10
U.S. Patent Aug. 2, 2022 Sheet 12 of 18 US 11,400,925 B2

12 0

1203

1206
1202
1205
201 Ww 12
.
FIG
1204
1207
f

120
1210 *
VY
U.S. Patent Aug. 2 , 2022 Sheet 13 of 18 US 11,400,925 B2

amW*ownrde
1306
1305
1302w

1303 13
.
FIG

1301

2
130
1310
U.S. Patent Aug. 2 , 2022 Sheet 14 of 18 US 11,400,925 B2

FIG
14
.
???
U.S. Patent Aug. 2 , 2022 Sheet 15 of 18 US 11,400,925 B2

1520

1521

1531 Warto1530

1532

FIG . 15
U.S. Patent Aug. 2 , 2022 Sheet 16 of 18 US 11,400,925 B2

1608 1609
16
FIG
.
1606 1607

09L
€

9091 1601
U.S. Patent Aug. 2 , 2022 Sheet 17 of 18 US 11,400,925 B2

1703

9026 17
.
FIG

1705
U.S. Patent Aug. 2 , 2022 Sheet 18 of 18 US 11,400,925 B2

1803
????

1805

1804

D 1802

1806

FIG . 18
 US 11,400,925 B2
1 2
PLANNING FOR UNKNOWN OBJECTS BY world and an unperceived world . The perceived world and
AN AUTONOMOUS VEHICLE the unperceived world are separated by a boundary.
Implementations may include detecting of the boundary.
CROSS - REFERENCE TO RELATED Detecting of the boundary uses data from one or more
APPLICATIONS 5
sensors to distinguish the observable ground from a fore
ground that obscures a portion of the ground . The one or
This application is a continuation application of, and more sensors comprise sensors of the vehicle, or sensors
claims priority to , U.S. patent application Ser. No. 15/451 , offboard the vehicle, or both .
734 , filed on Mar. 7 , 2017. The disclosure of the foregoing A location of the ego vehicle is determined based on a
application is incorporated herein by reference in its entirety. 10 road network database and one or more sensors . From aa road
BACKGROUND
network database , traffic lane information is also queried.
Stored data, acquired from any databases or from any
sensors, may be used to infer a possible location of the
In making driving decisions , typical autonomous vehicle
( AV)systems take accountofobjects such as other 15 hypothetical object.Determining a location ofthehypotheti
) — cal objects is based on querying traffic lane information from
vehicles and obstacles — that the AV system knows are in the a database and discretizing the traffic lane into discretized
environment of the AV, either because sensor systems on the points. Implementations may generate an unknown skeleton
AV observe the objects, or because the objects are identified of discretized points of aa lane that cannot be perceived by
by maps or other data sources . For making driving decisions, 20 sensors of the vehicle . The hypothetical object in the world
the AV systems may maintain world models that include the model is generated by iterating the following through dis
objects known to be in the AV's environment. Challenges to cretized points: generating a representative shape at a dis
good driving decisions also arise from vehicles and cretized point of the unknown skeleton , and evaluating
obstacles that the AV cannot perceive and does not otherwise whether the representative shape is completely within the
know to exist based on available data . 25 unperceived world . If the representative shape is completely
within the unperceived world , a representative shape is
SUMMARY OF THE TECHNOLOGIES treated as the hypothetical object.
Implementations may apply temporal filtering to deter
The technologies described in this document enable an mine a location of the hypothetical object . The filtering
AV system to plan for risks associated with objects that the 30 comprises smoothing an unknown skeleton by a forward
AV may have in its environment but be unaware of. The AV propagated unknown skeleton , wherein the forward propa
system then can make driving decisions — including driving
decisions that are relatively safe in view of potentially gated old
unknown skeleton is generated by moving forward an
unknown skeleton along a traffic lane .
unsafe scenarios — that take account of the unknown pos The hypothetical object in the world model is associated
sible objects in its environment. For this purpose , in some 35 with one or more attributes . One or more of the attributes are
implementations of these technologies, the AV system deter related to a possible motion state of the hypothetical object.
mines the boundaries between perceived worlds in the world
model and unperceived worlds. The AV system then hypoth The motion state may be a stationary condition , or a moving
esizes the presence or attributes of possible unknown objects condition, or a speed , or a moving direction , or a combina
( which we sometimes call “ dark objects ”) in the unperceived 40 tion of two or more of them . The speed is set to less than or
worlds, based on a variety of factors and approaches. These equal to a predetermined maximum value . The predeter
hypothetical objects and their attributes are then added to the mined maximum value comprises a speed limit . In some
world model for use in making driving decisions that cases , the predetermined maximum value comprises a quan
accommodate not only the known objects in the perceived tity derived from other objects concurrently or previously
worlds but also the unknown objects ( or so called , dark 45 observed in the environment. The predetermined maximum
objects) in the unperceived worlds . value may be a quantity derived from historical data, road
In general, in an aspect , a world model of an environment configuration, traffic rules , an event, a time , a weather
of a vehicle is maintained . Maintaining of the world model condition, or a combination of two or more of them .
accesses a database comprising road network information , The one or more sensors may comprise one or more of the
or uses data from one or more sensors , or both . The world 50 following: a radar sensor, a lidar sensor, and a camera sensor.
model includes a hypothetical object in the environment that The camera sensor includes a stereo camera sensor or a
cannot be perceived by sensors of the vehicle. A hypothetical monocular camera sensor, or both .
object comprises a moving object, or an object that uses a Implementations update a trajectory for the vehicle based
path of travel from which the vehicle is excluded, or both . on the world model and executes the trajectory for the
The hypothetical object may comprise at least one of the 55 vehicle. The vehicle comprises an autonomous vehicle .
following: a second vehicle, a bicycle , a bus , a train , a In general, in an aspect , data representing an observable
pedestrian , and an animal . Implementations of including the part of an environment of a vehicle is received from a sensor.
hypothetical object in the world model comprises selecting Data representing a non - observable part of the environment,
a type of the hypothetical object and an attribute of the including data representing at least one hypothetical object
hypothetical object probabilistically based on objects pre- 60 in the non - observable part of the environment, is generated .
viously observed in the environment. The attribute com- Commands for operation of the vehicle within the environ
prises a size or a speed or both . The world model in the ment is generated. The commands depend on the data
environment includes one or more known objects that are representing the observable part of the environment and on
perceived by sensors of the vehicle or are otherwise known. the data representing the hypothetical object in the non
The hypothetical objects and the known objects maintained 65 observable part of the environment. The hypothetical object
by the world model are in different parts of the environment. may comprise a moving object. In some cases , the hypo
The different parts of the environment comprise a perceived thetical object comprises an object that uses a path of travel
 US 11,400,925 B2
3 4
from which the vehicle is excluded . The hypothetical object autonomous vehicle colliding with another vehicle on the
can be at least one of: a vehicle, a bicycle , a bus , a train , a road network . The hypothetical speed and moving direction
pedestrian , and an animal. is probabilistically derived based on vehicles previously
Implementations of generating data representing a non- observed in the environment.
observable part of the environment comprise selecting a type 5 The observable part and the non - observable part are
of the hypothetical object and an attribute of the hypothetical separated by a boundary. Implementations include detecting
object probabilistically based on objects previously of the boundary. Detecting of the boundary uses data from
observed in the environment. In some examples, the hypo- one or more sensors to distinguish an observable ground
thetical object comprises a vehicle and the attribute com- from a foreground that obscures a portion of the ground. The
prises a size and a speed . 10 one or more sensors comprise sensors onboard the autono
The observable part and the non - observable part are mous vehicle, or sensors offboard the autonomous vehicle ,
separated by a boundary. The technologies comprise detect or both .
ing of the boundary. Detecting the boundary comprises using The current data representing a hypothetical speed and
data from the sensor to distinguish the observable ground moving direction of a hypothetical vehicle may be generated
from a foreground that obscures a portion of the ground. 15 based on known objects perceived by one or more sensors .
Generating data representing the non - observable part of the In some cases , the data generation includes one or more of
environment comprises one or more of the following data the following operations: querying traffic lane information
processing steps : ( 1 ) using stored data to infer a possible from a road network database , using stored data to infer a
location of the hypothetical object, ( 2 ) querying traffic lane possible location of the hypothetical vehicle , and determin
information from aa road network database, (3 ) determining 20 network
a location of the vehicle based on a road network database
ing a location of the autonomous vehicle based on a road
database and one or more sensors . Inferring the
and one or more sensors , (4 ) querying traffic lane informa- possible location of the hypothetical vehicle includes que
tion from a database and discretizing the traffic lane into rying traffic lane information from a database and discretiz
discretized points, ( 5 ) generating an unknown skeleton of ing the traffic lane into discretized points, generating an
discretized points of a lane that cannot be perceived by the 25 unknown skeleton of discretized points of a lane that cannot
sensor, ( 6 ) generating a representative shape at a discretized be perceived by sensors , generating a representative shape at
point of the unknown skeleton , and (7 ) evaluating if the a discretized point of the unknown skeleton , and evaluating
representative shape is completely within the non -observ- if the representative shape is completely within the unper
able part. A representative shape may be treated as the ceived world . A representative shape within the unperceived
hypothetical object. 30 world is treated as the hypothetical vehicle .
Implementations may apply temporal filtering to deter- The technologies may apply temporal filtering to deter
mine a location of the hypothetical object. The filtering mine the location of the hypothetical vehicle. The filtering
comprises smoothing an unknown skeleton by a forward process smoothes an unknown skeleton by a forward propa
propagated unknown skeleton , wherein the forward propa- gated unknown skeleton , wherein the forward propagated
gated unknown skeleton is generated by moving forward an 35 unknown skeleton is generated by moving forward an old
old unknown skeleton along a traffic lane . unknown skeleton along a traffic lane .
The hypothetical object is assigned one or more attributes. The hypothetical vehicle is assigned one or more attri
The one or more of the attributes are related to a possible butes. The one or more of the attributes are related to a
motion state of the hypothetical object. The motion state possible motion state of the hypothetical vehicle . The
comprises one or more of the following factors: a stationary 40 motion state comprises a stationary condition . The hypo
condition, a moving condition , a speed, and a moving thetical speed is set to less than or equal to a predetermined
direction . The speed is set to less than or equal to a maximum value . The predetermined maximum value may
predetermined maximum value . The predetermined maxi- be a speed limit or a computationally derived quant. The
mum value may be set to a speed limit . In some cases , the quantity can be derived from other objects concurrently or
predetermined maximum value is a quantity derived from 45 previously observed in the environment. The quantity can be
other objects concurrently or previously observed in the derived from historical data, road configuration, traffic rules,
environment. The predetermined maximum value can be an event, a time , a weather condition, or a combination of
derived from historical data , road configuration , traffic rules , two or more of them .
an event, a time , a weather condition, or a combination of Implementations have access a database with road net
two or more of them . 50 work information. Data from one or more sensors is used .
Implementations include accessing a database with road The one or more sensors include a radar sensor, or a lidar
network information , or using data from a second set of sensor, or a camera sensor, or a combination of two or more
sensors, or both . The sensor or a second set of sensors of them . The camera sensor comprises a stereo camera
comprises one or more of the following: a radar sensor, a sensor, or a monocular camera sensor, or both .
lidar sensor, and a camera sensor. A camera sensor may be 55 In general, in an aspect , technologies include an apparatus
a stereo camera sensor, a monocular camera sensor, or both . comprising an autonomous vehicle . The autonomous vehicle
Generating the commands for operating the vehicle comprises controllable devices configured to cause the
includes updating a trajectory for the vehicle, or executing autonomous vehicle to move on a road network ; aa controller
the trajectory for the vehicle, or both . The vehicle comprises to provide commands to the controllable devices ; and a
an autonomous vehicle . 60 computational element to update the commands in response
In general, in an aspect , technologies generate commands to current data representing a hypothetical speed and moving
to cause an autonomous vehicle to drive on a road network direction of a hypothetical vehicle also being driven on the
at specified speeds and make specified turns to reach a goal road network . The hypothetical speed and moving direction
position . The commands are updated in response to current is probabilistically derived based on vehicles previously
data representing a hypothetical speed and moving direction 65 observed in the environment.
of a hypothetical vehicle also being driven on the road The observable part and the non - observable part are
network The commands are updated to reduce aa risk of the separated by a boundary. The computational element detects
 US 11,400,925 B2
5 6
the boundary. Detecting of the boundary comprises using FIG . 18 shows an example of setting an expected future
data from one or more sensors to distinguish an observable trajectory.
ground from a foreground that obscures a portion of the
ground. The one or more sensors comprise sensors onboard DESCRIPTION
5
the autonomous vehicle , sensors offboard the autonomous
vehicle , or both . The phrase " environment of an AV ” is used in this
Generating the current data representing a hypothetical document to broadly include , for example, the area , geog
speed and moving direction of a hypothetical vehicle may be raphy, locale , vicinity , or road configuration in which the AV
based on known objects perceived by one or more sensors . is located or driving including the road network and features,
In some cases, the data generation includes one or more of 10 the
that built
are inenvironment, the . current
the environment conditionssometimes
This document , and objects
uses
the following operations: querying traffic lane information the term “ world ” interchangeably with “ environment.”
from a road network database , using stored data to infer a The term “ trajectory ” is used herein to broadly include
possible location of the hypothetical vehicle , and determin any path or route from one place to another, for example , a
ing a location of the autonomous vehicle based on a road 15 path from a pickup location to a drop off location .
network database and one or more sensors . Inferring the The term “ traffic lane” is used herein to broadly include
possible location of the hypotheticala vehicle includes que- any type of lane (e.g. , unpaved surface, sidewalk , crossings,
rying traffic lane information from a database and discretiz- pedestrian walks , road , street, highway, freeway , truckway ,
ing the traffic lane into discretized points, generating an vehicle lane, bicycle lane, bus lane , tram lane, rail road ,
unknown skeleton of discretized points of a lane that cannot 20 acceleration lane, merge lane, deceleration lane, turn lane,
be perceived by sensors , generating a representative shape at passing lane, climbing land , crawler lane, operational lane,
a discretized point of the unknown skeleton, and evaluating auxiliary lane, ramp, shoulder, emergency lane, breakdown
if the representative shape is completely within the unper- lane , transfer lane, express lane , collector lane , dedicated
ceived world . A representative shape within the unperceived lane , carpool lane , toll lane , parking lane , fire lane , and slow
world is treated as the hypothetical vehicle . 25 lane) for a moving object to travel.
The technologies may apply temporal filtering to deter- The term " object" is used in this document to broadly
mine the location of the hypothetical vehicle. The filtering include vehicles (e.g. , cars, wagons , trucks , buses , bicycles ,
process smoothes an unknown skeleton by a forward propa- motorcycles, trains, trams, watercrafts, aircrafts, and space
gated unknown skeleton , wherein the forward propagated crafts ), people , animals, signs, poles , curbs, traffic cones ,
unknown skeleton is generated by moving forward an old 30 barriers, mobile signs , trees, bushes , greenspaces, parks,
unknown skeleton along a traffic lane . railroads, worksites, stones , boulders, tombs, rivers, lakes ,
The hypothetical vehicle is assigned one or more attri- ponds , floods, logs , grasslands, snowbanks, deserts, sands,
butes . The one or more of the attributes are related to a buildings , and obstacles .
possible motion state of the hypothetical vehicle. The The term “ world model" is used in this document to
motion state comprises a stationary condition . The hypo- 35 broadly include a representation of an environment of an AV .
thetical speed is set to less than or equal to a predetermined The term “ area ” is used in this document broadly to
maximum value . The predetermined maximum value may include, for example, a physical region in an environment of
be a speed limit or a computationally derived quantity. The an AV, regardless of presence or absence of an object.
quantity can be derived from other objects concurrently or The term “ perceived world ” is used in this document to
previously observed in the environment. The quantity can be 40 broadly refer to areas or objects or attributes of areas or
derived from historical data , road configuration , traffic rules , objects or a combination of areas or objects or attributes that
an event, a time , a weather condition , or a combination of are perceived or observed or known in an environment.
two or more of them . The term “ unperceived world ” is used in this document to
The computational element accesses a database with road broadly refer to areas or objects or attributes of areas or
network information. Data from one or more sensors is used . 45 objects or a combination of areas or objects or attributes that
The one or more sensors include a radar sensor, or a lidar are unperceivable or non - observable or unknown in an
sensor, or a camera sensor, or a combination of two or more environment.
of them . The camera sensor comprises a stereo camera The term “ dark object ” is used herein to broadly include
sensor, or a monocular camera sensor, or both . an unknown object in the unperceived world . Information in
These and other aspects , features, and implementations 50 a world model about dark objects of the unperceived world
can be expressed as methods, apparatus, systems , compo- may be inferred or simulated or imagined or generated . This
nents, program products, methods of doing business , means document sometimes uses the term “ unknown object” inter
or steps for performing a function , and in other ways. changeably with “ dark object . ”
These and other aspects , features, and implementations The term “ goal” or “ goal position ” is used herein to
will become apparent from the following descriptions, 55 broadly include a place to be reached by an AV , including ,
including the claims . for example , an interim drop off location , a final drop off
location , and a destination , among others.
BRIEF DESCRIPTION OF THE DRAWINGS Although this document describes technologies based on
AVs, the technologies are equally applicable to semi-autono
FIG . 1 shows an example of an AV system . 60 mous vehicles, such as so - called Level 2 and Level 3
FIGS . 2 through 6 show examples of road scenarios. vehicles ( see SAE International's standard 13016 : Tax
FIGS . 7 through 9 show examples of processes . onomy and Definitions for Terms Related to On - Road Motor
FIG . 10 shows an example of a scanning process . Vehicle Automated Driving Systems , which is incorporated
FIGS . 11 through 15 show examples of boundary deter- by reference in its entirety, for more details on the classifi
minations. 65 cation of levels of autonomy in vehicles ) which attempt to
FIG . 16 shows an example of dark vehicle generation . control the steering or speed of a vehicle . The Level 2 and
FIG . 17 shows an example of temporal filtering. Level 3 systems may automate certain vehicle operations,
 US 11,400,925 B2
7 8
such as steering and braking, under certain driving condi- memory storage unit 32 on the vehicle or transmitted to
tions based on analysis of sensor inputs. Level 2 and Level the vehicle via wireless communications from a remote
3 systems in the market typically solely reason about the database 34 .
perceived world , for example, about obstacles that are 6. One or more data sources 36 for providing digital road
directly perceived by vehicle sensors during the decision 5 map data drawn from GIS databases , potentially
making process . The technologies described in this docu- including on or more of the following: high -precision
ment can benefit the semi-autonomous vehicles. Further, the maps of the roadway geometric properties ; maps
technologies described in this document also can assist describing road network connectivity properties ; maps
driving decisions of human -operated vehicles. describing roadway physical properties ( such as the
AVs 10 number of vehicular and cyclist traffic lanes , lane
As shown in FIG . 1 , a typical activity of an AV 10 is to width , lane traffic direction , lane marker type and
safely and reliably drive autonomously through an environ location ); and maps describing the spatial locations of
ment 12 to a goal position 14 , while avoiding vehicles, road features such as crosswalks , traffic signs of various
types ( e.g. , stop , yield ) and traffic signals of various
pedestrians, cyclists , and other obstacles 16 and obeying 15 types ( e.g. , red -yellow - green indicators, flashing yel
rules of the road . The AV's ability to perform this activity low or red indicators, right or left turn arrows ). Such
often is referred to as an autonomous driving capability. data may be stored on a memory storage unit 32 on the
The autonomous driving capability of an AV typically is AV, or transmitted to the AV by wireless communica
supported by an array of technologies 18 and 20 , ( e.g. , tion from a remotely located database , or a combination
hardware , software, and stored and real time data) that this 20 of the two.
document together refers to as an AV system 22. In some 7. One or more data sources 38 for providing historical
implementations, one or some or all of the technologies are information about driving properties (e.g. , typical
onboard the AV . In some cases , one or some or all of the speed and acceleration profiles) of vehicles that have
technologies are at another location such as at a server ( e.g. , previously traveled along local road sections at similar
in aa cloud computing infrastructure ). Components of an AV 25 times of day. Such data may be stored on a memory
system can include one or more or all of the following storage unit 32 on the AV, or transmitted to the AV by
( among others ). wireless communication from a remotely located data
1. Memory 32 for storing machine instructions and vari base 34 , or a combination of the two .
ous types of data . 8. One or more computer systems 40 located on the AV for
2. One or more sensors 24 for measuring or inferring or 30 executing algorithms (e.g. , processes 42 ) for the on - line
both properties of the AV's state and condition , such as ( that is , real -time on board ) generation of control
the vehicle's position , linear and angular velocity and actions based on both real - time sensor data and prior
information, allowing an AV execute its autonomous
acceleration , and heading (i.e. , orientation of the lead driving capability.
ing end of the AV ). For example, such sensors can 35 9. One or more interface devices 44 (e.g. , displays ,
include, but are not limited to : GPS ; inertial measure mouses , track points, keyboards, touchscreens, speak
ment units that measure both vehicle linear accelera ers , biometric readers, and gesture readers) coupled to
tions and angular rates; individual wheel speed sensors the computer system 40 for providing information and
for measuring or estimating individual wheel slip alerts of various types to , and to receive input from ,
ratios; individual wheel brake pressure or braking 40 occupants of the AV . The coupling may be wireless or
torque sensors ; engine torque or individual wheel wired . Any two or more of the interface devices may be
torque sensors ; and steering wheel angle and angular integrated into a single one .
rate sensors . 10. One or more wireless communication devices 46 for
3. One or more sensors 26 for measuring properties of the transmitting data from a remotely located database 34
AV's environment. For example, such sensors can 45 to the AV and to transmit vehicle sensor data or data
include, but are not limited to : LIDAR ; RADAR ; related to driving performance to a remotely located
monocular or stereo video cameras in the visible light, database 34 .
infrared and /or thermal spectra ; ultrasonic sensors ; 11. Functional devices and AV features 48 that are instru
time- of - flight (TOF ) depth sensors ; and temperature mented to receive and act on commands for driving
and rain sensors . 50 (e.g. , steering, acceleration , deceleration, gear selec
4. One or more devices 28 for communicating measured tion) and for auxiliary functions (e.g. , turn indicator
or inferred or both properties of other vehicles ' states activation ) from the computer system .
and conditions, such as positions , linear and angular 12. Reasoning algorithm 101 for analyzing information
velocities , linear and angular accelerations, and linear regarding perceived world and unperceived world .
and angular headings. These devices include Vehicle- 55 World Model
to - Vehicle (V2V) and Vehicle - to - Infrastructure (V2I ) In implementations involving the operation of an AV, the
communication devices, and devices for wireless com- AV is designed to be driven without direct human control or
munications over point- to -point or ad -hoc networks or supervisory input through an environment, while avoiding
both . The devices can operate across the electromag- collisions with obstacles and obeying the rules of the road
netic spectrum ( including radio and optical communi- 60 (e.g. , rules of operation or driving preferences ). To accom
cations ) or other media (e.g. , acoustic communica- plish such autonomous driving, the AV (or more specifically,
tions) the computer system or data processing equipment associ
5. One or more data sources 30 for providing historical , or ated with , in some cases attached to , the vehicle) or the AV
real- time, or predictive information , or a combination system first typically constructs a world model .
of any two or more of them about the environment 12 , 65 In some implementations, a world model includes a
including, for example, traffic congestion updates and representation of an environment of the AV, e.g. , constructed
weather conditions . Such data may be stored on a using data from a geolocation device, or a map , or a
 US 11,400,925 B2
9 10
geographic information system , or a combination of two or ronment. Areas beyond the sensing range of the sensors are
more of them , and from sensors that observe any areas or part of the unperceived world . However, the sensing range
objects. To construct the world model , an AV system collects of the AV's sensors may give the AV system enough infor
data from a variety of sensors (e.g. , LIDAR , monocular or mation of the AV's environment to make decisions , such as
stereoscopic cameras, and RADAR ) that are mounted to or 5 at what speed to travel and when to brake to avoid collision
attached to or placed inside the AV, or onboard the AV. In with an object. The lack of information about the unper
some cases , data is collected from some sensors that are not ceived world may not affect the decisions of the AV system
on or within the AV, for instance, from another vehicle , because a previously unobserved area or object in the
buildings , traffic lights, street light, a person's mobile phone, unperceived world that becomes known to the AV system as
or a combination of them , in the neighborhood or in a remote 10 it moves may be sufficiently far away from the AV, giving the
location. Then , the AV system analyzes the collected data to AV system enough time and distance to react safely once the
extract information ( e.g. , positions and motion properties ) area or object is observed .
about areas and objects in the environment. The AV may also FIG . 2 illustrates the first scenario . The values of the
rely on information gathered by onboard sensors , offboard speeds and distances described in this scenario are for
sensors, vehicle -to - vehicle communication, vehicle -to - in- 15 illustration , and the values may vary . In this example ,
frastructure communication , or information that is otherwise assume that the AV 201 has a sensor with a range 210 of 200
obtained from other data sources . m , is traveling at a rate of 30 m / s, and requires 150 m to
Given a world model , the AV system employs an algo- come to a complete stop (i.e. , stopping distance 211 ) . The
rithmic process to automatically generate and execute a shaded area 205 denotes the portion of the world that can be
trajectory through the environment toward a goal . The goal 20 observed by the sensor given its sensing range and lateral
is provided generally by another algorithmic process that field of view , and if there is no other sensor to be considered ,
may rely on a human input or on an automatic computational the shaded area denotes the perceived world of the AV . The
analysis. area outside of the boundary of the shaded area is non
In various applications, the world model comprises rep- observable by the sensor 201 and therefore is the unper
resentations of areas and objects in the perceived world and 25 ceived world . An object, such as the vehicle 202 , may be
in the unperceived world . Objects in the perceived world present in the unperceived world at the very edge of the
include objects that are observable by sensors onboard and sensor range ( e.g. , 200 meters away ). Even if the vehicle 202
offboard the AV , and objects about which the AV system has is stopped, the presence of such an object would not affect
information that is received from other data sources . the driving decisions of the AV system because the AV can
Sensors , regardless of types or onboard an AV or offboard 30 stop once it observes the object (as the object moves from
an AV, usually have a limited sensing range ; i.e. , sensors the unperceived world into the perceived world ) without a
>

only can observe areas or objects up to a certain extent of risk of collision with the stationary object 202 .
physical measurements, for example, distance , width , ver- The second scenario is illustrated in FIG . 3. The values of
tical extent, horizontal extent, orientation , speed , electro- the speeds and distances described in this scenario are for
magnetic amplitude and frequency , audio amplitude and 35 illustration, and the values may vary. This situation is similar
frequency, weight, and pressure . Areas or objects or attri- to FIG . 2 , but the sensing range of the AV system's sensors
butes beyond limited sensing ranges of sensors may not be 310 is shorter, say limited to 60 m . The shaded area 305
observable or may not be determined by the AV system . In represents the corresponding extent of the perceived world .
addition , because some types of sensors collect sensing data In this situation , a stationary object, for example the vehicle
along direct lines -of -sight, there may exist areas or objects 40 302 , that is located at the boundary of the perceived world
in the environment that are occluded from the view of the with the unperceived world may be 60 m away from the AV,
sensors due to the presence of other objects in the middle of which is less than the stopping distance 311 ( 150 m) that the
those lines- of - sight. AV requires in order to reach a full stop . In such aa situation,
In some implementations, an AV system generally has the current speed of the AV, the stopping distance , and the
information of static properties of the world (both perceived 45 sensing range of the sensors combine to put the AV at risk
world and unperceived world) , such as the road network , of being unable to stop before hitting an obstacle that is in
which typically come from one or more data sources ( e.g. , the unperceived world and about to enter the perceived
maps or geographic information systems or both ) and the world . To avoid such a risk , the AV system may decide to
dynamic properties of objects in the perceived world . In reduce the AV's speed even before the obstacle enters the
contrast, the AV system lacks information about the dynamic 50 perceived world , thus shortening its stopping distance to
status ( i.e. , the information about movable or moving or correspond to the sensing range of the sensors and thus be
changing objects such as vehicles, pedestrians, animals and able to stop before colliding with the object 302 that is
their attributes, e.g. , positions , orientations, and velocities ) located at the boundary of, or beyond, the unperceived
of the unperceived world , so the technologies in this docu- world . Therefore, the possibility that such an object will be
ment present methods to deal with this lack of information 55 present in the unperceived world affects the driving deci
about the dynamic status of the unperceived world . sions of the AV . Therefore, the technologies described in this
In some situations, the lack of information about the document include a reasoning algorithm 321 in the AV
unperceived world may not affect the decisions that the AV system to provide for the possible presence of unperceived
system makes . In other situations , the lack of information objects, even though these objects are unknown because
about the unperceived world may be critical for the AV 60 they lie beyond the sensor range of the AV .
system's decision -making. The following exemplary sce- The third scenario is that the sensing range of an AV
narios illustrate the situations. system's sensors is blocked by another moving object.
In the first scenario , an AV system may be traveling on a Referring to FIG . 4 , an AV 401 may be following another
straight road where the sensing range of its sensors in the vehicle 402 (called the “ leading vehicle ” ) that is immedi
direction of travel is not blocked by any objects. Neverthe- 65 ately ahead of the AV . In this situation, the leading vehicle
less , the limited sensing range of sensors implies that the may partially or fully block the sensing range of one or more
sensors can only perceive a limited part of the AV's envi- of the AV’s sensors , including restricting the view of traffic
 US 11,400,925 B2
11 12
in the opposing traffic lane , for example . The first shaded just beyond the hill crest (to the right of line 605 ) may lie in
area 405 represents a perceived world of the AV . The second the unperceived world , because that segment of the road
shaded area 406 represents an area that is part of the cannot be observed by the AV sensors . Therefore, an object,
unperceived world , but would have been part of the per- such as the vehicle 602 , on the unperceived road segment
ceived world had the sensing range and lateral field of view 5 may become an obstacle for the AV 601 , because the AV 601
of the AV system's sensors not been restricted by the leading may be unable to stop without colliding with the obstacle .
vehicle 402. In FIG . 4 , the vehicle 403 traveling the oppos- The technologies described in this document enable the AV
ing traffic lane belongs to the unperceived world . However, system to reason about the need to travel at a lower speed in
part of the vehicle 403 is present in the shaded area 406 , and
2 order to have sufficient time to react to obstacles that may be
therefore the vehicle 403 would have been perceived by the 10 present in the unperceived world .
AV 401 if the AV's sensors had not been partially occluded In general, although the AV system does not have infor
by the leading vehicle 402. Consider, for example, a situa- mation of the unperceived world and one or more objects
tion where the AV 401 needs to make a decision about may be present in the unperceived world, it is important for
whether to overtake the leading vehicle by temporarily the AV to reason about possible unknown objects in the
crossing over into the opposing lane of traffic. In this 15 unperceived world . The unknown objects may influence the
situation , information about the unperceived world , specifi- AV system's decision -making process and the trajectory that
cally the potential presence of an obstacle in the opposing will be chosen or adapted and executed by the AV system .
traffic lane, such as the vehicle 403 , may be critical to
2 This document describes technologies for, among other
making safe decision about the overtaking maneuver. things, reasoning about unknown objects that may poten
Therefore, the presence of such an object could affect the 20 tially be located in the unperceived world , with the aim of
driving decisions of the AV 401 , because the AV 401 may not improving the safety, comfort, or other aspects of the AV
be able to safely execute a passing maneuver without system's decision -making process and ultimately, the AV's
colliding with the obstacle . Therefore, the technologies operation.
described in this document include a reasoning algorithm Dark Objects Process
421 in the AV system to handle this scenario . In other words , 25 A broad idea of the technologies described in this docu
the AV system could make decisions based not only on ment is for the world model to systematically generate or
actual information of the perceived world , but also on infer the existence of hypothetical (that is , unknown ) objects
hypothetical information about the unperceived world . in the unperceived world . The dark objects are generated or
The fourth scenario is that the sensing range of an AV inferred , for instance, in a manner ( that is to say having
system's sensors is blocked by a non -moving object in its 30 assumed positions , orientations, and velocities , for example )
environment. For instance, there may be buildings , bill- that may change the AV's speed or trajectory or other aspects
boards, or other objects positioned along the road near an of the AV system's decision making process. The dark
intersection , which limit the ability of the AV system objects process is recursively executed through a time
observe, for example, cross - traffic traveling on the intersect- course . The technologies help the AV system to make
ing road . Referring to FIG . 5 , an AV 502 is approaching an 35 decisions in the face of uncertainty about the unperceived
intersection at which the conflicting traffic is not controlled world .
by a traffic signal , with the intention of proceeding straight In some implementations, the autonomous driving capa
along the trajectory 505. However, part of the trajectory 505 bility of an AV is achieved by the process illustrated in FIG .
may conflict with a trajectory 506 of another vehicle 503 7 , or a similar process. Some implementations may not use
2

traveling in another lane . Generally , the AV 502 may come 40 all of the operations illustrated in FIG . 7. Some implemen
to a stop if there is a STOP sign , check for and yield to any tations may divide tasks differently from the way illustrated
conflicting traffic , and then proceed straight. However, the in FIG . 7. Some implementations may include additional
presence of buildings or other objects near the road , such as processes beyond those illustrated in FIG . 7. In general, an
the built environment 501 , may partially occlude the sensing AV system starts with a perception process 701 , which uses
range and lateral field of view of the AV's sensors , resulting 45 data from onboard sensors , prior information stored in
in a reduced perceived world ( represented by the shaded databases, and other data sources to localize ( e.g. , determine
area 504 ) that does not contain potentially conflicting the current position of) the AV and to create a world model
vehicles such as 503. In such situations, the technologies 702 as of a current time . The AV system may also create one
described in this document may enable the AV 502 to or more predicted world models as of one or more future
override its normal practice of proceeding straight at normal 50 time steps . Next , the AV system operates a planning and
speed if no obstacle is observed in the perceived world and , decision -making process 703 , which uses the world model
instead , reason about how to deal with the lack of informa- generated from the perception process to plan a trajectory in
tion about potential objects in the unperceived world . The time - space to move the AV from its current position to its
technologies may, for example, enable the AV to adopt the goal position . The trajectory must satisfy requirements
strategy of slowly moving forward to increase its sensors ' 55 related to feasibility, safety, comfort, efficiency, or other
sensible range and thus increase the size of its perceived criteria or combinations of them . Next, the AV system
world . In this sense , it is therefore important for the AV 502 operates a control process 704 , which executes the trajectory
to be able to reason about when its information about the generated by the planning process by providing low -level
perceived world is sufficient, and when potential risks in the actuation signals needed to control the AV . These three
unperceived world are insignificant, for it to safely proceed 60 processes 701 , 703 and 704 are operated sequentially or
forward . simultaneously.
The fifth scenario is that the geometry of the road limits , The technologies described in this document include a
for example, the vertical aspect of the sensing range of dark objects process for analyzing and generating hypotheti
sensors regarding the environment of the perceived world . cal objects as part of the world model . The dark objects
Consider a situation illustrated in FIG . 6. An AV 601 may be 65 process runs as part of, or along with , the perception process .
driving on a road segment 603 where it is approaching a Referring to FIG . 8 , in some implementations, the dark
crest of a steep hill . In this situation , the segment of the road objects process 812 uses the same inputs as the perception
 US 11,400,925 B2
13 14
process 811 and modifies the world model 820 that is belonging to the road surface . If not , the point is assumed to
outputted by the perception process 811. The inputs com- be part of foreground. Such an analyzed pointcloud may be
prise one or more of the following : data from onboard called a “ semantic pointcloud .” For example, as shown in
sensors ( 801 ) ; data from offboard sensors , for instance on FIG . 10 , if a map used by the AV system 1001 has infor
other objects ( 802 ) ; and data about a road network from 5 mation about an exact elevation profile of the road surface
maps , databases, and other data sources (803 ) . This world 1004 , this information can be used to classify a given
model 820 serves as an input to the planning and decision LIDAR point as belonging to the road surface as follows.
making process in a manner that allows the planning and Given information of the current position and orientation of
decision making process to reason about the unperceived the AV, the position of the center 1002 of the LIDAR 1003
world and make decisions that account for unknown objects. 10 with respect to the center of the AV, and the exact elevation
Referring to FIG . 9 , in some implementations, a dark profile 1004 of the road surface or more generally the
objects process comprises a boundary determination process ground, the AV system 1001 computes a point ( 1009 , 1010 ,
901 and a dark object generation process 902. The boundary 1011 or 1012 ) at which an emitted LIDAR beam ( 1005 ,
determination process 901 uses data to determine a bound- 1006 , 1007 , or 1008 ) is expected to encounter the ground. If
ary between the perceived world and the unperceived world . 15 a point ( 1013 ) returned by a LIDAR beam ( 1008 ) is closer
The boundary between a perceived world and an unper- to the AV than the expected point ( 1012 ) by a predefined
ceived world is computed using data from , for example , difference, then the LIDAR beam may be assumed to have
onboard sensors, offboard sensors , maps, and other data encountered an object 1014 (e.g. , a point on the foreground ),
sources . The dark object generation process 902 generates and the point 1013 may be assumed not to belong to the
hypothetical dark objects for inclusion in the world model 20 ground. In some implementations, a machine learning based
and determines their attributes (e.g. , object type, position , approach ( for example, deep learning ) is used to perform the
orientation , direction, velocity, dynamics, and statics ) . task of foreground classification . Such an approach may also
Generation of aa dark object may comprise , for instance , fuse data from multiple sensors ( such as LIDAR and cam
considering, for example , a worst - case scenario for the dark eras) to improve the classification accuracy.
object being present in the unperceived world . The world 25 Consider a vertical slice of a LIDAR scan with Mx1
model is updated with the presence and attributes of these points that have been classified as described above . Assume
dark objects. Then , this updated world model is passed to the that a vertical slice comprises four LIDAR beams ( 1005 ,
planning and decision making process to generate and 1006 , 1007 , and 1008 ) , making M=4 in this case . The
execute a trajectory to the goal position . Such a trajectory is semantic label of each point in the semantic pointcloud is
planned and executed accounting for the presence of the 30 checked . It the semantic label of the first LIDAR point 1009
generated dark objects. The boundary determination process in this vertical slice is “ ground ” , it is safe to assume that the
and the dark objects generation process are described in space between the sensor origin and this LIDAR point is
more detail in the following sections . unobstructed . The subsequent points 1010 and 1011 will
Boundary Determination Process then be checked in order until a nearest foreground point
As described previously, for various reasons including 35 1013 is encountered . The space between the sensor origin
limited sensing range and field of view and presence of 1002 and this nearest foreground point 1013 is marked as
objects constraining the sensing range and field of view, an “ known ” in the world model . The perceived world expan
AV system's sensors may only be able to observe, or have sion for this vertical slice will stop at the nearest foreground
real - time information about , a limited part of the world point 1013 , and the areas behind the point 1013 will be
around it , referred to as the “ perceived world ” . The rest of 40 marked “ unperceived ” in the world model . However, if all
the world , that is , the complement of the perceived world , is points have been checked in order and no foreground point
referred to as the “ unperceived world ” . However, some is encountered, the final point of the scan is deemed a nearest
information about the unperceived world (e.g. , road con- foreground point, and the area beyond the final point is
figuration, traffic flow , traffic lights, peak time , and corre- marked " unperceived ".
sponding historical data ) may be still available from other 45 The process is performed for all N vertical slices that
data sources. constitute a LIDAR scan , and every foreground point is
The boundary determination process can be applied to determined for each slice as detailed above . The N nearest
analyzing any type of sensors and other data. Examples of foreground points represent sampled boundary points
sensors include LIDAR , radar, stereo vision cameras, mono between the perceived world and the unperceived world .
vision cameras , speed sensors , global positioning system 50 In some embodiments, a boundary in the 3 - D space can be
sensors , gyrosensors , and for aa combination of two or more constructed by interpolating and extrapolating the sampled
of those or other sensors , among others. Similar processes boundary points. Consider, for example, the situation illus
may be devised for other sensors and for information trated in FIG . 11 A LIDAR 1110 is mounted on an AV 1100 ,
collected from other vehicles or from sensors located in and the LIDAR 1110 makes N=3 scans and determines
infrastructure or other locations. 55 sampled boundary points 1101 , 1102 and 1103. An exem
Boundary Determination using Data from LIDAR Sen- plary 1 - D boundary ( denoted as 1101-1102-1103 ) may be
sors constructed by drawing lines 1104 and 1105. If the LIDAR
A typical 3D LIDAR sensor returns a pointcloud with is able to perform a 360 -degree scan, this boundary may be
MxN points in each scan , where M is the number of beams completed by joining the sampled boundary points as a
in a vertical direction , and N is the number of beams in the 60 contour. If the LIDAR does not perform a 360 -degree scan
horizontal direction . Each vertical slice of a scan is therefore ( such as the one shown in FIG . 11 ) , the boundary 1101
a subarea with Mx1 points along a specific bearing from the 1102-1103 may be extended by drawing a line 1106 from
sensor . Each beam emitted from the LIDAR returns the one end point 1103 of the 1 - D boundary to the center 1110
distance to the first object that the beam encounters . The of the LIDAR and another line 1107 from the center 1110 of
collection of such points is referred to as a pointcloud . 65 the LIDAR to the other end point 1101. The extended
A pointcloud can be analyzed to determine if each point boundary becomes a polygon, denoted as 1110-1101-1102
already has been classified in an available source of data as 1103-1110 . The area inside the boundary 1110-1101-1102
 US 11,400,925 B2
15 16
1103-1110 ( i.e. , the shaded area 1109 in FIG . 11 ) is deemed situation shown in FIG . 13 , where a RADAR 1310 mounted
to be a perceived world relative to the LIDAR and map data . on an AV 1300 shoots three beams 1301 , 1302 and 1303 .
The area 1111 outside of the boundary is the unperceived Each of these beams returns multiple foreground points ,
world . This boundary determination method is expected to denoted by the black dots ; for example, the beam 1303
perform well when the number of foreground points ( N ) is 5 receives returning points 1306 and 1307. Therefore , for
large, that is , when the LIDAR has a relatively high reso- individual beams 1301 , 1302 and 1303 , the AV system
lution. designates the closest points ( respectively 1304 , 1305 and
In some implementations, a boundary may be constructed 1306 ) as the sampled boundary points for determining the
using curves among foreground points instead of straight boundary between the perceived world and unperceived
lines . Various algorithms (e.g. , polynomial curve fitting, 10 world using the methods described above , for example, as
Bezier curves , etc. ) can be applied for boundary construc- illustrated by the shaded area 1309 in FIG . 13 .
tion . Boundary Determination using Data from Stereo Camera
While the above description of FIG . 11 constructs the 1 - D Sensors
boundary, the technologies can be extended to determining In comparison to RADAR and LIDAR sensors , stereo
a 2 - D boundary between perceived world and unperceived 15 cameras output denser pointclouds, especially for the over
world . For example, 2 - D patches (e.g. , flat or bendable ) can lapping part between a pair of acquired images . Semantic
be interpolated between points 1101 , 1102 , 1103 and 1110 , labeling of a pointcloud may be performed as described
and their union leads to a 2 - D boundary. earlier ( for example, using deep learning ) to distinguish
In some applications , a perceived world is constructed as foreground points and background points. A stereo camera
a union of multiple perceived areas , one for each vertical 20 setup comprises two or more cameras. One of the cameras
slice . The boundary is then defined as the boundary of the is designated as the origin of the light beams for the
union of the perceived areas . Consider for example a situ- semantically labeled pointcloud , and the pointcloud is pro
ation illustrated in FIG . 12 , where aa LIDAR 1210 is mounted cessed in a similar manner as for pointclouds from LIDAR
on an AV 1200. The LIDAR 1210 makes vertical slices and scans . Each of the labeled points may then be projected onto
identifies boundary sample points 1201 , 1202 and 1203. For 25 the ground surface, and sampled boundary points may then
each of these points, a buffer may be constructed in the be identified .
vicinity of the slice to represent the perceived area corre- FIG . 14 illustrates a boundary determination based on
sponding to that slice . For instance , the perceived area 1204 stereo camera sensors . Lines (e.g. , 1402 ) represent vertical
is constructed as a circular sector based on the point 1201 . slices going out of the stereo camera 1410. After the
The central radius of the sector is the line 1207 joining the 30 pointcloud has been semantically labeled and projected onto
point 1201 to the center of the LIDAR 1210. The sector is the ground plane, the black dots (e.g. , 1401 ) represent the
defined by sweeping a predefined angle theta on each side of nearest foreground point encountered along the correspond
this central radius. The angle of sweep , theta, may be ing slices (e.g. , point 1401 corresponding to the slice 1402 ) .
determined by various factors , including the horizontal Once the nearest foreground point for each slice has been
resolution of the LIDAR sensor 1210. This process is 35 determined , the techniques described previously may be
repeated for points 1202 and 1203 , resulting in perceived used to determine the boundary between the known and
areas 1205 and 1206 respectively. The complete perceived unperceived worlds, for example as illustrated by the shaded
world is then defined as the union of the three perceived area 1405 in FIG . 14 .
areas 1204 , 1205 and 1206 , and the boundary of this union Boundary Determination using Data from Monocular
may then be computed. 40 Camera Sensors
The boundary determination method can take into account The boundary determination process for monocular cam
additional factors or information. For instance , the boundary era images includes two steps as described below. The first
sample points are not limited to the nearest foreground step performs semantic labeling on an image to distinguish
points , but can be based on second-, third-, fourth-, or Z -th the ground ( e.g. , the road surface ) from the foreground. As
found foreground points. In some cases , a perceived area by 45 mentioned previously, this step can be based on classifica
another sensor, onboard AV or offboard AV, can be integrated tion and machine learning algorithms. The output of this step
into a perceived world ; for example, referring to FIG . 12 , the is an image in which pixels representing the observable road
area 1220 is determined to be a perceived area by a sensor surface are distinguished from pixels representing the fore
on another vehicle or by the AV 1200 itself at an earlier ground. The observable road surface represents a perceived
processing cycle , and the AV 1200 may expand its perceived 50 world, and everything else is assumed to belong to an
world by including the perceived area 1220 . unperceived world . Therefore, the boundary between the
Boundary Determination using Data from RADAR Sen- perceived world and unperceived world can be computed in
sors the pixel space.
RADAR measurements are similar to the LIDAR mea- The second step of the process is known as inverse
surements . The primary difference between LIDAR and 55 perspective mapping . Given information of the camera's
RADAR sensors is that while the LIDAR sensor returns the intrinsic properties ( e.g. , focal length ) and extrinsic proper
distance to the first object encountered by a beam , a RADAR ties (e.g. , the position and angle of the camera relative to the
sensor may be able to return the distance to more than one AV, and the AV’s position in the world model derived by
object in the path of a beam , because of the ability of localization ), a homograph transform can be performed to
RADAR beams to pass through certain objects. 60 map the 2 - D image plane in the pixel space to the road
Given these similarities, the method described above may surface plane in the 3 - D metric space . By performing inverse
also be used for boundary determination using RADAR perspective mapping , the boundary between the known and
measurements, by applying , for example, the following the unperceived worlds which was earlier estimated in the
adaptation: if a radar beam returns multiple foreground pixel space , is mapped to the 3 - D metric space .
points in response to encountering multiple objects, the 65 FIG . 15 illustrates this process . The scene 1510 represents
closest point may be designated as a sampled boundary point a typical image captured by a forward - facing monocular
for further boundary construction . For example, consider the camera . The scene 1520 represents the output of the seman
 US 11,400,925 B2
17 18
tic labeling step . The road surface pixels representing the or adjust the AV’s motion . In other words , the dark object
perceived world (shown as the shaded area 1521 ) are can be handled as a worst - case scenario for what might
retained, and the remaining pixels (which represent the reasonably exist in the unperceived world . By reasoning
unperceived world) are discarded . The scene 1530 shows the about a worst - case scenario that the AV system might
result of performing inverse perspective mapping on the 5 encounter, the AV system's planning and decision -making
image 1521. The scene 1530 corresponds to a top - down process can select an action that results in a conservative AV
view of the perceived world 1531 , i.e. , viewed from above motion .
the ground and looking straight down towards the ground. This document describes examples to illustrate generating
The thick contour 1532 represents the boundary between the dark objects in a manner that represents conservative sce
perceived world and unperceived world . The area outside 10 narios and allows an AV system to make a relatively safe
the boundary 1532 is the unperceived world . (e.g. , the safest possible) decision . This document describes
Boundary Determination using Data from Multiple Sen- methods for generating different types of dark objects, such
sors as vehicles , bicycles , buses , trains, pedestrians and animals ,
If an AV is equipped with multiple sensors , measurements as these are some of the most commonly encountered
from one or more of the sensors may be integrated to 15 objects on roads .
determine the boundary between the perceived world and Generation of Dark Objects
unperceived world . Integration of the sensor measurements An assumption is made that any unknown objects ( e.g. ,
can be employed as follows. dark vehicles ) in an unperceived world travel along existing ,
In some applications , measurements from two or more known traffic lanes on a road network and adhere to the rules
sensors are utilized and processed independently to generate 20 of the road . That is , they move within the speed limit ( or
a perceived world with respect to each individual sensor. within some other predefined upper bound that represents a
The perceived world may then be computed, for example , as commonly observed maximum speed for vehicles ) , within a
the union ( or intersection , or result of some other geospatial lane , and in the correct direction , and they observe any other
operation ) of these individual perceived areas, and the rules of the road , such as respecting traffic signs , traffic
boundary of the perceived world is the union of the bound- 25 lights, etc. Therefore, the generation process avoids gener
aries of individual perceived areas . ating dark objects that are moving above the speed limit , or
In some implementations, instead of forming individual in the wrong direction, or out of known traffic lanes. The
perceived areas from the measurements of individual sen- assumption ensures that the AV makes a safe and conserva
sors , measurements from two or more sensors are first fused tive decision , without being paralyzed in the face of uncer
together using data fusion techniques. The above described 30 tainty by having to consider the potential presence of
methods can then be applied to the fused data to determine unknown objects that are not moving according to the rules
the perceived world and evaluate the boundary between the of the road . However, it is also important to note that in
perceived world and unperceived world . This method may certain geographies, some rules of the road may be com
be useful, for example , for integrating RADAR and LIDAR monly flouted . For instance , vehicles in an area may com
sensors , because the measurements returned by these two 35 monly travel faster than a speed limit , and in such a case , a
types of sensors are similar, and the data processing methods dark vehicle may be generated with an average speed of the
are similar as well . vehicles in the area and does not have to be constrained by
Dark Object Generation Process the speed limit .
Dark objects are unknown or hypothetical or imaginary It is possible to generate dark objects of different types
objects. In a world model , dark objects are generated in the 40 and sizes and other attributes. The choice of which type and
unperceived world while observed objects are placed in the size of a dark object to generate, for example, depends on a
perceived world . A dark object generation process allows an number of factors including but not limited to the road
AV system to use the computationally derived boundary configurations, the fleet mix (that is , the mix of vehicles of
between perceived world and unperceived world to reason different attributes) observed in the neighborhood, the
about unknown objects, and plan the AV's movement ( e.g. , 45 vehicle type of the AV, the frequently observed objects, and
speed, direction , and trajectory ), for example in a safe and other considerations . For instance , a preferred rule of the
conservative manner, taking into account both observed dark object generation in an area where most objects are
objects in the perceived world and dark objects in the vans may be to generate a dark van . In an area where heavy
unperceived world . trucks are commonly observed, a preferred rule may be to
An unperceived world can be complicated. There may be 50 generate dark trucks. In an area where motorcycles are
many dark objects in an unperceived world . Further, dark commonly observed, a preferred rule may be to generate
objects may become present and later absent in the unper- dark motorcycles . The vehicle type of a dark object may also
ceived world . Dark objects can move in the unperceived be a random variable with a predefined distribution , and this
world , for example, approaching the AV . Therefore , in some distribution may be sampled to determine the type of a
implementations, the technologies described in this docu- 55 generated dark object.
ment can model pessimistic scenarios with respect to the To ease the illustrations, the following descriptions con
presence, absence , and motion of dark objects, among other sider objects as vehicles, but it is understood that other types
>

attributes. By modeling pessimistic scenarios , driving deci- of dark objects ( e.g. , dark pedestrians and dark animals ) can
sions for the AV system can be more conservative than be generated using the same method .
would otherwise be the case . However, in some implemen- 60 We assume that the AV system has access to detailed road
tations , a similar process may be employed to result in a less network information , which includes the configuration of
conservative driving decision or to satisfy other objectives. lanes on a roadway ( e.g. , number and position of lanes ,
The dark object generation process can generate dark position of centerlines , width , etc. ), and other relevant
objects in an unperceived world immediately adjacent to a information . Furthermore, the AV system is assumed to be
boundary with a perceived world and can assign reasonable 65 aware of its precise location derived from a localization
values to attributes (e.g. , position , speed, direction, and process that uses data from the AV system's sensors to
orientation ) of aa dark object that can be chosen to constrain estimate the precise position of the AV in the world model .
 US 11,400,925 B2
19 20
Among others, some or all of the following steps can be type ) of the dark vehicles, so that the dark vehicles can be
included in the dark vehicle generation process : inserted into a world model . In some implementations, a
1. Query the road network for lanes in the vicinity of the world model allows for inclusion of additional vehicle
AV system ( for example, within a range of 10 , 20 , 30 , attributes, such as , but not limited to : vehicle acceleration,
40 , 50 , 100 , 150 , 200 , 300 , 400 , or 500 meters ) and 5 status of turn indicators ( left turn or right turn indicators,
store the centerline of each lane as a sequence of also known as blinkers ), the expected future vehicle trajec
discretized points. tory ( for example , the expected trajectory of that vehicle for
2. Check the discretized centerline points of each lane to the next 1 second, 2 seconds , 3 seconds , 4 seconds , 5
see if they lie within the unperceived world that was seconds , 10 seconds , or 15 seconds , or the next 5 meters , 10
modeled in the step of boundary determination . Points 10 meters , 15 meters, 20 meters , 25 meters, or 30 meters ) , and
that lie within the unperceived world are labeled as part others. In some implementations, these and other attributes
of an " unknown skeleton " of that lane. are assigned values that conservatively (e.g. , most conser
3. For each lane , iterate through the points in its unknown vatively ) restrict the speed or trajectory or both of the AV.
skeleton . In some implementations, the iterations may FIG . 18 illustrates aa method for setting the expected future
be based on ordered distances between the points to the 15 trajectory of a generated dark vehicle . An AV 1801 is
AV . For a point under consideration , the dark vehicle approaching an intersection with the intention of turning
generation process creates a representative shape (e.g. , right along the trajectory 1804. The built environment 1802
a square , a rectangle, a triangle, a polygon, a circle , and restricts the sensor's sensing range and field of view , result
an ellipse ) centered at the point, oriented in the travel ing in the perceived area being limited as shown by the
direction of the lane , and of a size representing a 20 shaded polygon 1803. Following the procedure described
potential footprint of the dark vehicle . previously, a dark vehicle 1805 is generated as shown in the
4. If the potential footprint of the dark vehicle lies entirely unperceived world . To set the expected trajectory of this
within the unperceived world , generate a dark vehicle vehicle, the AV system considers the trajectories available to
centered at that point and deem it to be traveling in the the vehicle ( which may be obtained from the road network ).
known travel direction of that lane . Next, assign values 25 The dark vehicle 1805 may turn right along the trajectory
to the dark vehicle's attributes, e.g. , speed . The speed 1806 which is not in conflict with the AV's trajectory 1804 ,
assignment is based on various factors, for instance , or proceed straight along the trajectory 1807 which may
regulations ( e.g. , speed limits , traffic lights, and rule of conflict with the AV's trajectory 1804. Therefore, some
road ), road configurations ( e.g. , highway, intersection , implementations consider a conservative scenario by setting
bifurcation , oneway, four way stop , three way stop , 30 the trajectory 1807 as the expected future trajectory of the
T - intersection , and detour ,), weather conditions ( e.g. , dark vehicle 1805 as that scenario is more restrictive for
sunny, rainy, snowy, foggy, and smoky ), time ( e.g. , driving decisions of the AV system than the trajectory 1806 ,
peak hours, day time , night, and early morning ), and and is more representative of a conservative scenario .
events ( e.g. , accidents, protests , and parades ). For In general, when choosing between multiple trajectories
example , if the dark vehicle is driving in the same 35 for the dark vehicle, in implementations where the objective
direction as the AV , its speed may be set to 0 ; if the dark is to design a worst - case scenario , the trajectory of the dark
vehicle is driving in another direction, the dark vehi- vehicle that would cause the most delay for the AV could be
cle's speed may be set to a speed limit (or a predefined chosen . Other measures could also be used to select a
maximum speed contingent on such factors) of the trajectory for the dark vehicle from a set of possible trajec
lane. 40 tories , depending on the purpose for which the dark vehicles
5. Repeat the previous step for all lanes, until each lane are being generated.
has one or more dark vehicles generated for it , except In setting the attributes of the generated dark vehicles, it
that if no point in the unknown skeleton of a lane passes is possible to make use of detailed contextual information ,
the above check in item 2 , the lane will not have any
2 such as outputs from the perception process ( such as per
dark vehicles generated. 45 ceived or inferred states of traffic lights , and the locations ,
FIG . 16 illustrates an example of an AV 1601 following orientations and speeds of other objects ), detailed road
another vehicle 1602. The shaded polygon 1603 represents network information ( such as the positions of stop signs and
the perceived world perceived by one or more sensors . The turn restrictions), historical data on the behaviors of vehicles
road network is queried for lanes in the vicinity of the AV in the neighborhood, and other information . These addi
1601. The line 1605 represents the centerline of a lane, 50 tional data may be used to model the attributes of the dark
which is discretized into points. Points ( e.g. , 1606 ) outside vehicles in a richer manner, or for instance in a more
the perceived world 1603 constitute the unknown skeleton conservative manner (i.e. , in a way that places a greater
of the lane . The generation process iterates through the restriction on the speed and trajectory decisions of the AV
points on the unknown skeleton . For example, at the point system) . The following examples illustrate some scenarios.
1606 , a shape 1607 is created to represent a potential 55 If a dark vehicle is generated in a manner (e.g. , having a
footprint of a dark vehicle. As the potential footprint 1607 of position and orientation ) such that it is approaching a
the dark vehicle does not lie entirely outside the perceived traffic light, and if the status of that traffic light were
world 1603 , this point 1606 is disregarded, no corresponding known to be red , the speed of the dark vehicle may be
dark vehicle is generated , and the process moves to the set accordingly, instead of setting the speed to the speed
successive next points in the unknown skeleton, until reach- 60 limit of the lane, in this case , the speed may be set as
ing point 1608 and generating the footprint 1609 whose size O km /hr if the dark vehicle is located at the traffic light's
is completely in the unperceived world . A dark vehicle is stop line, or otherwise may be set to the maximum
then generated at the point 1608. The dark vehicle is oriented speed that the vehicle can drive at ( subject to the speed
in same direction as the lane's traffic flow , and its speed is limit of the lane) while being able to decelerate at a
set to the speed limit of the lane or another predefined limit . 65 reasonable rate and come to a stop before the stop line .
The process described above generates dark vehicles and A similar procedure may be adopted if the dark vehicle
sets attributes ( e.g. , the position, orientation , speed, size , and is approaching a stop sign .
 US 11,400,925 B2
21 22
If the dark vehicle is generated in a lane that also has other some or all of the following adaptations: instead of querying
known vehicles , the speed of the dark vehicle could be the road network for regular traffic lanes, the system queries
set to the known speed of the known vehicle immedi- for rail roads .
ately in front of the dark vehicle if the two are within Special Case : Dark Pedestrians
a certain distance of each other, or if not , to the 5 Pedestrians are generally not expected to walk in regular
maximum speed that the dark vehicle could drive at traffic lanes. Crosswalks or pedestrian crossings are
( subject to the speed limit of the lane ) while being able examples where pedestrians do share the same road space
to safely decelerate to the speed of the known vehicle with other vehicles and objects. In some cases , sidewalks are
in front of it . considered because pedestrians on a sidewalk may enter a
If historical data on vehicle behavior in a particular area 10 crosswalk . Therefore, it may be desirable to generate dark
shows that certain rules of the road are routinely pedestrians as part of the unperceived world . Similar to the
violated, then that behavior could also be applied to dark vehicle generation, the process queries sidewalks,
dark vehicles . For instance, if analysis of historical data crosswalks, pedestrian crossings, or a combination of them .
at a particular intersection shows that traffic light 15 Then , the dark object generation process is applied. Then a
violations are observed with a sufficiently high pre- dark pedestrian is assigned a reasonable walking speed or
defined probability, then aa dark vehicle being generated running speed, e.g. , 1 km /hr, 2 km /hr, 3 km /hr, 4 km /hr, 5
at the intersection may also be expected to violate the km /hr, 6 km /hr, 7 km /hr, 8 km /hr, 9 km /hr, 10 km /hr, 11
traffic light, with some predefined probability. For km /hr, 12 km /hr, 13 km /hr, 14 km /hr or 15 km /hr.
example, if a dark vehicle is generated at the intersec- 20 Temporal Filtering
tion just behind the traffic light stop line , and the status The methods described above for boundary determination
of the traffic light is known to be red, a traffic light and dark object generation can use instantaneous sensor
violation at this intersection could be provided for by readings, which may cause discontinuities in the boundary
setting the speed of the dark vehicle may be set to the and flickering of the dark objects for reasons that include , for
speed limit rather than to 0 km /hr. 25 example: noise in the sensor measurements and missing or
Special Case : Dark Bicycles delayed sensor measurements . Since boundary determina
The above method for generating dark vehicles may also tion and dark object generation are recursively executed
be applied to generating dark bicycles . This is particularly through a time course , a temporal filtering or smoothing
relevant for cities or areas that have exclusive bicycle lanes, 30
method may be employed to mitigate the issues by using
where there is road space that is not used by regular vehicles temporal relationships of dark objects generated at different
and is reserved for bicycles . In this case , the method for time steps. Temporal filtering or smoothing may be per
generating dark vehicles may be used with, for example, formed in many ways , and the following represents only
some or all of the following adaptations: instead of querying some implementations of this idea . The method below is
the road network for regular traffic lanes, the system queries 35 described from the point of view of the vehicle, and it may
for bicycle lanes. The speed of the bicycle may be set as high be adapted to other types of objects such as pedestrians and
as , faster than, or slower than the AV, up to a predefined animals. The steps used by a temporal filtering algorithm can
upper bound . If the bicycle lane is bidirectional , the gener- include some of or all of the following steps ( and others ).
ated dark bicycle may travel in one of two directions ; to be 1. The measurements from an instant time are processed
conservative, in some examples, the direction that results in 40 to compute the unknown skeletons for each lane , using
the bicycle moving closer to the AV can be used , as that is the methods described in the previous sections .
likely to result in a conservative scenario and thus a more 2. The points belonging to the unknown skeleton of each
conservative decision by the AV system . lane as of the previous time step are propagated for
Special Case : Dark Buses ward , that is , the points are moved forward along the
The dark vehicle generation process may also be applied 45 centerline of the lane in the travel direction of the lane .
to generating dark buses . This is particularly relevant for The distance by which the points are moved forward is
cities or areas that have exclusive bus lanes, i.e. , where there equal to the distance that a dark vehicle on that lane
is road space that is not used by regular vehicles and is would have traveled in that time step , that is , the speed
reserved for buses . In this case , the dark vehicle generation of a dark vehicle on that lane multiplied by the duration
process may be used with, for example, some or all of the 50 of the time step .
following adaptations : instead of querying the road network 3. The intersection of the two sets of unknown skeletons,
for regular traffic lanes , the process queries for bus lanes . i.e. , the unknown skeleton from the current time step's
The speed of the bus may be set as high as , faster than , or measurements ( computed in step 1 above) , and the
slower than the AV, up to a predefined maximum value . If the forward propagated unknown skeleton from the previ
bus lane is designed wider for bidirectional traffic , the 55 ous time step ( computed in step 2 above) , is then taken
direction that results in the bus moving closer to the AV can as the unknown skeleton for the current time step . In
be used , as that is likely to result in a conservative scenario some situations , it is possible that an unknown skeleton
and thus a more conservative decision by the AV system . is not computed from the current time step's measure
Special Case : Dark Trains ments ( for example, if a sensor returns no measure
Railroad crossings represent portions of the road network 60 ments or returns junk measurements in the current time
where trains or trams and other vehicles may potentially step ); in such situations, the forward propagated skel
come into conflict. The dark vehicle generation process may eton from the previous time step may be designated as
also be applied to generating dark trains . The dark train the unknown skeleton for the current time step . Simi
generation process may infer whether an AV system needs to larly, in some implementations, the forward propagated
slow down as it approaches a railroad crossing , or if it is safe 65 unknown skeleton from the previous time step may be
for a vehicle to cross a railroad crossing . The method for unavailable ( for example, during initialization or the
generating dark vehicles may be used with, for example, first time step ); in such implementations, the unknown
 US 11,400,925 B2
23 24
skeleton computed from the current time step's mea- determining, with the at least one processor, a nearest
surements may be designated as the unknown skeleton foreground point to a sensor ; and
for the current time step. marking a first area between an origin of the sensor and
This process is illustrated in FIG . 17. The sequence of the nearest foreground point as perceived and a second
points 1701 represents the unknown skeleton from the 5 area beyond the nearest foreground point as unper
previous time step , while the sequence of points 1703 ceived, where N nearest foreground points represent
represents the unknown skeleton computed from the current sample boundary points, and
measurements . The unknown skeleton from the previous determining, with the at least one processor, the boundary
time step 1701 is propagated forward, in the travel direction
of the lane 1705 , by a distance 1706 (that is equal to the 10 3. data based on the sample boundary points.
The method of claim 2 , wherein the pointcloud is
distance that would have been covered by a dark vehicle on output by a light detection and ranging (LiDAR) sensor.
that lane during that time step ), to produce the forward 4. The method of claim 1 , wherein the point is adjacent to
propagated unknown skeleton from the previous time step the boundary, and the attribute assigned to the at least one
1702. Finally, the sequence of points 1704 represents the
intersection of the unknown skeleton computed from the hypothetical object constrains or adjusts motion of the
15 vehicle
current measurements 1703 and the forward propagated .
unknown skeleton from the previous skeleton 1702 , and it is 5. The method of claim 1 , wherein the attribute assigned
used as the unknown skeleton for the current time step . The is at least one motion state of the at least one hypothetical
unknown skeleton is then used to generate dark objects in object.
the current time step after the dark objects generated in 20 6. The method of claim 5 , wherein the at least one motion
previous time steps are discarded from the world model. state is speed and the speed is set to less than or equal to a
While embodiments have been shown and described in speed limit .
this document, such embodiments are provided by way of 7. The method of claim 1 , wherein the attribute assigned
example only. Variations, changes, and substitutions will be causes a worst - case scenario that the vehicle might encoun
apparent. It should be understood that other implementations 25 ter.
are also within the scope of the claims . 8. The method of claim 1 , wherein the at least one
The invention claimed is : hypothetical object is constrained to travel along traffic lanes
1. A computer - implemented method comprising: on the road network and adhere to rules associated with the
generating, with at least one processor, boundary data road network .
representing a boundary dividing a perceived area and 30 9. The method of claim 1 , wherein a behavior of the at
an unperceived area of an environment of a vehicle, least one hypothetical object is based on historical data of
where the boundary data is generated based on sensor vehicles traveling on a portion of the road network where
data; rules of the road network are routinely violated .
generating, with the at least one processor , hypothetical 10. The method of claim 1 , wherein a type of the at least
object data representing at least one hypothetical object 35 one hypothetical object is determined based on an exclusive
at a particular location in the unperceived area of the lane of the road network .
environment, the generating comprising: 11. The method of claim 10 , wherein the type of the at
determining, with the at least one processor, whether at least one hypothetical object is a bicycle and the exclusive
least one lane of aa road network on which the vehicle lane is a bike lane .
is traveling lies within the unperceived area ; 40 12. The method of claim 10 , wherein the type of the at
in accordance with determining that the at least one lane least one hypothetical object is a bus and the exclusive lane
of the road network lies within the unperceived area of is a bus lane .
the environment of the vehicle , 13. The method of claim 10 , wherein the type of the at
generating, with the at least one processor, a representa- least one hypothetical object is a pedestrian and the exclu
tive shape for the at least one hypothetical object at a 45 sive lane is at least one of a pedestrian sidewalk or cross
point associated with the at least one lane ; walk .
determining, with the at least one processor, that the 14. The method of claim 10 , wherein a type of the at least
representative shape of the at least one hypothetical one hypothetical object is a train and the train is determined
object lies completely within the at least one lane in the based on presence of a railroad crossing light on the road
unperceived area of the environment; 50 network .
assigning, with the at least one processor, a value to at 15. The method of claim 10 , wherein the boundary and the
least one attribute of the at least one hypothetical at least one hypothetical object are generated based on
object; instantaneous sensor data that are temporally smoothed .
updating a world model with the generated hypothetical 16. A system comprising:
object; and 55 at least one sensor ;
generating, with the at least one processor, commands for at least one processor;
operation of the vehicle within the environment, the at least one computer -readable medium having stored
commands being based on the updated world model . thereon instructions, that when executed by the at least
2. The method of claim 1 , wherein the sensor data is a one processor, causes the at least one processor to
pointcloud and generating the boundary data further com- 60 perform operations comprising :
prises: generating boundary data representing a boundary divid
dividing, with the at least one processor, the pointcloud ing a perceived area and an unperceived area of an
into N vertical slices , where N is a positive integer environment of a vehicle , wherein the boundary data is
greater than one ; generated based on sensor data ;
for each slice : 65 generating hypothetical object data representing at least
determining, with the at least one processor, foreground one hypothetical object at a particular location in the
points in the slice ; unperceived area, the generating comprising:
 US 11,400,925 B2
25 26
determining whether at least one lane of a road network 19. The system of claim 16 , wherein the point is adjacent
on which the vehicle is traveling lies within the unper- to the boundary, and the attribute assigned to the at least one
ceived area ; hypothetical object constrains or adjusts motion of the
in accordance with determining that the at least one lane vehicle .
of the road network lies within the unperceived area of 5 is at20.least
The system of claim 16 , wherein the attribute assigned
one motion state of the at least one hypothetical
the environment of the vehicle , object.
generating a representative shape for the at least one 21. The system of claim 20 , wherein the at least one
hypothetical object at a point associated with the at motion state is speed and the speed is set to less than or equal
least one lane; 10 to a speed limit .
determining that the representative shape of the at least 22. The system of claim 16 , wherein the attribute assigned
one hypothetical object lies completely within the at generates a worst - case scenario that the vehicle might
least one lane in the unperceived area of the environ encounter.
ment of the vehicle; 23. The system of claim 16 , wherein the at least one
hypothetical object is constrained to travel along traffic lanes
assigning a value to at least one attribute of the at least one 15 on the road network and adhere to rules associated with the
hypothetical object;
updating a world model that comprises the perceived area road24.network .
The system of claim 16 , wherein a behavior of the at
and the unperceived area with the at least one hypo least one hypothetical object is based on historical data of
thetical object; and vehicles traveling on a portion of the road network where
generating commands for operation of the vehicle within 20 rules of the road network are routinely violated .
the environment, the commands being based on the 25. The system of claim 16 , wherein a type of the at least
updated world model. one hypothetical object is determined based on an exclusive
17. The system of claim 16 , wherein the sensor data lane of the road network .
comprises a pointcloud , and generating the boundary data 26. The system of claim 25 , wherein the type of the at
further comprises: 25 least one hypothetical object is a bicycle and the exclusive
dividing the pointcloud into N vertical slices , where N is lane is a bike lane .
a positive integer greater than one ; 27. The system of claim 25 , wherein the type of the at
for each slice :
least one hypothetical object is aa bus and the exclusive lane
is a bus lane.
determining foreground points in the slice ; 30 28. The system of claim 25 , wherein the type of the at
determining a nearest foreground point in the foreground least one hypothetical object is a pedestrian and the exclu
points in the slice to the at least one sensor ; and sive lane is at least one of a pedestrian sidewalk or cross
marking a first area between an origin of the at least one walk .
sensor and the nearest foreground point as perceived 29. The system of claim 25 , wherein a type of the at least
and aa second area beyond the nearest foreground point 35
one hypothetical object is aa train and the train is determined
as unperceived, where N nearest foreground points based on presence of a railroad crossing light on the road
represent sample boundary points ; and network .
determining the boundary data based on the sample 30. The system of claim 25 , wherein the boundary and the
boundary points. at least one hypothetical object is generated based on
18. The system of claim 17 , wherein the pointcloud is instantaneous sensor data that are temporally smoothed .
output by a light detection and ranging ( LiDAR) sensor.