KR in the age of Deep Learning

Tetherless World Constellation, RPI
KR in the age of
Deep Learning,
Watson,
and the Semantic Web
Jim Hendler
Tetherless World Professor of Computer, Web and Cognitive Sciences
Director, Institute for Data Exploration and Applications
Rensselaer Polytechnic Institute
http://www.cs.rpi.edu/~hendler
@jahendler (twitter)
Major talks at: http://www.slideshare.net/jahendler

Talk derives in part from a recent book
(More info at Springer booth)
(Thanks Alice!)

Outline
• Several important AI technologies have
moved through “knees in the curve”
bringing much of the attention to AI again
– Deep Learning (& ML in general)
– Watson (& “cognitive computing”)
– Semantic Web (& the knowledge graph)
• But what about KR
– What it is, why it still matters
• And how can these come together
– Which comes with a lot of important challenges

A) Deep Learning
“phase transition” in capabilities of neural networks
w/machine power

Trained on lots of categorized images
Imagenet: Duck Imagenet: Cat

Impressive results
Increasingly powerful techniques have yielded
incredible results in the past few years

B) Watson

The Watson DeepQA Pipeline

Watson is based on ”Associative knowledge”
© IBM, used with permission.

Impressive Results
Watson showed the power of “associative knowledge”

C) Semantic Web

From Semantic Web to the Knowledge Graph

Based on a large “knowledge graph” mined from
extracted and learned data

Impressive results
Google finds embedded metadata on >30% of its crawl – Guha, 2015
Google “knowledge vault” reported to have over 1.6 billion “facts” (links)

Summary: AI has done some way cool stuff
Summary (simplifying tremendously)
• Deep Learning: neural learning from data with high
quality, but imperfect results
• Watson: Associative learning from data with high
quality but imperfect results
• Semantic Web/Knowledge Graph: Graph links
formation from extraction, clustering and learning
As much as many of us “GOFAI” folks wish it, this stuff
cannot be ignored
but, there are still problems…

Many intermediate steps
(P. Norvig, WWW 2016, 4/16)

Why did knowledge graph need
“”Human Judgments”?
Association ≠ Correctness

GOFAI: Knowledge Representation?
• A knowledge representation (KR) is most fundamentally a surrogate, a
substitute for the thing itself, used to enable an entity to
determine consequences by thinking rather than acting, i.e., by
reasoning about the world rather than taking action in it.
• It is a set of ontological commitments, i.e., an answer to the question: In
what terms should I think about the world?
• It is a fragmentary theory of intelligent reasoning, expressed in terms of
three components: (i) the representation's fundamental conception of
intelligent reasoning; (ii) the set of inferences the representation
sanctions; and (iii) the set of inferences it recommends.
• It is a medium for pragmatically efficient computation, i.e., the
computational environment in which thinking is accomplished. One
contribution to this pragmatic efficiency is supplied by the guidance a
representation provides for organizing information so as to facilitate
making the recommended inferences.
• It is a medium of human expression, i.e., a language in which we
say things about the world.
R. Davis, H. Shrobe, P. Szolovits (1993)

KR: Human Expression
Cute kid story: first two words

Telling cats from ducks doesn’t need KR
!

“Saying things about the world” does
"If I was telling it to a
kid, I'd probably say
something like 'the cat
has fur and four legs and
goes meow, the duck is a
bird and it swims and
goes quack’. "
How would you explain the difference between a
duck and a cat to a child?
Woof

KR: Surrogate knowledge?
Which could you sit in?
What is most likely to bite what?
Which one is most likely to become a computer
scientist someday?
…

“Surrogate” knowledge
Which could you sit in?
What is most likely to bite what?
Which one is most likely to become a computer
scientist someday?
How would they go about doing it?

KR: Recommended vs. Possible inference
Which one would you save if the house was on fire?

Recommended vs. Possible inference
Which one would you save if the house was on fire?
Would you use a robot baby-sitter
without knowing which of the three
possibilities it would choose?

KR systems in AI need grounded symbols
• Logic- and rule- based systems
– Ground in “model theory” with a notion of truth
and falsity
• Probabilistic Reasoning
– P(A|B) requires A, B map to “meaningful”
concepts, P to be a “real” probability
• Constraint Satisfaction, etc
– Finding an interpretation satisfying a set of
boolean (T,F) constraints
(Note: Yes, I am simplifying, blurring distinctions, ignoring
much cutting edge work… happy to discuss later)

The challenge
• If we want to implement KR systems
on top of neural and associative
learners we have an issue
– The numbers coming out of Deep
Learning and Associative graphs are not
probabilities
– They don’t necessarily ground in
human-meaningful symbols
• ”sub-symbolic” learning …
• Association by clustering …
• Errorful extraction …

The challenges
• Can we avoid throwing out the
reasoning baby with the grounding
bathwater?
– We still need planning systems
– We still want to be able to define the
rules that a system should follow
– We want to be able to interact with and
understand these systems
• Even if computers don’t need to be symbolic
communicators, WE DO!!!

Starting Place: Rethinking grounding
– Formal Explanation vs. post hoc
justification
• Eg. Even if we cannot use a formal
decomposition to explain the reasoning, can
we produce a justification that explains it
– Reasoning systems that “know” some of
their axioms may be simply wrong
• Eg.F1 of .9 doesn’t mean answers are 90%
correct, it is (simplifying) more like 9 out of
10 answers are right, the others aren’t.
– Nailing context …

Vision Challenge 1
What is the relationship
between this man and
this woman?

Vision Challenge 1
this woman?
Deep learning produced Scene Graph w/relationships
(Klawonn & Heims, 2018)

Vision and Knowledge Challenge
this woman?
(Klawonn, 2018)

Vision and Knowledge Challenge
this woman?
(Klawonn, 2018)
Seeing the bride adds
significant information
that cannot be easily
learned w/o background
knowledge

This kind of context matters!

Human-Aware AI
• Context is key
– AI learning systems still perform best in
well-defined contexts (or trained
situations, or where their document set
is complete, etc.)
– Humans are good at recognizing context
and deciding when extraneous factors
don’t make sense
• Or add extra “inferencing” (the bride
example)

Learning inferences

In noisy data

Summary of talk (minus moose)
• Modern AI is making some huge strides
– Eg. DL, Associative Learning, Knowledge
Graphs, …
• But the need for KR has not gone away
– Eg. Surrogacy, Recommended Inference,
Human communication
• The integration challenge will require
rethinking some key AI ideas
– Grounding, explanation, context ….
• But we need to do it.

Questions?

KR in the age of Deep Learning

More Related Content

What's hot (20)

Similar to KR in the age of Deep Learning (20)

More from James Hendler (13)

Recently uploaded (20)

KR in the age of Deep Learning

Editor's Notes