Coherent Scene Understanding with 3D Geometric Reasoning

Coherent Scene Understanding with 3D Geometric Reasoning

Jiyan Pan12/3/2012

TaskDetect objects

Identify surface regions

Estimate ground plane

Infer gravity direction

Geometrically coherent in the

3D world

3D geometric context

O

xy

z

xbdb

dt

γ

nv

θ

xt

np

hp

ng

α Hf

ground plane

image plane(inverse) gravity

ground plane orientation

ground plane height

object vertical orientation

real world heightobject depthcamera center

focal length

object pitch and roll angles

object landmarks

Coordinate system

Deterministic relationships

Variables of global 3D geometries:

ng, np, hp

O

xy

z

xbdb

dt

γ

nv

θ

xt

np

hp

ng

α Hf

ground plane

image plane(inverse) gravity


ground plane height

object vertical orientation

real world heightobject depthcamera center

focal length

object pitch and roll angles

object landmarks

Coordinate system

Probabilistic relationships

Derived from appearance

Prior knowledge

Can we solve them all for a coherent solution?

• Non-linear• Non-deterministic• Even invalid equations from false detections

√

√

√

√

X

Global 3D context

Local 3D context

√

√

√

√

X

“Chicken and egg” problem: Local entities could be validated by global 3D context Global 3D context is induced from local entities

Global 3D context

Local 3D context

?

Possible solution (All in PGM)• Put both global 3D geometries and local entities in a PGM [1]

– Precision issue: Have to quantize continuous variables– Complexity issue: Pairwise potential would contain up to ~1e6 entries

[1] D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008

Ground

o1

o2

ok

Gravity

100(pitch) × 100 (roll) × 100 (height)

Possible solution (Fixed global geometries as hypotheses)

• Task much easier under a fixed hypothesis of global 3D geometries

Ground

o1

o2

ok

Gravity

× × × × × ×

• Task much easier under a fixed hypothesis of global 3D geometries

Possible solution (Fixed global geometries as hypotheses)

o1

o2

ok

ω1

ω2

ω3

How to generate global 3D geometry hypotheses?

Possible solution(Hypotheses by exhaustive search)

• Exhaustive search over the quantized space of global 3D geometries [2]

– Computational complexity tends to limit search space

[2] S. Bao et al. Toward coherent object detection and scene layout understanding. IVC, 2011

Possible solution(Hypotheses by Hough voting)

• Each local entity casts vote to the Hough voting space of the global 3D geometries and peaks are selected[3]

– False detections could corrupt the votes– Would applying EM help? Not likely, if false detections overwhelm

[3] M. Sun et al. Object detection with geometrical context feedback loop. BMVC, 2010

L1 L2 L3L5L4 L7L6

Our solution• We take a RANSAC-like approach: Randomly mix the

contributions of local entities

L1 L2 L3L5L4 L7L6

Our solution• We take a RANSAC-like approach: Randomly mix the

contributions of local entities– Compared to averaging over all local entities: More robust against outliers– Compared to directly using estimates from each single local entity: More robust against noise

L1 L2 L3L5L4 L7L6

0 5 10 15 20 25 30 35 40 45 501.6

1.8

2

2.2

2.4

2.6

2.8

3

Number of random mixtures

Min

imum

hyp

othe

sis

erro

r

Gravity Direction

IndividualMixtureAverage

0 5 10 15 20 25 30 35 40 45 501.6

1.8

2

2.2

2.4

2.6

2.8

3

3.2

Number of random mixtures

Min

imum

hyp

othe

sis

erro

r

Ground Plane Orientation

IndividualMixtureAverage

√

√

√

√

X

Local 3D context

Global 3D context


ground plane orientation valid

valid invalid (#1)

invalid (#1)invalid

(#1)

ground plane

#1: Common ground (global)


#2: Gravity direction (global)

(inverse) gravity

ground plane orientation invalid

(#2)

ground plane


#3: Depth ordering (local)

(inverse) gravity


incompatible (#3)

ground plane


#4: Space occupancy (local)

(inverse) gravity


incompatible (#4)

ground plane

2

345

6

1

2

345

6

1

Global geometric compatibility for an object:

Orientation:

Given a global 3D geometry hypothesis

2

345

6

1

Global geometric compatibility for an object:

Orientation:

Height:


2

345

6

1

Global geometric compatibility for a surface:

Orientation: local estimates vs. or

Location: horizontal surface region vs. ground horizon


2

345

6

1

Local geometric compatibility for two objects:

Depth ordering:

Space occupancy:


2

345

6

1

Objective function of the CRF:


0,01,5.0

ooss

o dg

else,0

1,,min,

)()(ji

ocpij

oclij

jiijooss

oo

√

√

√

√

X

Local 3D context

Global 3D context

Best hypothesis

3D reasoning agrees with raw detector

3D reasoning recovers detection rejected by raw detector

3D reasoning rejects detection accepted by raw detector

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

False Positive per Image

True

Pos

itive

Rat

eDeformable Part Model Detector

Baseline

Hoiem

Ours

3D geometric reasoning improves object detection performance

D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008

0 0.2 0.4 0.6 0.8 1 1.20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8


True

Pos

itive

Rat

eDalal-Triggs Detector

Baseline

Hoiem

Ours


D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008

Improvement in AP over baseline detector

Ours 10.4%

Hoiem 4.8%

Sun 5.1%

M. Sun et al. Object detection with geometrical context feedback loop. BMVC, 2010D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008


Horizon estimation median error

Ours 2.05⁰

Hoiem 3.15⁰

Sun 2.41⁰

M. Sun et al. Object detection with geometrical context feedback loop. BMVC, 2010D. Hoiem, A. A. Efros, and M. Hebert. Putting objects in perspective. IJCV, 2008

√

√

√

√

X

Local 3D context

Global 3D context

Best hypothesis

Contributions of different geometric context

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

0.1

0.2

0.3

0.4

0.5

0.6

0.7


True

Pos

itive

Rat

eDetection ROC Curve

Det

Det+IdvlGeo

Det+PairGeo

Det+FullGeo

Benefit is mutual

Error in gravity direction

Error in ground orientation

Vanishing points alone 2.62⁰ 4.85⁰

Whole system 2.05⁰ 2.21⁰

Extensions– Improved depth ordering constraint– Local geometric constraints involving vertical surfaces– Multiple supporting planes– Using more prior knowledge of objects– Utilizing semantic categories of surface regions

closer object

farther object

closer object farther object

occlusion mask of the farther object

intersection region of the two object masks

√

X

Fully cover?

Fully cover?

Occlusion: bottleneck in our system

– Missed detection– Erroneous estimation of local properties– Less effective depth ordering constraint

Generalized Hough voting: better at handle occlusions

K. Rematas et al. CORP 2011

B. Leibe et al. IJCV 2008

Occlusion-and-geometry-aware Hough voting

√

√

√

√

X

Local 3D context

Global 3D context

Best hypothesis

• So far we have treated the entire region labeled as "vertical" as a whole

Decompose vertical region into surface segments Occlusion boundary recovery (Hoiem et al. IJCV’11)Vanishing line sweeping (Lee et al. CVPR’09)

ground plane

inverse gravity

√

vertical surface candidate 1


ground plane


inverse gravity


X

ground plane

vertical surface candidateinverse gravity

object candidate

√

object candidate

ground plane

vertical surface candidateinverse gravity

X

Given object layout, erect surfaces one by one “Interpretation by synthesis” (Gupta et al. ECCV’10)

supporting plane 1

supporting plane 1

supporting plane 2

O

xy

z

ground plane

pn~

ph~

bx

vn~

bd

gn~

tx td tX

bt XX

bX

0H

w

l

β

pn~

ph~

• Spring 2013 (ICCV’13 submission)– Improved depth ordering constraint– Using more prior knowledge of objects– Multiple supporting planes

• Fall 2013 (CVPR’14 submission)– Local geometric constraints involving vertical surfaces– Utilizing semantic categories of surface regions

• During Spring Semester of 2014– Thesis writing

Expected Contributions

• Systematically model the relationships among global and local geometric variables

• Develop a RANSAC-CRF scheme to handle non-linear, non-deterministic, and possibly invalid relationships

• Occlusion-and-geometry-aware object detection for finer depth order reasoning

• Joint reasoning among global geometries, surface segments, and objects

Thank you!

Coherent Scene Understanding with 3D Geometric Reasoning

Documents

Transcript of Coherent Scene Understanding with 3D Geometric Reasoning