Time to Impact

unknown object size unknown object (camera) velocity

Time to Impact

3/7/11 ECEn631 2

€

z'= X fx'

=zxf⋅fax

=za

a =zz'

=x'x

=y'y

Time to Impact

3/7/11 ECEn631 3

Ignoringtheloca.onofthefocalcenter, x and y can be the width and height of the object.

The time to impact, τ, is the distance, z, divided by the velocity, v:

Time to Impact

3/7/11 ECEn631 4

If all we know is the frame rate, then this time to impact is measured as number of frames.

The expansion rate is not constant over time: for an object approaching the camera at a fixed velocity, v, a(t) is given by

, where z0 is the value of z at t = 0.

Time to Impact

3/7/11 ECEn631 5

The time to impact, τ, is the distance, z, divided by the velocity, v:

If we know the speed (or distance traveled between frames) that the camera is approaching the object (from onboard GPS or encoder), we can also estimate the distance to the object from the expansion rate.

Time to Impact

3/7/11 ECEn631 6

€

distance = τ × v = τ × (z − z' )

=aa−1

× (z'−z) =z

z'−z× (z'−z) = z

If the camera is calibrated and the object size is known, we can estimate the distance to the object from the object image size.

Time to Impact

3/7/11 ECEn631 7

€

z (mm) = f (pixel) X(mm)x (pixel)

Feature Identification

  Calculateimagederiva.ves,IxandIyusing3x3Sobelkernel(inhardware)

  Iden.fyfeaturesusingHarris’cornerdetectorforeachpixel:

3/7/11 ECEn631 8

Feature Selection

  Dividetheimageintosub‐regions  Select10bestfeaturesfromeachregion

3/7/11 ECEn631 9

Feature Tracking   Foreachframe,trackthehighestHarrisresponsewithinnpixelsofthepreviousfeature

3/7/11 ECEn631 10

3D Reconstruction   Foreachfeature,trackthedistanceofthefeaturefromtheop.calcenterforeachframeasvehiclemoves(centercanbeobtainedfromcameracalibra.onorusetheimagecenter)

3/7/11 ECEn631 11

3D Reconstruction   Foreachfeature,trackthedistanceofthefeaturefromtheop.calcenterforeachframeasvehiclemoves

3/7/11 ECEn631 12

3D Reconstruction   Foreachfeature,trackthedistanceofthefeaturefromtheop.calcenterforeachframeasvehiclemoves

3/7/11 ECEn631 13

3D Reconstruction   Foreachfeature,trackthedistanceofthefeaturefromtheop.calcenterforeachframeasvehiclemoves

3/7/11 ECEn631 14

3D Reconstruction

  Collectthedistancesfromtheop.calcenterasrobotmoves

3/7/11 ECEn631 15

Dis

tanc

e Fr

om O

ptic

al C

ente

r

Robot Distance Traveled (from wheel encoders)

3D Reconstruction

  Collectthedistancesfromtheop.calcenterasrobotmoves

3/7/11 ECEn631 16

Dis

tanc

e Fr

om O

ptic

al C

ente

r

Robot Distance Traveled (from wheel encoders)

3D Reconstruction

  Uselinearleast‐squarestogetclosed‐formapproxima.ontover.calasymptote

3/7/11 ECEn631 17

Dis

tanc

e Fr

om O

ptic

al C

ente

r

Robot Distance Traveled (from wheel encoders)

Distance to point

3D Reconstruction

  Giventhedistance(Z),intrinsicparameters(focallengthandop.calcenter),andtheimagepoints(x,y)wecandeterminethe3Dposi.onofthepoint(X,Y,Z)

3/7/11 ECEn631 18

Mapping the Environment   Usingthecalculatedpoints,wehaveasparsepointcloud

3/7/11 ECEn631 19

Mapping the Environment   Useaclippingvolumetoeliminatefeaturesonfloorandceiling

3/7/11 ECEn631 20

Tracked Features

3/7/11 ECEn631 21

Generated Map

3/7/11 ECEn631 22

Hardware Implementation

  Howdoyoucalculatereal‐.me.me‐to‐impactonasmallautonomousvehicle?– Real‐.me:15‐30fpsorhigherlivevideofeed

  GeneralPurposeCPUorGPU?– Toolarge,powerhungry,ornotfastenough

  CustomSiliconVisionProcessor– Tooexpensivetodeveloporproduceinsmallquan..es

3/7/11 ECEn631 23

Solution   FPGA‐basedRobo.cVisionPlaTorm

– OneormoreembeddedCPUs– Customhardwareaccelera.on– CompleteSystemonaProgrammableChip(SOPC)

3/7/11 ECEn631 24

Robot Platform Used in 2005

3/7/11 ECEn631 25

7”

3/7/11 ECEn631 26

Robot Platform   EmbeddedFPGACores

– XilinxMicroBlazeProcessor@75Mhz– CustomFloa.ngPointUnit(FPU)– 64KBRAMMemory– DDRSDRAMController– MotorController– CameraController– SerialInterfaces(UART,SPI)– TimersandInterruptControllers

3/7/11 ECEn631 27

Image Derivatives

3/7/11 ECEn631 28

Image Derivatives

3/7/11 ECEn631 29

Image Derivatives

3/7/11 ECEn631 30

Dual Port BRAM Buffer

External Camera

RAM Write

pixel FP

GA

Image Derivatives

3/7/11 ECEn631 31

DMA data

Dual Port BRAM Buffer

External Camera

RAM Write

RAM Read

pixel

Ix Pixel Matrix

Iy

FIFO

FIFO

FIFO

DMA Control

DMA address FPG

A

Image Derivatives

3/7/11 ECEn631 32

DMA data

Dual Port BRAM Buffer

External Camera

RAM Write

RAM Read

pixel

Ix Pixel Matrix

Iy

FIFO

FIFO

FIFO

DMA Control

DMA address FPG

A

Image Derivatives

3/7/11 ECEn631 33

DMA data

Dual Port BRAM Buffer

External Camera

RAM Write

RAM Read

pixel

Ix Pixel Matrix

Iy

FIFO

FIFO

FIFO

DMA Control

DMA address FPG

A

Image Derivatives

3/7/11 ECEn631 34

DMA data

Dual Port BRAM Buffer

External Camera

RAM Write

RAM Read

pixel

Ix Pixel Matrix

Iy

FIFO

FIFO

FIFO

DMA Control

DMA address FPG

A

Image Derivatives

3/7/11 ECEn631 35

DMA data

Dual Port BRAM Buffer

External Camera

RAM Write

RAM Read

pixel

Ix Pixel Matrix

Iy

FIFO

FIFO

FIFO

DMA Control

DMA address FPG

A

Resource Usage and Performance of Image

Processor   XilinxVirtex‐II(XC2V1000‐6FG456)

– 234Slices(4.6%)– 231LUTs(2.3%)– 1BRAM(2.5%)– MaximumFrequency:170Mhz– 75MHzsufficientfor320x240,30fps

PixelProcessoronly,excludingDMAControlleranditsFIFOs

3/7/11 ECEn631 36

Reconfigurable Platform   Systemcanbereconfiguredtosupportalternatealgorithms– Otherkernels(e.g.,7x7)– Blockmatching– Stereovision– Acombina.onofalgorithmsinparallel

3/7/11 ECEn631 37

3/7/11 ECEn631 38