Image Operator Learning - vision.ime.usp.br · Huge advances in image processing, computer vision,...

Post on 17-Aug-2020

7 views 0 download

Transcript of Image Operator Learning - vision.ime.usp.br · Huge advances in image processing, computer vision,...

SIBGRAPI 2016 – TUTORIAL

Image Operator Learningand Applications

Igor S. MontagnerNina S. T. Hirata

Roberto Hirata Jr.

Department of Computer ScienceInstitute of Mathematics and Statistics

University of São Paulo (USP)

October / 2016

Observed input

Expected output

Given pairs of images as the oneson the left...

Question:

– how to design an imageoperator Ψ that transformsthe input image to theexpected output image?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 2

Observed input

Expected output

Ψ =?

Given pairs of images as the oneson the left...

Question:

– how to design an imageoperator Ψ that transformsthe input image to theexpected output image?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 3

What about this transformation ?

Ψ = ?=⇒

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 4

And this one?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 5

This one?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 6

Or this one?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 7

Historical overview

1960s: Mathematical morphology was introduced by G.Matheron and J. Serra

1980s: Book by J. Serra (1982)Nonlinear filters (median, order statistic, stack), morphologicalfilters

1990s: E. R. Dougherty - Design of optimal morphologicalfilters

1994: Junior Barrera - Vision Lab created at IME/USPWe started working on designing morphological operators fromtraining images at that time !!

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 8

Today (almost 25 years since then)

Huge advances in image processing, computer vision,machine learning, hardware and computing technologies

High-level × low-level processing

– Deep learning (high level image understanding)

– Image operator learning (low level processing)

The second is a building block for the first. Thus, veryimportant.

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 9

Plan for this tutorial

1. Concepts and fundamentals

– you will know what an image operator is, and importantproperties that define a class of operators adequate for thetype of processing we are interested on

2. Optimal operators

– you will know how to characterize what a good imageoperator is

3. Learning image operators

– you will know how to learn image operators from trainingdata

4. Practice

– you will have the opportunity to do exercises and run code

5. Application examples

– you will see some application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 10

Plan for this tutorial

1. Concepts and fundamentals

– you will know what an image operator is, and importantproperties that define a class of operators adequate for thetype of processing we are interested on

2. Optimal operators

– you will know how to characterize what a good imageoperator is

3. Learning image operators

– you will know how to learn image operators from trainingdata

4. Practice

– you will have the opportunity to do exercises and run code

5. Application examples

– you will see some application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 11

Plan for this tutorial

1. Concepts and fundamentals

– you will know what an image operator is, and importantproperties that define a class of operators adequate for thetype of processing we are interested on

2. Optimal operators– you will know how to characterize what a good image

operator is

3. Learning image operators

– you will know how to learn image operators from trainingdata

4. Practice

– you will have the opportunity to do exercises and run code

5. Application examples

– you will see some application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 12

Plan for this tutorial

1. Concepts and fundamentals

– you will know what an image operator is, and importantproperties that define a class of operators adequate for thetype of processing we are interested on

2. Optimal operators– you will know how to characterize what a good image

operator is

3. Learning image operators– you will know how to learn image operators from training

data

4. Practice

– you will have the opportunity to do exercises and run code

5. Application examples

– you will see some application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 13

Plan for this tutorial

1. Concepts and fundamentals

– you will know what an image operator is, and importantproperties that define a class of operators adequate for thetype of processing we are interested on

2. Optimal operators– you will know how to characterize what a good image

operator is

3. Learning image operators– you will know how to learn image operators from training

data

4. Practice– you will have the opportunity to do exercises and run code

5. Application examples

– you will see some application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 14

Plan for this tutorial

1. Concepts and fundamentals

– you will know what an image operator is, and importantproperties that define a class of operators adequate for thetype of processing we are interested on

2. Optimal operators– you will know how to characterize what a good image

operator is

3. Learning image operators– you will know how to learn image operators from training

data

4. Practice– you will have the opportunity to do exercises and run code

5. Application examples– you will see some application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 15

Fundamentals on imageoperators

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 16

How to model image transformations ?

Image domain

E (subset of Z2)

Image

f : E→ K (K is the set of gray levels)

KE set of all images defined on E with gray-levels in K

Image operator

Ψ : KE → KE

[Ψ(f )](p) value of the processed image at p

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 17

Image operators (mathematical morphology view)

Morphological operators are built by composing basicelementary operators.

Examples of basic operators– erosion

– dilation

– hit-miss, ...

Structuring elements: parameter of the operatorsSimple shapes such as squares, discs, line segments, to locallyprobe an image and extract geometric structural information

Fundamentals: We will present concepts restricted to binaryimages, but they hold for gray-scale images too

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 18

Binary image operators = set operators

Binary image

f : E→ {0, 1}

Binary image = set

p ∈ Sf ⇐⇒ f (p) = 1

Operations on images = operations on sets

Set relations and operations: ⊆, ∪, ∩, ·cSet translation: Ap = {a + p : a ∈ A}

Set reflection: A = {−a : a ∈ A}

Set interval: [A, B] = {X : A ⊆ X ⊆ B}

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 19

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)

S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 20

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 21

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 22

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 23

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 24

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 25

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 26

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 27

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 28

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 29

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 30

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 31

Basic operator: erosion

εB(S) = {p ∈ E : Bp ⊆ S}

S εB(S)

S εB(S)

How does the erosion work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 32

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)

S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 33

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 34

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 35

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 36

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 37

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 38

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 39

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 40

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 41

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 42

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 43

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 44

Basic operator: dilation

δB(S) = {p ∈ E : Bp ∩ S 6= ∅}

S δB(S)

S δB(S)

How does the dilation work ?

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 45

Basic operator: erosion and dilation

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 46

Another basic operator: hit-miss

H(A,Bc)(S) = {p ∈ E : Ap ⊆ S and Bcp ⊆ Sc} = εA(S) ∩ εBc(Sc)

A Bc

places where A hits S

places where Bc hits thebackground (or, equiva-lently, misses S)

places detected by the hit-miss operator

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 47

Short break for an exercise

Exercise 1

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 48

Solution to exercise 1 – structuring element A

A Bc

In red, positions x at which Ax fits in the foreground

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 49

Solution to exercise 1 – structuring element Bc

A Bc

In red, positions x at which Bcx fits in the background

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 50

Solution to exercise 1 – Intersection

A Bc

Result: the holes

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 51

Many useful operators can be built by composing these andother simple operators

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 52

Example of an operator: Contour detection

f − εB(f )

S εB(S) S − εB(S)

Which properties of this operator

are interesting ?

• Translation invariance• Local definition

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 53

Example of an operator: Contour detection

f − εB(f )

S εB(S) S − εB(S)

Which properties of this operator

are interesting ?

• Translation invariance• Local definition

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 54

Example of an operator: Contour detection

f − εB(f )

S εB(S) S − εB(S)

Which properties of this operator

are interesting ?

• Translation invariance• Local definition

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 55

Translation invariance

[Ψ(f )]p = Ψ(fp)

Ψ(S)

Ψ(S)

[Ψ(S)]p

Ψ(S)

[Ψ(S)]p

Sp

Ψ(S)

[Ψ(S)]p

Sp

Ψ(Sp)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 56

Translation invariance

[Ψ(f )]p = Ψ(fp)

Ψ(S)

Ψ(S)

[Ψ(S)]p

Ψ(S)

[Ψ(S)]p

Sp

Ψ(S)

[Ψ(S)]p

Sp

Ψ(Sp)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 57

Translation invariance

[Ψ(f )]p = Ψ(fp)

Ψ(S)Ψ(S)

[Ψ(S)]p

Ψ(S)

[Ψ(S)]p

Sp

Ψ(S)

[Ψ(S)]p

Sp

Ψ(Sp)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 58

Translation invariance

[Ψ(f )]p = Ψ(fp)

Ψ(S)Ψ(S)

[Ψ(S)]p

Ψ(S)

[Ψ(S)]p

Sp

Ψ(S)

[Ψ(S)]p

Sp

Ψ(Sp)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 59

Local definition

Ψ is locally defined if there is a window W such that:

p ∈ Ψ(S)⇐⇒ p ∈ Ψ(S ∩W ′)

for every p ∈ E, S ∈ KE, and W ′ ⊇W

The red pixel is a contour point

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 60

Local definition

Ψ is locally defined if there is a window W such that:

p ∈ Ψ(S)⇐⇒ p ∈ Ψ(S ∩W ′)

for every p ∈ E, S ∈ KE, and W ′ ⊇W

The red pixel is a contour point

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 61

Local definition

Ψ is locally defined if there is a window W such that:

p ∈ Ψ(S)⇐⇒ p ∈ Ψ(S ∩W ′)

for every p ∈ E, S ∈ KE, and W ′ ⊇W

The red pixel is a contour point

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 62

Local definition

Ψ is locally defined if there is a window W such that:

p ∈ Ψ(S)⇐⇒ p ∈ Ψ(S ∩W ′)

for every p ∈ E, S ∈ KE, and W ′ ⊇W

The red pixel is a contour point

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 63

Window operator or W -operator

W -operator: translation invariance + local definition

There is a local function ψ that uniquely characterizes Ψ

S Ψ(S)

zz

Ψ(S)(z) = ψ( )

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 64

Back to contour detection

Contour detection operator: S − εB(S)– translation-invariant– locally defined

Local function:To decide if a pixel is a(n internal) contour point ornot, it suffices to check if it is in the foreground and ifthere is at least one pixel adjacent to it in thebackground.

Considering 4-adjacency, a cross-window is sufficient

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 65

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input Output

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 66

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input Output

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 67

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 68

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 69

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 70

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 71

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 72

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 73

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 74

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 75

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 76

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 77

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 78

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 79

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 80

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 81

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 82

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 83

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 84

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 85

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 86

Local function of the contour detection operator

x1

x1

x2x2 x3x3 x4x4 x5

x5

window array

Input OutputInput Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00101

Input Output

00101

Input Output

01001

Input Output

01001

Input Output

00000

Input Output

00000

Input Output

00011

Input Output

00011

Input Output

00111

Input Output

00111

Input Output

11111

Input Output

11111

Input Output

01101

Input Output

01101

Input Output

01001

Input Output

01001

Input Output

and so on

X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·

01001 0· · · · · ·

01101 1· · · · · ·

11011 0· · · · · ·

11110 111111 0

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 87

Short break for anotherexercise

Exercise 2

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 88

Solution to exercise 2 – structuring element A

A Bc

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 89

Solution to exercise 2 – structuring element Bc

A Bc

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 90

Solution to exercise 2 – Intersection

A Bc

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 91

Representation of an operator

How to represent an image operator ?

If Ψ is a W -operator it suffices to know ψ : {0, 1}W → {0, 1}.

Kernel of ΨK(Ψ) = {X ⊆W : ψ(X) = 1}

• What is the importance of the kernel ?

• Do we need to explicitly represent the kernel ?

Kernels are related to a canonical

representation of W -operators

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 92

Representation of an operator

How to represent an image operator ?

If Ψ is a W -operator it suffices to know ψ : {0, 1}W → {0, 1}.

Kernel of ΨK(Ψ) = {X ⊆W : ψ(X) = 1}

• What is the importance of the kernel ?

• Do we need to explicitly represent the kernel ?

Kernels are related to a canonical

representation of W -operators

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 93

Representation of operators

Hit-miss in an important piece in the representation ofW -operators

A hit-miss operator is equivalent to an intervalX is detected by hit-miss operator H(A,Bc) ⇐⇒ X ∈ [A, B]

W A Bc B

The window images detected by H(A,Bc) or in [A, B] are thesame:

(all templates that correspond top contour points)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 94

Decomposition theoremExample: intervals for the contour detection operator

, , , ,,,,{ }[[[[ ]]]]

Interval operator:

Λ(A,B)(X) = 1⇐⇒ X ∈ [A, B]

Basis of Ψ: B(Ψ) = maximal intervals in K(Ψ)

Minimal decomposition theorem

ψ(X) = max[A,B]∈B(Ψ)

Λ(A,B)(X)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 95

What would be the basis for the extreme point case ?

left endpoint

right endpoint

bottom endpoint

top endpoint

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 96

What would be the basis for the extreme point case ?

left endpoint

right endpoint

bottom endpoint

top endpoint

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 97

Summary of the fundamentals

You acquired knowledge on

– image operators

– W -operators, a broad class of image operators

– characterization by local functions

– basis representation: supremum of interval operators

– geometrical interpretation

We should also mention:

– binary case: equivalent to Boolean functions

– Similar results hold for gray-scale image operators

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 98

Learning image operators

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 99

Goal of image operator learning

Given observed images f and respective expectedtransformation g, we would like to find Ψ such thatΨ(f ) is a good approximation of g

MAE (mean absolute error)

MAE〈Ψ〉 = E[|[Ψ(f )](p) − g(p)|

]

Empirical MAE

Err =1|E|

∑p∈E

∣∣∣[Ψ(f )](p) − g(p)∣∣∣

Average of pixel-wise absolute difference

Binary images: pixel error rateSIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 100

Optimal W -operators

Window: W

Input-output pairs of images: (f , g)

Observations: (X , y)X is a window imagey is the corresponding output value

Optimal operator: (in the case of binary operators)

ψ(X) =

1, if P(1|X) > P(0|X),

0, if P(0|X) > P(1|X),

0 or 1 if P(0|X) = P(1|X) = 0.5

Probabilities P(y|X) can be estimated from training images

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 101

Counting occurrences of (X , y)

(0

0000

, 0)(0

0011

, 0)(0

0101

, 1)(0

1001

, 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 102

Counting occurrences of (X , y)

(0

0000

, 0)

(0

0011

, 0)(0

0101

, 1)(0

1001

, 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 103

Counting occurrences of (X , y)

(0

0000

, 0)

(0

0011

, 0)

(0

0101

, 1)(0

1001

, 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 104

Counting occurrences of (X , y)

(0

0000

, 0)(0

0011

, 0)

(0

0101

, 1)

(0

1001

, 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 105

Counting occurrences of (X , y)

(0

0000

, 0)(0

0011

, 0)(0

0101

, 1)

(0

1001

, 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 106

Example

X ψ(X)

27 09 03 30 77 10 30 80 11

00

0/110111

Optimal decision

ψ(X) =

0, if p(0|X) > p(1|X),

1, if p(0|X) < p(1|X),

×, if p(0|X) = p(1|X).

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 107

Short break for a new exerciseExercise 3

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 108

Solution to exercise 3

2 8

2

10 2

440

36 0

10

9 0

11

2 12

2

X0 1

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 109

Image operator designin the framework of

Machine Learning

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 110

Machine learning – Supervised classification

http://www.nltk.org/book/ch06.html

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 111

Learning the local function

S Ψ(S)

zz

Ψ(S)(z) = ψ( )

The local function can be understood as a classifier

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 112

Training samples

Positive example

Negative example

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 113

Training samples

Positive example

Negative example

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 114

Training samples

Positive example

Negative example

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 115

Feature extraction (to form the training set)

(0

0000

, 0)(0

0011

, 0)(0

0101

, 1)(0

1001

, 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 116

Feature extraction (to form the training set)

(0

0000

, 0)

(0

0011

, 0)(0

0101

, 1)(0

1001

, 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 117

Feature extraction (to form the training set)

(0

0000

, 0)

(0

0011

, 0)

(0

0101

, 1)(0

1001

, 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 118

Feature extraction (to form the training set)

(0

0000

, 0)(0

0011

, 0)

(0

0101

, 1)

(0

1001

, 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 119

Feature extraction (to form the training set)

(0

0000

, 0)(0

0011

, 0)(0

0101

, 1)

(0

1001

, 0)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 120

Overview

Training W−operator

S

I

Window

W

• W defines the feature space

• Training data is extracted from input-output images

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 121

The problem is not solved yet ...

Challenge: tradeoff between window size and generalizationerror

0.01 0.012 0.014 0.016 0.018

0.02 0.022 0.024 0.026 0.028

0.03

0 20 40 60 80 100 120 140

MA

E

Window size

1 training image5 training images

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 122

Two-level training

1 2 3W W W

same target pixel

distinct neighborhood

three results

pattern for the secondlevel classifier

target pixel

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 123

Input

=⇒

Results of 7operators

=⇒Superimposed

results(darker = less votes)

Combinationresult

(better than the

individual ones)

Why two-level training scheme is good?

Windows in the first level training

Their union is a larger window

Two-level training (indirectly) considers a larger neighborhood,without increasing window size.

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 126

Why two-level training scheme is good?

Windows in the first level training

Their union is a larger window

Two-level training (indirectly) considers a larger neighborhood,without increasing window size.

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 127

Application examples

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 128

Application examples

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 129

Application examples

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 130

Application examples

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 131

Application examples

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 132

Application examples

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 133

Application examples

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 134

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 135

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 136

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 137

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 138

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 139

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 140

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 141

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 142

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 143

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 144

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 145

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 146

Input Expected output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 147

Input Test output

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 148

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 149

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 150

Handwritten musical scores

Staff line removal:

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 151

Synthetically generated musical scores

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 152

PRACTICAL SESSION

TRIOSLibLaboratory session exercises 1 to 4

http://bit.ly/2dtL5en

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 153

TRIOSlib

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 154

TRIOSlib: research library containing implementations ofmany WOperator learning techniques.

– Integrates with Numpy, Scipy, scikit-learn.

– Extensible design: new techniques implemented withoutchanging the library.

– Error measuring, persistence on disk, parallelimplementations, ...

– Used in many published papers.

Available at https://sourceforge.net/projects/trioslib andinstallable using pip and conda.

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 155

Important classes:

– trios.Imageset: list of triples (input,output,mask).mask=None means that every pixel is processed.

– trios.WOperator: executes learning process;– trios.FeatureExtractor:

– raw pixels (RAWFeatureExtractor andRAWBitFeatureExtractor)

– two-level patterns (CombinationPattern)– aperture

– trios.Classifier:– Boolean function minimization (ISI)– models from scikit-learn (SKClassifier)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 156

Recent advances

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 157

Window determination

Empirical approach: Based on user experience; trial and error

Feature selection based methods

Window points ∼ features / Window design ∼ feature selection

– Interaction information: C. S. Santos, N. S. Hirata, and R. Hirata Jr., “An information

theory framework for two-stage binary image operator design,” Pattern Recognition Letters, 2010

– RELIEF (feature selection algorithm): I. Montagner, R. Hirata Jr., and N. S. T.

Hirata, Ä machine learning based method for staff removal,” ICPR 2014

Measures to rank windows: discriminative power of windows

– Entropy based ranking: M. Dornelles and N. S. T. Hirata, “Selection of Windows for

W-Operator Combination from Entropy Based Ranking,” SIBGRAPI 2015

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 158

Operator selection for combination

– Given a domain window W defined as a m × n = Nrectangle, there are 2N possible subwindows.

– Train a very strongly regularized sparse combinationusing all possible windows.

– Fast algorithms for linear sparse models(LASSO, L1-SVM).

Iterative solution is possible by checking optimality conditionsand random sampling of the operator space.

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 159

Processing gray scale images

n = |W |

There are:

– 22nbinary W-operators

– kkngray-level W-operators

Computational cost (memory, processing) and statisticalprecision are challenging issues

Some possible approaches:

– constrain operator space: two-level, stack filters, aperture

– reduce window image variability

Some details next

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 160

Gray-scale from binary

Threshold decomposition

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 161

Example of stack filter

Gray-scale median can be computed as the sum of themedians of the cross-sections

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 162

Test image and result

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 163

Aperture Operators

Class of operators to transform grayscale images.

Number of possible binary operators: 22|W |.

Number of possible grayscale operators: KK |W |.

Now we have to restrict W and K.

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 164

Aperture Operators

Besides translation invariance in the domain

o xt

h(x)

W

−t

xo

h (x)

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 165

Aperture Operators

we add translation invariance in grayscale

−t

Wu

−z

x

h (x)

o

o

u−z

| K

x

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 166

Aperture Operators

Translation invariance in the domain and in the range, locallydefined by a window W and a range K = {−k, . . . , 0, . . . , k}

Ψ(f )(t) = ζO(ut) +ψ(ut−ζI(ut)|K)

where ζO and ζI are functions defined from KW to N .

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 167

Kernel approximation

Main issue in training is generalization. Kernel SVMs offergood generalization, but require building a Gram matrixK ∈ RM×M . In WOperator learning, M >> 100, 000.

Nyström method:

– approximate K using a small set of m examples, m << M.

– works for any kernel;

– exact solution for m = M.

More details: Friday 14:00-15:45 at TS10!

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 168

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 169

Trios - training image operators from samples

Available: https://sourceforge.net/projects/trioslib

Igor S. Montagner igordsm@ime.usp.brNina S. T. Hirata nina@ime.usp.br

Roberto Hirata Jr. hirata@ime.usp.br

SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 170