SIBGRAPI 2016 – TUTORIAL
Image Operator Learningand Applications
Igor S. MontagnerNina S. T. Hirata
Roberto Hirata Jr.
Department of Computer ScienceInstitute of Mathematics and Statistics
University of São Paulo (USP)
October / 2016
Observed input
Expected output
Given pairs of images as the oneson the left...
Question:
– how to design an imageoperator Ψ that transformsthe input image to theexpected output image?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 2
Observed input
Expected output
Ψ =?
Given pairs of images as the oneson the left...
Question:
– how to design an imageoperator Ψ that transformsthe input image to theexpected output image?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 3
What about this transformation ?
Ψ = ?=⇒
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 4
And this one?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 5
This one?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 6
Or this one?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 7
Historical overview
1960s: Mathematical morphology was introduced by G.Matheron and J. Serra
1980s: Book by J. Serra (1982)Nonlinear filters (median, order statistic, stack), morphologicalfilters
1990s: E. R. Dougherty - Design of optimal morphologicalfilters
1994: Junior Barrera - Vision Lab created at IME/USPWe started working on designing morphological operators fromtraining images at that time !!
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 8
Today (almost 25 years since then)
Huge advances in image processing, computer vision,machine learning, hardware and computing technologies
High-level × low-level processing
– Deep learning (high level image understanding)
– Image operator learning (low level processing)
The second is a building block for the first. Thus, veryimportant.
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 9
Plan for this tutorial
1. Concepts and fundamentals
– you will know what an image operator is, and importantproperties that define a class of operators adequate for thetype of processing we are interested on
2. Optimal operators
– you will know how to characterize what a good imageoperator is
3. Learning image operators
– you will know how to learn image operators from trainingdata
4. Practice
– you will have the opportunity to do exercises and run code
5. Application examples
– you will see some application examples
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 10
Plan for this tutorial
1. Concepts and fundamentals
– you will know what an image operator is, and importantproperties that define a class of operators adequate for thetype of processing we are interested on
2. Optimal operators
– you will know how to characterize what a good imageoperator is
3. Learning image operators
– you will know how to learn image operators from trainingdata
4. Practice
– you will have the opportunity to do exercises and run code
5. Application examples
– you will see some application examples
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 11
Plan for this tutorial
1. Concepts and fundamentals
– you will know what an image operator is, and importantproperties that define a class of operators adequate for thetype of processing we are interested on
2. Optimal operators– you will know how to characterize what a good image
operator is
3. Learning image operators
– you will know how to learn image operators from trainingdata
4. Practice
– you will have the opportunity to do exercises and run code
5. Application examples
– you will see some application examples
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 12
Plan for this tutorial
1. Concepts and fundamentals
– you will know what an image operator is, and importantproperties that define a class of operators adequate for thetype of processing we are interested on
2. Optimal operators– you will know how to characterize what a good image
operator is
3. Learning image operators– you will know how to learn image operators from training
data
4. Practice
– you will have the opportunity to do exercises and run code
5. Application examples
– you will see some application examples
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 13
Plan for this tutorial
1. Concepts and fundamentals
– you will know what an image operator is, and importantproperties that define a class of operators adequate for thetype of processing we are interested on
2. Optimal operators– you will know how to characterize what a good image
operator is
3. Learning image operators– you will know how to learn image operators from training
data
4. Practice– you will have the opportunity to do exercises and run code
5. Application examples
– you will see some application examples
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 14
Plan for this tutorial
1. Concepts and fundamentals
– you will know what an image operator is, and importantproperties that define a class of operators adequate for thetype of processing we are interested on
2. Optimal operators– you will know how to characterize what a good image
operator is
3. Learning image operators– you will know how to learn image operators from training
data
4. Practice– you will have the opportunity to do exercises and run code
5. Application examples– you will see some application examples
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 15
Fundamentals on imageoperators
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 16
How to model image transformations ?
Image domain
E (subset of Z2)
Image
f : E→ K (K is the set of gray levels)
KE set of all images defined on E with gray-levels in K
Image operator
Ψ : KE → KE
[Ψ(f )](p) value of the processed image at p
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 17
Image operators (mathematical morphology view)
Morphological operators are built by composing basicelementary operators.
Examples of basic operators– erosion
– dilation
– hit-miss, ...
Structuring elements: parameter of the operatorsSimple shapes such as squares, discs, line segments, to locallyprobe an image and extract geometric structural information
Fundamentals: We will present concepts restricted to binaryimages, but they hold for gray-scale images too
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 18
Binary image operators = set operators
Binary image
f : E→ {0, 1}
Binary image = set
p ∈ Sf ⇐⇒ f (p) = 1
Operations on images = operations on sets
Set relations and operations: ⊆, ∪, ∩, ·cSet translation: Ap = {a + p : a ∈ A}
Set reflection: A = {−a : a ∈ A}
Set interval: [A, B] = {X : A ⊆ X ⊆ B}
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 19
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)
S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 20
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 21
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 22
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 23
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 24
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 25
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 26
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 27
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 28
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 29
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 30
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 31
Basic operator: erosion
εB(S) = {p ∈ E : Bp ⊆ S}
S εB(S)
S εB(S)
How does the erosion work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 32
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)
S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 33
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 34
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 35
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 36
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 37
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 38
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 39
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 40
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 41
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 42
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 43
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 44
Basic operator: dilation
δB(S) = {p ∈ E : Bp ∩ S 6= ∅}
S δB(S)
S δB(S)
How does the dilation work ?
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 45
Basic operator: erosion and dilation
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 46
Another basic operator: hit-miss
H(A,Bc)(S) = {p ∈ E : Ap ⊆ S and Bcp ⊆ Sc} = εA(S) ∩ εBc(Sc)
A Bc
places where A hits S
places where Bc hits thebackground (or, equiva-lently, misses S)
places detected by the hit-miss operator
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 47
Short break for an exercise
Exercise 1
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 48
Solution to exercise 1 – structuring element A
A Bc
In red, positions x at which Ax fits in the foreground
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 49
Solution to exercise 1 – structuring element Bc
A Bc
In red, positions x at which Bcx fits in the background
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 50
Solution to exercise 1 – Intersection
A Bc
Result: the holes
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 51
Many useful operators can be built by composing these andother simple operators
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 52
Example of an operator: Contour detection
f − εB(f )
S εB(S) S − εB(S)
Which properties of this operator
are interesting ?
• Translation invariance• Local definition
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 53
Example of an operator: Contour detection
f − εB(f )
S εB(S) S − εB(S)
Which properties of this operator
are interesting ?
• Translation invariance• Local definition
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 54
Example of an operator: Contour detection
f − εB(f )
S εB(S) S − εB(S)
Which properties of this operator
are interesting ?
• Translation invariance• Local definition
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 55
Translation invariance
[Ψ(f )]p = Ψ(fp)
Ψ(S)
Ψ(S)
[Ψ(S)]p
Ψ(S)
[Ψ(S)]p
Sp
Ψ(S)
[Ψ(S)]p
Sp
Ψ(Sp)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 56
Translation invariance
[Ψ(f )]p = Ψ(fp)
Ψ(S)
Ψ(S)
[Ψ(S)]p
Ψ(S)
[Ψ(S)]p
Sp
Ψ(S)
[Ψ(S)]p
Sp
Ψ(Sp)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 57
Translation invariance
[Ψ(f )]p = Ψ(fp)
Ψ(S)Ψ(S)
[Ψ(S)]p
Ψ(S)
[Ψ(S)]p
Sp
Ψ(S)
[Ψ(S)]p
Sp
Ψ(Sp)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 58
Translation invariance
[Ψ(f )]p = Ψ(fp)
Ψ(S)Ψ(S)
[Ψ(S)]p
Ψ(S)
[Ψ(S)]p
Sp
Ψ(S)
[Ψ(S)]p
Sp
Ψ(Sp)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 59
Local definition
Ψ is locally defined if there is a window W such that:
p ∈ Ψ(S)⇐⇒ p ∈ Ψ(S ∩W ′)
for every p ∈ E, S ∈ KE, and W ′ ⊇W
The red pixel is a contour point
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 60
Local definition
Ψ is locally defined if there is a window W such that:
p ∈ Ψ(S)⇐⇒ p ∈ Ψ(S ∩W ′)
for every p ∈ E, S ∈ KE, and W ′ ⊇W
The red pixel is a contour point
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 61
Local definition
Ψ is locally defined if there is a window W such that:
p ∈ Ψ(S)⇐⇒ p ∈ Ψ(S ∩W ′)
for every p ∈ E, S ∈ KE, and W ′ ⊇W
The red pixel is a contour point
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 62
Local definition
Ψ is locally defined if there is a window W such that:
p ∈ Ψ(S)⇐⇒ p ∈ Ψ(S ∩W ′)
for every p ∈ E, S ∈ KE, and W ′ ⊇W
The red pixel is a contour point
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 63
Window operator or W -operator
W -operator: translation invariance + local definition
There is a local function ψ that uniquely characterizes Ψ
S Ψ(S)
zz
Ψ(S)(z) = ψ( )
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 64
Back to contour detection
Contour detection operator: S − εB(S)– translation-invariant– locally defined
Local function:To decide if a pixel is a(n internal) contour point ornot, it suffices to check if it is in the foreground and ifthere is at least one pixel adjacent to it in thebackground.
Considering 4-adjacency, a cross-window is sufficient
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 65
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input Output
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 66
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input Output
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 67
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 68
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 69
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 70
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 71
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 72
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 73
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 74
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 75
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 76
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 77
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 78
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 79
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 80
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 81
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 82
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 83
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 84
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 85
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 86
Local function of the contour detection operator
x1
x1
x2x2 x3x3 x4x4 x5
x5
window array
Input OutputInput Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00101
Input Output
00101
Input Output
01001
Input Output
01001
Input Output
00000
Input Output
00000
Input Output
00011
Input Output
00011
Input Output
00111
Input Output
00111
Input Output
11111
Input Output
11111
Input Output
01101
Input Output
01101
Input Output
01001
Input Output
01001
Input Output
and so on
X ψ(X)00000 000001 000010 000011 000100 100101 100110 100111 1· · · · · ·
01001 0· · · · · ·
01101 1· · · · · ·
11011 0· · · · · ·
11110 111111 0
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 87
Short break for anotherexercise
Exercise 2
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 88
Solution to exercise 2 – structuring element A
A Bc
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 89
Solution to exercise 2 – structuring element Bc
A Bc
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 90
Solution to exercise 2 – Intersection
A Bc
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 91
Representation of an operator
How to represent an image operator ?
If Ψ is a W -operator it suffices to know ψ : {0, 1}W → {0, 1}.
Kernel of ΨK(Ψ) = {X ⊆W : ψ(X) = 1}
• What is the importance of the kernel ?
• Do we need to explicitly represent the kernel ?
Kernels are related to a canonical
representation of W -operators
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 92
Representation of an operator
How to represent an image operator ?
If Ψ is a W -operator it suffices to know ψ : {0, 1}W → {0, 1}.
Kernel of ΨK(Ψ) = {X ⊆W : ψ(X) = 1}
• What is the importance of the kernel ?
• Do we need to explicitly represent the kernel ?
Kernels are related to a canonical
representation of W -operators
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 93
Representation of operators
Hit-miss in an important piece in the representation ofW -operators
A hit-miss operator is equivalent to an intervalX is detected by hit-miss operator H(A,Bc) ⇐⇒ X ∈ [A, B]
W A Bc B
The window images detected by H(A,Bc) or in [A, B] are thesame:
(all templates that correspond top contour points)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 94
Decomposition theoremExample: intervals for the contour detection operator
, , , ,,,,{ }[[[[ ]]]]
Interval operator:
Λ(A,B)(X) = 1⇐⇒ X ∈ [A, B]
Basis of Ψ: B(Ψ) = maximal intervals in K(Ψ)
Minimal decomposition theorem
ψ(X) = max[A,B]∈B(Ψ)
Λ(A,B)(X)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 95
What would be the basis for the extreme point case ?
left endpoint
right endpoint
bottom endpoint
top endpoint
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 96
What would be the basis for the extreme point case ?
left endpoint
right endpoint
bottom endpoint
top endpoint
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 97
Summary of the fundamentals
You acquired knowledge on
– image operators
– W -operators, a broad class of image operators
– characterization by local functions
– basis representation: supremum of interval operators
– geometrical interpretation
We should also mention:
– binary case: equivalent to Boolean functions
– Similar results hold for gray-scale image operators
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 98
Learning image operators
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 99
Goal of image operator learning
Given observed images f and respective expectedtransformation g, we would like to find Ψ such thatΨ(f ) is a good approximation of g
MAE (mean absolute error)
MAE〈Ψ〉 = E[|[Ψ(f )](p) − g(p)|
]
Empirical MAE
Err =1|E|
∑p∈E
∣∣∣[Ψ(f )](p) − g(p)∣∣∣
Average of pixel-wise absolute difference
Binary images: pixel error rateSIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 100
Optimal W -operators
Window: W
Input-output pairs of images: (f , g)
Observations: (X , y)X is a window imagey is the corresponding output value
Optimal operator: (in the case of binary operators)
ψ(X) =
1, if P(1|X) > P(0|X),
0, if P(0|X) > P(1|X),
0 or 1 if P(0|X) = P(1|X) = 0.5
Probabilities P(y|X) can be estimated from training images
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 101
Counting occurrences of (X , y)
(0
0000
, 0)(0
0011
, 0)(0
0101
, 1)(0
1001
, 0)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 102
Counting occurrences of (X , y)
(0
0000
, 0)
(0
0011
, 0)(0
0101
, 1)(0
1001
, 0)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 103
Counting occurrences of (X , y)
(0
0000
, 0)
(0
0011
, 0)
(0
0101
, 1)(0
1001
, 0)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 104
Counting occurrences of (X , y)
(0
0000
, 0)(0
0011
, 0)
(0
0101
, 1)
(0
1001
, 0)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 105
Counting occurrences of (X , y)
(0
0000
, 0)(0
0011
, 0)(0
0101
, 1)
(0
1001
, 0)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 106
Example
X ψ(X)
27 09 03 30 77 10 30 80 11
00
0/110111
Optimal decision
ψ(X) =
0, if p(0|X) > p(1|X),
1, if p(0|X) < p(1|X),
×, if p(0|X) = p(1|X).
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 107
Short break for a new exerciseExercise 3
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 108
Solution to exercise 3
2 8
2
10 2
440
36 0
10
9 0
11
2 12
2
X0 1
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 109
Image operator designin the framework of
Machine Learning
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 110
Machine learning – Supervised classification
http://www.nltk.org/book/ch06.html
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 111
Learning the local function
S Ψ(S)
zz
Ψ(S)(z) = ψ( )
The local function can be understood as a classifier
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 112
Training samples
Positive example
Negative example
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 113
Training samples
Positive example
Negative example
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 114
Training samples
Positive example
Negative example
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 115
Feature extraction (to form the training set)
(0
0000
, 0)(0
0011
, 0)(0
0101
, 1)(0
1001
, 0)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 116
Feature extraction (to form the training set)
(0
0000
, 0)
(0
0011
, 0)(0
0101
, 1)(0
1001
, 0)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 117
Feature extraction (to form the training set)
(0
0000
, 0)
(0
0011
, 0)
(0
0101
, 1)(0
1001
, 0)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 118
Feature extraction (to form the training set)
(0
0000
, 0)(0
0011
, 0)
(0
0101
, 1)
(0
1001
, 0)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 119
Feature extraction (to form the training set)
(0
0000
, 0)(0
0011
, 0)(0
0101
, 1)
(0
1001
, 0)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 120
Overview
Training W−operator
S
I
Window
W
• W defines the feature space
• Training data is extracted from input-output images
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 121
The problem is not solved yet ...
Challenge: tradeoff between window size and generalizationerror
0.01 0.012 0.014 0.016 0.018
0.02 0.022 0.024 0.026 0.028
0.03
0 20 40 60 80 100 120 140
MA
E
Window size
1 training image5 training images
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 122
Two-level training
1 2 3W W W
same target pixel
distinct neighborhood
three results
pattern for the secondlevel classifier
target pixel
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 123
Input
=⇒
Results of 7operators
=⇒Superimposed
results(darker = less votes)
⇓
Combinationresult
(better than the
individual ones)
Why two-level training scheme is good?
Windows in the first level training
Their union is a larger window
Two-level training (indirectly) considers a larger neighborhood,without increasing window size.
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 126
Why two-level training scheme is good?
Windows in the first level training
Their union is a larger window
Two-level training (indirectly) considers a larger neighborhood,without increasing window size.
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 127
Application examples
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 128
Application examples
Input Expected output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 129
Application examples
Input Test output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 130
Application examples
Input Expected output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 131
Application examples
Input Test output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 132
Application examples
Input Expected output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 133
Application examples
Input Test output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 134
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 135
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 136
Input Expected output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 137
Input Test output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 138
Input Expected output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 139
Input Test output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 140
Input Expected output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 141
Input Test output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 142
Input Expected output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 143
Input Test output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 144
Input Expected output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 145
Input Test output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 146
Input Expected output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 147
Input Test output
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 148
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 149
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 150
Handwritten musical scores
Staff line removal:
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 151
Synthetically generated musical scores
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 152
PRACTICAL SESSION
TRIOSLibLaboratory session exercises 1 to 4
http://bit.ly/2dtL5en
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 153
TRIOSlib
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 154
TRIOSlib: research library containing implementations ofmany WOperator learning techniques.
– Integrates with Numpy, Scipy, scikit-learn.
– Extensible design: new techniques implemented withoutchanging the library.
– Error measuring, persistence on disk, parallelimplementations, ...
– Used in many published papers.
Available at https://sourceforge.net/projects/trioslib andinstallable using pip and conda.
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 155
Important classes:
– trios.Imageset: list of triples (input,output,mask).mask=None means that every pixel is processed.
– trios.WOperator: executes learning process;– trios.FeatureExtractor:
– raw pixels (RAWFeatureExtractor andRAWBitFeatureExtractor)
– two-level patterns (CombinationPattern)– aperture
– trios.Classifier:– Boolean function minimization (ISI)– models from scikit-learn (SKClassifier)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 156
Recent advances
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 157
Window determination
Empirical approach: Based on user experience; trial and error
Feature selection based methods
Window points ∼ features / Window design ∼ feature selection
– Interaction information: C. S. Santos, N. S. Hirata, and R. Hirata Jr., “An information
theory framework for two-stage binary image operator design,” Pattern Recognition Letters, 2010
– RELIEF (feature selection algorithm): I. Montagner, R. Hirata Jr., and N. S. T.
Hirata, Ä machine learning based method for staff removal,” ICPR 2014
Measures to rank windows: discriminative power of windows
– Entropy based ranking: M. Dornelles and N. S. T. Hirata, “Selection of Windows for
W-Operator Combination from Entropy Based Ranking,” SIBGRAPI 2015
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 158
Operator selection for combination
– Given a domain window W defined as a m × n = Nrectangle, there are 2N possible subwindows.
– Train a very strongly regularized sparse combinationusing all possible windows.
– Fast algorithms for linear sparse models(LASSO, L1-SVM).
Iterative solution is possible by checking optimality conditionsand random sampling of the operator space.
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 159
Processing gray scale images
n = |W |
There are:
– 22nbinary W-operators
– kkngray-level W-operators
Computational cost (memory, processing) and statisticalprecision are challenging issues
Some possible approaches:
– constrain operator space: two-level, stack filters, aperture
– reduce window image variability
Some details next
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 160
Gray-scale from binary
Threshold decomposition
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 161
Example of stack filter
Gray-scale median can be computed as the sum of themedians of the cross-sections
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 162
Test image and result
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 163
Aperture Operators
Class of operators to transform grayscale images.
Number of possible binary operators: 22|W |.
Number of possible grayscale operators: KK |W |.
Now we have to restrict W and K.
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 164
Aperture Operators
Besides translation invariance in the domain
o xt
h(x)
W
−t
xo
h (x)
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 165
Aperture Operators
we add translation invariance in grayscale
−t
Wu
−z
x
h (x)
o
o
u−z
| K
x
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 166
Aperture Operators
Translation invariance in the domain and in the range, locallydefined by a window W and a range K = {−k, . . . , 0, . . . , k}
Ψ(f )(t) = ζO(ut) +ψ(ut−ζI(ut)|K)
where ζO and ζI are functions defined from KW to N .
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 167
Kernel approximation
Main issue in training is generalization. Kernel SVMs offergood generalization, but require building a Gram matrixK ∈ RM×M . In WOperator learning, M >> 100, 000.
Nyström method:
– approximate K using a small set of m examples, m << M.
– works for any kernel;
– exact solution for m = M.
More details: Friday 14:00-15:45 at TS10!
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 168
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 169
Trios - training image operators from samples
Available: https://sourceforge.net/projects/trioslib
Igor S. Montagner [email protected] S. T. Hirata [email protected]
Roberto Hirata Jr. [email protected]
SIBGRAPI 2016 Tutorials — Learning Image Operators and Applications (I.S. Montagner, N.S.T. Hirata, R. Hirata Jr.) 170
Top Related