Post on 15-Jun-2020
/32
A Fast Approximation Algorithm for the Gaussian Filter Kentaro Imajo (Yamamoto Laboratory)
February 14, Master Thesis Defense 1
/32
I propose a fast approximation algorithm for the Gaussian filter, which computes one Gaussian-filtered pixel in O(1) in the area size n (∝ σ2). The naïve method requires O(n) time.
Key Algorithm
February 14, Master Thesis Defense
× Area size is n Target Pixel
2 Introduction
/32
Application: Binary Classifier Extending the key algorithm, • The proposed method speeds up prediction of SVMs with the Gaussian kernel,
• The proposed method speeds up training with small loss of accuracy. Training (pre-computation): O(n2) → O(4dnlogdn) Prediction: O(n) → O(4dlogdn) February 14, Master Thesis Defense 3 Introduction
/32
Outline 1. Key Algorithm: A fast approximation algorithm for the Gaussian filter Kentaro Imajo, "Fast Gaussian Filtering Algorithm Using Splines," The 21st International Conference on Pattern Recognition, 2012.
2. Application: Binary classification Kentaro Imajo, Otaki Keisuke, Yamamoto Akihiro, "Binary Classification Using Fast Gaussian Filtering Algorithm,” Algorithms for Large-Scale Information Processing in Knowledge Discovery, 2012.
3. Conclusions
February 14, Master Thesis Defense 4 Introduction
/32
1. Key Algorithm: Gaussian Filtering 2. Application: Binary Classification 3. Conclusions
February 14, Master Thesis Defense 5 Key Algorithm: Gaussian Filtering
/32
Summary of Key Algorithm It computes one Gaussian-filtered pixel in constant time in the area size n (∝ σ2) within 3% error. The Gaussian filter is convolution with the 2D Gaussian Function:
February 14, Master Thesis Defense 6
1�2��
exp��x2 + y2
2�2
�= where , the parameter σ is variance.
= * convolution
Key Algorithm: Gaussian Filtering
/32
Problem To compute one Gaussian-filtered pixel, • The Naïve method takes linear time in the area size of the Gaussian function.
• An FFT method cannot compute pixels apart because FFT depends on all pixels.
• A down-sampling method has 52% error.
February 14, Master Thesis Defense 7 Key Algorithm: Gaussian Filtering
/32
Solution We approximate the Gaussian function using a spline function: • A spline function becomes discrete if it is differentiated some times. – Convolution of an image and a discrete function is computed fast.
A spline function is a piecewise polynomial function that is written as , where (・)+ is max(・, 0). February 14, Master Thesis Defense 8
�̃(x) =�
i
ai (x� bi)n+
Key Algorithm: Gaussian Filtering
/32
�̃(x) = 3(x + 11)2+ � 11(x + 3)2++11(x� 3)2+ � 3(x� 11)2+
� 2.657� 102 exp�� x2
5.27202
�,
-15 -12.5 -10 -7.5 -5 -2.5 0 2.5 5 7.5 10 12.5 15
80
160
240
Approximation Function
February 14, Master Thesis Defense 9
× ×
× ×
- Approximation - Gaussian func. × Knots
where (・)+ is max(・, 0).
Key Algorithm: Gaussian Filtering
/32
�̃(x) = 3(x + 11)2+ � 11(x + 3)2++11(x� 3)2+ � 3(x� 11)2+
� 2.657� 102 exp�� x2
5.27202
�,
-15 -12.5 -10 -7.5 -5 -2.5 0 2.5 5 7.5 10 12.5 15
80
160
240
Approximation Function
February 14, Master Thesis Defense 10
× ×
× ×
- Approximation - Gaussian func. × Knots
where (・)+ is max(・, 0).
Key Algorithm: Gaussian Filtering
/32
Speed Up of Convolution The convolution can be reduced by differentiating a spline function until it becomes discrete. Intuition:
February 14, Master Thesis Defense 11
(� � I)(x) =��
d
dx�
��
��I(x)dx
��(x)
Integrated
Differentiated Becomes DISCRETE
�̃
This can be computed fast.
Key Algorithm: Gaussian Filtering
/32
Nature of a Spline Function Differentiating a spline function, we can get a discrete function: The Gaussian-like approximation function can also become a discrete function! February 14, Master Thesis Defense 12
-12 -8 -4 0 4 8 12
-240
-160
-80
80
160
240
-12 -8 -4 0 4 8 12
-25
25
-12 -8 -4 0 4 8 12
-10
-5
5
10
-12 -8 -4 0 4 8 12
-12
-8
-4
4
8
12
Differentiate 3-times
Key Algorithm: Gaussian Filtering
/32
Evaluation Approximation error is about 2%. Higher order improves more: Approximation error of 4th order approximation is about 0.3%.
February 14, Master Thesis Defense 13
�̃(x) = 70(x + 22)4+ � 624(x + 11)4+ + 1331(x + 4)4+�1331(x� 4)4+ + 624(x� 11)4+ � 70(x� 22)4+.
(�
|�̃(x)� �(x)|dx) / (�
|�(x)|dx)
Key Algorithm: Gaussian Filtering
/32
Approximation of 2D
February 14, Master Thesis Defense 14
using a discrete image J , which is written as:
J(x, y) =!
(!x,!y)!Z2+
!xn!ynI(x ! !x, y ! !y).
When a source image I is given, the image J can becomputed in O(n2|I|) time by iterations as well asEquation (6).
As a result, precomputing J at once, for any convo-lution of a source image I and an nth-order 2D splinefunction, a convoluted value (!̃II " I)(x, y) can be com-puted in O(m2) time.
6. Approximated Gaussian Functions
In this section, we present two good sets of param-eters below to approximate the 1D Gaussian function.Absolute values of parameters b of a good set shouldbe small for easy and fast computation, and absolutevalues of the parameters b of the following sets are notlarger than 22, which is small, compared with ordinaryimages.
Second-order approximation:
!̃2(x) = 3(x + 11)2+ ! 11(x + 3)2++11(x ! 3)2+ ! 3(x ! 11)2+, (7)
# 2.657 $ 102 exp"! x2
5.27202
#. (8)
Fourth-order approximation:
!̃4(x) = 70(x + 22)4+ ! 624(x + 11)4++1331(x + 4)4+ ! 1331(x ! 4)4++624(x ! 11)4+ ! 70(x ! 22)4+, (9)
# 7.621 $ 107 exp"! x2
7.767802
#. (10)
6.1. Evaluation of Approximated Functions
Next, we evaluate the approximated functions by cal-culating error of two types EI,!̃ and EII,!̃ , which areerror in 1D and in 2D respectively. Using an approxi-mation function !̃ and its targeted function !, they aredefined as:
EI,!̃ =
$
R
%%%!(x) ! !̃(x)%%% dx
$
R!(x)dx
, (11)
EII,!̃ =
$$
R2
%%%!(x)!(y) ! !̃(x)!̃(y)%%% dxdy
$$
R2|!(x)!(y)| dxdy
. (12)
0 10 20x
0
4 ! 104
8 ! 104
z
· · · Gaussian— Horizontal section- - - Oblique section
Figure 2. Second-order approximation
-15 0 15x
-15
0
15
y
$ $ $ $
$ $ $ $
$ $ $ $
$ $ $ $
· · · Gaussian — Approximation ! Control point
Figure 3. Elevation and control points
Applying the equations, we can calculate values oferror of the two approximations (7) and (9), whose tar-geted functions are (8) and (10) respectively, as follows:
EI,!̃2% 0.02009, EII,!̃2
% 0.03473,EI,!̃4
% 0.00305, EII,!̃4% 0.00583.
Figure 2 compares the 2D Gaussian function and twosections of the second-order approximated function intwo dimension. One of the sections is a horizontal sec-tion z = !̃2(x)!̃2(0), and the other one is an oblique
section z =&!̃2( x"
2)'2
.
Figure 3 shows elevations of every 104 for the Gaus-sian function and the approximation function in two di-mension. As the Gaussian function is rotatable, eleva-tions of the Gaussian function are perfect circles. Onthe other hand, elevations of the approximated Gaus-sian function, which are indicated by red solid lines, arevery similar to perfect circles. Therefore we can use theapproximated Gaussian function as an almost-rotatablefilter function.
�̃(x, y) = (3(x + 11)2+ � 11(x + 3)2+ + 11(x� 3)2+ � 3(x� 11)2+)(3(y + 11)2+ � 11(y + 3)2+ + 11(y � 3)2+ � 3(y � 11)2+)
Contour lines
Key Algorithm: Gaussian Filtering
/32
The 2D Gaussian function is decomposed: The approximation can be defined as: where
(�̃ � I)(x, y) =m�
i=1
ai
m�
j=1
ajJ(x� bi, y � bj)
2D Convolution
February 14, Master Thesis Defense 15
exp��x2 + y2
2�2
�= exp
�� x2
2�2
�exp
�� y2
2�2
�
O(m2) linear combination, where m is # of knots for each dimension
and m is equal to 4.
J(x, y) =�
(�x,�y)�Z2+
�xn�ynI(x��x, y ��y)
Key Algorithm: Gaussian Filtering
/32
Experiments
February 14, Master Thesis Defense 16
Method Precomputation Computation for all pixels Computation for one pixelNaı̈ve method No O(NM) O(M)FFT method No O(N log N) N/A
Proposed method O(N) at first O(N) O(1)
Table 1. Comparison of computational orderN and M is the size of a source image and the Gaussian filter respectively.
N/A indicates that the FFT method cannot compute values apart.
100 101 102 103 104
# of application pixels of the filter
10!8
10!7
10!6
10!5
10!4
Com
puta
tiona
ltim
e(s
ec.)
— Proposed method - - - Naı̈ve method
Figure 4. Computational time for one pixelon average
7. Experiments
Comparing with the naı̈ve method and the FastFourier Transform method, we evaluate the proposedmethod with a 1024 ! 1024 gray-scale image and theGaussian filter whose the parameter ! is 47.448 and theapplication area is 99 ! 99 pixels. We implementedthem in C++ and compiled them with GNU C++ Com-piler and execute them on Mac Pro (6-core 2.93GHzXeon). As the result of computation, the naı̈ve methodtook 13.7 seconds, and the FFT method took 0.373 sec-onds, and the proposed method took 0.102 seconds. Inthis case, the proposed method can do it more than 100times as fast as the naı̈ve method does.
Next, we evaluate computational time to figure outa filtered value of one pixel. Table 1 shows computa-tional order of the precomputation and computation ofall pixels and computation of one pixel for each method.Computational time of the naı̈ve method depends on ap-plication area of the filter. Figure 4 shows the proposedmethod is faster than the naı̈ve method if applicationarea of the filter is larger than 8 ! 8.
8. Conclusion
We proposed a fast method to apply the Gaussianfilter to a 2D image. In order to compute one Gaus-sian filtered pixel value, the naı̈ve method spends O(M)time, where M is the size of the Gaussian filter. Whilethe FFT method compute all Gaussian filtered pixel val-ues simultaneously in O(N log N) time, where N is thesize of a source image, it cannot compute them sparsely.The proposed method computes one Gaussian filteredpixel value in constant time on the size of a source im-age and the Gaussian filter.
The proposed method approximates the Gaussian fil-ter function using a spline function whose number of thecontrol points is small. Although precomputation takestime linear in the size of a source image, the precom-putation does not depend on the size of the Gaussianfilter, so we do not need to recalculate the precomputa-tion even if we use a different size of the Gaussian filter.
As a result, the proposed method is more than 100times as fast as the naı̈ve method when application areais 99! 99 pixels. Furthermore, unlike the FFT method,the proposed method can compute a filtered pixel valueat any position with any size of the Gaussian filter inconstant time.
References
[1] David G. Lowe. Object recognition from localscale-invariant features. The Proceedings of theSeventh IEEE International Conference on Com-puter Vision, 2:1150–1157, 1999.
[2] David G. Lowe. Distinctive image features fromscale-invariant keypoints. International Journal ofComputer Vision, 60(2):91–110, 2004.
[3] Herbert Bay, Andreas Ess, Tinne Tuytelaars, andLuc Van Gool. SURF: Speeded up robust fea-tures. European Conference on Computer Vision,110(3):346–359, 2006.
Figure Computational time for one pixel on average
For 70+ pixels our algorithm is faster
Diameter of 9 has 69 pixels
Key Algorithm: Gaussian Filtering
/32
Conclusions of Part 1 (1/2) We approximate the Gaussian function using a 2nd order spline function: • It has only 2% approximation error, • It has only 4 knots, • It has unequal intervals of control points while general approximation functions often have equal intervals. – It is sufficiently optimized as a spline.
February 14, Master Thesis Defense 17 Key Algorithm: Gaussian Filtering
/32
Conclusions of Part 1(2/2) Using spline approximation and an integrated image of an input image, we can compute every pixel apart in constant time to the area size.
February 14, Master Thesis Defense 18
* =
*=�…�
dx3dy3
Key Algorithm: Gaussian Filtering
/32 February 14, Master Thesis Defense 19
1. Key Algorithm: Gaussian Filtering 2. Application: Binary Classification 3. Conclusions
Application: Binary Classification
/32
Summary of Application Extending the key algorithm, • The proposed method speeds up prediction of SVMs with the Gaussian kernel, – O(n) → O(4dlogdn)
• It speeds up training with small loss of accuracy. – O(n2) → O(4dnlogdn)
February 14, Master Thesis Defense
Judges which is nearer to
20 Application: Binary Classification
/32
Problem An SVM is a good binary classifier, but – prediction takes O(n) time, – training takes O(n2)~O(n3) time.
We cannot use SVMs if # of data gets large. Solution • For prediction, we use the key algorithm. • For training, we treat data equally with small loss of accuracy.
February 14, Master Thesis Defense 21 Application: Binary Classification
/32
Every pixel has a value
Few pixels have a value
Key Algorithm It assumes that an image is given. Since we cannot integrate an image, we propose to use range trees.
February 14, Master Thesis Defense 22
Image Binary Classification
Application: Binary Classification
/32
Range Tree A range tree [Bentley 1979] is a data structure that can compute the total sum in a box x1-x2 in O(logdn) time with O(nlogdn) space.
February 14, Master Thesis Defense 23 Application: Binary Classification
2k
Balanced tree
We can calculate the total sum with the sum of O(logdn) regions x1
x2
/32
The Proposed Method The proposed method can decide the class of any point in O(4dlogdn) time, and the pre-computation is O(4dnlogdn) time.
February 14, Master Thesis Defense 24
Range Trees -15 0 15
x
-15
0
15
y
! ! ! !
! ! ! !
! ! ! !
! ! ! !
· · · Gaussian — Approximation ! Control point
Figure 3: Elevation and control points
10
Key Algorithm
Application: Binary Classification
/32
Formulation A range tree can give: Using 9 range trees, J can be computed with:
February 14, Master Thesis Defense 25
Binary Classification Using Fast Gaussian Filtering Algorithm 3
3 FGFA for Sparse Data
FGFA assumes that an input data consists of dense data such as an image. Theprecomputation of FGFA takes linear time on the number of pixels of an inputimage. If we deal with continuous values for coordinates as is, a large numberof pixels are required to approximate. Therefore, this section extends the d-dimensional range query algorithm to calculate integrated values of sparse data,and it applies the extension to FGFA.
3.1 Extension of d-dimensional Range Tree
The d-dimensional range tree algorithm provides a method to calculate sum-mation of a rectangular area of d-dimensional spaces. Using the method, thissection shows a method to calculate J in Equation 6. The d-dimensional rangetree algorithm can compute a value Ki,j(x, y) in O(logd n)-time that is writtenas:
Ki,j(x, y) =!
(!x,!y)!{(0,0),···,(x,y)}
!xi!yjI(!x,!y). (7)
Using the values K, J can be rewritten as:
J(x, y) =
"
#x2
!2x1
$
%
"
#y2
!2y1
$
%T"
#K0,0(x, y) K1,0(x, y) K2,0(x, y)K0,1(x, y) K1,1(x, y) K2,1(x, y)K0,2(x, y) K1,2(x, y) K2,2(x, y)
$
% . (8)
Therefore, J can be computed in O(log2d n) time.
3.2 Extension of FGFA
Since the extension of d-dimensional range tree o!ers a method to calculate J inO(log2d n) time, we reveal that FGFA can be computed in O(4d log2d n) time.
3.3 Proposed Classifier
Using the extension of FGFA, the proposed classifier approximates the followingfunction ytest for binary classification:
ytest = sgn
&n!
i=1
yi exp
&! |xtest ! xi|2
"2
''. (9)
J(x, y) =
�
�y2
�2y1
�
�T �
�K0,0(x, y) K1,0(x, y) K2,0(x, y)K0,1(x, y) K1,1(x, y) K2,1(x, y)K0,2(x, y) K1,2(x, y) K2,2(x, y)
�
�
�
�x2
�2x1
�
� .
Application: Binary Classification
/32
Experiments on Time Time consumption of prediction: The proposed method is faster when # of training data is over 150,000.
February 14, Master Thesis Defense 26
Binary Classification Using Fast Gaussian Filtering Algorithm 5
Table 1. Accuracy of classification of the Iris flower data set(Classification between two species with 5-fold cross validation)
Proposed C-SVMMethod C = 0.1 C = 1 C = 10 C = 106
! = 0.1 93% 58% 68% 69% 69%! = 0.3 93% 83% 94% 93% 93%! = 1 93% 93% 95% 93% 92%! = 3 86% 90% 94% 95% 89%! = 10 84% 85% 90% 93% 92%
Table 2. Computational time to predict a class(Every vector has two real numbers between 0 and 10, ! is 1)
# of training data 1, 000 10, 000 100, 000 1, 000, 000
Proposed Prediction 2.43 ms 2.48 ms 2.55 ms 2.96 msmethod (Precomputation) (0.13 s) (1.48 s) (17.8 s) (203 s)
LIBSVMPrediction 0.02 ms 0.22 ms 2.29 ms 23.6 ms(Training) (0.05 s) (4.47 s) (442 s) (1000+ s)
predicts a class by calculating all the distances between a target and trainingdata. Precomputation of the proposed method takes nearly linear on the numberof training data. Training of LIBSVM takes time that is nearly proportional tothe square of the number of training data.
6 Conclusions
This paper showed that FGFA can be applied to sparse data using the d-dimensional range tree algorithm. This is the same classification as a C-SVMwhose parameter C approaches 0, and it has as good accuracy as C-SVMs. Ad-ditionally, we revealed that this can reduce time of prediction of C-SVMs whenthe number of support vectors is larger than 10,000.
References
1. Kentaro Imajo. Fast Gaussian Filtering Algorithm Using Splines. In The 21stInternational Conference on Pattern Recognition, 2012.
2. Koji Tsuda. Overview of Support Vector Machine. The Journal of the Institute ofElectronics, Information, and Communication Engineers, 83(6):460–466, 2000.
3. Christopher M. Bishop. Pattern Recognition and Machine Learning (InformationScience and Statistics). Springer-Verlag New York, Inc., 2006.
4. Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A Library for Support VectorMachines. ACM Transactions on Intelligent Systems and Technology, 2(3):27, 2011.
5. Jon Louis Bentley. Decomposable Searching Problems. Information ProcessingLetters, 8(5):244–251, 1979.
6. R.A. Fisher. The Use of Multiple Measurements in Taxonomic Problems. Annalsof Human Genetics, 7(2):179–188, 1936.
Table Time of Training and Prediction for 2D Data
Application: Binary Classification
/32
Experiments on Accuracy If we treat input data equally, accuracy of a new classifier and LIBSVM: The loss of accuracy is small. February 14, Master Thesis Defense 27
Binary Classification Using Fast Gaussian Filtering Algorithm 5
Table 1. Accuracy of classification of the Iris flower data set(Classification between two species with 5-fold cross validation)
Proposed C-SVMMethod C = 0.1 C = 1 C = 10 C = 106
! = 0.1 93% 58% 68% 69% 69%! = 0.3 93% 83% 94% 93% 93%! = 1 93% 93% 95% 93% 92%! = 3 86% 90% 94% 95% 89%! = 10 84% 85% 90% 93% 92%
Table 2. Computational time to predict a class(Every vector has two real numbers between 0 and 10, ! is 1)
# of training data 1, 000 10, 000 100, 000 1, 000, 000
Proposed Prediction 2.43 ms 2.48 ms 2.55 ms 2.96 msmethod (Precomputation) (0.13 s) (1.48 s) (17.8 s) (203 s)
LIBSVMPrediction 0.02 ms 0.22 ms 2.29 ms 23.6 ms(Training) (0.05 s) (4.47 s) (442 s) (1000+ s)
predicts a class by calculating all the distances between a target and trainingdata. Precomputation of the proposed method takes nearly linear on the numberof training data. Training of LIBSVM takes time that is nearly proportional tothe square of the number of training data.
6 Conclusions
This paper showed that FGFA can be applied to sparse data using the d-dimensional range tree algorithm. This is the same classification as a C-SVMwhose parameter C approaches 0, and it has as good accuracy as C-SVMs. Ad-ditionally, we revealed that this can reduce time of prediction of C-SVMs whenthe number of support vectors is larger than 10,000.
References
1. Kentaro Imajo. Fast Gaussian Filtering Algorithm Using Splines. In The 21stInternational Conference on Pattern Recognition, 2012.
2. Koji Tsuda. Overview of Support Vector Machine. The Journal of the Institute ofElectronics, Information, and Communication Engineers, 83(6):460–466, 2000.
3. Christopher M. Bishop. Pattern Recognition and Machine Learning (InformationScience and Statistics). Springer-Verlag New York, Inc., 2006.
4. Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A Library for Support VectorMachines. ACM Transactions on Intelligent Systems and Technology, 2(3):27, 2011.
5. Jon Louis Bentley. Decomposable Searching Problems. Information ProcessingLetters, 8(5):244–251, 1979.
6. R.A. Fisher. The Use of Multiple Measurements in Taxonomic Problems. Annalsof Human Genetics, 7(2):179–188, 1936.
Table Accuracy for Iris Flower Data Set
Application: Binary Classification
/32
Conclusions of Part 2 • It speeds up prediction of SVM: O(n) → O(4dlogdn).
• With small loss of accuracy, it speeds up training: O(n2) → O(4dlogdn). The proposed method consists of • The key algorithm: A fast approximation algorithm for the Gaussian filter,
• Range trees.
February 14, Master Thesis Defense 28 Application: Binary Classification
/32 February 14, Master Thesis Defense 29
1. Key Algorithm: Gaussian Filtering 2. Application: Binary Classification 3. Conclusions
Conclusions
/32
Conclusions Key Algorithm: A fast approximation algorithm for the Gaussian filter: • It uses a spline function for approximation. • It speeds up Gaussian filtering, which is often used in image processing. Application: Binary classification • It extends the key algorithm with range trees. • It speeds up prediction of SVMs. • It speeds up training with small loss of accuracy.
February 14, Master Thesis Defense 30 Conclusions
/32
Related Achievements International Conference (Reviewed): • Kentaro Imajo, Otaki Keisuke, Yamamoto Akihiro, "Binary Classification Using Fast Gaussian Filtering Algorithm,” ALSIP 2012.
• Kentaro Imajo, "Fast Gaussian Filtering Algorithm Using Splines,” ICPR 2012.
February 14, Master Thesis Defense 31 Conclusions
/32
Other Achievements International Conference (Reviewed): • Takuya Akiba, Kentaro Imajo, Daisuke Okanohara, "Engineering Parallel String Sorting Algorithms,” Algorithms for Large-Scale Information Processing in Knowledge Discovery, 2011.
Journal: • Kentaro Imajo, "Contest Environment Using Wireless Networks: A Case Study From Japan,” Olympiad in Informatics 5, 26―31, 2011.
• Mahito Sugiyama, Kentaro Imajo, Keisuke Otaki, Akihiro Yamamoto, "Semi-Supervised Ligand Finding Using Formal Concept Analysis,” IPSJ Transactions on Mathematical Modeling and Its Applications 5(2), 39―48, 2012.
Programming Contests: • ICFP Programming Contest, the 2nd place, September 2011. • ACM/ICPC Asia Regional Contest in Fukuoka, Japan, the 5th place, November 2011. • ACM/ICPC Asia Regional Contest in Hsinchu, Taiwan, the 3rd place, November 2011. • ACM/ICPC World Finals 2012, an 18th place tie, May 2012.
February 14, Master Thesis Defense 32 Conclusions