This work was supported by NASA EPSCoR.

Content Based Retrieval for Remotely Sensed Imagery

Badrinarayan Raghunathan and Scott T. ActonThe Oklahoma Imaging Laboratory

School of Electrical and Computer EngineeringOklahoma State University

Stillwater, OK 74078Email: raghuna@fajita.ecen.okstate.edu,sacton@ceat.okstate.edu

AbstractWe present a framework for content based retrieval (CBR)of remotely sensed imagery. The main focus of ourresearch is the segmentation step in CBR. A bank of gaborfilters is used to extract regions of homogeneous texture.These filter responses are utilized in a multiscaleclustering technique to yield the final segmentation. Novelarea morphological filters are utilized for the purpose ofscaling. The resultant segmentation yields regions that arehomogeneous in terms of texture and are significant interms of scale. These regions are used for the purpose ofextracting shape and textural features (on a global andlocal basis) that provide important similarity cues in CBRof remotely sensed imagery. In comparison to solutionswhich use region merging, the segmentation from thetexture / scale space does not require heuristic post-processing, nor knowledge of the number of significantregions.

1. INTRODUCTION

To manage and utilize large multimedia databases,content based retrieval (CBR) tools for diverseapplications like automated inspection, databasemanagement and web based searches [1], [2] have emergedrecently. Due to increases in the number of satellites,available bandwidth, and in commercial applications ofremote sensing, image databases containing remotelysensed images have been expanding rapidly. Remotesensing data are finding applications in many diverse areasincluding agriculture, meteorology, geology and urbanplanning. Traditionally these databases have been accessedby means of metadata which contain keyword descriptorsof the database entities. Obvious issues such as access

automation and metadata relevancy limit the effectivenessof such keyword based search methods.

In this paper we present a CBR method for remotelysensed imagery based on the efficient extraction of textureand shape features. Our work hinges on an imagesegmentation technique that exploits both texture andscale. This approach explores the clustering of imagepositions in a 4-D texture / scale space. We also study animportant aspect of matching large-scale texturallyhomogeneous regions. Finally we present our featuresimilarity computation strategies and give experimentalresults involving a prototype database consisting ofLandsat TM imagery.

2. TEXTURE ANALYSIS FORSEGMENTATION

Our CBR approach models each remotely sensedimage as a union of segments with various texture andshape properties. A vast literature exists on texturesegmentation and classification including gray level co-occurrence matrices, Gabor filters, moments, entropy,Moran autocorrelation functions, fractal methods andprincipal component analysis. The Gabor filterbankapproach provides a versatile model for texture descriptionand optimizes the balance between localization in thespatial and frequency domains [3]. Moreover, Gabor filtersare known to mimic the biological perception of texture.The texture analysis presented in this paper utilizes a bankof Gabor filters to create a texture space. Within thetexture space, we can perform both texture-based featureextraction and segmentation.

A 2-D Gabor function is a complex sinusoidal gratingmodulated by a 2-D Gaussian function:

)](2exp[),(),( '' VyUxjyxgyxh += π ,

( )

+−

=

2

22

2

2 2exp

2

1),(

σλ

πλσ

yxyxg

where )cossin,sincos(),( '' φφφφ yxyxyx +−+=are rotated coordinates, and ),( yxg is a 2-D Gaussian

function with aspect ratio λ , scale parameter σ , and

major axis oriented at an angle φ from the x-axis. The pair

(U, V) gives the center frequencies of the Gabor filter.Textures at different orientations and frequencies can beextracted by means of applying a number of Gabor filterstuned at different frequencies and orientations. Thesefrequencies and orientations can be specified in terms ofthe Gabor half-peak frequency and orientation bandwidths[3]. Ideally, several Gabor filters (e.g., 41 filters) would beused to provide coverage and to discriminate between thepossible textures. In our experiments we have used areduced set of g Gabor filters to balance the tradeoffsbetween computation time and efficient feature extraction,where g ranges from 5 to 17. The texture space is realizedby convolving each of the Gabor filters with the inputimage.

Recently a fuzzy c-means (FCM) based clusteringtechnique was introduced to segment images through ascale-space [4]. (A scale-space contains a set of scaledimages varying from coarse to fine.) A similar vector FCMbased technique could be used here to cluster vectorswithin the texture space. However, segmentation resultsbased solely on the texture space may contain insignificantor spurious regions in terms of area, which lead to errorsand increased computational burden in CBR. Our emphasiswithin the context of CBR has been on matching largescale regions that are similar in terms of texture. Thesolution presented here overcomes the difficulties withsmall scale regions by utilizing a combined texture / scalespace. For each texture space layer (each Gabor-convolvedresult), a set of scaled representations is created via areamorphological operations.

Specifically, the area open-close operation [6] is used toremove connected components within the image level sets(thresholded image representations) that do not meet thespecified minimum area (where the minimum area dependson the sensor resolution and the particular remote sensingapplication). For a level set B, we can define the area openoperation by

(x, y) ∈ s$ (B) if |CB(x, y)| ≥ s,

where |CB(x, y)| is the cardinality (area) of the connectedcomponent at location (x, y), and s is the minimum area.

On the other hand, the area open implies that (x, y) ∉ s$ (B)

if |CB((x, y))| < s. The area close operation is similarlyconstructed, using instead the Boolean complement of B.

For grayscale imagery, each level set is area open-closed independently, and the grayscale result is computedby a stacking operation. Traditionally these operationshave been cumbersome and time-consuming due toconnected component labeling at each level set. Thedrawback of computational cost has been overcome bymeans of a fast algorithm for this area morphologicalprocess. Given a fast algorithm for the area openalgorithm, we can produce an area close result byemploying the Boolean complement of the input level setsused in the area open operation.

In the fast algorithm used here, we have utilized thestandard open filter as the starting point. Any connectedcomponents within the level sets that (partially) survive theopen operation are fully reconstructed. In the morphologyliterature, the original image has been termed the maskimage I , the opened image a marker image M , and the finalproduct the grayscale reconstruction R. Although the fastalgorithm improves vastly upon the computational costrequired, it is not equivalent to the area open operation. Itshould be regarded instead as an approximate algorithm.This is because the opening by reconstruction is notequivalent to an area opening, since some connectedcomponents that exceed the area criterion may not survivethe opening. We now elaborate on the fast algorithm forarea open-close.

2.1. Marker image creation using the open filter

In order to ensure that connected components ofinsufficient area do no survive the filtering process weapply a square or circular structuring element K of totalsize of a. In this case, a is the minimum area parameter.This marker image is used as a precursor to thereconstruction process discussed below.

2.2. Reconstruction by geodesic dilation

The process of reconstruction involves taking a partialconnected component (within any level set, obtained frommarker image M ) and reconstructing the entire connectedcomponent, based on intensities in the input image I . Thereconstruction of the connected components is achieved byselectively dilating these components (one pixel at a time).Essentially, if a connected component in the marker imagehas a neighboring pixel (in 4-connectivity) of lowerintensity than that of the input image, then that pixel isincreased to the minimum intensity between the dilatedimage and the input image:

( )( ) ( ) yxIyxKyxR tt ,,,min),( 1+

− ⊕= R

where MR =0 , and +K is a 3x3 cross-shaped

structuring element with the origin at the center. Uponcomplete reconstruction of each of the marked connectedcomponents the update in the above equation stabilizes.This requires T total updates. This value T is bound by themaximum geodesic distance between a boundary pixel of aconnected component in M and the boundary in thereconstructed connected component in R. Further detailsregarding a formal computational complexity analysis ofsuch fast algorithms for area morphology can be found in[6].

For each position (x, y) in the input image, each scales and each texture layer t, we have intensities I(x, y, s, t) inthe 4-D texture / scale space. These intensities are used tocluster vectors for each position (x, y) using the FCMalgorithm. We can consider a vector I (x, y) as theevolution of the pixel intensity at (x, y) through scale s andtexture t. The fuzzy clustering technique is based onminimizing an objective functional that quantifies thedistance between cluster centers and the data within thevarious clusters. This objective functional is given by

∑ ∑Ω =

=C

ii

mim yxdyxuUJ

1

2||),(||)),((),( µ

Here, U is the fuzzy C – class partition of the texture/scale-space, where C is the number of classes. µ is the set ofcluster centers, and Ω is the domain over which the

clustering is done, i.e. ( ) Ω∈yx, . Given a texture / scale

space vector I (x,y) at location (x, y), the measure

||),(||||),(|| ii yxyxd µ−= Iis the distance between the texture / scale space vector and

the i th cluster center iµ . The distance is weighed by the

fuzzy membership value (of each texture / scale space

vector) ),( yxui corresponding to i th class. The fuzzy

exponent m has the range [ ]∞∈ ,1m . The objective

functional is iteratively minimized. This iteration is subjectto the conditions

∑ ∑Ω =

=l

ii yxu

11),( ,

∑Ω

Ω<< ),(0 yxui

and 0),( ≥yxui .

At each iteration the fuzzy membership value for eachtexture / scale-space vector I (x,y) is computed by

∑ ∑Ω =

−

=

C

j

m

j

ii yxd

yxdyxu

1

)1/(2

]),(),(

/[1),(

Initially this membership value is computed using auniformly-distributed random number generator. At each

iteration, the cluster center iµ is updated according to

∑

∑

Ω

Ω=m

i

mi

i

yxu

yxyxu

)),((

),()),(( Iµ

The clustering proceeds in this iterative fashion untilconvergence, where convergence is defined byinsignificant changes in the observed objective functionalbetween two consecutive iterations.

The resultant segmentation provides region that arehomogeneous in terms of texture and are significant interms of minimum scale. In contrast to the solutions whichuse heuristic region merging, the segmentation fromtexture / scale space does not require post-processing, norknowledge of the number of significant regions. Themotivation for multi-scale, multi-texture clustering hasbeen that it improves classification by clustering membersof similar objects more effectively than a fixed scaleclassifier [4]. An example of the superiority of the texture /scale space approach over the segmentation approach thatonly utilizes texture is given in the results.

3. GLOBAL AND LOCAL FEATUREEXTRACTION

In addition to segmentation, the texture / scale space isused for texture feature extraction. The types of texturefeatures extracted are the mean, standard deviation andentropy of the Gabor-filtered images. Global and localfeatures are computed. The global features include theglobal texture features and the number of segments, whilethe local features are the segment specific texture andshape features. In terms of matching regions ofhomogeneous texture, it is not sufficient to simply matchby texture (granularity and directionality). Our CBRengine also takes segment shape into account. Kauppinenet al. [5] have described an experimental comparison ofvarious shape classification techniques and have shownresults that characterize Fourier based shape features ashaving a high degree of accuracy. We have used Fourierbased features as well as simple features such as thecompactness parameter (Area/Perimeter2) for matchingsegments based on shape.

The Fourier based features are derived from thesegment boundary representations. The boundaryrepresentations used in our experiments include thecurvature, radius, complex coordinate and affine invariantrepresentations [5]. The actual Fourier based features areappropriately weighted versions of the Fourier transform

coefficients of the boundary positions [5]. An affineinvariant shape feature has also been incorporated with aview of matching remote sensing images of overlappingland areas (w.r.t. a query image) taken at different satellitepositions (inferring a change in viewpoints, viz. affinetransformations). Notice that the circular objects in theoriginal image I in Figure 1 are a result of center pivotirrigation systems (CPIS). For example, these circularsegments can provide strong shape cues in a CBR querywhere the user is interested in retrieving images withsimilarly irrigated fields. Figures 1 and 2 show the sampleimage I and the corresponding scaled Gabor filterresponses.

Figure 3 provides an example of the importance of thetexture / scale space in segmentation. In Figure 3, aLandsat TM image (band 3) is shown, along with thecorresponding segmentations achieved by clustering in thetexture space alone and by clustering in the texture / scalespace. The resultant segments from the texture / scalespace can be used for texture and shape based CBR.

4. FEATURE SIMILARITYCOMPUTATION AND QUERYPROCESSING

An important aspect of our research includes theactual computation of feature similarity. Simple matchingbased on combination of local feature errors can beundesirable in view of one feature error dwarfing anotherfeature error and thus skewing the total error magnitude.Moreover, appropriate weighting of the texture and shapefeatures involved in CBR is still an evolving area ofresearch. We have used a novel sieve methodology tocombine different features intelligently. The sieve methodessentially trims the initial search space by using acombination of the global features and a local feature – thecompactness parameter. A cutoff is imposed on thematches from the first stage, and then those matches areprovided to the second stage search using local texture andFourier based shape features.

The second stage matches are presented as the finalmatches. Segmentation and feature extraction areperformed off-line in view of the considerable size of thedatabase. The actual query processing is executed on-line.The user specifies a query image while browsing thedatabase library. The query may be specified in the form ofa specific image or in the form of a specific segment /region in a specific image. Figures 4 and 5 show somesample CBR queries and results using the texture / scalespace approach.

5. CONCLUSIONS

A general texture / shape based framework has beenpresented for content based retrieval of remotely sensedimagery. A texture based segmentation scheme has beenutilized effectively to extract and match large texturallyhomogeneous regions. Potential applications of the remotesensing CBR engine include precision agriculture,hydrology, meteorology and retrieval of cloud-freeimagery for image analysis.

REFERENCES[1]. Badrinarayan Raghunathan and S.T. Acton, “A Content

Based Retrieval Engine for Circuit Board Inspection ,”Proc. of the IEEE Int. Conf. on Image Processing, Kobe,Japan, Oct. 25-29, 1999.

[2]. W. Y. Ma and B. S. Manjunath, “NeTra: A Toolbox forNavigating Large Image Databases,” Proc. of the IEEE Int.Conf. on Image Processing, Santa Barbara, CA, pp. 568-571, Oct. 26-29, 1997.

[3]. A.C. Bovik, M. Clark, and W.S. Geisler, “Multichanneltexture Analysis Using Localized Spatial Filters,” IEEETrans. PAMI, vol. 12, no. 1, pp. 55-73, Jan. 1990.

[4]. D.P. Mukherjee and S.T. Acton, “Document pagesegmentation using multiscale clustering,” Proc. IEEE Int.Conf. On Image Processing, Kobe, Japan, Oct. 25-29, 1999.

[5]. H. Kauppinen, T. Seppnaaen and M. Pietikainen, “Anexperimental comparison of autoregressive and Fourierbased descriptors in 2D shape classification,” IEEE Trans.PAMI, vol. 17, no. 2, pp. 201- 207, 1995.

[6]. S.T. Acton, “Fast Algorithms for Area Morphology,”submitted to Digital Signal Processing: A Review Journal.

Figure 1 . (from left to right) Original image I and itsscaled gabor filter response (area=25)

Figure 2 . (from left to right) Scaled gabor filter

responses of image I from Fig. 1, Areas=100 and225

Figure 3. (from left to right) Texture Space

segmentation, Texture / Scale Spacesegmentation of image from Fig. 1

Figure 4. CPIS Query Image, Best matches – indecreasing order of similarity (left to right, top to

bottom)

Figure 5. Riverbank query image, Best matches in

decreasing order of similarity (left to right, top tobottom)