1

Query Expansion Based on Visual Content

Lazaros T. Tsochatzidis

Athanasios Ch. Kapoutsis

Nikolaos I. Dourvas

Savvas A. Chatzichristofis

Yiannis S. Boutalis

Monday, May 1, 2023

Department of Electrical and Computer Engineering Democritus University of Thrace

Monday, May 1, 2023 2

How the today’s web search engines work?

Keywords (from the user)Metadata (keywords, tags, any other

information)

compare

Is everything working fine?

Monday, May 1, 2023 3

The Problems

Ordinary image search is based on metadata, sometimes irrelative or missing

Very difficult to describe(using textual query) the image exactly as it is

Monday, May 1, 2023 4

Our goal… is to reduce the sematic gap between “What” the user wants to find and “How” he describes it.

Monday, May 1, 2023 5

How does TsoKaDo work?

Monday, May 1, 2023 6

CEDD(Color and Edge Directivity Descriptor)

Monday, May 1, 2023 7

What is it?

Belongs to the family of Compact Composite Descriptors (CCD’s)

Describes color and texture of an image.Used for searching and retrieving images

from multimedia databases.

WHY?The results are very effective.It is very light and fast as it used only 54

bytes per image

Monday, May 1, 2023 8

Function of CEDD For the color:2 fuzzy systems map the color of the

image in a 24-color palette.

For the texture:Using a fuzzy version of the five digital

filters proposed by MPEG-7 Edge Histogram Descriptor

Monday, May 1, 2023 9

Classification

Monday, May 1, 2023 10

ClassificationSGONGThe Self Growing and Self Organized Neural Gas Network (SGONG) Innovative neural classifier Dynamically defines the number of classes

needed

K-means N observations into k clusters Criterion: Each observation belongs to the

cluster with the nearest mean.

Monday, May 1, 2023 11

The proposed tags

Monday, May 1, 2023 12

Semantic similaritySemantic similarityThe definition of it relates to

computing the conceptual similarity between terms that are not lexicography similar

Why we use it?It reduces the amount of proposed

tags and gives more corresponding results.

Monday, May 1, 2023 13

WORDNETA lexical database which provides a large

repository of English lexical terms.

How it works?Parts Of Speech (POS) →

noun,verb,adjective, adverb.Establishes connection between POS and

groups them into “synsets”.“Sense” → A group of synset. .Represents

different meaning of the same term.

Monday, May 1, 2023 14

The “Path length” measurement

Wu and Palmer method

Monday, May 1, 2023 15

Tsokado in action

Monday, May 1, 2023 16

Problems And ObstaclesCEDD is based only in color and texture

of the image

Noisy tags at Flickr (human imported): Irrelevant tags

Time consuming when using Wordnet.

Monday, May 1, 2023 17

Solutions and future workUse of Visual Words instead of CEDD

for better semantic classification

Fetching of tags from more reliable sources (e.g. Wikipedia)

Use of EuroWordnet, a multilanguage version.

Monday, May 1, 2023 18

Are there any ?'s

Please visit:http://tsokado.nonrelevant.net