IASTED conf.

"Computer Science and Technology", May 19-21, 2003 Cancum, Mexico, 214-218

USING IMAGE MINING FOR IMAGE RETRIEVAL

Peter Stanchev
Kettering University
Flint, Michigan, USA
[email protected]
www.kettering.edu/~pstanche

ABSTRACT methods ranging from functional transformations such as

Fourier descriptors to structural transformations such as
chain codes and curvature scale space feature vectors.
In this paper a new method for image retrieval using high Spatial features are presented: as a topological set of
level semantic features is proposed. It is based on relations among image-objects; as a vector set of
extraction of low level color, shape and texture relations, which considers the relevant positions of the
characteristics and their conversion into high level image-objects; as a metric set of relations; 2D-strings;
semantic features using fuzzy production rules, derived geometry-based θR-strings; spatial orientation graphs;
with the help of an image mining technique. Dempster- quadtree-based spatial arrangements of feature points.
Shafer theory of evidence is applied to obtain a list of High level image semantic representation
structures containing information for the image high level techniques are based on the idea of developing a model of
semantic features. Johannes Itten theory is adopted for each object to be recognized and identifying image
acquiring high level color features. regions which might contain examples of the image
objects. One early system aimed at tackling this problem
is GRIM_DBMS [4]. The system analyzed object
KEY WORDS drawings, and use grammar structures to derive likely
interpretations of the scene. Another technique for scene
analysis, using low-frequency image components to train
Image mining, Semantic features, Multimedia databases
a neural network is presented in [5]. The concept of the
semantic visual template is introduced by Chang et al [6].
The user is asked to identify a possible range of color,
1. INTRODUCTION texture, shape or motion parameters to express his or her
query, which is then refined using relevance feedback
techniques. When the user is satisfied, the query is given a
More and more audio-visual information is available in semantic label (such as “sunset”) and stored in a query
digital form, in various places around the world. MPEG- database for later use. The use of the subjective
7, formally called "Multimedia Content Description characteristics of color (such as warm or cold) to allow
Interface", was created to describe multimedia documents. retrieval of images is described in [7].
The most used features for image description are: color, Image mining deals with the extraction of
texture, shape and spatial features. Many of the existing knowledge, image data relationship, or other patterns not
image databases allow users to formulate queries by explicitly stored in the images [8]. It uses methods from
submitting an example image. The system then identifies computer vision, image processing, image retrieval, data
those stored images whose feature values match those of mining, machine learning, database, and artificial
the query most closely, and displays them. Color features intelligence. Rule mining has been applied to large image
are usually represented as a histogram of intensity of the databases [9, 10, 11]. There are two main approaches. The
pixel colors. Same system, such as Color-WISE [1], first approach is to mine from large collections of images
partitions the image into blocks and each block is indexed alone and the second approach is to mine from the
by its dominant hue and saturation values. Color and combined collections of images and associated
spatial distribution can be also captured by an anglogram alphanumeric data. [11] presents an image mining
data structure [2]. The most used texture features are the algorithm using blob needed to perform the mining of
Gabor filters [3]. Other texture measurements are: Tamura associations within the context of images. [10] uses rule
features, Unser’s sum and difference histogram, mining to discover associations between structures and
Galloway’s run-length based features, Chen’s geometric functions of human brain. In this paper we use image
features, Laine’s texture energy. Shape feature mining to define rules for converting low level semantic
techniques are represented from primitive measures such characteristics into high level features.
as area and circularity to more sophisticated measures of
various moment invariants; and transformation-based
 In this paper a method for image retrieval, based 2.2. Texture characteristics
on high level image semantic features is presented. For
example “Baroque period”, era in the history of the
Western arts roughly coinciding with the 17th century can The Quasi-Gabor filter [14] is explored to present the
be defined as: “The work that distinguishes the Baroque image texture features. The image is characterized with
period is stylistically complex, even contradictory. In 42 values by calculation of the energy for each block
general, however, the desire to evoke emotional states by defined by a combination of one of 6 frequencies (f = 1, 2,
appealing to the senses, often in dramatic ways, underlies 4, 8, 16 and 32) and one of 7 orientations (q = 0°, 36°,
its manifestations. Some of the qualities most frequently 72°, 108°, 144°, 45° and 135°). We take the average value
associated with the Baroque are grandeur, sensuous of the magnitude of the filtered image in each block.
richness, drama, vitality, movement, tension, emotional
exuberance, and a tendency to blur distinctions between
the various arts.” We try in our search to use such high 2.3. Shape characteristics
level terms. The layout of the paper is as follows. In
section 2 we explain the image feature extraction
mechanism. In section 3 we describe image retrieval For shape representations a procedure based on [15] is
based on high level semantic features. In section 4 we adopted. The image is converting into binary. Polygonal
detail our experiments, and finally in section 5 the approximation that uses straight-line, Bézier curve and B-
conclusions of this paper are presented. Spline are applied. As a result the image is presented as a
set of straight lines, arcs and curves.

2. IMAGE FEATURE EXTRACTION

MECHANISM 2.4 Multidimensional association-rules
mining
The proposed mechanism transfers low level image
characteristics into high level semantic features using For a given image database we construct a database with
fuzzy production rules with degree of recognition and records containing the following structure: (imageID, C1,
image interpretation, based on the Dempster-Shafer C2, …, Cn, T1, T2, …, Tm, S1, S2, …, Sk, F1, F2, …, Fl),
theory of evidence. For the low level image where imageID is a unique identification of the image; C1,
characteristics the following color, shape and texture C2, …, Cn, are the values of the color characteristics; T1,
features are calculated. T2, …, Tm, are the values of texture characteristics; S1, S2,
…, Sk, are the values of shape characteristics; F1, F2, …,
Fl are the high level semantic features, given by an expert
2.1. Color characteristics in the field. The mining process is divided into two steps.
First we find the frequent multidimensional value
combinations and find the corresponding frequent features
The color feature extraction procedure includes color in the database. The combination of attribute values that
image segmentation. For this purpose ideas from the occurs two or more times are called multidimensional
procedure described in [7] are adopted. Fist the standard pattern [16]. For mining such pattern a modified BUC
RGB image is converted as L*u*v* (extended algorithm [17] is used. The second step includes mining
chromaticity) image, where L* is luminance, u* is the frequent features for each multidimensional pattern.
redness–greenness, and v* is approximately blueness– They constitute the obtain rule base set for the high level
yellowness [12]. Twelve hues are used as fundamental semantic features.
colors. There are yellow, red, blue, orange, green, purple,
and six colors obtained as linear combinations of them.
Five levels of luminance and three levels of saturation are 2.5. Low level feature translation into high
identified. This results that every color is transferred into
one of 180 references colors. After that clustering in the level image semantic
3-dimensional feature space is performed using the K-
means algorithm [13]. After this step the image is
segmented as N regions, every of which is presented in The purpose of this phase is to compose more complex
extended chromaticity space. image semantic interpretation from the derived through
the low-level image analysis features. It is accomplished
by applying methods for extracting high level features and
recursively applied production rules from a set defined for
the correspondent application domain. The rules are
defining also the degree of recognition (RD) of a high
level semantic feature as a distance between features
implied in the rule and those found in the image. RD is Johannes Itten in 1960 [20]. In this theory, seven types of
calculated with the help of fuzzy measures. An contrast are defined:
interference mechanism based on backward chaining tries 1. Contrast of hue
to derive from the low level features more general 2. Light-dark contract
features and to give a recognition degree to the features 3. Cold-warm contrast (Yellow through red–purple
recognized. In this phase a generalized inference give the filing of ‘‘warm’’, yellow–green
mechanism is used. After this step a sequence in the form through purple is find as ‘‘cold’’)
(1) is obtained: 4. Complementary contrast
(1) O1 1(m1 1, l1 1), ..., O1 S1(m1 S 1, l1 S 1), ... On 5. Simultaneous contrast
1(mn 1 , ln 1), ..., On Sn(mn S n, ln S n). 6. Contrast of saturation
Such a sequence describes an image with n 7. Contrast of extension
distinct high level semantic features. The unit Oi Harmony is defined as a combination of colors
resulting in a gray mix that generates stability effect onto
j(mi j, li j) is a semantic representation of the image the human eyes. Non-harmonic combinations are called
feature i (i=1,2, … ,n) in the j-th (j=1,2, …, sj) expressive. Itten’s model is adopted for defining fuzzy
recognition. mi j and li j are respectively the RD and the production rules that are used to translate the low level
list of attributes of the i-th semantic feature in the j-th semantic features into sentences qualifying warmth
recognition. A logic programming language Prolog is degree, and contrasts among colors.
chosen to express the feature recognition rules. In this
case the Prolog's inference mechanism is used to perform 3.2. Retrieval by high level texture properties
the high level feature recognition.
To reduce the sequence (1) a procedure similar Transforming the low level texture characteristics into
to Barnett's scheme [18], based on the Dempster-Shafer high level semantic features such as texture of wood,
theory of evidence [19] is applied. The results obtained rock, wall-paper, etc. is made by calculation the low level
from applying the production rules are converted into a texture characteristic of a typical set of corresponding
list of new structures containing information for each textures and finding the “cluster center” values which is
semantic feature: used in the fuzzy production rules.
(2) O11([Bel(O11, 1-Bel(not O11)], l11), ...,
O1q1([Bel(O1 q1, 1-Bel(not O1 q1)], l1 q1), ..., 3.3. Retrieval by high level shape properties
On1([Bel(On1, 1-Bel(not On1)], ln1), ...,
O q1([Bel(On qn, 1-Bel(not On qn)], ln qn),
A set of typical shapes characterizing the domain specific
where qi ≤ si (i=1, 2, ..., n). objects are defined. Fuzzy production rules are used for
The function Bel(Oij, 1-Bel(not Oij)] is a belief calculation similarity between the search shape and given
object shape. They are obtained after image mining.
function. In such sequence, features interpretations with
low belief are omitted. The belief function Bel(Oi) (i=1,2,
…,n) gives the total amount of belief committed to the
features Oi after all evidence bearing on Oi has been
3.4. Retrieval by high level semantic features
pooled. The function Bel provides additional information
about Oi, namely Bel(not Oi), the extent to which the A set of high level semantic features which are defining in
evidence supports the negation of Oi, i.e. not Oij. the image mining process are used. They combine high
level color, texture and shape properties and high level
semantic features defined by the expert during the image
3. RETRIEVAL BASED ON HIGH LEVEL mining.
SEMANTIC FEATURES
4. THE EXPERIMENTS
In this section we discuss image retrieval based on high
level color, texture, shape and semantic features.
The proposed method is in process of realization in a
system named “Flint”. In our experiments we use an
3.1. Retrieval by high level color properties image database with images from Bulgaria. After low
level image properties extraction image mining was made
for obtaining associate rules, describing the high level
The spatial arrangement of chromatic contents in the image semantic features.
image is obtained using the theory formulated by
 ==

Figure 1. Sea, sky images with a regain with worm color Figure 2. House images

In our example for the query “Find a sea or sky

images with a regain with worm color” the following 5. CONCLUSIONS
images are retrieved (Figure 1.) The result from the query
“Find a house images” is given in the Figure 2. In the first
query we use color description to find images with color The main advantage of the proposed method is the
regions satisfy worm contract and textures of sea and sky. possibility of retrieval using high level image semantic
In the second query we use shape descriptors for the features. After the full system realization we will be able
house forms. to obtain statistic characteristics about the usefulness of
the suggested method.
REFERENCES [12.] Carter, R., Carter, E., CIELUV color difference
equations for self-luminous displays, Color Res. &
Appl., 8(4), 1983, 252–553.
[1.] Sethi, I., Coman, I., Day, B., Jiang, F., Li, D.,
Segovia-Juarez, J., Wei, G., and You, B., Color- [13.] Jain, A., Algorithms for clustering data.
WISE: A System for Image Similarity Retrieval Englewood Cliffs, NJ, Prentice Hall, 1991.
Using Color, Proceedings of SPIE Storage and
Retrieval for Image and Video Databases, Volume [14.] Mira P., Jesse J., Laurence W., Fast Content-
3312, February 1998, 140-149. Based Image Retrieval Using Quasi-Gabor Filter and
Reduction of Image Feature Dimension. SSIAI, 2002,
[2.] Grosky W., Stanchev P., An Image Data Model, in 178-182.
Advances in Visual Information Systems, Laurini, R.
(edt.), Lecture Notes in Computer Science 1929, [15.] Mori K., Wada K., Toraichi K., Function
2000, 14-25. Approximated Shape Representation using Dynamic
Programing with Multi-Resolution Analysis, ICSPAT
[3.] Manjuanth, B., Ma, W., Texture features for '99, 1999.
browsing and retrieval of image data, IEEE
Transactions on Pattern Analysis and Machine [16.] Kantardzic M., Data Mining, Wiley-Interscience,
Intelligence, 18(8), 1996, 837-842. 2003.
[4.] Rabbitti, F and Stanchev, P., GRIM_DBMS: a [17.] Beyer K., and Ramakrishnan, R., Bottom-Up
graphical image database management system, in Computation of Sparse and Iceberg CUBEs.
Visual Database Systems, Kunii, T. (edt.), Elsevier, SIGMOD’99.
Amsterdam, 1989, 415-430.
[18.] Barnett J., Computational Methods for a
[5.] Oliva, A et al., Real-world scene categorization by a Mathematical Theory of Evidence, Proc. 7-th Inter.
self-organizing neural network, Perception, supp 26, Joint Conf. on Artificial Intelligence, Vancouver, BC,
19, 1997. 1982, 868-875.
[6.] Chang, S., et al., Semantic visual templates: linking [19.] Gordon J., Shortliffe J., The Dempster-Shafer
visual features to semantics, in IEEE International Theory of Evidence in Rule-Based Expert Systems,
Conference on Image Processing (ICIP’98), Chicago, in B. Buchanan, E. Shortliffe (edt.), Mycin
Illinois, 1998, 531-535. Experiments of the Stanford Heuristic Programming
Project, Addison-Wesley Publishing Company, 1984,
[7.] Corridoni, J, Bimbo A., Vicario E., Image retrieval 272-292.
by color semantics with incomplete knowledge,
Journal of the American Society for Information [20.] Itten, J., Kunst der Farbe. Ravensburg, Otto
Science 49(3), 1998, 267-282. Maier Verlag. 1961.
[8.] Zhang Ji, Hsu, Mong, Lee, Image Mining: Issues,
Frameworks And Techniques, Proceedings of the
Second International Workshop on Multimedia Data
Mining (MDM/KDD'2001), in conjunction with
ACM SIGKDD conference. San Francisco, USA,
August 26, 2001.

[9.] Ordonez C. and Omiecinski E., Discovering

association rules based on image content.
Proceedings of the IEEE Advances in Digital
Libraries Conference (ADL'99), 1999.

[10.] Megalooikonomou, V., Davataikos C. and

Herskovits, E., Mining lesion-deficit associations in a
brain image database. KDD, San Diego, CA USA,
1999.

[11.] Zaiane, O., Han J., et al. Mining MultiMedia

Data. CASCON'98: Meeting of Minds, pp 83-96,
Toronto, Canada, November 1998.