NEES TEQNIKES AXIOLOGHSHS STHN ANAKTHSH...

110
ΠΑΝΕΠΙΣΤΗΜΙΟ ΠΑΤΡWΝ ΤΜΗΜΑ ΜΗΧΑΝΙΚWΝ Η/Υ ΚΑΙ ΠΛΗΡΟΦΟΡΙΚΗΣ ΜΕΤΑΠΤΥΧΙΑΚΟΠΡΟΓΡΑΜΜΑ <<ΕΠΙΣΤΗΜΗ ΚΑΙ ΤΕΧΝΟΛΟΓΙΑ ΥΠΟΛΟΓΙΣΤWΝ>> ΜΕΤΑΠΤΥΧΙΑΚΗ DΙΠΛWΜΑΤΙΚΗ ΕΡΓΑΣΙΑ ΝΕΕΣ ΤΕΧΝΙΚΕΣ ΑΞΙΟΛΟΓΗΣΗΣ ΣΤΗΝ ΑΝΑΚΤΗΣΗ ΠΛΗΡΟΦΟΡΙΑΣ ΕΥΑΓΓΕΛΟΠΟΥΛΟΣ Γ. ΞΕΝΟΦWΝ Α.Μ: 974 ΤΡΙΜΕΛΗΣ ΕΠΙΤΡΟΠΗ ΜΑΚΡΗΣ ΧΡΗΣΤΟΣ (ΕΠΙΒΛΕΠWΝ) ΤΣΑΚΑΛΙDΗΣ ΑΘΑΝΑΣΙΟΣ ΧΑΤΖΗΛΥΓΕΡΟΥDΗΣΙWΑΝΝΗΣ ΜΑΪΟΣ 2015

Transcript of NEES TEQNIKES AXIOLOGHSHS STHN ANAKTHSH...

/

.

.: 974

()

2015

..

.

- - ( ) - , . - , . , - , , -. -

, . , , - . , , . , , - /, . , , , ., -

, -. Bayes .

i

ABSTRACT

Information retrieval constitutes an important scientific area of the com-puter science, that focuses on the extraction of amounts of unstructuredinformation (usually text from documents) from large collections (corpora,etc.) according to a special information need of a user. Over the last years,one major task of information retrieval research is the evaluation of the re-trieval process. As a result, a vast amount of evaluation metrics and usermodels have been developed, trying to best model users behaviour duringthe search.

In this thesis we propose a new evaluation metric which aims at the bestevaluation of search process from the perspective of users behaviour. A con-ventional approach when estimating the relevance of a document is by usingrelevance judgements from assessors that are responsible to assess whether adocument is relevant according to a specific query. However, relevance judge-ments do not always reflect the opinion of every user, rather from a smallproportion only. Our evaluation metric introduces a novel factor of relevance,document popularity which can be seen as users vote for a document. Thus,by employing a linear combination of relevance judgements and popularity,we achieve a better explanation of users behaviour.

Additionally, we present a novel click user model which by the best mod-elling of users navigational behaviour, aims at the best estimation of the rel-evance of a document. This particular user model, is based on the dynamicBayesian networks theory and employs the notion of popularity in order tobetter estimate actual document relevance, rather perceived relevance, thatmost other models do.

ii

-, . , , . .

. , ., ,

.

iii

1 11.1 . . . . . . . . . . . . . . 61.2 . . . . . . . . . . . . . . . . . . . . . . . . . . 61.3 . . . . . . . . . . . . . . . 71.4 . . . . . . . . . . . . . . . . . 81.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.6 . . . . . . . . . . . . . . . . . . . . . . . 11

1.6.1 . . . . . . . . . . . . . . . . . . 111.6.2 . . . . . . . . . . 13

1.7 . . . . . . . . . . . . . . . . . . . . . 14

2 162.1 . . . . . 17

2.1.1 - TREC . . . 192.2 . . . . . . . . . . . . . . . . . . . . . . . 22

2.2.1 . . . . . . . . . . . . . . . . . . 232.2.2 . . . . . . . . . . . . . . . . . 26

2.3 (DCG) . . . . . . . . . 292.3.1 Rank Biased Precision . . . . . . . . . . . . . . . . . . 322.3.2 Bpref . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342.3.3 Expected Reciprocal Rank . . . . . . . . . . . . . . . . 36

2.4 . . . . . . . . . . . . . 392.5 . . . . . . . . . . . . . . . . . 412.6 . . . . . . . . . . . . . . 42

2.6.1 Crowdsourcing . . . . . . . . . . . . . . . . . . . . . . 432.6.2 / . . . . . . . . . . . . . . . . . . . . . . . 44

3 453.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.2 Reciprocal Rank using Document Popularity . . . . . . . . . . 47

iv

3.2.1 . . . . . . . . . . . . . . . . . . . . . 473.2.2 . . . . . . . . . . . . . . . . . . 483.2.3 . . . . . . . . . . . . . . . . . . 49

3.3 . . . . . . . . . . . . . . . . . . . . . . 513.3.1 . . . . . . . . . . . . . . 51

3.4 . . . . . . . . . . . . . . . . . . . 543.4.1 . . . . 56

4 594.1 . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4.1.1 COEC . . . . . . . . . . . . . . . . . . . . 614.1.2 . . . . . . . . . . . . . . . . . . . 624.1.3 . . . . . . . . . . . . . . . . . . . 62

4.2 cascade . . . . . . . . . . . . . . . . . . . . . . . . 624.3 UBM . . . . . . . . . . . . . . . . . . . . . . . . . 634.4 DCM . . . . . . . . . . . . . . . . . . . . . . . . . 654.5 DBN . . . . . . . . . . . . . . . . . . . . . . . . . 68

5 - 735.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 735.2 . . . . . . . . . . . . . . . . . . . . . . . . 755.3 . . . . . . . . . . . . . . . . . . . . . . 77

5.3.1 . . . . . . . . . . . . . . 775.3.2 . . . . . . . . . . . . . . . . 78

6 806.1 . . . . . . . . . . . . 826.2 . . . . . . . . . . . . . . . . . . . . 83

Bayes 85.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 86.2 . . . . . . . . . . . . . . . . . . . . . . 90

v

1.1 Yahoo!, http://www.yahoo.com, 1995. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2 . . . . . . . . . . . . . . . . 51.3 XML . . . . . . . . . . . . . . . . . . . . . 71.4 .

, - back-end , - - . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.5 Google 1998. . . . . . . . . . . . . 121.6 . . . . . . . . . . . 131.7 snippet . . . . . 14

2.1 Cranfield. , . , - ., , stopwords , - ( [CMK66]). . . . 18

2.2 TREC TREC 7 1998. Ellen Voorhees, . . . . . . . . . . . . . . . . . . 22

2.3 TREC . . . . . . . . . . . . . . 232.4 I.

R - F . D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.5 2.1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

vi

2.6 . - 10. 6, 4 . . . . . . . . . . . . . . . . . . . 27

2.7 RBP . [08]. . . . . . . . . 322.8 RBP TREC-5 50 -

. p , 10 . - 100 p. [08]. . . . . . . . . . . . 34

2.9 - . [BV04]. . . . . . . 36

2.10 . ERR . 1, 0. [CMZG09]. . . . . . . . . . . . . . . . . 39

2.11 Kendalls - 61 TREC-5 . . . . . . . . . . . . . . . . . . 40

2.12 /. . . . . . . . . . . . . . . . . 44

3.1 , TREC Web Tracks. . . . . . . . . . . . . . . . . . . . . . . . 52

3.2 . . . . . . . . . . . 57

3.3 41 . . . . . . 58

3.4 83 .. . . . . 58

4.1 DCM . . . . . . . . . . . . . . . . . . . . . . . . . 674.2 -

. . . . . . . . . . 684.3 DBN . . . . . . . . . . . . . . . . . . 704.4 CTR

1 () KL- (). . . . . . . . . . . . . . . . . . . . . . 71

4.5 NDCG DBN . . . . . . . . 72

vii

5.1 . - - . . . . . . . . . . . . . . . . . . . . . . . 76

5.2 Log-Likelihood . DBN . . . . . . 79

.1 HMM DBN , 3 . 87.2 HMM (

) . - : P (X1 = i) = (i), P (Xt = j|Xt1 = i) =A(i, j), P (Yt = j|Xt = i) = B(i, j). . . . . . . . . . . . . . 88

.3 HMM Gaussians . . . . . . . . . 89.4 Auto-regressive HMM. . . . . . . . . . . . . . . . . . . . . . . 89

viii

2.1 - . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.1 4 . . . . . . . . . . 493.2 . 553.3

. . . . . . . . . . . . . . 56

ix

1

, , , . , . , -

, Taube [Tau54], , -. , . Cranfield , Cleverdon - .

.., . - Cleverdon 18.000 1.200 - . , , -.

, , , - 1952 Cleverdon , -

1

. -

, . - . (relevance judgments ) , - . 1990, -

, . , , . , - , , . ,

, . , , - - . , , , , . , -

, . , - . , , , , , .,

, . ,

2

, - . , [KT03] - . -

, - [ABDR06]. . [CZTR08], [DP08] - ., -

, , . , - , . (click-logs) .

, , . , - -, . ( 1.1)

Boolean , , -. , - -. , Salton[Sal68] , -, .

-

3

1.1: Yahoo!, http://www.yahoo.com, 1995.

. ( ) , [MRS08] ( 1.2). , -

. - , . , , ,

4

1.2:

.

- , , , , . - 1990. (). - , - ., ,

. -, , . , - , - . , , . ,

5

- . . -

, , - , - . , - .

1.1

. , , , . , -

. , , - , . , , -, . , - .

1.2

. - , . ; -. , . - . , , .

6

, . , - , ., -

. - . , , , , .. - , . XML - ( 1.3).

1.3: XML .

1.3

, . , - . ( ), . , - , -. - . ,

7

, , - . , , Q1 Q2.

1.4

. . (hypertext) -, 1990. - ( ), , , , HTML . , - , . - , .

1.5

.html , - , . , - http://ceid.upatras.gr/contact.html , . URL (Univer-sal Resource Locator) HTTP webserver . browsers , - URL . HTML ,

8

. web . , . - , . -, , .

, - . :

Altavista, Excite Infoseek.

Yahoo!.

- - . . , . , - - , , - . , - . , - . , . , -

, - . . - . ,

9

- . () - . , . , . , . -

, . 1995, Altavista - 30 . . -, , . - , 1.4. URL ? . , , 1995, Al-tavista .

1.4: . , back-end , .

10

1.6

, - . , . , - . , - - . , - ( ) , . - - 2 3, , Boolean , .

Google ( 1.5). , Google , , . . Google . , , user interface . Google , , , ( 1.6).

1.6.1

:

,

,

11

1.5: Google 1998.

.

, (.. Facebook ), - , - . , . , , , . , - , . , .

. . , - .

12

1.6: .

1.6.2

- . , . . . , URL . 1.7.

. - . - , - (examination model ). . - , . , -

.

13

1.7: snippet .

cascade . , . - . , .

1.7

, 2 - . Cranfield , - . , . 3 -

, relevance judg-ments . TREC - click-through - .

14

4 -, , . - , . 5, -

. Bayesian , . - , ., 6

, .

15

2

. 1990, , . - , . - -, . - , , . , -

, 1990, - . 2.1 - . Cranfield , - , - , TREC. 2.2 , , nDCG , Expected Reciprocal Rank .. ,

16

2.3, click-through .

2.1 -

, - [BYRN99]. , 1952, Cranfield -, Cleverdon . - Mortimer Taube , Uniterm. , -. , - , Uniterm . Cleverdon ,

, Cranfield-1 , , Uniterm . Cleverdon 18.000 1.200 . , , - . , Cleverdon -

. , Cranfield-2 , , -, ( 2.1). - . Cranfield-2 ,

17

, - . , , Cleverdon . , , -, .

2.1: Cranfield. , . , . , , stopwords , ( [CMK66]).

- , , , qrels .

18

, . qrels , - , . , ,

, - , . - , Cranfield Cranfield [Voo02], [VH05],[BV04].

, Cranfield -, . Cranfield - . , , , . - -, ., - . , Cranfield , .

2.1.1 - TREC

Cranfiled-2 , - . (reference collections ). - D, , I rj , - . , rj = 0, , rj = 1, -

19

. , , 0 1. , 5 , . 5 0 4, 5 - : , , , , [Kek05]. -

, - ., -, - . , ( - ), . , - , . ,

. - . , - . k , - . TREC. 1992 -

, , - . (NIST) , - , TREC . TREC , , -. TREC TIPSTER 750.000 , . - 2GB

20

, [Har93]. TREC ,

, . , . - ( 2.2) , . , TREC -. . TREC ,

( ), , , . TREC [AMWZ09]. - , TREC . -, . , . , TREC-9 -

, -. - ad-hoc TREC . - ( 2.3). TREC .

, . ad-hoc - . , .

21

2.2: TREC TREC 7 1998. Ellen Voorhees, .

. 2005, 117 TREC,

7 , , -, spam. , -. TREC, NTCIR, 1999, [KKK+99], (CLEF) 2000, - -, XML (INEX), 2002 [GK02].

2.2

, -

22

2.3: TREC .

, .

2.1. - .[BYRN99].

- . - 0 1

.rel = rd1 , rd2 , rd3 , ..., rdn,

rdi {0, 1} , , .

2.2.1

- . D I. - D, F . F

23

R - I, 2.2 :

Precision =|R F||F|

(2.1)

Recall =|R F||R|

(2.2)

- . - Cleverdon , , , . , , , . - - -, .

2.4: I. R - F . D .

- - , -. 2.1

24

2.1: - - .

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1 1 1 0.6 0.6 0.6 0.6 0.6 0.44 0.44 0.44

. 2.5 2.1. - . , .

2.5: 2.1.

, - . , -, ., ., Nq , . :

P (rj) =

Nqi=1

Pi(rj)

Nq(2.3)

25

P (rj) rj Pi(rj) rj i- .

- . - . , - . , .

2.2.2

, , - . , . .

n , , . , , 5, 10 20 . Cranfield-2 , -

. , . , - [GJG04], . , precision at 5 (P@5), precision

at 10 (P@10), precision at 20 (P@20) - .

26

Average Precision (AP) . - , . 0. -. , :

AP@k(r) =1

R

kd=1

rd

di=1

ri

d(2.4)

k , , rd . , . 2.6.

2.6: . - 10. 6, 4 .

R- - R-. - R- , R . R- :

RPrec(r) =1

R

Ri=1

ri (2.5)

27

R-Precision -. R-, R - .

- -, van Rijsbergen [Rij79], - - . - :

E(j) = 1 1 + b2

b2

rj++ 1

P (j)

(2.6)

r(i) i- P (i) i- . b (b 0) , -. b - b - . -, b 1, . , b 1, .

F - - F -. F - , . -, F -, - 1 . F - , . F - :

F (j) =2

1r(j)

+ 1P (j)

(2.7)

28

2.3 (DCG)

. - . - Cleverdon , , , . , , , . - -, . - . , .

, - . [JK02] - . , - . , - . , ,

, k , . , , G

, . ,

G = {3, 1, 2, 3, 2, 0, 3, 1, 0, 0, 2, ...}.

G - i, 1 i. CG,

29

:

CG[i] =

{G[1], if i = 1CG[i 1] +G[i], otherwise. (2.8)

, G, :

CG = {3, 4, 6, 9, 11, 11, 14, 15, 15, 15, 17, ...}.

, [Kek05] , . , . . , , , . :

DCG[i] =

{CG[1], if i < bDCG[i 1] +G[i]/ logb i, if i b.

(2.9)

b . , 2. , . -

. , - 100% . , DCG . DCG DCG.

. , . , -

:IG = {3, 3, 3, 3, 3, 2, 2, 2, 2, 1, 1, 1, 1, 0, 0, 0, ...}.

30

, - (ICG) (IDCG) . -

DCG, - , . [Kek05] nDCG. - . nDCG , -

DCG IDCG . [BYRN99], Nq , - ( overlineDCG) :

DCG[i] =1

Nq

Nqj=1

DCGj[i] (2.10)

:

IDCG[i] =1

Nq

Nqj=1

IDCGj[i] (2.11)

nDCG qi DCG qi IDCG . :

nDCG[i] =DCG[i]

IDCG[i](2.12)

, , , nDCG . -, nDCG . ,

31

, - , .

2.3.1 Rank Biased Precision

- , . . , - . , . - , 4. Moffat Zobel [MZ08]

, . , - . , -

. , , p 1 p. , , , p, p2, 1p . 2.7 BRP .

2.7: RBP . [08].

, q,

32

R = r1, r2, ..., rd i -, :

di=1

ri pi1

i .

,

, i=1

i pi1

(1 p) = 11p ,

. RBP , 0.0 1.0, :

RBP = (1 p) di=1

ri pi1. (2.13)

RBP , . - 1, 2, 6, 11 17. p 0.50, RBP 2 11, 2.13 0.5000 0.0010 . , p = 0.80, 0.8000 0.1074. , p

. p 1.0, - -. , p , . , 10 p = 0.5, . , RBP

p. p .

33

p, , . , p , RBP . 2.8 RBP TREC-5 p 10. : 10, p 0.7 , 100, p 0.95 .

2.8: RBP TREC-5 50 -. p , 10 . 100 p. [08].

2.3.2 Bpref

- . , . -

34

, . , - . , , . Buckley - Voorhees [BV04]

bpref , , , . , dj dk, . bpref . - , , , ., R , bpref

:

bpref =1

R

r

1 |n ranked higher than r|R

(2.14)

r n R - . bpref ,

, . bpref bpref 10, 10 . , bpref 10 :

bpref =1

R

r

1 |n ranked higher than r|10 +R

(2.15)

n 10 +R . 2.9 -

, 10, R- bpref 10, - . - . TREC-8 . ,

35

, bpref 10 .

2.9: - . [BV04].

2.3.3 Expected Reciprocal Rank

2.2.3, - NDCG . , . , - NDCG , ., nDCG,

. , ., [CZTR08], [CZ09] i, . , Chappelle et al. [CZ09] -

,

36

. ExpectedReciprocal Rank (ERR), - . Cascade user model -. , Cascade user model - , i, , Ri. , . . Cascade user model (ERR)

:

ERR =nr=1

1

rP (user stops at position r).

, (r) (reciprocal rank) , , (1) = 1 (r) 0 R ., cascade , R :

P (user stops at position r) =r1i=1

(1Ri)Rr. (2.16)

- . , Ri , - 2.16 . , ERR :

ERR =nr=1

1

r

r1i=1

(1Ri)Rr, (2.17)

n . - 1 ERR . O(n2) .

37

Algorithm 1 Algorithm to compute ERR metric in linear time

Require: Relevance grades gi, 1 i n, and a mapping function R.p 1, ERR 0.for r=1 to n do

R R(gr)ERR ERR + p R/rp p (1R)

end forreturn ERR

, . , ., ERR . , , ., -

. , . , , . Radlinski et al. [RKJ08], -

, ERR, ERR . 2.10 ,

ERR , UCTR,Min,Max and Mean Reciprocal Ranks, Search Success, Precision at LowestRank . - ERR ERR , .

38

2.10: . ERR . 1, 0. [CMZG09].

2.4

, - , - . , - , . , . - , . .

Kendalls

Kendalls , - [YAR08]. n, C ( ) A - ( ). , Kendalls

39

:

=C D

N(N 1)/2(2.18)

N(N 1)/2 . Kendalls , - , 1 . , Kendalls , - . , Kendalls 1, - , Kendalls -1, , Kendalls 0. 2.11 -

Kendalls . , 61 TREC-5 . 1.0 , .

2.11: Kendalls - 61TREC-5 .

Pearson

2.2.6 , . ,

40

- . [HH07], - , . - 2.3., , -

Pearson . Pearson , X Y , [1, 1], -1 , 0 1 . Pearson :

rxy =

ni=1

(xi x)(yi y)ni=1

(xi x)

ni=1

(yi y)(2.19)

, n , xi yi x y .

2.5

. , , , . , - , . , - . ,

, . - , ,

41

[JFM97], [LF07]. - . , Fox et al. [FKM+05] , - , . - .

2.6

2.1.1 ., , . . - . , . , - .

. - ( ) -. Joachims [Joa03], - , . , - , . -

, (click-through-rate (CTR)) . , ,

42

, - ., .,

. , . . , - , . , -

[CJRY12] . - r , . , - , .

2.6.1 Crowdsourcing

, , . , -, Yahoo!, Google, Microsoft .., crowd-

sourcing , . - . crowdsourcing ( - ) , . .

43

Amazon Ama-zon Mechanical Turk (AMT), crowd-sourcing - , . , , . - - , .

2.12: /.

2.6.2 /

/ - , . crowdsourcing , . / - , , . , -, . . 2.12 /. .

44

3

? AIAI 2014 : Xenophon Evangelopou-los, Christos Makris, and Yannis Plegas. Reciprocal rank using web pagepopularity. AIAI, Rhodes 2014

3.1

, , . Cranfield [Cyr91] .

- . , - - , , , NDCG Jarvelin - Kekalainen [Kek05], ERR[CMZG09] [YSCR10] .,

45

. - , . - - 5 . , -

, , -, . , - , click-through [CJRY12]. , - [RDR07a]., [HH07], [SZ05]. click-through -

, [HH07]., Carterette Allan [CA05], Sanderson - Joho

[SJ04] . . Buckley - Voorhees [BV04] , - bpref , - . , Sakai , [Sak07]. -

. [CZTR08]. , (position models) cascade [CZ09]. , . cascade , . - NDCG RBP [MZ08], ERR cascade .

46

- . cascade , ( ) . 3.2 - . 3.3 , . ,

. , . Di, , web traffic . - click-metrics . , .

3.2 Reciprocal Rank using Document Popularity

3.2.1

- . . , . - , , cascade [CZTR08], Bayes [CZ09] . cascade -

. , , . , . Ri i - -

.

47

-. Craswell et al. [CZTR08], Ri . , . , , . , (daily pageviews) , .

, . Ri , i, i, .

3.2.2

, - . Cho et al. [CRA05] - V (p,t) p - t. , :

3.1. (P (p,t)) ( -) d, (pv) t.

, , . , - -. . , 3.1,

,

48

3.1: 4

WebSite Daily Page views Popularity Gradehttp://google.com 584.640.000 4http://wikipedia.com 30.451.680 3http://ceid.upatras.gr 11.228 1http://sample-site.wordpress.com 11 0

- . , pv u, :

pu =

ln pvu

5

(3.1)

(3.1) - . , pv - ( ) 500.000.000 ( Google ) . - , . , 5 0 4, . (3.1) 0

4 : , , , , . 3.1 .

3.2.3

- , . , cascade - . . gi i pi

49

i, 3.2.2. , gi pi.

Ri = R(gi, pi), (3.2) f -

, . f :

f(r) =2r 12rmax

, (3.3)

r =g + p

2, r {0, ..., rmax}. (3.4)

(3.4) - . - . - . , , . , -

. - , ( ) - . , . , (g = 1), (p = 4), , . ,

. , , , 1 0. , (1) = 1 (R) 0 r +. (r) = 1/r :

RRP =nr=1

1

r

r1i=1

(1Ri)Rr. (3.5)

50

3.3

- . , - . -, [CMZG09],[CSdR13] . , [RKJ08] . , , click-through .

3.3.1

, click-through . - , crowdsourcing . - Indri? , TREC Web Tracks 2008-2012. , .

TREC Web Tracks 2008-2012 ClueWeb09, . - 1.000 , 25 (5 ) . 2009. , ClueWeb09 ,

?http://www.lemurproject.org/. The lemur project. University of Massachusetts andCarnegie Mellon University.

51

http://www.lemurproject.org/

50.000.000 . , (topics ) MSN Microsoft Research.

3.1: , TREC Web Tracks.

200 TREC Web Tracks , .html . , .aspx C # , , 20 ( Indri ) -. , , URL ( 3.3). .

Indri Indri [SMTC04]

52

- . Indri - :

, -

-.

, - . , , , , Indri - . . Indri ,

. , - . , ,, , . , In-

dri . ., , Indri

, , . Indri . Indri , click-through

. - TRECWeb Tracks (0 4). ,

53

, 3.2. , (0 4) .

- . TREC Web Tracks , - -. , 167 .

3.4

- - . , - ( ) . - :

Normalized Discounted Cumulative Gain

Average Precision

Expected Reciprocal Rank

Reciprocal Rank using Document Popularity

:

Precision at Lowest Rank

Max, Min and Mean Reciprocal Ranks of the clicks

UCTR

3.2 ( - ) . Pearson , .

54

3.2: -.

PLC MeanRR MinRR MaxRR UCTRnDCG 0.498 0.497 0.503 0.445 -0.024AP 0.402 0.417 0.395 0.396 -0.004ERR 0.528 0.512 0.517 0.459 0.064RRP 0.559 0.554 0.588 0.472 0.041

3.2, - . , RRP - nDCG cascade ERR. -, ERR, RRP , Mean, Max, Min Reciprocal Rank PLC 3.2. , ERR RRP reciprocalrank 1/r. ,

UCTR . UCTR , ., -

, . , r - gi pi ( 3.4). , - . , - , . :

r = 0.7 gi + 0.3 pi, (3.6)

. : , , . 3.3 -

55

3.3: - .

PLC MeanRR MinRR MaxRR UCTRnDCG 0.498 0.497 0.503 0.445 -0.024AP 0.402 0.417 0.395 0.396 -0.004ERR 0.528 0.512 0.517 0.459 0.064RRP 0.578 0.562 0.588 0.490 0.083

(3.7) . , RRP , 3.2. , - .

3.4.1

- . , - . , . Buckley - Voorhees[BV04] ( 2), , Sakai , [Sak07]. , -

, - . , -, . ., 167 ,

56

, 41 167 - 83 167 . . 3.2, 3.3 3.4 . 3.3, , - . , , 3.4 ( ), - . , - .

3.2: .

57

3.3: 41 .

3.4: 83 ..

58

4

. , -, - . , - , . ,

, - . . , - , . , - ., -

, , - . , , . - , .

, - , ,

59

[ABDR06] .. 3. click-through -, . (CTR ) - , .

. - - , . , , -, . , - url , , . . . ,

, - , -, . - . , -

, , . - . , , cascade . , - , . - 4.1 , 4.2 cascade 4.3, 4.4 4.5 - . User Browsing Model , Dependent Click Model Dynamic Bayesian Network Click Model.

60

4.1

, , - [DP08], [RDR07b]. , , , . , - d p [CZTR08]:

P (C = 1|d, p) = adbp,

ad d bp - p, . , . , ad , , - . , b1 = 1, , ad (CTR) 1. , , COEC , .

4.1.1 COEC

, bp CTR N p [ZJ07] . ci p i, COEC ( ) D :

ad =

Nn=1

ci

Nn=1

bpi

(4.1)

- . ,

61

, COEC , .

4.1.2

- ad. , bp, ad :

ad = argmaxn

Ni=1

ci log(abpi) + (1 ci) log(1 abpi). (4.2)

bp - ad bp. , , , - , . ad > 1. , ad , 1 Expectation Maximization .

4.1.3

. , . :

P (C = 1|d, p) = 11 + exp(ad bp)

. (4.3)

4.2 cascade

, - , -. , cascade [CZTR08] - . , cascade p . , .

62

Ri, , . , , r:

r1i=1

(1Ri)Rr (4.4)

i -. cascade 1[CMZG09]. , cascade .

Algorithm 2 Cascade model algorithm

Require: R1, R2, ..., Ri.i = 1User examines position i.if random(0,1) Ri then User is satisfied with document in position iand stops.else i = i+ 1; go to 2.end if

4.3 UBM

, - [DP08]. , , . cascade

, . cascade , User Browsing Model (UBM)

63

. , , . cascade , . , , . -

a e . [CZTR08] - , , . , .

ds , p. , - . d (a) q P (a|d, q), - Bernoulli :

P (a|d, q) = aad,q(1 ad,g)1a (4.5)

ad,q - d q. , :

P (e|p, ds) = ep,ds(1 p,ds)1e (4.6)

p,ds ds p. P (c, a, e, d, q, dr, p). - :

P (c, a, e|d, q, ds, p) = P (c|a, e)P (e|ds, p)P (a|d, q)= P (c|a, e)ep,ds(1 p,ds)1eaad,q(1 ad,g)1a

(4.7)

P (c|a, e) - .

64

, . , (CTR) . , (c, d, q, ds) . , c = 1 , , (a = 1) (e = 1). , 4.7 :

P (c = 1|d, q, ds, p) = ad,qp,dsP (c = 0|d, q, ds, p) = 1 ad,qp,ds

a . cascade .,

. , - . UBM . - . m,

:

P (e|p, dis,m) = ep,ds,m(1 p,ds,m)1e (4.8)

Expectation Maximization a, m.

4.4 DCM

- , cascade, , . , -, ,

65

. - , . , Guo et al. [GLK+09]

. , - , . , :

1, - .

.

:edi,i = 1, cdi,i = rdi . (4.9)

. , IndependentClick Model (ICM). , . , C1, C2, ..., CM d1, d2, ..., dM , :

LICM =Mi=1

(Ci log rdi + (1 Ci) log (1 rdi)). (4.10)

Ci Bernoulli . , - , rd - 4.10. rd

rd = number of clicks on d/measured relevance of d.

, -

66

. , - , i. 4.9

, . :

cdi,i = edi,irdi (4.11)

edi+1,i+1 = icdi,i + (edi,i cdi,i) (4.12)

[GLW09] , Dependent Click Model. 4.1. , - . :

LICM l

i=1

(Ci log rdi + (1 Ci) log (1 rdi)) +l1i=1

(Ci log i + log (1 i)),

(4.13)

4.1: DCM .

-.

67

ICM . :

rd =number of clicks on d

measured relevance of d at position l.

i = number of query sessions when last click occurs at i

number of query sessions when position i is clicked.

, ICM DCM - ( 4.2). (8% - ), DCM ICM . .

4.2: - .

4.5 DBN

([CZ09], [GLK+09]) ,

68

. , -, , . , 3

1 2 , 3 . , 1 2 , [08] 3 . Chapelle et al. [CZ09] -

, , . Ci i ( ). , Ei, i, Ai, - i Si i. , -

, . Ai i, . , Si i, , , . , , -

, i . , cascade , . , . , 1 . , cascade . , i, .

69

4.3: DBN .

, 4.3.

Ai = 1, Ei = 1 C = 1

P (Ai = 1) = ad

P (Si = 1|Ci = 1) = sdCi = 0 Si = 0

Si = 1 Ei+1 = 0

P (Ei+1 = 1|Ei = 1, Si = 0) =

Ei = 0 Ei+1 = 0

4.14, DBN ad sd, - . , . ad sd ExpectationMaximization , .

70

, . . -, CTR 1, - , , , ad. 4.4 KL- .

4.4: CTR 1 () KL- ().

, - . , sd , - NDCG. . 4.5 NDCG - . DBN , .

71

4.5: NDCG DBN .

72

5

? SPIRE : Xenophon Evangelopoulos andChristos Makris, Modeling Clicks using Document Popularity. SPIRE, Lon-don 2015.

5.1

, - , - . - , , , , [RKJ08]. ,

-, , -. , .

73

, , . , ,

. , . , - , , , , [KT03]. , -

- . [GLW09], [GLK+09] . Dupret et al. [DP08] UBM , -, , ds -. -

DBN [CZ09], - . - -. cascade , DBN - . , , . , -

. 5.2 5.3 - .

74

5.2

, - , . , . , -

. . 5.1. [Mur02].

10 . 5.1 2 , 8 . :

Ei: i

Ci: i

Pi: i

Ri: i

-. , Ci, . (rd) , -. , -

, . cascade , . , - . , . ,

75

5.1: . - - .

. [EMP14] , () [0, 4], , pi . (pi) -: 0, 1, 2, 3 4, 0.5, 0.6, 0.7, 0.8 0.9. ,

, -. , , , 1 , . - , . : - . pi . -

76

:

=Ni=1

piN, (5.1)

. :

P (Ci = 1|Ei = 0) = 0 (5.2)

P (Ei+1 = 1|Ei = 0) = 0 (5.3)

P (Ri = 1|Ci = 1) = rd (5.4)

P (Ei+1 = 1|Ei = 1, Ri = 0, Pi = 1) = (5.5)

P (Ei+1 = 1|Ei = 1, Ri = 1, Pi = 0) = 1 (5.6)

ru, - Expectation Maximization .

5.3

- state-of-the-art. , , DBN -.

5.3.1

. , 10 . , crowdsourcing TREC Web Tracks , [EMP14].

-. , ,

77

-. , - , -. MATLAB R2009b, -

BNT ?.

5.3.2

- (Log-Likelihood). LL , . - LL . 5.2 DBN

LL. . , DBN . 110 -3.21, DBN -3.29. , 50 , LL -3.11, DBN -3.13. , state-of-the-art.

?https://code.google .com/p/BNT/ Bayes Toolbox for Matlab. By Kevin Murphy,1997 - -2002

78

https://code.google.com/p/bnt/

5.2: Log-Likelihood . DBN .

79

6

- . , - . , , - , - , ( ) . ,

- , . . - , , , . . , . , -

, , . ,

80

. , , , . , , .

, - . , - , , . , , . ,

:

.

- , -. , - . , , - . , , . , , - . , , - , .

81

. Bayes . . , .

6.1

, . , , . . , , . -

, - : , , . - . , -. , , . -

, . , - , micro-blogging. -, , .

82

, . - , , click-through , - . , - , , , , .. ,

. - , AOL 2008. - , , . , - - .

6.2

. , . , , - . , , , . , , crowdsourcing -. , , . 5,

. ,

83

, .,

, . .

84

Bayes

(DBN) [DK89], [Mur02] - Bayes , - - . Z1, Z2, ..., , Zt = (Ut, Xt, Yt) , - . , t . , , . DBN (B1, B), B1

Bayesian P (Z1) B Bayes (2TBN) P (Zt|Zt1) :

P (Zt|Zt1) =Ni=1

P (Zit |Pa(Zit))

Zit i- t, Pa(Zit)

Zit . 2TBN , 2TBN (CPD) , P (Zit |Pa(Zit)) t > 1. , Pa(Zit),

. , .

85

Zit1 Zit ,

. , DAG., -

. , CPD -

, . , . , - , . DBN 2TBN

T . :

P (Z1:T ) =Tt=1

Ni=1

P (Zti|Pa(Zti))

Hidden Markov Models (HMMs) Baysian . DBN HMM DBN -, Xt1, . . . , QtNh , , . , HMM , Xt.

.1

HMM DBN , .1. -

. , - , -. .1 : Xt+1 Xt1|Xt ( Markov ) Yt Yt|Xt, t 6= t. Bayes , -

(CPD) . .1, P (X1),P (Xt|Xt1) P (Yt|Xt). (CPD) P (X1) - , ,, P (X1 = i) = (i), 0 (i) 1

i (i) = 1.

(CPD) P (Xt|Xt1) ,

86

.1: HMM DBN , 3 .

, P (Xt = j|Xt1 = i) = A(i, j) 1. (CPD) P (Yt|Xt) . Yt - , - , :P (Yt = j|Xt = i) = B(i, j). Yt , Gaussian Gaussians. , -

P (X1), P (X2|X1) P (Y1|X1). CPDs . . .2. HMM DBN . , ,

P (Yt|Xt = i) Gaus-sians i. Gaussians CPD , .3. CPDs Y M :

P (Yt|Xt = i,Mt = m) = N (yt;i,m,i,m)P (Mt = m|Xt = i) = C(i,m)

, CPD -

87

.2: HMM ( ) . : P (X1 =i) = (i), P (Xt = j|Xt1 = i) = A(i, j), P (Yt = j|Xt = i) = B(i, j).

, ,

P (Yt|Xt = i,Mt = m) = N (Ayt;i,m,i,m)

( t) [KA96]. , . - , ( ) ( [KA96] ). HMM Yt Yt|Xt, ,

, .4. - HMM (ARHMM). , [Rab89]. - HMM DBN , - .

88

.3: HMM Gaussians .

.4: Auto-regressive HMM.

ARHMM Xt, Yt1 Yt. . Y , CPD Y .

Y , CPD Y

P (Yt = yt|Xt = i, Yt1 = yt1) = N (yt;Wiyt1 + i,i)

Wi , Xt i. HMM -, Gaussian HMM , Markov [Ham90].

89

.2

- . , - . . -

. HMM , , , CPD P (Xt|Xt1). , - [Bra99] .

. -, . - , . , Expec-tation Maximization .

Online offline . offline , - ( online ). online - . , online - offline .

. (MLE) CPD . , . - D = {D1, . . . , DM} , :

L = logMm=1

Pr(Dm|G) =ni=1

Mm=1

logP (Xi|Pa(Xi), Dm)

90

Pa(Xi) Xi. . ( - : .)

CPD Dm(Xi, Pa(Xi)). ( CPD , :

mDm(Xi, Pa(Xi)).)

- , . -

, - . Expectation Maximization (EM) . Jensen

[CT91] , . Jensen , f ,

f

(j

jyj

)j

jf(yj)

j j = 1. , f f , f . , Jensen :

L =m

logh

P(H = h, Vm)

=m

logh

q(h|Vm)P(H = h, Vm)

q(h|Vm)

m

h

q(h|Vm) logP(H = h, Vm)

q(h|Vm)

=m

h

q(h|Vm) logP(H = h, Vm)m

h

q(h|Vm) log q(h|Vm)

91

q

h q(h|Vm) = 1 0 q(h|Vm) 1, . q

q(h|Vm) = P(h|Vm) E (expectation) , . , [NH98]. -

.

lc()q =m

h

q(h|Vm) logP(H = h, Vm)

M (maximization) . , , q . q(h|Vm) = P(h|Vm), EM ,

:

Q(|) =m

h

P (h|Vm, ) logP (h, Vm|)

Dempster et al. [DLR77]

Q(|) > Q(|) P (D|) > P (D|), - , - . q(h|Vm) = P(h|Vm) , - . , . CPDs ,

:

Q(|) =ijk

E[Nijk] log ijk

ENijk =

m P (Xi = k, Pa(Xi) = j|Dm, ), , :=arg max Q(

|),

ijk =ENijkk ENijk

92

[ABDR06] Eugene Agichtein, Eric Brill, Susan Dumais, and Robert Ragno.Learning user interaction models for predicting web search re-sult preferences. In Proceedings of the 29th Annual InternationalACM SIGIR Conference on Research and Development in Infor-mation Retrieval, SIGIR 06, pages 310, New York, NY, USA,2006. ACM.

[AMWZ09] Timothy G. Armstrong, Alistair Moffat, William Webber, andJustin Zobel. Improvements that dont add up: Ad-hoc retrievalresults since 1998. In Proceedings of the 18th ACM Conferenceon Information and Knowledge Management, CIKM 09, pages601610, New York, NY, USA, 2009. ACM.

[Bra99] M. Brand. Structure learning in conditional probability modelsvia an entropic prior and parameter extinction. Neural Compu-tation, 11:11551182, 1999.

[BV04] Chris Buckley and Ellen M. Voorhees. Retrieval evaluation withincomplete information. In Proceedings of the 27th Annual Inter-national ACM SIGIR Conference on Research and Developmentin Information Retrieval, SIGIR 04, pages 2532, New York,NY, USA, 2004. ACM.

[BYRN99] Ricardo A. Baeza-Yates and Berthier Ribeiro-Neto. Modern In-formation Retrieval. Addison-Wesley Longman Publishing Co.,Inc., Boston, MA, USA, 1999.

[CA05] B. Carterette and J. Allan. : Incremental test collections. In:Proceedings of the 14th ACM International Conference on In-formation and Knowledge Management, CIKM 2005, 2005.

[CJRY12] Olivier Chapelle, Thorsten Joachims, Filip Radlinski, andYisong Yue. Large-scale validation and analysis of interleaved

93

search evaluation. ACM Trans. Inf. Syst., 30(1):6:16:41, March2012.

[CMK66] Cyril Cleverdon, Jack Mills, and Michael Keen. Aslib cranfieldresearch project: factors determining the performance of index-ing systems, 1966.

[CMZG09] Olivier Chapelle, Donald Metlzer, Ya Zhang, and PierreGrinspan. Expected reciprocal rank for graded relevance. InProceedings of the 18th ACM Conference on Information andKnowledge Management, CIKM 09, pages 621630, New York,NY, USA, 2009. ACM.

[CRA05] J. Cho, S. Roy, and R. E. Adams. : Page quality: In search ofa unbiased web ranking. In: Proceedings of the 2005 ACM SIG-MOD International Conference on Management of Data, SIG-MOD 2005, 2005.

[CSdR13] A. Chuklin, P. Serdyukov, and M. de Rijke. : Click model-basedinformation retrieval metric. In: Proceedings of the 36th Inter-national ACM SIGIR Conference on Research and Developmentin Information Retrieval, SIGIR 20013, 2013.

[CT91] T. M. Cover and J. A. Thomas. Elements of Information Theory.John Wiley, 1991.

[Cyr91] W. Cyril. Cleverdon. The significance of the cranfield test on in-dex languages. In: Proceedings of the 14th Annual Inter-nationalACM SIGIR Conference on Research and Development in Infor-mation Retrieval, SIGIR 1991, 1991.

[CZ09] Olivier Chapelle and Ya Zhang. A dynamic bayesian networkclick model for web search ranking. In Proceedings of the 18thInternational Conference on World Wide Web, WWW 09, pages110, New York, NY, USA, 2009. ACM.

[CZTR08] Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey.An experimental comparison of click position-bias models. InProceedings of the 2008 International Conference on Web Searchand Data Mining, WSDM 08, pages 8794, New York, NY, USA,2008. ACM.

[DK89] T. Dean and K. Kanazawa. A model for reasoning about persis-tence and causation. Artificial Intelligence, 93(12):127, 1989.

94

[DLR77] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum like-lihood from incomplete data via the EM algorithm. J. of theRoyal Statistical Society, Series B, 34:138, 1977.

[DP08] Georges E. Dupret and Benjamin Piwowarski. A user brows-ing model to predict search engine click data from past obser-vations. In Proceedings of the 31st Annual International ACMSIGIR Conference on Research and Development in InformationRetrieval, SIGIR 08, pages 331338, New York, NY, USA, 2008.ACM.

[EMP14] Xenophon Evangelopoulos, Christos Makris, and Yannis Plegas.Reciprocal rank using web page popularity. In Lazaros Iliadis,Ilias Maglogiannis, Harris Papadopoulos, Spyros Sioutas, andChristos Makris, editors, Artificial Intelligence Applications andInnovations, volume 437 of IFIP Advances in Information andCommunication Technology, pages 116125. Springer Berlin Hei-delberg, 2014.

[FKM+05] Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais,and Thomas White. Evaluating implicit measures to improveweb search. ACM Trans. Inf. Syst., 23(2):147168, April 2005.

[GJG04] Laura A. Granka, Thorsten Joachims, and Geri Gay. Eye-tracking analysis of user behavior in www search. In Proceedingsof the 27th Annual International ACM SIGIR Conference onResearch and Development in Information Retrieval, SIGIR 04,pages 478479, New York, NY, USA, 2004. ACM.

[GK02] Norbert Govert and Gabriella Kazai. Overview of the initiativefor the evaluation of xml retrieval (inex) 2002. In IN: PROC. OFTHE FIRST WORKSHOP OF THE INITIATIVE FOR THEEVALUATION OF XML RETRIEVAL (INEX), DAGSTUHL,2002, pages 117, 2002.

[GLK+09] Fan Guo, Chao Liu, Anitha Kannan, Tom Minka, Michael Tay-lor, Yi-Min Wang, and Christos Faloutsos. Click chain model inweb search. In Proceedings of the 18th International Conferenceon World Wide Web, WWW 09, pages 1120, New York, NY,USA, 2009. ACM.

[GLW09] Fan Guo, Chao Liu, and Yi Min Wang. Efficient multiple-clickmodels in web search. In Proceedings of the Second ACM Inter-

95

national Conference on Web Search and Data Mining, WSDM09, pages 124131, New York, NY, USA, 2009. ACM.

[Ham90] J. Hamilton. Analysis of time series subject to changes in regime.J. Econometrics, 45:3970, 1990.

[Har93] Donna Harman. Overview of the first trec conference. In Proceed-ings of the 16th Annual International ACM SIGIR Conferenceon Research and Development in Information Retrieval, SIGIR93, pages 3647, New York, NY, USA, 1993. ACM.

[HH07] S. B. Huffman and M. Hochster. : How well does result rele-vance predict session satisfaction? In: Proceedings of the 30thAnnual International ACM SIGIR Conference on Research andDevelopment in Information Retrieval SIGIR 2007, 2007.

[JFM97] Thorsten Joachims, Dayne Freitag, and Tom Mitchell. Web-watcher: A tour guide for the world wide web. In INPROCEEDINGS OF THE FIFTEENTH INTERNATIONALJOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE,pages 770775. Morgan Kaufmann, 1997.

[JK02] Kalervo Jarvelin and Jaana Kekalainen. Cumulated gain-basedevaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422446, October 2002.

[Joa03] Thorsten Joachims. Evaluating retrieval performance usingclickthrough data, 2003.

[KA96] N. Kumar and A. G. Andreou. A generalization of linear dis-criminant analysis in maximum likelihood framework. In Proc.of the Joint Statistical Meeting, Statistical Computing section,1996.

[Kek05] Jaana Kekalainen. Binary and graded relevance in ir evaluations-comparison of the effects on ranking of ir systems. Inf. Process.Manage., 41(5):10191033, September 2005.

[KKK+99] Noriko Kando Kazuko, Noriko K, Kazuko Kuriyama, ToshihikoNozue, Koji Eguchi, Hiroyuki Kato, Soichiro Hidaka, and JunAdachi. The ntcir workshop : the first evaluation workshop onjapanese text retrieval and cross-lingual information retrieval,1999.

96

[KT03] Diane Kelly and Jaime Teevan. Implicit feedback for inferringuser preference: A bibliography. SIGIR Forum, 37(2):1828,September 2003.

[LF07] Yiqun Liu and Yupeng Fu. Automatic search engine performanceevaluation with click-through data analysis. In In Proceedings ofthe 16th international conference on World Wide Web (WWW,2007.

[MRS08] Christopher D. Manning, Prabhakar Raghavan, and HinrichSchutze. Introduction to Information Retrieval. Cambridge Uni-versity Press, New York, NY, USA, 2008.

[Mur02] Kevin Patrick Murphy. Dynamic bayesian networks: Represen-tation, inference and learning, 2002.

[MZ08] Alistair Moffat and Justin Zobel. Rank-biased precision formeasurement of retrieval effectiveness. ACM Trans. Inf. Syst.,27(1):2:12:27, December 2008.

[NH98] R. M. Neal and G. E. Hinton. A new view of the EM algorithmthat justifies incremental and other variants. In M. Jordan, edi-tor, Learning in Graphical Models. MIT Press, 1998.

[Rab89] L. R. Rabiner. A tutorial on Hidden Markov Models and selectedapplications in speech recognition. Proc. of the IEEE, 77(2):257286, 1989.

[RDR07a] M. Richardson, E. Dominowska, and R. Ragno. : Predictingclicks: Estimating the click-through rate for new ads. In: Pro-ceedings of the 16th International Conference on World WideWeb, WWW 2007, 2007.

[RDR07b] Matthew Richardson, Ewa Dominowska, and Robert Ragno.Predicting clicks: Estimating the click-through rate for new ads.In Proceedings of the 16th International Conference on WorldWide Web, WWW 07, pages 521530, New York, NY, USA,2007. ACM.

[Rij79] C. J. Van Rijsbergen. Information Retrieval. Butterworth-Heinemann, Newton, MA, USA, 2nd edition, 1979.

[RKJ08] Filip Radlinski, Madhu Kurup, and Thorsten Joachims. Howdoes clickthrough data reflect retrieval quality? In Proceedings of

97

the 17th ACM Conference on Information and Knowledge Man-agement, CIKM 08, pages 4352, New York, NY, USA, 2008.ACM.

[Sak07] T. Sakai. : Alternatives to bpref. In: Proceedings of the 30thAnnual International ACM SIGIR Conference on Research andDevelopment in Information Retrieval, SIGIR 2007, 2007.

[Sal68] Gerard. Salton. Automatic Information Organization and Re-trieval. McGraw Hill Text, 1968.

[SJ04] M. Sanderson and H. Joho. : Forming test collections with nosystem pooling. In: Proceedings of the 27th annual internationalACM Conference on Research and Development in InformationRetrieval, SIGIR 2004, 2004.

[SMTC04] T. Strohman, D. Metzler, H. Turtle, and W. B. Croft. Indri: Alanguage model-based search engine for complex queries. Pro-ceedings of the International Conference on Intelligence Analy-sis, 2004.

[SZ05] M. Sanderson and J. Zobel. : Information retrieval system eval-uation: Effort, sensitivity, and reliability. In: Proceedings of the28th Annual International ACM SIGIR Conference on Re-searchand Development in Information Retrieval, SIGIR 2005, 2005.

[Tau54] M. et al. Taube. The uniterm coordinate indexing of reports. InThe Technical Report. New York, Reinhold, 1954.

[VH05] Ellen M. Voorhees and Donna K. Harman. TREC: Experimentand Evaluation in Information Retrieval (Digital Libraries andElectronic Publishing). The MIT Press, 2005.

[Voo02] Ellen M. Voorhees. The philosophy of information retrievalevaluation. In Revised Papers from the Second Workshop ofthe Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, CLEF 01, pages 355370, London, UK, UK, 2002. Springer-Verlag.

[YAR08] Emine Yilmaz, Javed A. Aslam, and Stephen Robertson. A newrank correlation coefficient for information retrieval. In Proceed-ings of the 31st Annual International ACM SIGIR Conferenceon Research and Development in Information Retrieval, SIGIR08, pages 587594, New York, NY, USA, 2008. ACM.

98

[YSCR10] E. Yilmaz, M. Shokouhi, N. Craswell, and S. Robertson. : Ex-pected browsing utility for web search evaluation. In: Proceedingsof the 19th ACM International Conference of Information andknowledge management, CIKM 2010, 2010.

[ZJ07] V. Zhang and R. Jones. Comparing click logs and editorial labelsfor training query rewriting. in query log analysis: Social andtechnological challenges. In Workshop of the 2007 InternationalWorld Wide Web Conference, 2007.

99

- TREC

(DCG)Rank Biased PrecisionBprefExpected Reciprocal Rank

Crowdsourcing A/B

o Reciprocal Rank using Document Popularity

COEC

cascade UBM DCM DBN

Bayes