Ανάκτηση Πληροφορίας Αποτίμηση Αποτελεσματικότητας

Click here to load reader

  • date post

    18-Mar-2016
  • Category

    Documents

  • view

    38
  • download

    0

Embed Size (px)

description

Ανάκτηση Πληροφορίας Αποτίμηση Αποτελεσματικότητας. Μέτρα Απόδοσης. Precision = # σχετικών κειμένων που επιστρέφονται # κειμένων που επιστρέφονται Recall = # σχετικών κειμένων που επιστρέφονται # συνολικών σχετικών κειμένων Ο στόχος είναι να μεγιστοποιηθούν και τα δύο. - PowerPoint PPT Presentation

Transcript of Ανάκτηση Πληροφορίας Αποτίμηση Αποτελεσματικότητας

  • Precision = # #

    Recall = # #

    ., (Relevance)

  • Recall Precision

  • ?Test reference collections:TIPSTER/TRECCACMCISICystic Fibrosis

  • Recall = |Ra| / |R| Precision = |Ra| / |A|CollectionAnswer set (A)Relevantdocuments (R)relevant &retrieved (Ra)

  • -

    R

    Ar

    R

    Ar

    R

    Ar

    R

    Ar

  • R Relevance: ; Partial Matching background

  • Precision/ Recall Precision - Recall trade-off Precision Recallprecisionrecallxxxx

  • Precision/ Recall :precisionrecallxxxx

  • Ark1

    RankDocRel

    00%0%

    1a10%100%11

    210%50%12

    3a20%67%23

    420%50%24

    520%40%25

    6a30%50%36

    730%43%37

    830%38%38

    930%33%39

    10a40%40%410

    1140%36%411

    1240%33%412

    1340%31%413

    1440%29%414

    15a50%33%515

    Ark2

    Ark3

    Chart1

    100

    67

    50

    40

    33

    Precision

    Recall

    Precision

    Ark1

    RankDocRel

    00%0%

    1a10%100%111010050

    210%50%1220675040

    3a20%67%23

    420%50%24

    520%40%25

    6a30%50%36

    730%43%37

    830%38%38

    930%33%39

    10a40%40%410

    1140%36%411

    1240%33%412

    1340%31%413

    1440%29%414

    15a50%33%515

    10100

    105010100

    20672067

    20503050

    20404040

    30505033

    3043

    3038

    3033

    4040

    4036

    4033

    4031

    4029

    5033

    Ark1

    Precision

    Recall

    Precision

    Ark2

    Precision

    Recall

    Precision

    Ark3

  • :

    - r. Nq - Pi(r) r i- .

  • 11 recall:0%, 10%, 20%, , 100%

    j- recall. ,

  • 11

    Chart2

    1

    1

    0.67

    0.5

    0.4

    0.33

    0

    0

    0

    0

    0

    Recall

    Precision

    Ark1

    RankDocRel

    00%0%

    1a10%100%111010050

    210%50%1220675040

    3a20%67%23

    420%50%24

    520%40%25

    6a30%50%36

    730%43%37

    830%38%38

    930%33%39

    10a40%40%410

    1140%36%411

    1240%33%412

    1340%31%413

    1440%29%414

    15a50%33%515

    10100

    105010100

    20672067

    20503050

    20404040

    30505033

    3043

    3038

    3033

    4040

    4036

    4033

    4031

    4029

    5033

    Ark1

    Precision

    Recall

    Precision

    Ark1 (2)

    Precision

    Recall

    Precision

    Ark2

    LevelIterpolatedPrecision

    0%100%

    10%100%011010050

    20%67%0220675040

    30%50%13

    40%40%14

    50%33%15

    60%0%16

    70%0%17

    80%0%28

    90%0%29

    100%0%210

    10

    1010100

    202067

    203050

    204040

    305033

    30

    30

    30

    40

    40

    40

    40

    40

    50

    Ark2

    #REF!

    Recall

    Precision

    Ark3

    Recall

    Precision

    Ark1

    RankDocRel

    00%0%

    1a10%100%111010050

    210%50%1220675040

    3a20%67%23

    420%50%24

    520%40%25

    6a30%50%36

    730%43%37

    830%38%38

    930%33%39

    10a40%40%410

    1140%36%411

    1240%33%412

    1340%31%413

    1440%29%414

    15a50%33%515

    10100

    105010100

    20672067

    20503050

    20404040

    30505033

    3043

    3038

    3033

    4040

    4036

    4033

    4031

    4029

    5033

    Ark1

    Precision

    Recall

    Precision

    Ark1 (2)

    Precision

    Recall

    Precision

    Ark2

    LevelIterpolatedPrecision

    0%100%

    10%100%011010050

    20%67%0220675040

    30%50%13

    40%40%14

    50%33%15

    60%0%16

    70%0%17

    80%0%28

    90%0%29

    100%0%210

    10

    1010100

    202067

    203050

    204040

    305033

    30

    30

    30

    40

    40

    40

    40

    40

    50

    Ark2

    #REF!

    Recall

    Precision

    Ark3

    Recall

    Precision

  • : :top 5, top 10, top 20, top 50, top 100, top 500 precision high precision

  • Average Precision vs Recall . ;

    Single value measures:Average Precision at Seen Relevant Documents;R-Precision;Precision Histograms;Summary Table Statistics.

  • The harmonic mean combines Recall & Precision into a single number ranging from 0 to 1:

    P(j) - precision of j-th document in ranking;r(j) recall of j-th document in ranking;

    If F(j) = 0 no relevant docs have been retrieved;If F(j) = 1 all ranked docs are relevant;The harmonic mean assumes high value only when both recall & precision are high.

  • E-MeasureCombine Precision and Recall into one number and allow user to specify whether s/he is more interested in recall or precision:

    P(j) precision of j-th document in ranking;r(j) recall of j-th document in ranking;b - measure of relative importance of precision or recall:b > 1 emphasizes precision, b < 1 emphasizes recall

  • Different users might have a different interpretation of which document is relevant or notUser-oriented measures:Coverage ratio:

    Novelty ratio:

    Relative ratio the ratio between the # of relevant docs found and the # of relevant docs the user expected to findRecall effort the ratio between the # of relevant docs the user expected to find and the # of docs examined in an attempt to find the expected relevant docs.

  • Answer set (A)Relevantdocuments (R)Relevant Docsknown to user (U)Relevant Docsknown to user which were retrieved (Rk)Relevant Docspreviously unknown to user which were retrieved (Ru)

  • TREC collection (Text REtrieval Conference) 5.8 GB, >1.5 Million Docs.CACM - computer science 3024 articles.ISI (CISI) library science 1460 docs.Cystic Fibrosis (CF) - medicine 1239 docs.CRAN aeronautics 1400 docs.Time general articles 423 docs.NPL electrical engineering 11429 docs.

  • IR : Recall & Precision .

    Recall and Precision, as defined above, assume that all documents in the answer set A have been examined. However, the user is not usually presented with all the documents in the answer set A at once.Recall and Precision measures vary as the user proceeds with his examination of the answer ser A.