Evidence from Behavior

Evidence from BehaviorLBSC 796/INFM 719RDouglas W. OardSession 7, October 22, 2007

Picture of Relevance FeedbackxxxxoooRevised queryx non-relevant documentso relevant documentsoooxxxxxxxxxxxxxxInitial queryx

Rocchio Formulaqm = modified query vector; q0 = original query vector;,,: weights (hand-chosen or set empirically); Dr = set of known relevant doc vectors; Dnr = set of known irrelevant doc vectors

Rocchio Example040800124001201104-16370-30408002480028044016Original queryPositive FeedbackNegative feedback(+)(-)New queryTypically, <

Using Positive InformationSource: Jon Herlocker, SIGIR 1999

SmallWorld

SpaceMtn

Mad Tea Pty

Dumbo

Speed-way

CntryBear

Joe

D

A

B

D

?

?

Ellen

A

F

D

F

Mickey

A

A

A

A

A

A

Goofy

D

A

C

John

A

C

A

C

A

Ben

F

A

F

Nathan

D

A

A

Using Negative InformationSource: Jon Herlocker, SIGIR 1999

SmallWorld

SpaceMtn

Mad Tea Pty

Dumbo

Speed-way

CntryBear

Joe

D

A

B

D

?

?

Ellen

A

F

D

F

Mickey

A

A

A

A

A

A

Goofy

D

A

C

John

A

C

A

C

A

Ben

F

A

F

Nathan

D

A

A

Rating-Based RecommendationUse ratings as to describe objectsPersonal recommendations, peer review,

Beyond topicality:Accuracy, coherence, depth, novelty, style,

Has been applied to many modalitiesBooks, Usenet news, movies, music, jokes, beer,

Motivations to Provide RatingsSelf-interestUse the ratings to improve systems user model

Economic benefitIf a market for ratings is created

Altruism

Explicit Feedback: AssumptionsA1: User has sufficient knowledge for a reasonable initial query

A2: Selected examples are representative

A3: The user will try giving feedback

A4: The user will keep giving feedback

A1: Good Initial Query?Two problems:User may not have sufficient initial knowledgeFew or no relevant documents may be retrieved

Examples:Misspellings (Brittany Speers)Cross-language information retrievalVocabulary mismatch (e.g., cosmonaut/astronaut)

A2: Representative Examples?There may be several clusters of relevant documents

Examples:Burma/MyanmarContradictory government policiesOpinions

A3: Will People Use It?EfficiencyLonger queries require more processing time

UnderstandabilityHarder to see why subsequent documents retrieved

RiskUsers are reluctant to provide negative feedback

A4: Self-Interest Decreases!Number of RatingsValue of ratingsMarginal value to raterMarginal value to communityFewLotsMarginal cost

Solving the Cost vs. Value ProblemMaximize the valueProvide for continuous user model adaptation

Minimize the costsUse implicit feedback rather than explicit ratingsMinimize privacy concerns through encryptionBuild an efficient scalable architectureLimit the scope to noncompetitive activities

Solution: Reduce the Marginal CostNumber of RatingsMarginal value to raterMarginal value to communityFewLotsMarginal cost

View Listen Select Print Bookmark Save Purchase Delete Subscribe Copy / paste Quote Forward Reply Link Cite Mark up Tag Publish Organize Type Edit

Estimating Authority from LinksAuthorityAuthorityHub

Anchor Text IndexingSometimes yields unexpected results: Google ranked Microsoft #1 for evil empireAdapted from: Manning and Raghavan

Collecting Click StreamsBrowsing histories are easily capturedMake all links initially point to a central siteEncode the desired URL as a parameterBuild a time-annotated transition graph for each userCookies identify users (when they use the same machine)Redirect the browser to the desired page

Reading time is correlated with interestCan be used to build individual profilesUsed to target advertising by doubleclick.com

020406080100120140160180NoInterestLowInterestModerateInterestHighInterestRatingReading Time (seconds)Full Text Articles (Telecommunications)50325843

More Complete ObservationsUser selects an articleInterpretation: Summary was interestingUser quickly prints the articleInterpretation: They want to read itUser selects a second articleInterpretation: another interesting summaryUser scrolls around in the articleInterpretation: Parts with high dwell time and/or repeated revisits are interestingUser stops scrolling for an extended periodInterpretation: User was interrupted

NoInterestNoInterestLowInterestModerateInterestHighInterestAbstracts (Pharmaceuticals)42555251

Chart2

56

30

66

45

67

32

140

27

118

72

67

93

6

7

4

18

53

17

95

52

68

105

11

46

75

64

45

77

30

29

15

4

12

10

6

3

13

4

24

50

40

70

47

175

80

50

56

125

91

25

53

66

76

105

75

108

27

23

137

62

72

21

59

54

25

46

43

41

32

16

176

35

102

58

15

100

39

35

30

21

38

166

23

41

3

5

20

48

77

75

54

38

39

36

22

79

21

15

33

37

8

64

40

35

23

10

22

107

69

72

67

84

88

87

70

106

95

78

60

37

100

96

13

19

38

21

9

10

27

76

24

37

46

55

40

27

27

14

36

23

29

33

16

5

107

73

81

71

79

123

99

17

12

Rating

Reading Time

z_ab_to

zool_3101.0010:34:2256.00CNT1685KCH1623WIZ1624ITP16581.00F1.00ab.05464-.55905

zool_3101.0010:36:2030.00CNT2820KCH1623WIZ2760ITP2794.00F1.00ab-.53380-1.30297

zool_3101.0010:37:3766.00CNT2841KCH1623WIZ2760ITP27942.00P2.00ab.28096.18488




zool_385.0010:31:45140.00CNT1685KCH1623WIZ1624ITP16582.00F1.00ab1.95575.18488


zool_385.0010:40:42118.00CNT1708KCH1623WIZ1624ITP1668.00F1.00ab1.45784-1.30297

zool_385.0010:45:4572.00CNT3001KCH1623WIZ2760ITP28152.00F1.00ab.41675.18488

zool_388.0010:32:1367.00CNT1685KCH1623WIZ1624ITP1658.00F1.00ab.30359-1.30297


zool_389.0010:27:596.00CNT1919KCH1623WIZ1813ITP18572.00F1.00ab-1.07697.18488

zool_389.0010:32:367.00CNT3632KCH1623WIZ3548ITP35921.00F1.00ab-1.05434-.55905



zool_391.0010:38:0253.00CNT2171KCH1623WIZ1624ITP16793.00P2.00ab-.01326.92881



zool_395.0010:36:4052.00CNT4988KCH1623WIZ4905ITP49602.00F1.00ab-.03589.18488


zool_485.0012:34:56105.00CNT5195KCH1623WIZ4905ITP49491.00F1.00ab1.16362-.55905

zool_485.0012:37:3911.00CNT2276KCH1623WIZ1813ITP18681.00P2.00ab-.96381-.55905

zool_485.0012:39:0346.00CNT2255KCH1623WIZ1813ITP18681.00F1.00ab-.17168-.55905








zool_488.0012:42:214.00CNT4966KCH1623WIZ4905ITP4939.00F1.00ab-1.12224-1.30297




zool_489.0012:25:413.00CNT2841KCH1623WIZ2760ITP27942.00F1.00ab-1.14487.18488








































zool_6103.0014:46:42100.00CNT2276KCH1623WIZ1813ITP18682.00P2.00ab1.05046.18488

























zool_691.0014:45:588.00CNT2234KCH1623WIZ1813ITP18682.00P2.00ab-1.03171.18488











zool_7101.0015:0484.00CNT1685KCH1623WIZ1624ITP16581.00F1.00ab.68834-.55905

zool_7101.0032:0688.00CNT2213KCH1623WIZ1624ITP16792.00F1.00ab.77887.18488



zool_7103.0014:45106.00CNT1685KCH1623WIZ1624ITP16582.00F1.00ab1.18625.18488

zool_7103.0024:3895.00CNT2213KCH1623WIZ1624ITP1679.00F1.00ab.93730-1.30297



zool_7103.0039:5837.00CNT2820KCH1623WIZ2760ITP27941.00F1.00ab-.37537-.55905

zool_7103.0042:05100.00CNT2841KCH1623WIZ2760ITP27943.00P2.00ab1.05046.92881


zool_784.0019:1113.00CNT1708KCH1623WIZ1624ITP16682.00F1.00ab-.91855.18488




zool_784.0037:369.00CNT4966KCH1623WIZ4905ITP49393.00F1.00ab-1.00908.92881

















zool_789.0030:415.00CNT4988KCH1623WIZ4905ITP4960.00F1.00ab-1.09961-1.30297







zool_795.0041:2199.00CNT2841KCH1623WIZ2760ITP2794.00F1.00ab1.02783-1.30297

zool_797.0035:1517.00CNT4988KCH1623WIZ4905ITP49601.00P2.00ab-.82802-.55905

zool_797.0039:3512.00CNT2892KCH1623WIZ2760ITP28043.00P2.00ab-.94118.92881

Recommending w/Implicit FeedbackEstimateRatingUserModelRatings ServerUser RatingsCommunityRatingsPredictedRatingsUserObservationsUser RatingsUserModelEstimateRatingsObservations ServerPredictedObservationsCommunityObservationsPredictedRatings

UserObservations

Critical IssuesProtecting privacyWhat absolute assurances can we provide?How can we make remaining risks understood?

Scalable rating serversIs a fully distributed architecture practical?

Non-cooperative usersHow can the effect of spamming be limited?

Gaining Access to ObservationsObserve public behaviorHypertext linking, publication, citing,

Policy protectionEU: Privacy lawsUS: Privacy policies + FTC enforcement

Statistical assurance of privacyDistributed architectureModel and mitigate privacy risks

Search EngineQuery LogsA: Southeast Asia (Dec 27, 2004) B: Indonesia (Mar 29, 2005) C; Pakistan (Oct 10, 2005) D; Hawaii (Oct 16, 2006) E: Indonesia (Aug 8, 2007) F: Peru (Aug 16, 2007)

Search Engine Query Logshttp://hannu.biz/aolsearch/

AOL User 4417749

11627

Evidence from Behavior

Documents

Transcript of Evidence from Behavior