Question: Below is a table showing how two human judges rated the relevance of a set of 12 documents to a particular information need (0 =

 Below is a table showing how two human judges rated the

Below is a table showing how two human judges rated the relevance of a set of 12 documents to a particular information need (0 = nonrelevant, 1-relevant). Let us assume that you've written an IR system that for this query returns the set of documents 4, 5, 6,7,8) Document ID 2 4 0 10 0 12 Calculate the kappa measure between the two judges. (5 points) Calculate precision, recall, and Fl of your system if a document is considered relevant only if the two judges agree. (5 points) Calculate precision, recall, and Fl of your system if a document is considered relevant if either judge thinks it is relevant. (5 points) a. b. c

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Databases Questions!