Question: Below is a table showing how two human judges rated the relevance of a set of 12 documents to a particular information need (0 =

Below is a table showing how two human judges rated the relevance of a set of 12 documents to a particular information need (0 = nonrelevant, 1-relevant). Let us assume that you've written an IR system that for this query returns the set of documents 4, 5, 6,7,8) Document ID 2 4 0 10 0 12 Calculate the kappa measure between the two judges. (5 points) Calculate precision, recall, and Fl of your system if a document is considered relevant only if the two judges agree. (5 points) Calculate precision, recall, and Fl of your system if a document is considered relevant if either judge thinks it is relevant. (5 points) a. b. c
Step by Step Solution
There are 3 Steps involved in it
Get step-by-step solutions from verified subject matter experts
