Question: Problem 5 . EvaluationBelow is a table showing how two human judges rated the relevance of a set of 1 2 documents to a particular

Problem 5. EvaluationBelow is a table showing how two human judges rated the relevance of a set of 12 documents to a particular information need (0= nonrelevant, 1= relevant). Let us assume that youve written an IR system that for this query returns the set of documents {3,4,5,6,7,8}.docIDJudge1Judge21002003114115106017108109011011110112011) Calculate the kappa measure between the two judges;2) Calculate precision, recall, and F1 of your system if a document is considered relevantonly if the two judges agree;3) Calculate precision, recall, and F1 of your system if a document is considered relevantif either judge thinks it is relevant.(Hint: create a confusion matrix)

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Programming Questions!