3. Problem Solving, NDCG, P@K, MAP. (3 Points!) Consider an information need for which there are 5 relevant documents in the collection, which are judged as very relevant, relevant, somewhat relevant (VR, R, SR) vs. nonrelevant (N). For computation of NDCG, treat these ordinal values as the numbers (relevance values) 3, 2, 1, and 0, respectively. Contrast two systems running a query for the information need on this collection. Their top 10 results were judged for relevance as follows (the leftmost item is the top ranked search result): System 1: VR, R, R, N, SR, VR, N, N, N, N System 2: VR, VR, R, R, N, N, N, SR, N, N a). (1 point) What is the NDCG of each system? Which has the higher NDCG? Show calculations. Note: For part b and c, we collapse distinctions to a 2-way relevance assessment (R vs. N) b). (1 Point, 1/3 point per item.). (b1) Is one list better than the other based on Precision @3? (b2) Is one list better than the other based on Precision @5? (b3) What is the MAP of each system? Which has a higher MAP? c). (1 Point) Intuitively, which system seems better for web search? Why? 3. Problem Solving, NDCG, P@K, MAP. (3 Points!) Consider an information need for which there are 5 relevant documents in the collection, which are judged as very relevant, relevant, somewhat relevant (VR, R, SR) vs. nonrelevant (N). For computation of NDCG, treat these ordinal values as the numbers (relevance values) 3, 2, 1, and 0, respectively. Contrast two systems running a query for the information need on this collection. Their top 10 results were judged for relevance as follows (the leftmost item is the top ranked search result): System 1: VR, R, R, N, SR, VR, N, N, N, N System 2: VR, VR, R, R, N, N, N, SR, N, N a). (1 point) What is the NDCG of each system? Which has the higher NDCG? Show calculations. Note: For part b and c, we collapse distinctions to a 2-way relevance assessment (R vs. N) b). (1 Point, 1/3 point per item.). (b1) Is one list better than the other based on Precision @3? (b2) Is one list better than the other based on Precision @5? (b3) What is the MAP of each system? Which has a higher MAP? c). (1 Point) Intuitively, which system seems better for web search? Why