$1.[$Points $15]$ Given candidate translation sentences and their references below. Please

compute brevity penalty, modified BLEU$-2$ score for the following examples. Here

n$=2.$ Note: No partial grading for wrong calculation$/$answer$.$

Candidate Sentences:

Candidate $1$: "The boy is playing with a red ball."

Candidate $2$: "A child plays with a ball outside."

Reference Sentences:

Reference $1$: "The boy is outside playing with a ball."

Reference $2$: "A young child is playing with a red ball."

Reference $3$: "The boy plays with a ball in the park."

$2.[$Points $10]$ Describe issue of BLEU Score? Write down the difference between BLEU

and ROGUE$-$S score. Can you use both scores to evaluate both text summarization and

machine translation? Provide your rationale in supporting your answer.

$3.[$Points $20]$ We are given the system summary and human gold standard summary for the

definition summary of spinach below. Answer the following questions.

System Summary:

"Artificial intelligence is revolutionizing industries by improving efficiency, automating

tasks, and enabling data$-$driven decision$-$making."

Reference Summary Sentences:

R$1$: $"$AI is transforming sectors by streamlining operations, automating processes, and

providing actionable insights from data."

R$2$: "The integration of artificial intelligence in industries enhances productivity,

reduces manual effort, and leverages big data analytics."

R$3$: $"$AI technologies are driving innovation by optimizing workflows, reducing human

intervention, and supporting smarter decisions."

a$.[$Points $20]$ Compute skip$-$bigram ROGUE$-2$ score for the above summarized

examples. Please show details of your generated skip$-$bigrams in your answer.

$4.[$Points $25]$ Given the query and candidate answer, rank the answer using the following

ranking mechanism. $($ignore case sensitivity$).$ Answer the below question.

Given relevance score Equation below.

R $($Q$,$ Ai$)=$ W$1*$ BM$25($Q$,$ Ai$)+$ W$2*$ COS$($Q$,$ Ai$)+$ W$3*$ AnsLen$($Ai$)$

Where R$(,)$ is the relevance score, BM$25(,)$ score, COS$(,)$ is the cosine similarity score,

and AnsLen$()$ is the answer length. Hyperparameters W$1,$ W$2,$ and W$3$ are given $0\mathrm{.}3,0\mathrm{.}4,$

$0\mathrm{.}3,$ and constraints W$1+$ W$2+$ W$3=1.$ For BM$25,$ k$=1\mathrm{.}2,$ b$=0\mathrm{.}85$ are given.

Query, Q: Query:

"What are the effects of deforestation?"

Candidate Answers:

A$1$: "Deforestation leads to loss of biodiversity."

A$2$: "Cutting down trees causes soil erosion and disrupts ecosystems."

A$3$: "Forest conservation helps mitigate climate change."

$5.[$Points $10]$ Which of the following is an example of a context$-$based question$-$answering

system? Provide explanation for each of the options and show correct answer$($s$).$ Note:

No partial grading for wrong computation and$/$or answer.

$($a$)$ Chatbot that answers general queries.

$($b$)$ Search engine that retrieves web pages.

$($c$)$ System that reads a paragraph and answers questions based on it$.$

$($d$)$ Text summarization tool.

$6.[$Points $10]$ Which of the following is an example of an extractive question$-$answering

system? Provide explanation for each of the options and show correct answer$($s$).$ Note: No

partial grading for wrong computation and$/$or answer.

$($a$)$ Summarizing a paragraph into key points.

$($b$)$ Generating a detailed answer based on a provided context.

$($c$)$ Extracting a specific phrase or sentence from the text as the answer.

$($d$)$ Translating a question into another language before answering.

$7.[$Points $10]$ Which of the following metrics is commonly used to evaluate extractive

question$-$answering systems? Provide explanation for each of the options and show

correct answer$($s$).$ Note: No partial grading for wrong computation and$/$or answer.

$($a$)$ BLEU Score

$($b$)$ Exact Match $($EM$)$

$($c$)$ Perplexity

$($d$)$ F$1$ Score Query, Q: Query:

"What are the effects of deforestation?"

Candidate Answers:

A$1$: "Deforestation leads to loss of biodiversity."

A$2$: "Cutting down trees causes soil erosion and disrupts ecosystems."

A$3$: "Forest conservation helps mitigate climate change."

Given Embeddings:

Compute the relevance score for each sentence and select the best$-$ranked answer. Provide detailed computations. Note: No partial grading for wrong computation and$/$or answer.