Question $2$: Hallucinations in LLMs: Identification and Mitigation

Comparative Study Across LLMs: Select any two publicly available LLMs $($e$.$g$.,$ GPT$-4,$ LLaMA, BLOOM, Claude, etc.$)$ and compare their hallucination patterns in three distinct domains: history, technology$/$science$,$ and medicine.

$1.$ Design $5$ complex, multi$-$part prompts for each domain $(15$ total prompts$).$ The prompts must challenge the model with facts, reasoning, and synthesis, probing areas where hallucinations are likely to occur.

$2.$ Identify at least three types of hallucinations:

a$.$ Factual Hallucinations: Incorrect information.

b$.$ Logical Hallucinations: Errors in reasoning.

c$.$ Contradictory Hallucinations: Instances where the model contradicts itself within the same or multiple responses.

$3.$ Quantify the frequency of each type of hallucination across both models and domains. Develop a hallucination taxonomy to categorize and understand the variations in hallucination behavior.

Novel Mitigation Strategy

Based on your findings, propose a novel method for hallucination mitigation. Your method must include:

$1.$ A novel prompt design or external augmentation strategy $($e$.$g$.,$ introducing external fact verification, reasoning chains, or context$-$specific fine$-$tuning$).$

$2.$ Justify the effectiveness of your proposed method and compare it with two existing mitigation approaches, and cite them.

$3.$ How would you quantitatively measure the generated output for the hallucination?

$[$Bonus$]$

$1.$ Hallucination Detection Framework: Propose a lightweight hallucination detection framework that could be incorporated into LLM deployment pipelines. This framework should work in real time to flag potentially hallucinated responses based on the patterns identified in your study.

LLM$-$Assisted Evaluation: Use Perplexity$-$LLM for the same prompts and check the alignment with the output of your model.