Mining Unstructured Data: A Big Data Perspective

$($Provide steps$/$pseudo code and detailed algorithm for every question. Draw diagrams depicting your understanding. Also support the discussion with worked out example$/$s$.)$

Q$1$: Decision tree and Market basket analysis:

a$.$ How can one use market basket analysis to calculate entropy and information gain?

b$.$ Perform market basket analysis on following table, form Apriori rules and calculate information gain.

c$.$ A shop increases the items in a particular rack. What will be impact of this move on information gain and entropy?

d$.$ What is meant by antecedent, consequence and confidence in case of it$?$ How could you link it to Bayesian conditional probability? Combine Apriori, Bayesian and Decision Tree based on entropy pattern.

e$.$ If this table is set of two transactions by eight individuals $$ then what will be impact on information gain

f$.$ Develop recommender system based on the tree techniques $($decision tree, Bayesian, Market basket analysis$).$ Write pseudo code and work out on above example. You will have to come up with algorithm to decide minimum support. Use entropy related concept for the same. Work out this example in detail on this table $($Consider both cases: $1.$ sixteen different transactions and $2.$ two transactions each by eight individuals$)$

g$.$ Also write pseudocode to find negative patterns in table$-$one. Write two negative patterns as rules.

Table One Transaction ID Items list

$1$ Banana, Apple, Spinach, Milk, Sandwich

$2$ Bottled water, Cornflakes, Beans, Spinach, Pizza, Salad, Egg

$3$ Rice crackers, lemon green tea, Bottled water, Spinach, Sandwich, Yogurt, Egg

$4$ Cornflakes, Pie, Pizza, Salad, Soda, Egg

$5$ Cornflakes, ice cream, pie, pizza, salad, soda, Egg

$6$ Chocolate shake, Banana, Apple, Spinach, milk, sandwich

$7$ Rice crackers, lemon green tea, chocolate shake, Banana, Apple, milk, yogurt

$8$ Bottled water, cornflakes, beans, chocolate shake, Spinach, pie, pizza, salad, soda

$9$ Rice crackers, lemon green tea, bottled water, Spinach, milk, pizza, salad, yogurt

$10$ Chocolate shake, Banana, Apple, Spinach, milk, sandwich

$11$ Rice crackers, lemon green tea, cornflakes, salad, Egg

$12$ Banana, Apple, Spinach, milk, sandwich, yogurt

$13$ Rice crackers, lemon green tea, bottled water, Spinach, pie, pizza, sandwich, Egg $14$ Banana, Apple, Spinach, milk, sandwich

$15$ Bottled water, cornflakes, beans, Spinach, pie, pizza, salad $16$ Banana, Apple, Spinach