Modeling/Analyzing Big Data

a) What is the over-fitting problem in analyzing big data sets? Illustrate the problem in the case of tree models.

b) How do neural net models compare to tree models in analyzing big data?

c) You read a paper that says the best way to predict profitability of new investments is to do an ensemble analysis. It estimates different models from linear regression to trees to neural nets to logistic equations that link economic data and tweets to whether a past investment was profitable (=1) or not (=0). It applies the estimated models to predict whether new investments are likely to profitable or not and invests in the projects getting the most votes. Why might this voting procedure work?

d) Professor Critical says This class is about rare discontinuous events that affect economies but big data means having large numbers of observations on many small events. How can big data illuminate rare events?