Project$-3$: Predicting Algae Blooms

Problem Description and Objectives

High concentrations of certain harmful algae in rivers constitute a serious ecological problem with a strong impact not only on river lifeforms, but also on water quality. Being able to monitor and perform an early forecast of algae blooms is essential to improving the quality of rivers.

With the goal of addressing this prediction problem, several water samples were collected in different European rivers at different times during a period of approximately $1$ year. For each water sample, different chemical properties were measured as well as the frequency of occurrence of seven harmful algae. Some other characteristics of the water collection process were also stored, such as the season of the year, the river size, and the river speed.

One of the main motivations behind this application lies in the fact that chemical monitoring is cheap and easily automated, while the biological analysis of the samples to identify the algae that are present in the water involves microscopic examination, requires trained manpower, and is therefore both expensive and slow. As such, obtaining models that can accurately predict the algae frequencies based on chemical properties would facilitate the creation of cheap and automated systems for monitoring harmful algae blooms.

Another objective of this study is to provide a better understanding of the factors influencing the algae frequencies. Namely, we want to understand how these frequencies are related to certain chemical attributes of water samples as well as other characteristics of the samples $($like season of the year, type of river, etc.$).$

$2.$ Data Description

Training Data: The training data $($Analysis$.$txt$)$ consists of data for $200$ water samples. Each observation contains information on $11$ variables. Three of these variables are nominal and describe the season of the year when the water samples to be aggregated were collected, as well as the size and speed of the river in question. The eight remaining variables are values of different chemical parameters measured in the water samples forming the aggregation, namely:

Maximum $pH$ value

Minimum value of $O2($oxygen$)$

Mean value of $Cl($chloride$)$

Mean value of $N{O}_3^{-}($nitrates$)$

Mean value of $N{H}_4^{+}($ammonium$)$

Mean of $P{O}_4^{3-}($orthophosphate$)$

Mean of total PO$4($phosphate$)$

Mean of chlorophyll

Associated with each of these parameters are seven frequency numbers of different harmful algae found in the respective water samples. No information is given regarding the names of the algae that were identified.

$1$

The second dataset $($Eval$.$txt$)$ contains information on $140$ extra observations. It uses the same basic structure, but it does not include information concerning the seven harmful algae frequencies. These extra observations can be regarded as a kind of test set. The main goal of our study is to predict the frequencies of the seven algae for these $140$ water samples. This means that we are facing a predictive data mining task. This is one among the diverse set of problems tackled in data mining. In this type of task, our main goal is to obtain a model that allows us to predict the value of a certain target variable given the values of a set of predictor variables. This model may also provide indications on which predictor variables have a larger impact on the target variable; that is$,$ the model may provide a comprehensive description of the factors that influence the target variable.

Note: Unknown values are indicated with the string $"$XXXXXXX$".$

$3.$ The Experiment

In this project you must do the following tasks $($either individually or with a partner$).$

Task$-1$: Convert the text files into CSV files and convert them into Pandas' data frames: Analysis.txt into training data frame and Eval.txt into testing data frame.

Task$-2$: Remove the unknown values $($indicated with the string $"$XXXXXXX$")$ and make them as NA values in your data frames.

Task$-3$: Use the statistical summary to conduct exploratory data analysis. Summarize your findings about the dataset $($including normality$).$

Task$-4$: Use the graphical plots $($Histogram$,$ Q$-$Q plot, Boxplot, etc.$)$ to conduct exploratory data analysis. Summarize your findings about the dataset $($normality$,$ outliers, equality of variance, etc.,$).$

Task$-5$: Remove the NA values in the data: Use mean value to replace the NA if the data is normally distributed; if the data is skewed then use the median to replace the data.

Task$-6$: Use the training dataset to build models to find the frequencies of each $7$ Algae types based on the $8$ chemical components present in the water.

Task$-7$: Using the trained model to find the frequencies of each $7$ Algae types present in the testing data set for the given chemical composition of the water.