Advanced-Data Manipulation and Exploratory Data Analysis (EDA)

Objectives: In this lab sheet, you will continue working with Pandas and dive into more advanced data manipulation techniques. Then, you will perform Exploratory Data Analysis (EDA).

1. Data Loading/Collection: Pandas provides simple methods to load data from various sources, including CSV files, Excel spreadsheets, databases, and web APIs, into a structured format.
2. Data Cleaning: It offers tools to handle missing data, duplicate records, and outliers, ensuring that the dataset is clean and ready for analysis.
3. Data Exploration: You can quickly explore data by calculating summary statistics, visualizing distributions, and identifying correlations between variables.
4. Data Transformation: Pandas enables you to reshape and transform data easily. You can filter, sort, making it suitable for modeling or visualization.
5. Data Integration: Pandas allows you to merge, join, or concatenate datasets, which is crucial when working with multiple data sources.(We will discuss in next labsheet when you work with multiple files)

1. Data Loading:

Task 1 Load a dataset from a CSV file into a Pandas DataFrame for EDA.

2. Data Cleaning

Task 2: Handle duplicate rows by detecting and removing them.

Task 3: Handle missing or null values by dropping rows with missing data.

Task 4: Detect and handle outliers using the IQR score technique.

3. Data Exploration:

Task 5: Check the data types of columns in the DataFrame to ensure data consistency.

Task 6: Calculate correlations between different features in the dataset.

Task 7: Calculate Interquartile Range (IQR)

Task 8: Create various visualizations, including histograms, bar plots, and heatmaps, to explore and understand the data.

4. Data Transformation

Task 8: Sorting the Data and Selecting Data

Task 9: Rename columns to improve readability.