Question: Issue with python code used for correlation (person) using databricks. This is what I'm receiving: **I have to use df = spark.read.format(csv).option(header, False).load ** CODE

Issue with python code used for correlation (person) using databricks.

This is what I'm receiving:

Issue with python code used for correlation (person) using databricks. This is

**I have to use df = spark.read.format("csv").option("header", "False").load **

CODE

import pandas as pd

## DataFrameReader

df = spark.read.format("csv").option("header", "False").load("dbfs:/FileStore/tables/users_no_labels.txt").toPandas()

##find correlation

correlation = df.corr(method = 'pearson')

##to display full correlation dataframe

with pd.option_context('display.max_rows', None, 'display.max_columns', None):

print(correlation)

**********************************************************************************

users_no_labels.txt Data:

0 , 1 , 0 1 , 0 , 0 1 , 1 , 0 0 , 1 , 1 1 , 0 , 1 0 , 1 , 1

(2) Spark Jobs - Job 72 View (Stages: 1/1) - Job 73 View (Stages: 1/1) Empty DataFrame Columns: [] Index: []

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Databases Questions!