Question: I have this code to run pyspark notebook. How do I display the original log entries? And what is regexp_extract doing? import re from pyspark.sql.types

I have this code to run pyspark notebook. How do I display the original log entries?

And what is regexp_extract doing?

import re

from pyspark.sql.types import *

from pyspark.sql.functions import *

inputPath = "/databricks-datasets/sample_logs/"

df = sqlContext.read.text(inputPath)

converted = df.select(unix_timestamp(regexp_extract(df["value"], ".+\[(.+) -", 1), "dd/MMM/yyyy:HH:mm:ss") \

.cast(TimestampType()),

split(df["value"], "")[8])

display(converted.take(10))

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Programming Questions!