Question: I have this code to run pyspark notebook. How do I display the original log entries? And what is regexp_extract doing? import re from pyspark.sql.types
I have this code to run pyspark notebook. How do I display the original log entries?
And what is regexp_extract doing?
import re
from pyspark.sql.types import *
from pyspark.sql.functions import *
inputPath = "/databricks-datasets/sample_logs/"
df = sqlContext.read.text(inputPath)
converted = df.select(unix_timestamp(regexp_extract(df["value"], ".+\[(.+) -", 1), "dd/MMM/yyyy:HH:mm:ss") \
.cast(TimestampType()),
split(df["value"], "")[8])
display(converted.take(10))
Step by Step Solution
There are 3 Steps involved in it
Get step-by-step solutions from verified subject matter experts
