Question: Description: Given a time series data which is a clickstream of user activity is stored in any flat flies, ask is to enrich the data

Description: Given a time series data which is a clickstream of user activity is stored in any flat flies, ask is to enrich the data with session id.

Session Definition: Session expires after inactivity of 30 mins, because of inactivity no clickstream record will be generated.

Session remains active for a total duration of 2 hours Steps:

Load Data in any flat file format.

Read the data and use spark batch (pyspark/scala) to do the computation.

Save the results in parquet with enriched data.

Note: Please do not use direct spark-sql.

Given Dataset: timestamp userid

2018-01-01T11:00:00Z u1

2018-01-01T12:00:00Z u1

2018-01-01T11:00:00Z u2

2018-01-02T11:00:00Z u2

2018-01-01T12:15:00Z u1

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Databases Questions!

Q:

Description: Given a time series data which is a clickstream of user activity is stored in any flat flies, ask is to enrich the data with session id. Session Definition: Session expires after...

Q:

Address the interaction of laws and new technologies and how they have evolved each other in recent years based on the below article... BIG DATA "Privacy Versus Progress" : A Necessary Sacrifice, By...

Q:

Case Study: Quick Fix Dental Practice Technology requirements Application must be built using Visual Studio 2019 or Visual Studio 2017, professional or enterprise. The community edition is not...

Q:

Demand and Supply Discussion Question: Applications-Ubereconomics Please read the article below and respond to the follow-up question: Using Big Data to Estimate Consumer Surplus: The Case of Uber...

Q:

Business Research Methodology- Question Bank 1 1. When the marketing department of an organization attempts to determine the amount of time the managers in this department spend at their computers in...

Q:

10. An experimenter has some degree of control over the: a. independent variable. b. correlative variable. c. history effect. d. All of the above, if the experiment is conducted properly. 11. If a...

Q:

Business Research MethodologyQuestion Bank 1 1. When the marketing department of an organization attempts to determine the amount of time the managers in this department spend at their computers in...

Q:

London School of Science & Technology Qualification Unit number and title BTEC Level 5 HND Diploma Business UNIT 6: Business Decision Making Student name and ID number Assessor name Al Hassan Barrie...

Q:

uestion: 1. IN A STOCK ACQUISITION ACCOUNTED FOR BY THE EQUITY METHOD, A PORTION OF THE PURCHASE PRICE OFTEN IS ATTRIBUTABLE TO GOODWILL OR TO SPECIFIC ASSETS OR LIABILITIES. HOW OR UNDER WHAT BASIS...

Q:

Deliverables [100 marks]. This assignment in essence is a 'condensed' research and data analysis project. So, we will create the report based on a simplified research report template. Your report...

Q:

The table below shows the productivity levels of producing beer and sardines in Canada and Mexico. What should be the Mexican productivity per worker in the sardine industry for no advantage to be...

Q:

Using the law of conservation of momentum, explain how a colliding beam reaction (in which both particles move toward a head-on collision) requires less incident kinetic energy than a reaction in...

Q:

While the value of a stock is independent of the call holder's behavior, the value of an R&D investment is not independent of the investor's behavior.

Q:

Seved Help 14 Wisconsin Snowmobile Corp. is considering a switch to level production Cost efficiencies would occur under level production, and aftertax costs would decline by $31,500, but inventory...

Q:

Know why loyalty programs and customer relationship management (CRM) systems are important enablers of delivering loyalty strategies.

Q:

Recognize the actions that customers may take in response to service failures.

Q:

Understand the part played by CRM systems in delivering customized services and building loyalty.