Question: I'm getting an error when running scala: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties. How do I fix this? Code below: I've changed the configuration, but

I'm getting an error when running scala: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties.

How do I fix this? Code below:

I'm getting an error when running scala: Using Spark's default log4j profile:I've changed the configuration, but it still didn't work. I'm just wondering how to get rid of the error.

A1 A Project 01 - build.sbtx O ScalaTest.scala untitled -/Documents/KSU /FALL 21/SWE 6623/ 1 package com.learningspark.example. WordCount > .bsp 2 Dimport org.apache.spark.- > idea 3 import org.apache.log4j.- > project (untitled-build] sources root 4 -import org.apache.spark.SparkContext. - v src 5 | main 6 Fobject ScalaTest { v scala 7 /** Our main function where the action happens */ com.learningspark.example.WordCou 8 ScalaTest def main(args: Array[String]) { > test 9 // Set the log level to only print errors > target 10 Logger.getLogger( name = "org").setLevel(Level. ERROR) build.sbt 11 // Create a SparkContext using every core of the local machine > Milli External Libraries 12 val sc = new SparkContext(new Sparkconf().setAppName("Spark Word Count").setMaster("local")) Scratches and Consoles 13 14 // Read each line of my book into an RDD 15 val input = sc.textFile( path = "news-2016-2017.txt") 16 // Split into words separated by a space character; flatMap one to many transformation 17 val words = input.flatmap(x => x.split( regex = " ")) 18 // Count up the occurrences of each word 19 val wordCounts = words.countByValue() 20 // Print the results. 21 wordCounts.foreach(printin) 22 23 } 24 Run: a Word Count E /Users/sloanedavidson/Library/Java/JavaVirtualMachines/openjdk-16/Contents/Home/bin/java ... Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Exception in thread "main" java.lang.IllegalAccessError Create breakpoint : class org.apache.spark.storage.StorageUtils$ (in unnamed module @ox4ddced80) cannot access class sun.nio.ch.DirectBuffer (in modu at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala:213) at org.apache.spark.storage. BlockManager MasterEndpoint.(BlockManager MasterEndpoint.scala:110) at org.apache.spark.SparkEnv$.$anonfun$create$9 (SparkEnv.scala:348) at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287) at org.apache.spark.SparkEnv$.create (SparkEnv.scala:336) at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191) at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277) at org.apache.spark. SparkContext.(SparkContext.scala:460)

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Databases Questions!