22/08/22 17:02:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/08/22 17:02:40 INFO SecurityManager: Changing view acls to: nathan
22/08/22 17:02:40 INFO SecurityManager: Changing modify acls to: nathan
22/08/22 17:02:40 INFO SecurityManager: Changing view acls groups to: 
22/08/22 17:02:40 INFO SecurityManager: Changing modify acls groups to: 
22/08/22 17:02:40 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(nathan); groups with view permissions: Set(); users  with modify permissions: Set(nathan); groups with modify permissions: Set()
22/08/22 17:02:41 INFO HiveConf: Found configuration file file:/Users/nathan/spark/spark-3.1.1-bin-hadoop3.2/conf/hive-site.xml
22/08/22 17:02:41 INFO SparkContext: Running Spark version 3.1.1
22/08/22 17:02:41 INFO ResourceUtils: ==============================================================
22/08/22 17:02:41 INFO ResourceUtils: No custom resources configured for spark.driver.
22/08/22 17:02:41 INFO ResourceUtils: ==============================================================
22/08/22 17:02:41 INFO SparkContext: Submitted application: sparklyr
22/08/22 17:02:41 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
22/08/22 17:02:41 INFO ResourceProfile: Limiting resource is cpu
22/08/22 17:02:41 INFO ResourceProfileManager: Added ResourceProfile id: 0
22/08/22 17:02:41 INFO SecurityManager: Changing view acls to: nathan
22/08/22 17:02:41 INFO SecurityManager: Changing modify acls to: nathan
22/08/22 17:02:41 INFO SecurityManager: Changing view acls groups to: 
22/08/22 17:02:41 INFO SecurityManager: Changing modify acls groups to: 
22/08/22 17:02:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(nathan); groups with view permissions: Set(); users  with modify permissions: Set(nathan); groups with modify permissions: Set()
22/08/22 17:02:41 INFO Utils: Successfully started service 'sparkDriver' on port 49961.
22/08/22 17:02:41 INFO SparkEnv: Registering MapOutputTracker
22/08/22 17:02:41 INFO SparkEnv: Registering BlockManagerMaster
22/08/22 17:02:41 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/08/22 17:02:41 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/08/22 17:02:41 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
22/08/22 17:02:41 INFO DiskBlockManager: Created local directory at /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/blockmgr-c8627b5a-d816-4285-99e0-da192652200d
22/08/22 17:02:41 INFO MemoryStore: MemoryStore started with capacity 912.3 MiB
22/08/22 17:02:41 INFO SparkEnv: Registering OutputCommitCoordinator
22/08/22 17:02:42 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/08/22 17:02:42 INFO SparkUI: Bound SparkUI to 127.0.0.1, and started at http://localhost:4040
22/08/22 17:02:42 INFO SparkContext: Added JAR file:/Library/Frameworks/R.framework/Versions/4.2/Resources/library/sparklyr/java/sparklyr-master-2.12.jar at spark://localhost:49961/jars/sparklyr-master-2.12.jar with timestamp 1661184161332
22/08/22 17:02:42 INFO Executor: Starting executor ID driver on host localhost
22/08/22 17:02:42 INFO Executor: Fetching spark://localhost:49961/jars/sparklyr-master-2.12.jar with timestamp 1661184161332
22/08/22 17:02:42 INFO TransportClientFactory: Successfully created connection to localhost/127.0.0.1:49961 after 24 ms (0 ms spent in bootstraps)
22/08/22 17:02:42 INFO Utils: Fetching spark://localhost:49961/jars/sparklyr-master-2.12.jar to /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-8301cce4-c59e-4d64-b5ee-8e10e424a5e7/userFiles-92b9a507-5ad4-4e9b-9c2b-a5e56b6f87f8/fetchFileTemp8391619000067175710.tmp
22/08/22 17:02:42 INFO Executor: Adding file:/private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-8301cce4-c59e-4d64-b5ee-8e10e424a5e7/userFiles-92b9a507-5ad4-4e9b-9c2b-a5e56b6f87f8/sparklyr-master-2.12.jar to class loader
22/08/22 17:02:42 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 49963.
22/08/22 17:02:42 INFO NettyBlockTransferService: Server created on localhost:49963
22/08/22 17:02:42 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/08/22 17:02:42 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, localhost, 49963, None)
22/08/22 17:02:42 INFO BlockManagerMasterEndpoint: Registering block manager localhost:49963 with 912.3 MiB RAM, BlockManagerId(driver, localhost, 49963, None)
22/08/22 17:02:42 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, localhost, 49963, None)
22/08/22 17:02:42 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, localhost, 49963, None)
22/08/22 17:02:43 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/nathan/Documents/work/NEDataLtd/projects/catalog/inst/tinytest/spark-warehouse').
22/08/22 17:02:43 INFO SharedState: Warehouse path is 'file:/Users/nathan/Documents/work/NEDataLtd/projects/catalog/inst/tinytest/spark-warehouse'.
22/08/22 17:02:47 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.7 using Spark classes.
22/08/22 17:02:47 INFO HiveConf: Found configuration file file:/Users/nathan/spark/spark-3.1.1-bin-hadoop3.2/conf/hive-site.xml
22/08/22 17:02:48 INFO SessionState: Created HDFS directory: /tmp/hive/nathan
22/08/22 17:02:48 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/nathan
22/08/22 17:02:48 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/8ce2a849-0e0f-4572-87d2-1bfdb9a4b0cc
22/08/22 17:02:48 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/nathan/8ce2a849-0e0f-4572-87d2-1bfdb9a4b0cc
22/08/22 17:02:48 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/8ce2a849-0e0f-4572-87d2-1bfdb9a4b0cc/_tmp_space.db
22/08/22 17:02:48 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is file:/Users/nathan/Documents/work/NEDataLtd/projects/catalog/inst/tinytest/spark-warehouse
22/08/22 17:02:49 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
22/08/22 17:02:49 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
22/08/22 17:02:49 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
22/08/22 17:02:49 INFO ObjectStore: ObjectStore, initialize called
22/08/22 17:02:50 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
22/08/22 17:02:50 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
22/08/22 17:02:52 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
22/08/22 17:02:55 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
22/08/22 17:02:55 INFO ObjectStore: Initialized ObjectStore
22/08/22 17:02:55 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
22/08/22 17:02:55 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore nathan@127.0.0.1
22/08/22 17:02:55 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
22/08/22 17:02:55 INFO HiveMetaStore: Added admin role in metastore
22/08/22 17:02:55 INFO HiveMetaStore: Added public role in metastore
22/08/22 17:02:55 INFO HiveMetaStore: No user is added in admin role, since config is empty
22/08/22 17:02:56 INFO HiveMetaStore: 0: get_all_functions
22/08/22 17:02:56 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_all_functions	
22/08/22 17:02:56 INFO HiveMetaStore: 0: get_database: default
22/08/22 17:02:56 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
22/08/22 17:02:56 INFO HiveMetaStore: 0: get_database: global_temp
22/08/22 17:02:56 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: global_temp	
22/08/22 17:02:56 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
22/08/22 17:02:56 INFO HiveMetaStore: 0: get_database: default
22/08/22 17:02:56 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
22/08/22 17:02:56 INFO HiveMetaStore: 0: get_database: default
22/08/22 17:02:56 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
22/08/22 17:02:56 INFO HiveMetaStore: 0: get_tables: db=default pat=*
22/08/22 17:02:56 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_tables: db=default pat=*	
22/08/22 17:02:58 INFO CodeGenerator: Code generated in 262.140029 ms
22/08/22 17:02:58 INFO SparkContext: Starting job: collect at utils.scala:26
22/08/22 17:02:58 INFO DAGScheduler: Got job 0 (collect at utils.scala:26) with 1 output partitions
22/08/22 17:02:58 INFO DAGScheduler: Final stage: ResultStage 0 (collect at utils.scala:26)
22/08/22 17:02:58 INFO DAGScheduler: Parents of final stage: List()
22/08/22 17:02:58 INFO DAGScheduler: Missing parents: List()
22/08/22 17:02:58 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at collect at utils.scala:26), which has no missing parents
22/08/22 17:02:58 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 6.8 KiB, free 912.3 MiB)
22/08/22 17:02:58 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.3 KiB, free 912.3 MiB)
22/08/22 17:02:58 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:49963 (size: 3.3 KiB, free: 912.3 MiB)
22/08/22 17:02:58 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1383
22/08/22 17:02:58 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at collect at utils.scala:26) (first 15 tasks are for partitions Vector(0))
22/08/22 17:02:58 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0
22/08/22 17:02:58 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (localhost, executor driver, partition 0, PROCESS_LOCAL, 4689 bytes) taskResourceAssignments Map()
22/08/22 17:02:59 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
22/08/22 17:02:59 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1397 bytes result sent to driver
22/08/22 17:02:59 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 348 ms on localhost (executor driver) (1/1)
22/08/22 17:02:59 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
22/08/22 17:02:59 INFO DAGScheduler: ResultStage 0 (collect at utils.scala:26) finished in 0.604 s
22/08/22 17:02:59 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
22/08/22 17:02:59 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
22/08/22 17:02:59 INFO DAGScheduler: Job 0 finished: collect at utils.scala:26, took 0.702282 s
22/08/22 17:02:59 INFO CodeGenerator: Code generated in 17.391597 ms
22/08/22 17:02:59 INFO SparkContext: Starting job: collect at utils.scala:26
22/08/22 17:02:59 INFO DAGScheduler: Got job 1 (collect at utils.scala:26) with 1 output partitions
22/08/22 17:02:59 INFO DAGScheduler: Final stage: ResultStage 1 (collect at utils.scala:26)
22/08/22 17:02:59 INFO DAGScheduler: Parents of final stage: List()
22/08/22 17:02:59 INFO DAGScheduler: Missing parents: List()
22/08/22 17:02:59 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[5] at collect at utils.scala:26), which has no missing parents
22/08/22 17:02:59 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 6.8 KiB, free 912.3 MiB)
22/08/22 17:02:59 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.3 KiB, free 912.3 MiB)
22/08/22 17:02:59 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:49963 (size: 3.3 KiB, free: 912.3 MiB)
22/08/22 17:02:59 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1383
22/08/22 17:02:59 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[5] at collect at utils.scala:26) (first 15 tasks are for partitions Vector(0))
22/08/22 17:02:59 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks resource profile 0
22/08/22 17:02:59 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1) (localhost, executor driver, partition 0, PROCESS_LOCAL, 4689 bytes) taskResourceAssignments Map()
22/08/22 17:02:59 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
22/08/22 17:02:59 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1397 bytes result sent to driver
22/08/22 17:02:59 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 5 ms on localhost (executor driver) (1/1)
22/08/22 17:02:59 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
22/08/22 17:02:59 INFO DAGScheduler: ResultStage 1 (collect at utils.scala:26) finished in 0.014 s
22/08/22 17:02:59 INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job
22/08/22 17:02:59 INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished
22/08/22 17:02:59 INFO DAGScheduler: Job 1 finished: collect at utils.scala:26, took 0.017161 s
22/08/22 17:02:59 INFO CodeGenerator: Code generated in 50.503579 ms
22/08/22 17:02:59 INFO CodeGenerator: Code generated in 20.311167 ms
22/08/22 17:02:59 INFO CodeGenerator: Code generated in 17.104536 ms
22/08/22 17:03:00 INFO SparkContext: Invoking stop() from shutdown hook
22/08/22 17:03:00 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:49963 in memory (size: 3.3 KiB, free: 912.3 MiB)
22/08/22 17:03:00 INFO SparkUI: Stopped Spark web UI at http://localhost:4040
22/08/22 17:03:00 INFO BlockManagerInfo: Removed broadcast_0_piece0 on localhost:49963 in memory (size: 3.3 KiB, free: 912.3 MiB)
22/08/22 17:03:00 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/08/22 17:03:00 INFO MemoryStore: MemoryStore cleared
22/08/22 17:03:00 INFO BlockManager: BlockManager stopped
22/08/22 17:03:00 INFO BlockManagerMaster: BlockManagerMaster stopped
22/08/22 17:03:00 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/08/22 17:03:00 INFO SparkContext: Successfully stopped SparkContext
22/08/22 17:03:00 INFO ShutdownHookManager: Shutdown hook called
22/08/22 17:03:00 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-96724e8b-da06-45f1-b9d6-867ce70689f7
22/08/22 17:03:00 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-8301cce4-c59e-4d64-b5ee-8e10e424a5e7
22/08/22 17:03:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/08/22 17:03:03 INFO SecurityManager: Changing view acls to: nathan
22/08/22 17:03:03 INFO SecurityManager: Changing modify acls to: nathan
22/08/22 17:03:03 INFO SecurityManager: Changing view acls groups to: 
22/08/22 17:03:03 INFO SecurityManager: Changing modify acls groups to: 
22/08/22 17:03:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(nathan); groups with view permissions: Set(); users  with modify permissions: Set(nathan); groups with modify permissions: Set()
22/08/22 17:04:03 INFO ShutdownHookManager: Shutdown hook called
22/08/22 17:04:03 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-9ad8740c-d622-48a9-8aab-7e50d4082ef0
