21/03/18 20:33:44 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/03/18 20:33:46 INFO SparkContext: Running Spark version 2.4.3
21/03/18 20:33:46 INFO SparkContext: Submitted application: sparklyr
21/03/18 20:33:46 INFO SecurityManager: Changing view acls to: nathan
21/03/18 20:33:46 INFO SecurityManager: Changing modify acls to: nathan
21/03/18 20:33:46 INFO SecurityManager: Changing view acls groups to: 
21/03/18 20:33:46 INFO SecurityManager: Changing modify acls groups to: 
21/03/18 20:33:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(nathan); groups with view permissions: Set(); users  with modify permissions: Set(nathan); groups with modify permissions: Set()
21/03/18 20:33:46 INFO Utils: Successfully started service 'sparkDriver' on port 64426.
21/03/18 20:33:46 INFO SparkEnv: Registering MapOutputTracker
21/03/18 20:33:46 INFO SparkEnv: Registering BlockManagerMaster
21/03/18 20:33:46 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/03/18 20:33:46 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/03/18 20:33:46 INFO DiskBlockManager: Created local directory at /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/blockmgr-6cc395aa-ed11-43c1-8c9a-cf84b132c851
21/03/18 20:33:46 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
21/03/18 20:33:46 INFO SparkEnv: Registering OutputCommitCoordinator
21/03/18 20:33:46 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/03/18 20:33:46 INFO SparkUI: Bound SparkUI to 127.0.0.1, and started at http://localhost:4040
21/03/18 20:33:46 INFO SparkContext: Added JAR file:/Library/Frameworks/R.framework/Versions/4.0/Resources/library/sparklyr/java/sparklyr-2.4-2.11.jar at spark://localhost:64426/jars/sparklyr-2.4-2.11.jar with timestamp 1616096026878
21/03/18 20:33:46 INFO Executor: Starting executor ID driver on host localhost
21/03/18 20:33:47 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 64427.
21/03/18 20:33:47 INFO NettyBlockTransferService: Server created on localhost:64427
21/03/18 20:33:47 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/03/18 20:33:47 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, localhost, 64427, None)
21/03/18 20:33:47 INFO BlockManagerMasterEndpoint: Registering block manager localhost:64427 with 912.3 MB RAM, BlockManagerId(driver, localhost, 64427, None)
21/03/18 20:33:47 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, localhost, 64427, None)
21/03/18 20:33:47 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, localhost, 64427, None)
21/03/18 20:33:47 INFO SharedState: loading hive config file: file:/Users/nathan/spark/spark-2.4.3-bin-hadoop2.7/conf/hive-site.xml
21/03/18 20:33:47 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse').
21/03/18 20:33:47 INFO SharedState: Warehouse path is 'file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse'.
21/03/18 20:33:48 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
21/03/18 20:33:51 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
21/03/18 20:33:52 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
21/03/18 20:33:52 INFO ObjectStore: ObjectStore, initialize called
21/03/18 20:33:52 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
21/03/18 20:33:52 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
21/03/18 20:33:54 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
21/03/18 20:33:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:33:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:33:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:33:56 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:33:56 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
21/03/18 20:33:56 INFO ObjectStore: Initialized ObjectStore
21/03/18 20:33:56 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
21/03/18 20:33:56 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
21/03/18 20:33:57 INFO HiveMetaStore: Added admin role in metastore
21/03/18 20:33:57 INFO HiveMetaStore: Added public role in metastore
21/03/18 20:33:57 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/03/18 20:33:57 INFO HiveMetaStore: 0: get_all_databases
21/03/18 20:33:57 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_all_databases	
21/03/18 20:33:57 INFO HiveMetaStore: 0: get_functions: db=default pat=*
21/03/18 20:33:57 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_functions: db=default pat=*	
21/03/18 20:33:57 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:33:58 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/ef24fc23-77db-4c76-8209-be3b2737ea53_resources
21/03/18 20:33:58 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/ef24fc23-77db-4c76-8209-be3b2737ea53
21/03/18 20:33:58 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/nathan/ef24fc23-77db-4c76-8209-be3b2737ea53
21/03/18 20:33:58 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/ef24fc23-77db-4c76-8209-be3b2737ea53/_tmp_space.db
21/03/18 20:33:58 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse
21/03/18 20:33:58 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:33:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:33:58 INFO HiveMetaStore: 0: get_database: global_temp
21/03/18 20:33:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: global_temp	
21/03/18 20:33:58 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
21/03/18 20:33:58 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:33:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:33:58 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:33:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:33:58 INFO HiveMetaStore: 0: get_tables: db=default pat=*
21/03/18 20:33:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_tables: db=default pat=*	
21/03/18 20:34:00 INFO SparkContext: Starting job: collect at utils.scala:61
21/03/18 20:34:00 INFO DAGScheduler: Got job 0 (collect at utils.scala:61) with 1 output partitions
21/03/18 20:34:00 INFO DAGScheduler: Final stage: ResultStage 0 (collect at utils.scala:61)
21/03/18 20:34:00 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:34:00 INFO DAGScheduler: Missing parents: List()
21/03/18 20:34:00 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[4] at map at utils.scala:54), which has no missing parents
21/03/18 20:34:00 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 6.0 KB, free 912.3 MB)
21/03/18 20:34:00 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.4 KB, free 912.3 MB)
21/03/18 20:34:00 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:64427 (size: 3.4 KB, free: 912.3 MB)
21/03/18 20:34:00 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:00 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[4] at map at utils.scala:54) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:00 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
21/03/18 20:34:01 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7893 bytes)
21/03/18 20:34:01 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/03/18 20:34:01 INFO Executor: Fetching spark://localhost:64426/jars/sparklyr-2.4-2.11.jar with timestamp 1616096026878
21/03/18 20:34:01 INFO TransportClientFactory: Successfully created connection to localhost/127.0.0.1:64426 after 25 ms (0 ms spent in bootstraps)
21/03/18 20:34:01 INFO Utils: Fetching spark://localhost:64426/jars/sparklyr-2.4-2.11.jar to /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-c6dc1d5f-a8e9-4cdf-a115-cfb683ce166d/userFiles-3451756c-289c-4839-9dcd-b0c8290c8aec/fetchFileTemp4491128065805710888.tmp
21/03/18 20:34:01 INFO Executor: Adding file:/private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-c6dc1d5f-a8e9-4cdf-a115-cfb683ce166d/userFiles-3451756c-289c-4839-9dcd-b0c8290c8aec/sparklyr-2.4-2.11.jar to class loader
21/03/18 20:34:01 INFO CodeGenerator: Code generated in 245.355392 ms
21/03/18 20:34:01 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 970 bytes result sent to driver
21/03/18 20:34:01 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 730 ms on localhost (executor driver) (1/1)
21/03/18 20:34:01 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
21/03/18 20:34:01 INFO DAGScheduler: ResultStage 0 (collect at utils.scala:61) finished in 1.625 s
21/03/18 20:34:01 INFO DAGScheduler: Job 0 finished: collect at utils.scala:61, took 1.747284 s
21/03/18 20:34:02 INFO ContextCleaner: Cleaned accumulator 0
21/03/18 20:34:02 INFO BlockManagerInfo: Removed broadcast_0_piece0 on localhost:64427 in memory (size: 3.4 KB, free: 912.3 MB)
21/03/18 20:34:02 INFO CodeGenerator: Code generated in 19.016555 ms
21/03/18 20:34:02 INFO CodeGenerator: Code generated in 20.644784 ms
21/03/18 20:34:02 INFO SparkContext: Starting job: collect at utils.scala:135
21/03/18 20:34:02 INFO DAGScheduler: Got job 1 (collect at utils.scala:135) with 1 output partitions
21/03/18 20:34:02 INFO DAGScheduler: Final stage: ResultStage 1 (collect at utils.scala:135)
21/03/18 20:34:02 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:34:02 INFO DAGScheduler: Missing parents: List()
21/03/18 20:34:02 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[9] at collect at utils.scala:135), which has no missing parents
21/03/18 20:34:02 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 6.5 KB, free 912.3 MB)
21/03/18 20:34:02 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.3 KB, free 912.3 MB)
21/03/18 20:34:02 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:64427 (size: 3.3 KB, free: 912.3 MB)
21/03/18 20:34:02 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:02 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[9] at collect at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:02 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
21/03/18 20:34:02 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, PROCESS_LOCAL, 8080 bytes)
21/03/18 20:34:02 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
21/03/18 20:34:02 INFO CodeGenerator: Code generated in 9.416818 ms
21/03/18 20:34:02 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1286 bytes result sent to driver
21/03/18 20:34:02 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 33 ms on localhost (executor driver) (1/1)
21/03/18 20:34:02 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
21/03/18 20:34:02 INFO DAGScheduler: ResultStage 1 (collect at utils.scala:135) finished in 0.044 s
21/03/18 20:34:02 INFO DAGScheduler: Job 1 finished: collect at utils.scala:135, took 0.046990 s
21/03/18 20:34:03 INFO CodeGenerator: Code generated in 18.318532 ms
21/03/18 20:34:03 INFO CodeGenerator: Code generated in 14.261939 ms
21/03/18 20:34:03 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:34:03 INFO DAGScheduler: Registering RDD 12 (count at utils.scala:135)
21/03/18 20:34:03 INFO DAGScheduler: Got job 2 (count at utils.scala:135) with 1 output partitions
21/03/18 20:34:03 INFO DAGScheduler: Final stage: ResultStage 3 (count at utils.scala:135)
21/03/18 20:34:03 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
21/03/18 20:34:03 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 2)
21/03/18 20:34:03 INFO DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[12] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:03 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 8.4 KB, free 912.3 MB)
21/03/18 20:34:03 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 4.5 KB, free 912.3 MB)
21/03/18 20:34:03 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:64427 (size: 4.5 KB, free: 912.3 MB)
21/03/18 20:34:03 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:03 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[12] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:03 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
21/03/18 20:34:03 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, executor driver, partition 0, PROCESS_LOCAL, 8069 bytes)
21/03/18 20:34:03 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
21/03/18 20:34:03 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1542 bytes result sent to driver
21/03/18 20:34:03 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 57 ms on localhost (executor driver) (1/1)
21/03/18 20:34:03 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 
21/03/18 20:34:03 INFO DAGScheduler: ShuffleMapStage 2 (count at utils.scala:135) finished in 0.071 s
21/03/18 20:34:03 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:34:03 INFO DAGScheduler: running: Set()
21/03/18 20:34:03 INFO DAGScheduler: waiting: Set(ResultStage 3)
21/03/18 20:34:03 INFO DAGScheduler: failed: Set()
21/03/18 20:34:03 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[15] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:03 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 7.1 KB, free 912.3 MB)
21/03/18 20:34:03 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 3.8 KB, free 912.3 MB)
21/03/18 20:34:03 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:64427 (size: 3.8 KB, free: 912.3 MB)
21/03/18 20:34:03 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:03 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[15] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:03 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
21/03/18 20:34:03 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:34:03 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
21/03/18 20:34:03 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:34:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 7 ms
21/03/18 20:34:03 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1782 bytes result sent to driver
21/03/18 20:34:03 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 49 ms on localhost (executor driver) (1/1)
21/03/18 20:34:03 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 
21/03/18 20:34:03 INFO DAGScheduler: ResultStage 3 (count at utils.scala:135) finished in 0.061 s
21/03/18 20:34:03 INFO DAGScheduler: Job 2 finished: count at utils.scala:135, took 0.158133 s
21/03/18 20:34:03 INFO SparkContext: Starting job: collect at utils.scala:135
21/03/18 20:34:03 INFO DAGScheduler: Got job 3 (collect at utils.scala:135) with 1 output partitions
21/03/18 20:34:03 INFO DAGScheduler: Final stage: ResultStage 4 (collect at utils.scala:135)
21/03/18 20:34:03 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:34:03 INFO DAGScheduler: Missing parents: List()
21/03/18 20:34:03 INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[18] at collect at utils.scala:135), which has no missing parents
21/03/18 20:34:03 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 6.5 KB, free 912.3 MB)
21/03/18 20:34:03 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 3.3 KB, free 912.3 MB)
21/03/18 20:34:03 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:64427 (size: 3.3 KB, free: 912.3 MB)
21/03/18 20:34:03 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:03 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[18] at collect at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:03 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
21/03/18 20:34:03 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, localhost, executor driver, partition 0, PROCESS_LOCAL, 8080 bytes)
21/03/18 20:34:03 INFO Executor: Running task 0.0 in stage 4.0 (TID 4)
21/03/18 20:34:03 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4). 1286 bytes result sent to driver
21/03/18 20:34:03 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 11 ms on localhost (executor driver) (1/1)
21/03/18 20:34:03 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 
21/03/18 20:34:03 INFO DAGScheduler: ResultStage 4 (collect at utils.scala:135) finished in 0.018 s
21/03/18 20:34:03 INFO DAGScheduler: Job 3 finished: collect at utils.scala:135, took 0.026891 s
21/03/18 20:34:03 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:34:03 INFO DAGScheduler: Registering RDD 21 (count at utils.scala:135)
21/03/18 20:34:03 INFO DAGScheduler: Got job 4 (count at utils.scala:135) with 1 output partitions
21/03/18 20:34:03 INFO DAGScheduler: Final stage: ResultStage 6 (count at utils.scala:135)
21/03/18 20:34:03 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 5)
21/03/18 20:34:03 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 5)
21/03/18 20:34:03 INFO DAGScheduler: Submitting ShuffleMapStage 5 (MapPartitionsRDD[21] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:03 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 8.4 KB, free 912.2 MB)
21/03/18 20:34:03 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 4.5 KB, free 912.2 MB)
21/03/18 20:34:03 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:64427 (size: 4.5 KB, free: 912.3 MB)
21/03/18 20:34:03 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:03 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 5 (MapPartitionsRDD[21] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:03 INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
21/03/18 20:34:03 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, localhost, executor driver, partition 0, PROCESS_LOCAL, 8069 bytes)
21/03/18 20:34:03 INFO Executor: Running task 0.0 in stage 5.0 (TID 5)
21/03/18 20:34:03 INFO Executor: Finished task 0.0 in stage 5.0 (TID 5). 1499 bytes result sent to driver
21/03/18 20:34:03 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 10 ms on localhost (executor driver) (1/1)
21/03/18 20:34:03 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
21/03/18 20:34:03 INFO DAGScheduler: ShuffleMapStage 5 (count at utils.scala:135) finished in 0.018 s
21/03/18 20:34:03 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:34:03 INFO DAGScheduler: running: Set()
21/03/18 20:34:03 INFO DAGScheduler: waiting: Set(ResultStage 6)
21/03/18 20:34:03 INFO DAGScheduler: failed: Set()
21/03/18 20:34:03 INFO DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[24] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:03 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 7.1 KB, free 912.2 MB)
21/03/18 20:34:03 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 3.8 KB, free 912.2 MB)
21/03/18 20:34:03 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:64427 (size: 3.8 KB, free: 912.3 MB)
21/03/18 20:34:03 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:03 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 6 (MapPartitionsRDD[24] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:03 INFO TaskSchedulerImpl: Adding task set 6.0 with 1 tasks
21/03/18 20:34:03 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 6, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:34:03 INFO Executor: Running task 0.0 in stage 6.0 (TID 6)
21/03/18 20:34:03 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:34:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/03/18 20:34:03 INFO Executor: Finished task 0.0 in stage 6.0 (TID 6). 1782 bytes result sent to driver
21/03/18 20:34:03 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 6) in 6 ms on localhost (executor driver) (1/1)
21/03/18 20:34:03 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 
21/03/18 20:34:03 INFO DAGScheduler: ResultStage 6 (count at utils.scala:135) finished in 0.012 s
21/03/18 20:34:03 INFO DAGScheduler: Job 4 finished: count at utils.scala:135, took 0.035717 s
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 206
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 67
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 172
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 201
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 97
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 42
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 126
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 166
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 74
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 167
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 171
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 34
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 177
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 31
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 95
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 181
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 164
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 103
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 124
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 33
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 196
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 79
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 91
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 142
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 84
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 94
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 147
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 77
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 168
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 190
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 86
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 59
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 26
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 207
21/03/18 20:34:04 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:64427 in memory (size: 3.3 KB, free: 912.3 MB)
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 176
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 83
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 63
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 175
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 96
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 187
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 49
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 50
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 99
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 114
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 80
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 56
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 204
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 151
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 37
21/03/18 20:34:04 INFO BlockManagerInfo: Removed broadcast_3_piece0 on localhost:64427 in memory (size: 3.8 KB, free: 912.3 MB)
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 200
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 180
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 184
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 36
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 134
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 139
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 122
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 179
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 32
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 53
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 194
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 89
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 161
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 160
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 156
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 60
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 54
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 69
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 162
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 92
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 46
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 123
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 144
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 163
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 159
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 44
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 110
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 137
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 135
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 29
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 153
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 65
21/03/18 20:34:04 INFO BlockManagerInfo: Removed broadcast_5_piece0 on localhost:64427 in memory (size: 4.5 KB, free: 912.3 MB)
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 100
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 183
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 188
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 197
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 141
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 68
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 193
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 210
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 202
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 185
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 170
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 117
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 76
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 111
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 174
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 208
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 150
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 71
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 75
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 133
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 165
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 45
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 106
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 182
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 57
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 154
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 81
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 143
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 148
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 27
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 132
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 173
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 195
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 169
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 152
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 198
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 104
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 39
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 102
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 108
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 28
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 115
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 85
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 30
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 38
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 55
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 82
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 125
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 101
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 209
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 145
21/03/18 20:34:04 INFO BlockManagerInfo: Removed broadcast_4_piece0 on localhost:64427 in memory (size: 3.3 KB, free: 912.3 MB)
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 128
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 58
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 158
21/03/18 20:34:04 INFO ContextCleaner: Cleaned shuffle 1
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 186
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 113
21/03/18 20:34:04 INFO BlockManagerInfo: Removed broadcast_6_piece0 on localhost:64427 in memory (size: 3.8 KB, free: 912.3 MB)
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 61
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 129
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 140
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 121
21/03/18 20:34:04 INFO BlockManagerInfo: Removed broadcast_2_piece0 on localhost:64427 in memory (size: 4.5 KB, free: 912.3 MB)
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 107
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 109
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 47
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 192
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 146
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 138
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 70
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 52
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 120
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 93
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 157
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 199
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 119
21/03/18 20:34:04 INFO CodeGenerator: Code generated in 84.342607 ms
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 48
21/03/18 20:34:04 INFO ContextCleaner: Cleaned shuffle 0
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 155
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 149
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 43
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 51
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 35
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 131
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 189
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 64
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 41
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 127
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 205
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 88
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 105
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 178
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 40
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 191
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 130
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 72
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 66
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 112
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 62
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 90
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 116
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 203
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 78
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 136
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 98
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 73
21/03/18 20:34:04 INFO ContextCleaner: Cleaned accumulator 87
21/03/18 20:34:04 INFO CodeGenerator: Code generated in 18.994347 ms
21/03/18 20:34:04 INFO CodeGenerator: Code generated in 14.585215 ms
21/03/18 20:34:04 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:34:04 INFO DAGScheduler: Registering RDD 28 (count at utils.scala:135)
21/03/18 20:34:04 INFO DAGScheduler: Got job 5 (count at utils.scala:135) with 1 output partitions
21/03/18 20:34:04 INFO DAGScheduler: Final stage: ResultStage 8 (count at utils.scala:135)
21/03/18 20:34:04 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 7)
21/03/18 20:34:04 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 7)
21/03/18 20:34:04 INFO DAGScheduler: Submitting ShuffleMapStage 7 (MapPartitionsRDD[28] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:04 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 7.9 KB, free 912.3 MB)
21/03/18 20:34:04 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 4.2 KB, free 912.3 MB)
21/03/18 20:34:04 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:64427 (size: 4.2 KB, free: 912.3 MB)
21/03/18 20:34:04 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:04 INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 7 (MapPartitionsRDD[28] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
21/03/18 20:34:04 INFO TaskSchedulerImpl: Adding task set 7.0 with 4 tasks
21/03/18 20:34:04 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 7, localhost, executor driver, partition 0, PROCESS_LOCAL, 8034 bytes)
21/03/18 20:34:04 INFO TaskSetManager: Starting task 1.0 in stage 7.0 (TID 8, localhost, executor driver, partition 1, PROCESS_LOCAL, 8051 bytes)
21/03/18 20:34:04 INFO TaskSetManager: Starting task 2.0 in stage 7.0 (TID 9, localhost, executor driver, partition 2, PROCESS_LOCAL, 8051 bytes)
21/03/18 20:34:04 INFO TaskSetManager: Starting task 3.0 in stage 7.0 (TID 10, localhost, executor driver, partition 3, PROCESS_LOCAL, 8051 bytes)
21/03/18 20:34:04 INFO Executor: Running task 0.0 in stage 7.0 (TID 7)
21/03/18 20:34:04 INFO Executor: Running task 1.0 in stage 7.0 (TID 8)
21/03/18 20:34:04 INFO Executor: Running task 2.0 in stage 7.0 (TID 9)
21/03/18 20:34:04 INFO Executor: Running task 3.0 in stage 7.0 (TID 10)
21/03/18 20:34:04 INFO Executor: Finished task 0.0 in stage 7.0 (TID 7). 1499 bytes result sent to driver
21/03/18 20:34:04 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 7) in 15 ms on localhost (executor driver) (1/4)
21/03/18 20:34:04 INFO Executor: Finished task 3.0 in stage 7.0 (TID 10). 1499 bytes result sent to driver
21/03/18 20:34:04 INFO Executor: Finished task 1.0 in stage 7.0 (TID 8). 1499 bytes result sent to driver
21/03/18 20:34:04 INFO TaskSetManager: Finished task 3.0 in stage 7.0 (TID 10) in 14 ms on localhost (executor driver) (2/4)
21/03/18 20:34:04 INFO Executor: Finished task 2.0 in stage 7.0 (TID 9). 1499 bytes result sent to driver
21/03/18 20:34:04 INFO TaskSetManager: Finished task 1.0 in stage 7.0 (TID 8) in 15 ms on localhost (executor driver) (3/4)
21/03/18 20:34:04 INFO TaskSetManager: Finished task 2.0 in stage 7.0 (TID 9) in 14 ms on localhost (executor driver) (4/4)
21/03/18 20:34:04 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 
21/03/18 20:34:04 INFO DAGScheduler: ShuffleMapStage 7 (count at utils.scala:135) finished in 0.025 s
21/03/18 20:34:04 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:34:04 INFO DAGScheduler: running: Set()
21/03/18 20:34:04 INFO DAGScheduler: waiting: Set(ResultStage 8)
21/03/18 20:34:04 INFO DAGScheduler: failed: Set()
21/03/18 20:34:04 INFO DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[31] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:04 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 7.1 KB, free 912.3 MB)
21/03/18 20:34:04 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 3.8 KB, free 912.3 MB)
21/03/18 20:34:04 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:64427 (size: 3.8 KB, free: 912.3 MB)
21/03/18 20:34:04 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:04 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 8 (MapPartitionsRDD[31] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:04 INFO TaskSchedulerImpl: Adding task set 8.0 with 1 tasks
21/03/18 20:34:04 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 11, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:34:04 INFO Executor: Running task 0.0 in stage 8.0 (TID 11)
21/03/18 20:34:04 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
21/03/18 20:34:04 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/03/18 20:34:04 INFO Executor: Finished task 0.0 in stage 8.0 (TID 11). 1782 bytes result sent to driver
21/03/18 20:34:04 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 11) in 9 ms on localhost (executor driver) (1/1)
21/03/18 20:34:04 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 
21/03/18 20:34:04 INFO DAGScheduler: ResultStage 8 (count at utils.scala:135) finished in 0.016 s
21/03/18 20:34:04 INFO DAGScheduler: Job 5 finished: count at utils.scala:135, took 0.048519 s
21/03/18 20:34:04 INFO SparkContext: Invoking stop() from shutdown hook
21/03/18 20:34:04 INFO SparkUI: Stopped Spark web UI at http://localhost:4040
21/03/18 20:34:04 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/03/18 20:34:04 INFO MemoryStore: MemoryStore cleared
21/03/18 20:34:04 INFO BlockManager: BlockManager stopped
21/03/18 20:34:04 INFO BlockManagerMaster: BlockManagerMaster stopped
21/03/18 20:34:04 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/03/18 20:34:04 INFO SparkContext: Successfully stopped SparkContext
21/03/18 20:34:04 INFO ShutdownHookManager: Shutdown hook called
21/03/18 20:34:04 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-c6dc1d5f-a8e9-4cdf-a115-cfb683ce166d
21/03/18 20:34:04 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-99bf9fcc-8711-4e72-93ba-422f436395dc
21/03/18 20:34:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/03/18 20:34:08 INFO SparkContext: Running Spark version 2.4.3
21/03/18 20:34:08 INFO SparkContext: Submitted application: sparklyr
21/03/18 20:34:08 INFO SecurityManager: Changing view acls to: nathan
21/03/18 20:34:08 INFO SecurityManager: Changing modify acls to: nathan
21/03/18 20:34:08 INFO SecurityManager: Changing view acls groups to: 
21/03/18 20:34:08 INFO SecurityManager: Changing modify acls groups to: 
21/03/18 20:34:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(nathan); groups with view permissions: Set(); users  with modify permissions: Set(nathan); groups with modify permissions: Set()
21/03/18 20:34:08 INFO Utils: Successfully started service 'sparkDriver' on port 64587.
21/03/18 20:34:08 INFO SparkEnv: Registering MapOutputTracker
21/03/18 20:34:08 INFO SparkEnv: Registering BlockManagerMaster
21/03/18 20:34:08 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/03/18 20:34:08 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/03/18 20:34:08 INFO DiskBlockManager: Created local directory at /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/blockmgr-d660fdc3-5363-41e4-8f27-7c329144736f
21/03/18 20:34:08 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
21/03/18 20:34:08 INFO SparkEnv: Registering OutputCommitCoordinator
21/03/18 20:34:08 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/03/18 20:34:08 INFO SparkUI: Bound SparkUI to 127.0.0.1, and started at http://localhost:4040
21/03/18 20:34:08 INFO SparkContext: Added JAR file:/Library/Frameworks/R.framework/Versions/4.0/Resources/library/sparklyr/java/sparklyr-2.4-2.11.jar at spark://localhost:64587/jars/sparklyr-2.4-2.11.jar with timestamp 1616096048951
21/03/18 20:34:09 INFO Executor: Starting executor ID driver on host localhost
21/03/18 20:34:09 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 64589.
21/03/18 20:34:09 INFO NettyBlockTransferService: Server created on localhost:64589
21/03/18 20:34:09 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/03/18 20:34:09 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, localhost, 64589, None)
21/03/18 20:34:09 INFO BlockManagerMasterEndpoint: Registering block manager localhost:64589 with 912.3 MB RAM, BlockManagerId(driver, localhost, 64589, None)
21/03/18 20:34:09 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, localhost, 64589, None)
21/03/18 20:34:09 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, localhost, 64589, None)
21/03/18 20:34:09 INFO SharedState: loading hive config file: file:/Users/nathan/spark/spark-2.4.3-bin-hadoop2.7/conf/hive-site.xml
21/03/18 20:34:09 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse').
21/03/18 20:34:09 INFO SharedState: Warehouse path is 'file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse'.
21/03/18 20:34:10 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
21/03/18 20:34:12 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
21/03/18 20:34:12 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
21/03/18 20:34:12 INFO ObjectStore: ObjectStore, initialize called
21/03/18 20:34:12 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
21/03/18 20:34:12 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
21/03/18 20:34:14 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
21/03/18 20:34:15 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:15 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:16 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:16 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:16 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
21/03/18 20:34:16 INFO ObjectStore: Initialized ObjectStore
21/03/18 20:34:17 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
21/03/18 20:34:17 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
21/03/18 20:34:17 INFO HiveMetaStore: Added admin role in metastore
21/03/18 20:34:17 INFO HiveMetaStore: Added public role in metastore
21/03/18 20:34:17 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/03/18 20:34:17 INFO HiveMetaStore: 0: get_all_databases
21/03/18 20:34:17 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_all_databases	
21/03/18 20:34:17 INFO HiveMetaStore: 0: get_functions: db=default pat=*
21/03/18 20:34:17 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_functions: db=default pat=*	
21/03/18 20:34:17 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:17 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/f6fa4dd8-1db4-4b6a-9c3b-b570964eb02f_resources
21/03/18 20:34:17 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/f6fa4dd8-1db4-4b6a-9c3b-b570964eb02f
21/03/18 20:34:17 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/nathan/f6fa4dd8-1db4-4b6a-9c3b-b570964eb02f
21/03/18 20:34:17 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/f6fa4dd8-1db4-4b6a-9c3b-b570964eb02f/_tmp_space.db
21/03/18 20:34:17 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse
21/03/18 20:34:17 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:17 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:17 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:17 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:17 INFO HiveMetaStore: 0: get_database: fake_database
21/03/18 20:34:17 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: fake_database	
21/03/18 20:34:17 WARN ObjectStore: Failed to get database fake_database, returning NoSuchObjectException
21/03/18 20:34:17 INFO HiveMetaStore: 0: get_databases: *
21/03/18 20:34:17 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_databases: *	
21/03/18 20:34:17 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:17 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:17 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:17 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:18 INFO CodeGenerator: Code generated in 454.265218 ms
21/03/18 20:34:19 INFO CodeGenerator: Code generated in 17.189681 ms
21/03/18 20:34:19 INFO CodeGenerator: Code generated in 18.412767 ms
21/03/18 20:34:20 INFO CodeGenerator: Code generated in 25.838167 ms
21/03/18 20:34:20 INFO CodeGenerator: Code generated in 15.879459 ms
21/03/18 20:34:20 INFO CodeGenerator: Code generated in 10.641992 ms
21/03/18 20:34:20 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:34:20 INFO DAGScheduler: Registering RDD 3 (count at utils.scala:135)
21/03/18 20:34:20 INFO DAGScheduler: Got job 0 (count at utils.scala:135) with 1 output partitions
21/03/18 20:34:20 INFO DAGScheduler: Final stage: ResultStage 1 (count at utils.scala:135)
21/03/18 20:34:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
21/03/18 20:34:20 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
21/03/18 20:34:20 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:20 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 7.9 KB, free 912.3 MB)
21/03/18 20:34:20 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 4.2 KB, free 912.3 MB)
21/03/18 20:34:20 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:64589 (size: 4.2 KB, free: 912.3 MB)
21/03/18 20:34:20 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:20 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:20 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
21/03/18 20:34:20 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 8017 bytes)
21/03/18 20:34:20 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/03/18 20:34:20 INFO Executor: Fetching spark://localhost:64587/jars/sparklyr-2.4-2.11.jar with timestamp 1616096048951
21/03/18 20:34:20 INFO TransportClientFactory: Successfully created connection to localhost/127.0.0.1:64587 after 29 ms (0 ms spent in bootstraps)
21/03/18 20:34:20 INFO Utils: Fetching spark://localhost:64587/jars/sparklyr-2.4-2.11.jar to /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-ea650d55-9fa3-49aa-9024-013c14aeb75e/userFiles-da99043b-4c33-4cf0-8892-894adc8415c3/fetchFileTemp3489694474120782485.tmp
21/03/18 20:34:20 INFO ContextCleaner: Cleaned accumulator 1
21/03/18 20:34:20 INFO Executor: Adding file:/private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-ea650d55-9fa3-49aa-9024-013c14aeb75e/userFiles-da99043b-4c33-4cf0-8892-894adc8415c3/sparklyr-2.4-2.11.jar to class loader
21/03/18 20:34:21 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1585 bytes result sent to driver
21/03/18 20:34:21 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 516 ms on localhost (executor driver) (1/1)
21/03/18 20:34:21 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
21/03/18 20:34:21 INFO DAGScheduler: ShuffleMapStage 0 (count at utils.scala:135) finished in 0.781 s
21/03/18 20:34:21 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:34:21 INFO DAGScheduler: running: Set()
21/03/18 20:34:21 INFO DAGScheduler: waiting: Set(ResultStage 1)
21/03/18 20:34:21 INFO DAGScheduler: failed: Set()
21/03/18 20:34:21 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[6] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:21 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 7.1 KB, free 912.3 MB)
21/03/18 20:34:21 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.8 KB, free 912.3 MB)
21/03/18 20:34:21 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:64589 (size: 3.8 KB, free: 912.3 MB)
21/03/18 20:34:21 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:21 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[6] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:21 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
21/03/18 20:34:21 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:34:21 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
21/03/18 20:34:21 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:34:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 10 ms
21/03/18 20:34:21 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1825 bytes result sent to driver
21/03/18 20:34:21 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 58 ms on localhost (executor driver) (1/1)
21/03/18 20:34:21 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
21/03/18 20:34:21 INFO DAGScheduler: ResultStage 1 (count at utils.scala:135) finished in 0.074 s
21/03/18 20:34:21 INFO DAGScheduler: Job 0 finished: count at utils.scala:135, took 0.959193 s
21/03/18 20:34:21 INFO HiveMetaStore: 0: get_database: global_temp
21/03/18 20:34:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: global_temp	
21/03/18 20:34:21 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
21/03/18 20:34:21 INFO HiveMetaStore: 0: create_database: Database(name:catalog_test_database_db, description:, locationUri:file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse/catalog_test_database_db.db, parameters:{})
21/03/18 20:34:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=create_database: Database(name:catalog_test_database_db, description:, locationUri:file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse/catalog_test_database_db.db, parameters:{})	
21/03/18 20:34:21 WARN ObjectStore: Failed to get database catalog_test_database_db, returning NoSuchObjectException
21/03/18 20:34:21 INFO FileUtils: Creating directory if it doesn't exist: file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse/catalog_test_database_db.db
21/03/18 20:34:21 INFO CodeGenerator: Code generated in 8.500392 ms
21/03/18 20:34:21 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:34:21 INFO DAGScheduler: Registering RDD 10 (count at utils.scala:135)
21/03/18 20:34:21 INFO DAGScheduler: Got job 1 (count at utils.scala:135) with 1 output partitions
21/03/18 20:34:21 INFO DAGScheduler: Final stage: ResultStage 3 (count at utils.scala:135)
21/03/18 20:34:21 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
21/03/18 20:34:21 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 2)
21/03/18 20:34:21 INFO DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[10] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:21 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 7.9 KB, free 912.3 MB)
21/03/18 20:34:21 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 4.2 KB, free 912.3 MB)
21/03/18 20:34:21 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:64589 (size: 4.2 KB, free: 912.3 MB)
21/03/18 20:34:21 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:21 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[10] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:21 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
21/03/18 20:34:21 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, executor driver, partition 0, PROCESS_LOCAL, 7882 bytes)
21/03/18 20:34:21 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
21/03/18 20:34:21 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1499 bytes result sent to driver
21/03/18 20:34:21 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 13 ms on localhost (executor driver) (1/1)
21/03/18 20:34:21 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 
21/03/18 20:34:21 INFO DAGScheduler: ShuffleMapStage 2 (count at utils.scala:135) finished in 0.021 s
21/03/18 20:34:21 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:34:21 INFO DAGScheduler: running: Set()
21/03/18 20:34:21 INFO DAGScheduler: waiting: Set(ResultStage 3)
21/03/18 20:34:21 INFO DAGScheduler: failed: Set()
21/03/18 20:34:21 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[13] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:21 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 7.1 KB, free 912.3 MB)
21/03/18 20:34:21 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 3.8 KB, free 912.3 MB)
21/03/18 20:34:21 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:64589 (size: 3.8 KB, free: 912.3 MB)
21/03/18 20:34:21 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:21 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[13] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:21 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
21/03/18 20:34:21 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:34:21 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
21/03/18 20:34:21 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:34:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/03/18 20:34:21 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1775 bytes result sent to driver
21/03/18 20:34:21 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 9 ms on localhost (executor driver) (1/1)
21/03/18 20:34:21 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 
21/03/18 20:34:21 INFO DAGScheduler: ResultStage 3 (count at utils.scala:135) finished in 0.017 s
21/03/18 20:34:21 INFO DAGScheduler: Job 1 finished: count at utils.scala:135, took 0.043734 s
21/03/18 20:34:21 INFO HiveMetaStore: 0: get_database: catalog_test_database_db
21/03/18 20:34:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: catalog_test_database_db	
21/03/18 20:34:21 INFO HiveMetaStore: 0: get_database: catalog_test_database_db
21/03/18 20:34:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: catalog_test_database_db	
21/03/18 20:34:21 INFO HiveMetaStore: 0: get_database: catalog_test_database_db
21/03/18 20:34:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: catalog_test_database_db	
21/03/18 20:34:21 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:21 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:21 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:21 INFO HiveMetaStore: 0: get_database: catalog_test_database_db
21/03/18 20:34:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: catalog_test_database_db	
21/03/18 20:34:21 INFO HiveMetaStore: 0: drop_database: catalog_test_database_db
21/03/18 20:34:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=drop_database: catalog_test_database_db	
21/03/18 20:34:21 INFO HiveMetaStore: 0: get_all_tables: db=catalog_test_database_db
21/03/18 20:34:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_all_tables: db=catalog_test_database_db	
21/03/18 20:34:21 INFO HiveMetaStore: 0: get_functions: db=catalog_test_database_db pat=*
21/03/18 20:34:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_functions: db=catalog_test_database_db pat=*	
21/03/18 20:34:21 INFO ObjectStore: Dropping database catalog_test_database_db along with all tables
21/03/18 20:34:22 INFO hivemetastoressimpl: deleting  file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse/catalog_test_database_db.db
21/03/18 20:34:22 INFO deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
21/03/18 20:34:22 INFO TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
21/03/18 20:34:22 INFO hivemetastoressimpl: Deleted the diretory file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse/catalog_test_database_db.db
21/03/18 20:34:22 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:34:22 INFO DAGScheduler: Registering RDD 17 (count at utils.scala:135)
21/03/18 20:34:22 INFO DAGScheduler: Got job 2 (count at utils.scala:135) with 1 output partitions
21/03/18 20:34:22 INFO DAGScheduler: Final stage: ResultStage 5 (count at utils.scala:135)
21/03/18 20:34:22 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4)
21/03/18 20:34:22 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 4)
21/03/18 20:34:22 INFO DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[17] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:22 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 7.9 KB, free 912.2 MB)
21/03/18 20:34:22 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 4.2 KB, free 912.2 MB)
21/03/18 20:34:22 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:64589 (size: 4.2 KB, free: 912.3 MB)
21/03/18 20:34:22 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:22 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[17] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:22 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
21/03/18 20:34:22 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, localhost, executor driver, partition 0, PROCESS_LOCAL, 7882 bytes)
21/03/18 20:34:22 INFO Executor: Running task 0.0 in stage 4.0 (TID 4)
21/03/18 20:34:22 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4). 1499 bytes result sent to driver
21/03/18 20:34:22 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 11 ms on localhost (executor driver) (1/1)
21/03/18 20:34:22 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 
21/03/18 20:34:22 INFO DAGScheduler: ShuffleMapStage 4 (count at utils.scala:135) finished in 0.021 s
21/03/18 20:34:22 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:34:22 INFO DAGScheduler: running: Set()
21/03/18 20:34:22 INFO DAGScheduler: waiting: Set(ResultStage 5)
21/03/18 20:34:22 INFO DAGScheduler: failed: Set()
21/03/18 20:34:22 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[20] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:22 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 7.1 KB, free 912.2 MB)
21/03/18 20:34:22 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 3.8 KB, free 912.2 MB)
21/03/18 20:34:22 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:64589 (size: 3.8 KB, free: 912.3 MB)
21/03/18 20:34:22 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:22 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[20] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:22 INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
21/03/18 20:34:22 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:34:22 INFO Executor: Running task 0.0 in stage 5.0 (TID 5)
21/03/18 20:34:22 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:34:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/03/18 20:34:22 INFO Executor: Finished task 0.0 in stage 5.0 (TID 5). 1775 bytes result sent to driver
21/03/18 20:34:22 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 8 ms on localhost (executor driver) (1/1)
21/03/18 20:34:22 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
21/03/18 20:34:22 INFO DAGScheduler: ResultStage 5 (count at utils.scala:135) finished in 0.016 s
21/03/18 20:34:22 INFO DAGScheduler: Job 2 finished: count at utils.scala:135, took 0.041356 s
21/03/18 20:34:22 INFO SparkContext: Invoking stop() from shutdown hook
21/03/18 20:34:22 INFO SparkUI: Stopped Spark web UI at http://localhost:4040
21/03/18 20:34:22 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/03/18 20:34:22 INFO MemoryStore: MemoryStore cleared
21/03/18 20:34:22 INFO BlockManager: BlockManager stopped
21/03/18 20:34:22 INFO BlockManagerMaster: BlockManagerMaster stopped
21/03/18 20:34:22 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/03/18 20:34:22 INFO SparkContext: Successfully stopped SparkContext
21/03/18 20:34:22 INFO ShutdownHookManager: Shutdown hook called
21/03/18 20:34:22 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-ea650d55-9fa3-49aa-9024-013c14aeb75e
21/03/18 20:34:22 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-734ece7a-5d78-42e0-8ea6-6fb0aa593732
21/03/18 20:34:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/03/18 20:34:25 INFO SparkContext: Running Spark version 2.4.3
21/03/18 20:34:25 INFO SparkContext: Submitted application: sparklyr
21/03/18 20:34:25 INFO SecurityManager: Changing view acls to: nathan
21/03/18 20:34:25 INFO SecurityManager: Changing modify acls to: nathan
21/03/18 20:34:25 INFO SecurityManager: Changing view acls groups to: 
21/03/18 20:34:25 INFO SecurityManager: Changing modify acls groups to: 
21/03/18 20:34:25 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(nathan); groups with view permissions: Set(); users  with modify permissions: Set(nathan); groups with modify permissions: Set()
21/03/18 20:34:26 INFO Utils: Successfully started service 'sparkDriver' on port 64650.
21/03/18 20:34:26 INFO SparkEnv: Registering MapOutputTracker
21/03/18 20:34:26 INFO SparkEnv: Registering BlockManagerMaster
21/03/18 20:34:26 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/03/18 20:34:26 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/03/18 20:34:26 INFO DiskBlockManager: Created local directory at /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/blockmgr-6921fcae-4403-485a-9bfb-aba8e6c75035
21/03/18 20:34:26 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
21/03/18 20:34:26 INFO SparkEnv: Registering OutputCommitCoordinator
21/03/18 20:34:26 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/03/18 20:34:26 INFO SparkUI: Bound SparkUI to 127.0.0.1, and started at http://localhost:4040
21/03/18 20:34:26 INFO SparkContext: Added JAR file:/Library/Frameworks/R.framework/Versions/4.0/Resources/library/sparklyr/java/sparklyr-2.4-2.11.jar at spark://localhost:64650/jars/sparklyr-2.4-2.11.jar with timestamp 1616096066424
21/03/18 20:34:26 INFO Executor: Starting executor ID driver on host localhost
21/03/18 20:34:26 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 64651.
21/03/18 20:34:26 INFO NettyBlockTransferService: Server created on localhost:64651
21/03/18 20:34:26 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/03/18 20:34:26 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, localhost, 64651, None)
21/03/18 20:34:26 INFO BlockManagerMasterEndpoint: Registering block manager localhost:64651 with 912.3 MB RAM, BlockManagerId(driver, localhost, 64651, None)
21/03/18 20:34:26 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, localhost, 64651, None)
21/03/18 20:34:26 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, localhost, 64651, None)
21/03/18 20:34:26 INFO SharedState: loading hive config file: file:/Users/nathan/spark/spark-2.4.3-bin-hadoop2.7/conf/hive-site.xml
21/03/18 20:34:26 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse').
21/03/18 20:34:26 INFO SharedState: Warehouse path is 'file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse'.
21/03/18 20:34:27 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
21/03/18 20:34:29 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
21/03/18 20:34:30 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
21/03/18 20:34:30 INFO ObjectStore: ObjectStore, initialize called
21/03/18 20:34:31 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
21/03/18 20:34:31 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
21/03/18 20:34:32 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
21/03/18 20:34:33 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:33 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:34 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:34 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:34 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
21/03/18 20:34:34 INFO ObjectStore: Initialized ObjectStore
21/03/18 20:34:34 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
21/03/18 20:34:34 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
21/03/18 20:34:34 INFO HiveMetaStore: Added admin role in metastore
21/03/18 20:34:34 INFO HiveMetaStore: Added public role in metastore
21/03/18 20:34:34 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/03/18 20:34:34 INFO HiveMetaStore: 0: get_all_databases
21/03/18 20:34:34 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_all_databases	
21/03/18 20:34:34 INFO HiveMetaStore: 0: get_functions: db=default pat=*
21/03/18 20:34:34 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_functions: db=default pat=*	
21/03/18 20:34:34 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:34 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/97b0a393-7e76-45fb-b2bc-af539167fb55_resources
21/03/18 20:34:34 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/97b0a393-7e76-45fb-b2bc-af539167fb55
21/03/18 20:34:34 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/nathan/97b0a393-7e76-45fb-b2bc-af539167fb55
21/03/18 20:34:34 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/97b0a393-7e76-45fb-b2bc-af539167fb55/_tmp_space.db
21/03/18 20:34:34 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse
21/03/18 20:34:35 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:35 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:35 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:35 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:35 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:35 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:35 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:35 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:35 INFO HiveMetaStore: 0: get_function: default.catalog_fake_function
21/03/18 20:34:35 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_function: default.catalog_fake_function	
21/03/18 20:34:35 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:35 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:35 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:35 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:35 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:35 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:35 INFO HiveMetaStore: 0: get_functions: db=default pat=*
21/03/18 20:34:35 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_functions: db=default pat=*	
21/03/18 20:34:36 INFO CodeGenerator: Code generated in 242.986215 ms
21/03/18 20:34:36 INFO CodeGenerator: Code generated in 19.897839 ms
21/03/18 20:34:36 INFO CodeGenerator: Code generated in 15.53041 ms
21/03/18 20:34:37 INFO CodeGenerator: Code generated in 39.898516 ms
21/03/18 20:34:37 INFO CodeGenerator: Code generated in 13.788436 ms
21/03/18 20:34:37 INFO CodeGenerator: Code generated in 7.413068 ms
21/03/18 20:34:37 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:34:37 INFO DAGScheduler: Registering RDD 3 (count at utils.scala:135)
21/03/18 20:34:37 INFO DAGScheduler: Got job 0 (count at utils.scala:135) with 1 output partitions
21/03/18 20:34:37 INFO DAGScheduler: Final stage: ResultStage 1 (count at utils.scala:135)
21/03/18 20:34:37 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
21/03/18 20:34:37 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
21/03/18 20:34:37 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:37 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 7.9 KB, free 912.3 MB)
21/03/18 20:34:38 INFO ContextCleaner: Cleaned accumulator 1
21/03/18 20:34:38 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 4.2 KB, free 912.3 MB)
21/03/18 20:34:38 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:64651 (size: 4.2 KB, free: 912.3 MB)
21/03/18 20:34:38 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:38 INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
21/03/18 20:34:38 INFO TaskSchedulerImpl: Adding task set 0.0 with 4 tasks
21/03/18 20:34:38 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 9258 bytes)
21/03/18 20:34:38 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 9275 bytes)
21/03/18 20:34:38 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, executor driver, partition 2, PROCESS_LOCAL, 9258 bytes)
21/03/18 20:34:38 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, executor driver, partition 3, PROCESS_LOCAL, 9275 bytes)
21/03/18 20:34:38 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/03/18 20:34:38 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
21/03/18 20:34:38 INFO Executor: Running task 3.0 in stage 0.0 (TID 3)
21/03/18 20:34:38 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
21/03/18 20:34:38 INFO Executor: Fetching spark://localhost:64650/jars/sparklyr-2.4-2.11.jar with timestamp 1616096066424
21/03/18 20:34:38 INFO TransportClientFactory: Successfully created connection to localhost/127.0.0.1:64650 after 21 ms (0 ms spent in bootstraps)
21/03/18 20:34:38 INFO Utils: Fetching spark://localhost:64650/jars/sparklyr-2.4-2.11.jar to /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-933c0793-365f-47c3-937f-3e2d8259253f/userFiles-12131268-b68c-43ed-9e4f-fc545dbd5f85/fetchFileTemp2423524123755880576.tmp
21/03/18 20:34:38 INFO Executor: Adding file:/private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-933c0793-365f-47c3-937f-3e2d8259253f/userFiles-12131268-b68c-43ed-9e4f-fc545dbd5f85/sparklyr-2.4-2.11.jar to class loader
21/03/18 20:34:38 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1542 bytes result sent to driver
21/03/18 20:34:38 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 1542 bytes result sent to driver
21/03/18 20:34:38 INFO Executor: Finished task 3.0 in stage 0.0 (TID 3). 1542 bytes result sent to driver
21/03/18 20:34:38 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 1542 bytes result sent to driver
21/03/18 20:34:38 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 296 ms on localhost (executor driver) (1/4)
21/03/18 20:34:38 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 303 ms on localhost (executor driver) (2/4)
21/03/18 20:34:38 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 332 ms on localhost (executor driver) (3/4)
21/03/18 20:34:38 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 301 ms on localhost (executor driver) (4/4)
21/03/18 20:34:38 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
21/03/18 20:34:38 INFO DAGScheduler: ShuffleMapStage 0 (count at utils.scala:135) finished in 0.838 s
21/03/18 20:34:38 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:34:38 INFO DAGScheduler: running: Set()
21/03/18 20:34:38 INFO DAGScheduler: waiting: Set(ResultStage 1)
21/03/18 20:34:38 INFO DAGScheduler: failed: Set()
21/03/18 20:34:38 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[6] at count at utils.scala:135), which has no missing parents
21/03/18 20:34:38 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 7.1 KB, free 912.3 MB)
21/03/18 20:34:38 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.8 KB, free 912.3 MB)
21/03/18 20:34:38 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:64651 (size: 3.8 KB, free: 912.3 MB)
21/03/18 20:34:38 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:38 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[6] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:38 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
21/03/18 20:34:38 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 4, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:34:38 INFO Executor: Running task 0.0 in stage 1.0 (TID 4)
21/03/18 20:34:38 INFO ShuffleBlockFetcherIterator: Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
21/03/18 20:34:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 8 ms
21/03/18 20:34:38 INFO Executor: Finished task 0.0 in stage 1.0 (TID 4). 1782 bytes result sent to driver
21/03/18 20:34:38 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 4) in 53 ms on localhost (executor driver) (1/1)
21/03/18 20:34:38 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
21/03/18 20:34:38 INFO DAGScheduler: ResultStage 1 (count at utils.scala:135) finished in 0.066 s
21/03/18 20:34:38 INFO DAGScheduler: Job 0 finished: count at utils.scala:135, took 0.998054 s
21/03/18 20:34:38 INFO SparkContext: Invoking stop() from shutdown hook
21/03/18 20:34:38 INFO SparkUI: Stopped Spark web UI at http://localhost:4040
21/03/18 20:34:38 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/03/18 20:34:38 INFO MemoryStore: MemoryStore cleared
21/03/18 20:34:38 INFO BlockManager: BlockManager stopped
21/03/18 20:34:38 INFO BlockManagerMaster: BlockManagerMaster stopped
21/03/18 20:34:38 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/03/18 20:34:38 INFO SparkContext: Successfully stopped SparkContext
21/03/18 20:34:38 INFO ShutdownHookManager: Shutdown hook called
21/03/18 20:34:38 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-933c0793-365f-47c3-937f-3e2d8259253f
21/03/18 20:34:38 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-cd70a902-580a-4668-9572-4ef2d6d99be7
21/03/18 20:34:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/03/18 20:34:43 INFO SparkContext: Running Spark version 2.4.3
21/03/18 20:34:43 INFO SparkContext: Submitted application: sparklyr
21/03/18 20:34:43 INFO SecurityManager: Changing view acls to: nathan
21/03/18 20:34:43 INFO SecurityManager: Changing modify acls to: nathan
21/03/18 20:34:43 INFO SecurityManager: Changing view acls groups to: 
21/03/18 20:34:43 INFO SecurityManager: Changing modify acls groups to: 
21/03/18 20:34:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(nathan); groups with view permissions: Set(); users  with modify permissions: Set(nathan); groups with modify permissions: Set()
21/03/18 20:34:43 INFO Utils: Successfully started service 'sparkDriver' on port 64732.
21/03/18 20:34:43 INFO SparkEnv: Registering MapOutputTracker
21/03/18 20:34:43 INFO SparkEnv: Registering BlockManagerMaster
21/03/18 20:34:43 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/03/18 20:34:43 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/03/18 20:34:43 INFO DiskBlockManager: Created local directory at /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/blockmgr-19b20597-3f73-47f3-a374-bec9f0d24861
21/03/18 20:34:43 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
21/03/18 20:34:43 INFO SparkEnv: Registering OutputCommitCoordinator
21/03/18 20:34:44 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/03/18 20:34:44 INFO SparkUI: Bound SparkUI to 127.0.0.1, and started at http://localhost:4040
21/03/18 20:34:44 INFO SparkContext: Added JAR file:/Library/Frameworks/R.framework/Versions/4.0/Resources/library/sparklyr/java/sparklyr-2.4-2.11.jar at spark://localhost:64732/jars/sparklyr-2.4-2.11.jar with timestamp 1616096084268
21/03/18 20:34:44 INFO Executor: Starting executor ID driver on host localhost
21/03/18 20:34:44 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 64733.
21/03/18 20:34:44 INFO NettyBlockTransferService: Server created on localhost:64733
21/03/18 20:34:44 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/03/18 20:34:44 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, localhost, 64733, None)
21/03/18 20:34:44 INFO BlockManagerMasterEndpoint: Registering block manager localhost:64733 with 912.3 MB RAM, BlockManagerId(driver, localhost, 64733, None)
21/03/18 20:34:44 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, localhost, 64733, None)
21/03/18 20:34:44 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, localhost, 64733, None)
21/03/18 20:34:44 INFO SharedState: loading hive config file: file:/Users/nathan/spark/spark-2.4.3-bin-hadoop2.7/conf/hive-site.xml
21/03/18 20:34:44 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse').
21/03/18 20:34:44 INFO SharedState: Warehouse path is 'file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse'.
21/03/18 20:34:45 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
21/03/18 20:34:48 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
21/03/18 20:34:49 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
21/03/18 20:34:49 INFO ObjectStore: ObjectStore, initialize called
21/03/18 20:34:50 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
21/03/18 20:34:50 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
21/03/18 20:34:52 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
21/03/18 20:34:53 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:53 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:54 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:54 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:54 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
21/03/18 20:34:54 INFO ObjectStore: Initialized ObjectStore
21/03/18 20:34:54 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
21/03/18 20:34:54 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
21/03/18 20:34:54 INFO HiveMetaStore: Added admin role in metastore
21/03/18 20:34:54 INFO HiveMetaStore: Added public role in metastore
21/03/18 20:34:54 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/03/18 20:34:54 INFO HiveMetaStore: 0: get_all_databases
21/03/18 20:34:54 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_all_databases	
21/03/18 20:34:55 INFO HiveMetaStore: 0: get_functions: db=default pat=*
21/03/18 20:34:55 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_functions: db=default pat=*	
21/03/18 20:34:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:34:55 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/b7b1006d-4fc5-4509-88d4-b14e15c2b2c8_resources
21/03/18 20:34:55 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/b7b1006d-4fc5-4509-88d4-b14e15c2b2c8
21/03/18 20:34:55 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/nathan/b7b1006d-4fc5-4509-88d4-b14e15c2b2c8
21/03/18 20:34:55 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/b7b1006d-4fc5-4509-88d4-b14e15c2b2c8/_tmp_space.db
21/03/18 20:34:55 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse
21/03/18 20:34:55 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:55 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:55 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:34:55 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:34:56 INFO FileSourceStrategy: Pruning directories with: 
21/03/18 20:34:56 INFO FileSourceStrategy: Post-Scan Filters: (length(trim(value#0, None)) > 0)
21/03/18 20:34:56 INFO FileSourceStrategy: Output Data Schema: struct<value: string>
21/03/18 20:34:56 INFO FileSourceScanExec: Pushed Filters: 
21/03/18 20:34:56 INFO CodeGenerator: Code generated in 231.460228 ms
21/03/18 20:34:56 INFO CodeGenerator: Code generated in 49.182181 ms
21/03/18 20:34:56 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 284.3 KB, free 912.0 MB)
21/03/18 20:34:56 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.9 KB, free 912.0 MB)
21/03/18 20:34:56 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:64733 (size: 23.9 KB, free: 912.3 MB)
21/03/18 20:34:56 INFO SparkContext: Created broadcast 0 from createTable at NativeMethodAccessorImpl.java:0
21/03/18 20:34:56 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
21/03/18 20:34:57 INFO SparkContext: Starting job: createTable at NativeMethodAccessorImpl.java:0
21/03/18 20:34:57 INFO DAGScheduler: Got job 0 (createTable at NativeMethodAccessorImpl.java:0) with 1 output partitions
21/03/18 20:34:57 INFO DAGScheduler: Final stage: ResultStage 0 (createTable at NativeMethodAccessorImpl.java:0)
21/03/18 20:34:57 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:34:57 INFO DAGScheduler: Missing parents: List()
21/03/18 20:34:57 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at createTable at NativeMethodAccessorImpl.java:0), which has no missing parents
21/03/18 20:34:57 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 8.8 KB, free 912.0 MB)
21/03/18 20:34:57 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 4.5 KB, free 912.0 MB)
21/03/18 20:34:57 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:64733 (size: 4.5 KB, free: 912.3 MB)
21/03/18 20:34:57 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:57 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at createTable at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:57 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
21/03/18 20:34:57 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 8359 bytes)
21/03/18 20:34:57 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/03/18 20:34:57 INFO Executor: Fetching spark://localhost:64732/jars/sparklyr-2.4-2.11.jar with timestamp 1616096084268
21/03/18 20:34:57 INFO TransportClientFactory: Successfully created connection to localhost/127.0.0.1:64732 after 28 ms (0 ms spent in bootstraps)
21/03/18 20:34:57 INFO Utils: Fetching spark://localhost:64732/jars/sparklyr-2.4-2.11.jar to /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-8d1c28c4-fc73-4925-82fc-2cd6119308a5/userFiles-51d677b3-f4d0-44ac-b9a1-52468c0e7d7c/fetchFileTemp9050883508434799197.tmp
21/03/18 20:34:57 INFO Executor: Adding file:/private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-8d1c28c4-fc73-4925-82fc-2cd6119308a5/userFiles-51d677b3-f4d0-44ac-b9a1-52468c0e7d7c/sparklyr-2.4-2.11.jar to class loader
21/03/18 20:34:57 INFO FileScanRDD: Reading File path: file:///var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/RtmpxYR9UR/file4c90b98648, range: 0-1303, partition values: [empty row]
21/03/18 20:34:57 INFO CodeGenerator: Code generated in 18.420765 ms
21/03/18 20:34:57 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1372 bytes result sent to driver
21/03/18 20:34:57 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 412 ms on localhost (executor driver) (1/1)
21/03/18 20:34:57 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
21/03/18 20:34:57 INFO DAGScheduler: ResultStage 0 (createTable at NativeMethodAccessorImpl.java:0) finished in 0.567 s
21/03/18 20:34:57 INFO DAGScheduler: Job 0 finished: createTable at NativeMethodAccessorImpl.java:0, took 0.817437 s
21/03/18 20:34:57 INFO FileSourceStrategy: Pruning directories with: 
21/03/18 20:34:57 INFO FileSourceStrategy: Post-Scan Filters: 
21/03/18 20:34:57 INFO FileSourceStrategy: Output Data Schema: struct<value: string>
21/03/18 20:34:57 INFO FileSourceScanExec: Pushed Filters: 
21/03/18 20:34:57 INFO CodeGenerator: Code generated in 9.69223 ms
21/03/18 20:34:58 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 284.3 KB, free 911.7 MB)
21/03/18 20:34:58 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 23.9 KB, free 911.7 MB)
21/03/18 20:34:58 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:64733 (size: 23.9 KB, free: 912.2 MB)
21/03/18 20:34:58 INFO SparkContext: Created broadcast 2 from createTable at NativeMethodAccessorImpl.java:0
21/03/18 20:34:58 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
21/03/18 20:34:58 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:58 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:34:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:34:58 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:58 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:34:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:34:58 WARN HiveExternalCatalog: Couldn't find corresponding Hive SerDe for data source provider csv. Persisting data source table `default`.`mtcars` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive.
21/03/18 20:34:58 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:34:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:34:58 INFO HiveMetaStore: 0: create_table: Table(tableName:mtcars, dbName:default, owner:nathan, createTime:1616096087, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col, type:array<string>, comment:from deserializer)], location:file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse/mtcars-__PLACEHOLDER__, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{path=file:/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/RtmpxYR9UR/file4c90b98648, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{EXTERNAL=TRUE, spark.sql.sources.schema.part.0={"type":"struct","fields":[{"name":"_c0","type":"string","nullable":true,"metadata":{}},{"name":"_c1","type":"string","nullable":true,"metadata":{}},{"name":"_c2","type":"string","nullable":true,"metadata":{}},{"name":"_c3","type":"string","nullable":true,"metadata":{}},{"name":"_c4","type":"string","nullable":true,"metadata":{}},{"name":"_c5","type":"string","nullable":true,"metadata":{}},{"name":"_c6","type":"string","nullable":true,"metadata":{}},{"name":"_c7","type":"string","nullable":true,"metadata":{}},{"name":"_c8","type":"string","nullable":true,"metadata":{}},{"name":"_c9","type":"string","nullable":true,"metadata":{}},{"name":"_c10","type":"string","nullable":true,"metadata":{}}]}, spark.sql.sources.schema.numParts=1, spark.sql.sources.provider=csv, spark.sql.create.version=2.4.3}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{}, groupPrivileges:null, rolePrivileges:null))
21/03/18 20:34:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=create_table: Table(tableName:mtcars, dbName:default, owner:nathan, createTime:1616096087, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col, type:array<string>, comment:from deserializer)], location:file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse/mtcars-__PLACEHOLDER__, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{path=file:/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/RtmpxYR9UR/file4c90b98648, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{EXTERNAL=TRUE, spark.sql.sources.schema.part.0={"type":"struct","fields":[{"name":"_c0","type":"string","nullable":true,"metadata":{}},{"name":"_c1","type":"string","nullable":true,"metadata":{}},{"name":"_c2","type":"string","nullable":true,"metadata":{}},{"name":"_c3","type":"string","nullable":true,"metadata":{}},{"name":"_c4","type":"string","nullable":true,"metadata":{}},{"name":"_c5","type":"string","nullable":true,"metadata":{}},{"name":"_c6","type":"string","nullable":true,"metadata":{}},{"name":"_c7","type":"string","nullable":true,"metadata":{}},{"name":"_c8","type":"string","nullable":true,"metadata":{}},{"name":"_c9","type":"string","nullable":true,"metadata":{}},{"name":"_c10","type":"string","nullable":true,"metadata":{}}]}, spark.sql.sources.schema.numParts=1, spark.sql.sources.provider=csv, spark.sql.create.version=2.4.3}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{}, groupPrivileges:null, rolePrivileges:null))	
21/03/18 20:34:58 INFO FileUtils: Creating directory if it doesn't exist: file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse/mtcars-__PLACEHOLDER__
21/03/18 20:34:58 INFO HiveMetaStore: 0: get_database: global_temp
21/03/18 20:34:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: global_temp	
21/03/18 20:34:58 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
21/03/18 20:34:58 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:34:58 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:34:59 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:34:59 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:34:59 INFO CodeGenerator: Code generated in 19.481848 ms
21/03/18 20:34:59 INFO CodeGenerator: Code generated in 20.46652 ms
21/03/18 20:34:59 INFO SparkContext: Starting job: collect at utils.scala:135
21/03/18 20:34:59 INFO DAGScheduler: Got job 1 (collect at utils.scala:135) with 1 output partitions
21/03/18 20:34:59 INFO DAGScheduler: Final stage: ResultStage 1 (collect at utils.scala:135)
21/03/18 20:34:59 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:34:59 INFO DAGScheduler: Missing parents: List()
21/03/18 20:34:59 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[13] at collect at utils.scala:135), which has no missing parents
21/03/18 20:34:59 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 6.5 KB, free 911.7 MB)
21/03/18 20:34:59 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 3.3 KB, free 911.7 MB)
21/03/18 20:34:59 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:64733 (size: 3.3 KB, free: 912.2 MB)
21/03/18 20:34:59 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1161
21/03/18 20:34:59 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[13] at collect at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:34:59 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
21/03/18 20:34:59 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, PROCESS_LOCAL, 8080 bytes)
21/03/18 20:34:59 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
21/03/18 20:34:59 INFO CodeGenerator: Code generated in 8.661339 ms
21/03/18 20:34:59 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1329 bytes result sent to driver
21/03/18 20:34:59 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 30 ms on localhost (executor driver) (1/1)
21/03/18 20:34:59 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
21/03/18 20:34:59 INFO DAGScheduler: ResultStage 1 (collect at utils.scala:135) finished in 0.039 s
21/03/18 20:34:59 INFO DAGScheduler: Job 1 finished: collect at utils.scala:135, took 0.044181 s
21/03/18 20:35:00 INFO CodeGenerator: Code generated in 21.86945 ms
21/03/18 20:35:00 INFO CodeGenerator: Code generated in 13.315958 ms
21/03/18 20:35:00 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:35:00 INFO DAGScheduler: Registering RDD 16 (count at utils.scala:135)
21/03/18 20:35:00 INFO DAGScheduler: Got job 2 (count at utils.scala:135) with 1 output partitions
21/03/18 20:35:00 INFO DAGScheduler: Final stage: ResultStage 3 (count at utils.scala:135)
21/03/18 20:35:00 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
21/03/18 20:35:00 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 2)
21/03/18 20:35:00 INFO DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[16] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:00 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 8.4 KB, free 911.7 MB)
21/03/18 20:35:00 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 4.5 KB, free 911.7 MB)
21/03/18 20:35:00 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:64733 (size: 4.5 KB, free: 912.2 MB)
21/03/18 20:35:00 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:00 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[16] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:00 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
21/03/18 20:35:00 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, executor driver, partition 0, PROCESS_LOCAL, 8069 bytes)
21/03/18 20:35:00 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
21/03/18 20:35:00 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1542 bytes result sent to driver
21/03/18 20:35:00 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 89 ms on localhost (executor driver) (1/1)
21/03/18 20:35:00 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 
21/03/18 20:35:00 INFO DAGScheduler: ShuffleMapStage 2 (count at utils.scala:135) finished in 0.108 s
21/03/18 20:35:00 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:35:00 INFO DAGScheduler: running: Set()
21/03/18 20:35:00 INFO DAGScheduler: waiting: Set(ResultStage 3)
21/03/18 20:35:00 INFO DAGScheduler: failed: Set()
21/03/18 20:35:00 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[19] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:00 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 7.1 KB, free 911.7 MB)
21/03/18 20:35:00 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 3.8 KB, free 911.7 MB)
21/03/18 20:35:00 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:64733 (size: 3.8 KB, free: 912.2 MB)
21/03/18 20:35:00 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:00 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[19] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:00 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
21/03/18 20:35:00 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:35:00 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
21/03/18 20:35:00 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:35:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 11 ms
21/03/18 20:35:00 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1782 bytes result sent to driver
21/03/18 20:35:00 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 67 ms on localhost (executor driver) (1/1)
21/03/18 20:35:00 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 
21/03/18 20:35:00 INFO DAGScheduler: ResultStage 3 (count at utils.scala:135) finished in 0.083 s
21/03/18 20:35:00 INFO DAGScheduler: Job 2 finished: count at utils.scala:135, took 0.225182 s
21/03/18 20:35:00 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:00 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:00 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:00 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:00 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:00 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:00 INFO FileSourceStrategy: Pruning directories with: 
21/03/18 20:35:00 INFO FileSourceStrategy: Post-Scan Filters: 
21/03/18 20:35:00 INFO FileSourceStrategy: Output Data Schema: struct<_c0: string, _c1: string, _c2: string, _c3: string, _c4: string ... 9 more fields>
21/03/18 20:35:00 INFO FileSourceScanExec: Pushed Filters: 
21/03/18 20:35:00 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:00 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:00 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:00 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 55
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 44
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 98
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 103
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 125
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 49
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 68
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 113
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 71
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 59
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 111
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 96
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 126
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 118
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 104
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 43
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 119
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 115
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 42
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 79
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 66
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 58
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 92
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 46
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 36
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 47
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 76
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 67
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 122
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 100
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 45
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 52
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 39
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 73
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 89
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 41
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 75
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 83
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 38
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 53
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 116
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 102
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 72
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 50
21/03/18 20:35:00 INFO BlockManagerInfo: Removed broadcast_4_piece0 on localhost:64733 in memory (size: 4.5 KB, free: 912.2 MB)
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 54
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 95
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 56
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 109
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 40
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 60
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 94
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 37
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 117
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 110
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 63
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 99
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 84
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 123
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 93
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 108
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 88
21/03/18 20:35:00 INFO BlockManagerInfo: Removed broadcast_5_piece0 on localhost:64733 in memory (size: 3.8 KB, free: 912.2 MB)
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 61
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 69
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 106
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 114
21/03/18 20:35:00 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:64733 in memory (size: 4.5 KB, free: 912.3 MB)
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 81
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 120
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 121
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 86
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 124
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 48
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 74
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 80
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 105
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 62
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 101
21/03/18 20:35:00 INFO BlockManagerInfo: Removed broadcast_3_piece0 on localhost:64733 in memory (size: 3.3 KB, free: 912.3 MB)
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 65
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 112
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 57
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 78
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 127
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 87
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 82
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 77
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 85
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 70
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 107
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 91
21/03/18 20:35:00 INFO ContextCleaner: Cleaned shuffle 0
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 90
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 51
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 64
21/03/18 20:35:00 INFO ContextCleaner: Cleaned accumulator 97
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 284.5 KB, free 911.4 MB)
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 23.9 KB, free 911.4 MB)
21/03/18 20:35:01 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:64733 (size: 23.9 KB, free: 912.2 MB)
21/03/18 20:35:01 INFO SparkContext: Created broadcast 6 from count at NativeMethodAccessorImpl.java:0
21/03/18 20:35:01 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
21/03/18 20:35:01 INFO SparkContext: Starting job: count at NativeMethodAccessorImpl.java:0
21/03/18 20:35:01 INFO DAGScheduler: Registering RDD 27 (count at NativeMethodAccessorImpl.java:0)
21/03/18 20:35:01 INFO DAGScheduler: Got job 3 (count at NativeMethodAccessorImpl.java:0) with 1 output partitions
21/03/18 20:35:01 INFO DAGScheduler: Final stage: ResultStage 5 (count at NativeMethodAccessorImpl.java:0)
21/03/18 20:35:01 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4)
21/03/18 20:35:01 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 4)
21/03/18 20:35:01 INFO DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[27] at count at NativeMethodAccessorImpl.java:0), which has no missing parents
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 25.9 KB, free 911.4 MB)
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 12.2 KB, free 911.4 MB)
21/03/18 20:35:01 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:64733 (size: 12.2 KB, free: 912.2 MB)
21/03/18 20:35:01 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:01 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[27] at count at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:01 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
21/03/18 20:35:01 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, localhost, executor driver, partition 0, PROCESS_LOCAL, 8348 bytes)
21/03/18 20:35:01 INFO Executor: Running task 0.0 in stage 4.0 (TID 4)
21/03/18 20:35:01 INFO FileScanRDD: Reading File path: file:///var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/RtmpxYR9UR/file4c90b98648, range: 0-1303, partition values: [empty row]
21/03/18 20:35:01 INFO CodeGenerator: Code generated in 21.554163 ms
21/03/18 20:35:01 INFO MemoryStore: Block rdd_22_0 stored as values in memory (estimated size 4.1 KB, free 911.4 MB)
21/03/18 20:35:01 INFO BlockManagerInfo: Added rdd_22_0 in memory on localhost:64733 (size: 4.1 KB, free: 912.2 MB)
21/03/18 20:35:01 INFO CodeGenerator: Code generated in 6.841671 ms
21/03/18 20:35:01 INFO CodeGenerator: Code generated in 29.098609 ms
21/03/18 20:35:01 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4). 1914 bytes result sent to driver
21/03/18 20:35:01 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 218 ms on localhost (executor driver) (1/1)
21/03/18 20:35:01 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 
21/03/18 20:35:01 INFO DAGScheduler: ShuffleMapStage 4 (count at NativeMethodAccessorImpl.java:0) finished in 0.259 s
21/03/18 20:35:01 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:35:01 INFO DAGScheduler: running: Set()
21/03/18 20:35:01 INFO DAGScheduler: waiting: Set(ResultStage 5)
21/03/18 20:35:01 INFO DAGScheduler: failed: Set()
21/03/18 20:35:01 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[30] at count at NativeMethodAccessorImpl.java:0), which has no missing parents
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 7.1 KB, free 911.3 MB)
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 3.8 KB, free 911.3 MB)
21/03/18 20:35:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:64733 (size: 3.8 KB, free: 912.2 MB)
21/03/18 20:35:01 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:01 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[30] at count at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:01 INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
21/03/18 20:35:01 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:35:01 INFO Executor: Running task 0.0 in stage 5.0 (TID 5)
21/03/18 20:35:01 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:35:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/03/18 20:35:01 INFO Executor: Finished task 0.0 in stage 5.0 (TID 5). 1782 bytes result sent to driver
21/03/18 20:35:01 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 7 ms on localhost (executor driver) (1/1)
21/03/18 20:35:01 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
21/03/18 20:35:01 INFO DAGScheduler: ResultStage 5 (count at NativeMethodAccessorImpl.java:0) finished in 0.015 s
21/03/18 20:35:01 INFO DAGScheduler: Job 3 finished: count at NativeMethodAccessorImpl.java:0, took 0.282999 s
21/03/18 20:35:01 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:01 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:01 INFO SparkContext: Starting job: count at NativeMethodAccessorImpl.java:0
21/03/18 20:35:01 INFO DAGScheduler: Registering RDD 35 (count at NativeMethodAccessorImpl.java:0)
21/03/18 20:35:01 INFO DAGScheduler: Got job 4 (count at NativeMethodAccessorImpl.java:0) with 1 output partitions
21/03/18 20:35:01 INFO DAGScheduler: Final stage: ResultStage 7 (count at NativeMethodAccessorImpl.java:0)
21/03/18 20:35:01 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 6)
21/03/18 20:35:01 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 6)
21/03/18 20:35:01 INFO DAGScheduler: Submitting ShuffleMapStage 6 (MapPartitionsRDD[35] at count at NativeMethodAccessorImpl.java:0), which has no missing parents
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 25.9 KB, free 911.3 MB)
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 12.1 KB, free 911.3 MB)
21/03/18 20:35:01 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:64733 (size: 12.1 KB, free: 912.2 MB)
21/03/18 20:35:01 INFO SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:01 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 6 (MapPartitionsRDD[35] at count at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:01 INFO TaskSchedulerImpl: Adding task set 6.0 with 1 tasks
21/03/18 20:35:01 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 6, localhost, executor driver, partition 0, PROCESS_LOCAL, 8348 bytes)
21/03/18 20:35:01 INFO Executor: Running task 0.0 in stage 6.0 (TID 6)
21/03/18 20:35:01 INFO BlockManager: Found block rdd_22_0 locally
21/03/18 20:35:01 INFO Executor: Finished task 0.0 in stage 6.0 (TID 6). 1871 bytes result sent to driver
21/03/18 20:35:01 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 6) in 15 ms on localhost (executor driver) (1/1)
21/03/18 20:35:01 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 
21/03/18 20:35:01 INFO DAGScheduler: ShuffleMapStage 6 (count at NativeMethodAccessorImpl.java:0) finished in 0.032 s
21/03/18 20:35:01 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:35:01 INFO DAGScheduler: running: Set()
21/03/18 20:35:01 INFO DAGScheduler: waiting: Set(ResultStage 7)
21/03/18 20:35:01 INFO DAGScheduler: failed: Set()
21/03/18 20:35:01 INFO DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[38] at count at NativeMethodAccessorImpl.java:0), which has no missing parents
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 7.1 KB, free 911.3 MB)
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 3.8 KB, free 911.3 MB)
21/03/18 20:35:01 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:64733 (size: 3.8 KB, free: 912.2 MB)
21/03/18 20:35:01 INFO SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:01 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 7 (MapPartitionsRDD[38] at count at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:01 INFO TaskSchedulerImpl: Adding task set 7.0 with 1 tasks
21/03/18 20:35:01 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 7, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:35:01 INFO Executor: Running task 0.0 in stage 7.0 (TID 7)
21/03/18 20:35:01 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:35:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/03/18 20:35:01 INFO Executor: Finished task 0.0 in stage 7.0 (TID 7). 1782 bytes result sent to driver
21/03/18 20:35:01 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 7) in 8 ms on localhost (executor driver) (1/1)
21/03/18 20:35:01 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 
21/03/18 20:35:01 INFO DAGScheduler: ResultStage 7 (count at NativeMethodAccessorImpl.java:0) finished in 0.014 s
21/03/18 20:35:01 INFO DAGScheduler: Job 4 finished: count at NativeMethodAccessorImpl.java:0, took 0.052976 s
21/03/18 20:35:01 INFO MapPartitionsRDD: Removing RDD 22 from persistence list
21/03/18 20:35:01 INFO BlockManager: Removing RDD 22
21/03/18 20:35:01 INFO FileSourceStrategy: Pruning directories with: 
21/03/18 20:35:01 INFO FileSourceStrategy: Post-Scan Filters: 
21/03/18 20:35:01 INFO FileSourceStrategy: Output Data Schema: struct<_c0: string, _c1: string, _c2: string, _c3: string, _c4: string ... 9 more fields>
21/03/18 20:35:01 INFO FileSourceScanExec: Pushed Filters: 
21/03/18 20:35:01 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:01 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 284.5 KB, free 911.0 MB)
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 23.9 KB, free 911.0 MB)
21/03/18 20:35:01 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:64733 (size: 23.9 KB, free: 912.2 MB)
21/03/18 20:35:01 INFO SparkContext: Created broadcast 11 from count at NativeMethodAccessorImpl.java:0
21/03/18 20:35:01 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
21/03/18 20:35:01 INFO SparkContext: Starting job: count at NativeMethodAccessorImpl.java:0
21/03/18 20:35:01 INFO DAGScheduler: Registering RDD 46 (count at NativeMethodAccessorImpl.java:0)
21/03/18 20:35:01 INFO DAGScheduler: Got job 5 (count at NativeMethodAccessorImpl.java:0) with 1 output partitions
21/03/18 20:35:01 INFO DAGScheduler: Final stage: ResultStage 9 (count at NativeMethodAccessorImpl.java:0)
21/03/18 20:35:01 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 8)
21/03/18 20:35:01 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 8)
21/03/18 20:35:01 INFO DAGScheduler: Submitting ShuffleMapStage 8 (MapPartitionsRDD[46] at count at NativeMethodAccessorImpl.java:0), which has no missing parents
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 25.9 KB, free 911.0 MB)
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 12.1 KB, free 911.0 MB)
21/03/18 20:35:01 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:64733 (size: 12.1 KB, free: 912.2 MB)
21/03/18 20:35:01 INFO SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:01 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 8 (MapPartitionsRDD[46] at count at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:01 INFO TaskSchedulerImpl: Adding task set 8.0 with 1 tasks
21/03/18 20:35:01 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 8, localhost, executor driver, partition 0, PROCESS_LOCAL, 8348 bytes)
21/03/18 20:35:01 INFO Executor: Running task 0.0 in stage 8.0 (TID 8)
21/03/18 20:35:01 INFO FileScanRDD: Reading File path: file:///var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/RtmpxYR9UR/file4c90b98648, range: 0-2539, partition values: [empty row]
21/03/18 20:35:01 INFO MemoryStore: Block rdd_41_0 stored as values in memory (estimated size 4.9 KB, free 911.0 MB)
21/03/18 20:35:01 INFO BlockManagerInfo: Added rdd_41_0 in memory on localhost:64733 (size: 4.9 KB, free: 912.2 MB)
21/03/18 20:35:01 INFO Executor: Finished task 0.0 in stage 8.0 (TID 8). 1871 bytes result sent to driver
21/03/18 20:35:01 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 8) in 57 ms on localhost (executor driver) (1/1)
21/03/18 20:35:01 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 
21/03/18 20:35:01 INFO DAGScheduler: ShuffleMapStage 8 (count at NativeMethodAccessorImpl.java:0) finished in 0.068 s
21/03/18 20:35:01 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:35:01 INFO DAGScheduler: running: Set()
21/03/18 20:35:01 INFO DAGScheduler: waiting: Set(ResultStage 9)
21/03/18 20:35:01 INFO DAGScheduler: failed: Set()
21/03/18 20:35:01 INFO DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[49] at count at NativeMethodAccessorImpl.java:0), which has no missing parents
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 7.1 KB, free 911.0 MB)
21/03/18 20:35:01 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 3.8 KB, free 910.9 MB)
21/03/18 20:35:01 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:64733 (size: 3.8 KB, free: 912.2 MB)
21/03/18 20:35:01 INFO SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:01 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 9 (MapPartitionsRDD[49] at count at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:01 INFO TaskSchedulerImpl: Adding task set 9.0 with 1 tasks
21/03/18 20:35:01 INFO TaskSetManager: Starting task 0.0 in stage 9.0 (TID 9, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:35:01 INFO Executor: Running task 0.0 in stage 9.0 (TID 9)
21/03/18 20:35:01 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:35:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/03/18 20:35:01 INFO Executor: Finished task 0.0 in stage 9.0 (TID 9). 1782 bytes result sent to driver
21/03/18 20:35:01 INFO TaskSetManager: Finished task 0.0 in stage 9.0 (TID 9) in 9 ms on localhost (executor driver) (1/1)
21/03/18 20:35:01 INFO TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 
21/03/18 20:35:01 INFO DAGScheduler: ResultStage 9 (count at NativeMethodAccessorImpl.java:0) finished in 0.016 s
21/03/18 20:35:01 INFO DAGScheduler: Job 5 finished: count at NativeMethodAccessorImpl.java:0, took 0.090045 s
21/03/18 20:35:01 INFO SparkContext: Invoking stop() from shutdown hook
21/03/18 20:35:01 INFO SparkUI: Stopped Spark web UI at http://localhost:4040
21/03/18 20:35:01 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/03/18 20:35:02 INFO MemoryStore: MemoryStore cleared
21/03/18 20:35:02 INFO BlockManager: BlockManager stopped
21/03/18 20:35:02 INFO BlockManagerMaster: BlockManagerMaster stopped
21/03/18 20:35:02 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/03/18 20:35:02 INFO SparkContext: Successfully stopped SparkContext
21/03/18 20:35:02 INFO ShutdownHookManager: Shutdown hook called
21/03/18 20:35:02 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-ef4d7d0c-c210-4ceb-9799-bd42d9b89c40
21/03/18 20:35:02 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-8d1c28c4-fc73-4925-82fc-2cd6119308a5
21/03/18 20:35:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/03/18 20:35:06 INFO SparkContext: Running Spark version 2.4.3
21/03/18 20:35:06 INFO SparkContext: Submitted application: sparklyr
21/03/18 20:35:06 INFO SecurityManager: Changing view acls to: nathan
21/03/18 20:35:06 INFO SecurityManager: Changing modify acls to: nathan
21/03/18 20:35:06 INFO SecurityManager: Changing view acls groups to: 
21/03/18 20:35:06 INFO SecurityManager: Changing modify acls groups to: 
21/03/18 20:35:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(nathan); groups with view permissions: Set(); users  with modify permissions: Set(nathan); groups with modify permissions: Set()
21/03/18 20:35:06 INFO Utils: Successfully started service 'sparkDriver' on port 64870.
21/03/18 20:35:06 INFO SparkEnv: Registering MapOutputTracker
21/03/18 20:35:06 INFO SparkEnv: Registering BlockManagerMaster
21/03/18 20:35:06 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/03/18 20:35:06 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/03/18 20:35:06 INFO DiskBlockManager: Created local directory at /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/blockmgr-471f241e-ff04-42f5-85eb-5c620e8d5316
21/03/18 20:35:06 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
21/03/18 20:35:06 INFO SparkEnv: Registering OutputCommitCoordinator
21/03/18 20:35:06 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/03/18 20:35:06 INFO SparkUI: Bound SparkUI to 127.0.0.1, and started at http://localhost:4040
21/03/18 20:35:06 INFO SparkContext: Added JAR file:/Library/Frameworks/R.framework/Versions/4.0/Resources/library/sparklyr/java/sparklyr-2.4-2.11.jar at spark://localhost:64870/jars/sparklyr-2.4-2.11.jar with timestamp 1616096106850
21/03/18 20:35:06 INFO Executor: Starting executor ID driver on host localhost
21/03/18 20:35:07 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 64880.
21/03/18 20:35:07 INFO NettyBlockTransferService: Server created on localhost:64880
21/03/18 20:35:07 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/03/18 20:35:07 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, localhost, 64880, None)
21/03/18 20:35:07 INFO BlockManagerMasterEndpoint: Registering block manager localhost:64880 with 912.3 MB RAM, BlockManagerId(driver, localhost, 64880, None)
21/03/18 20:35:07 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, localhost, 64880, None)
21/03/18 20:35:07 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, localhost, 64880, None)
21/03/18 20:35:07 INFO SharedState: loading hive config file: file:/Users/nathan/spark/spark-2.4.3-bin-hadoop2.7/conf/hive-site.xml
21/03/18 20:35:07 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse').
21/03/18 20:35:07 INFO SharedState: Warehouse path is 'file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse'.
21/03/18 20:35:08 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
21/03/18 20:35:11 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
21/03/18 20:35:12 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
21/03/18 20:35:12 INFO ObjectStore: ObjectStore, initialize called
21/03/18 20:35:12 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
21/03/18 20:35:12 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
21/03/18 20:35:13 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
21/03/18 20:35:14 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:35:14 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:35:15 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:35:15 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:35:15 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
21/03/18 20:35:15 INFO ObjectStore: Initialized ObjectStore
21/03/18 20:35:15 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
21/03/18 20:35:15 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
21/03/18 20:35:15 INFO HiveMetaStore: Added admin role in metastore
21/03/18 20:35:15 INFO HiveMetaStore: Added public role in metastore
21/03/18 20:35:15 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/03/18 20:35:16 INFO HiveMetaStore: 0: get_all_databases
21/03/18 20:35:16 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_all_databases	
21/03/18 20:35:16 INFO HiveMetaStore: 0: get_functions: db=default pat=*
21/03/18 20:35:16 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_functions: db=default pat=*	
21/03/18 20:35:16 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:35:16 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/08590854-401b-44e1-bb4a-983fb4c83971_resources
21/03/18 20:35:16 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/08590854-401b-44e1-bb4a-983fb4c83971
21/03/18 20:35:16 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/nathan/08590854-401b-44e1-bb4a-983fb4c83971
21/03/18 20:35:16 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/08590854-401b-44e1-bb4a-983fb4c83971/_tmp_space.db
21/03/18 20:35:16 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse
21/03/18 20:35:16 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:16 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:16 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:16 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:16 INFO FileSourceStrategy: Pruning directories with: 
21/03/18 20:35:16 INFO FileSourceStrategy: Post-Scan Filters: (length(trim(value#0, None)) > 0)
21/03/18 20:35:16 INFO FileSourceStrategy: Output Data Schema: struct<value: string>
21/03/18 20:35:16 INFO FileSourceScanExec: Pushed Filters: 
21/03/18 20:35:17 INFO CodeGenerator: Code generated in 284.487481 ms
21/03/18 20:35:17 INFO CodeGenerator: Code generated in 17.498354 ms
21/03/18 20:35:17 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 284.3 KB, free 912.0 MB)
21/03/18 20:35:17 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.9 KB, free 912.0 MB)
21/03/18 20:35:17 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:64880 (size: 23.9 KB, free: 912.3 MB)
21/03/18 20:35:17 INFO SparkContext: Created broadcast 0 from createTable at NativeMethodAccessorImpl.java:0
21/03/18 20:35:17 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
21/03/18 20:35:17 INFO SparkContext: Starting job: createTable at NativeMethodAccessorImpl.java:0
21/03/18 20:35:17 INFO DAGScheduler: Got job 0 (createTable at NativeMethodAccessorImpl.java:0) with 1 output partitions
21/03/18 20:35:17 INFO DAGScheduler: Final stage: ResultStage 0 (createTable at NativeMethodAccessorImpl.java:0)
21/03/18 20:35:17 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:35:17 INFO DAGScheduler: Missing parents: List()
21/03/18 20:35:17 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at createTable at NativeMethodAccessorImpl.java:0), which has no missing parents
21/03/18 20:35:17 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 8.8 KB, free 912.0 MB)
21/03/18 20:35:17 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 4.5 KB, free 912.0 MB)
21/03/18 20:35:17 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:64880 (size: 4.5 KB, free: 912.3 MB)
21/03/18 20:35:17 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:17 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at createTable at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:17 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
21/03/18 20:35:17 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 8361 bytes)
21/03/18 20:35:18 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/03/18 20:35:18 INFO Executor: Fetching spark://localhost:64870/jars/sparklyr-2.4-2.11.jar with timestamp 1616096106850
21/03/18 20:35:18 INFO TransportClientFactory: Successfully created connection to localhost/127.0.0.1:64870 after 26 ms (0 ms spent in bootstraps)
21/03/18 20:35:18 INFO Utils: Fetching spark://localhost:64870/jars/sparklyr-2.4-2.11.jar to /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-a1b647e2-3864-437e-ac90-db2e66e7fc93/userFiles-3e113b71-d72d-4e16-ba44-014db096aa8b/fetchFileTemp3392709851163014165.tmp
21/03/18 20:35:18 INFO Executor: Adding file:/private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-a1b647e2-3864-437e-ac90-db2e66e7fc93/userFiles-3e113b71-d72d-4e16-ba44-014db096aa8b/sparklyr-2.4-2.11.jar to class loader
21/03/18 20:35:18 INFO FileScanRDD: Reading File path: file:///var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/RtmpxYR9UR/file4c9011f87f12, range: 0-1303, partition values: [empty row]
21/03/18 20:35:18 INFO CodeGenerator: Code generated in 29.514071 ms
21/03/18 20:35:18 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1329 bytes result sent to driver
21/03/18 20:35:18 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 581 ms on localhost (executor driver) (1/1)
21/03/18 20:35:18 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
21/03/18 20:35:18 INFO DAGScheduler: ResultStage 0 (createTable at NativeMethodAccessorImpl.java:0) finished in 0.694 s
21/03/18 20:35:18 INFO DAGScheduler: Job 0 finished: createTable at NativeMethodAccessorImpl.java:0, took 0.879717 s
21/03/18 20:35:18 INFO FileSourceStrategy: Pruning directories with: 
21/03/18 20:35:18 INFO FileSourceStrategy: Post-Scan Filters: 
21/03/18 20:35:18 INFO FileSourceStrategy: Output Data Schema: struct<value: string>
21/03/18 20:35:18 INFO FileSourceScanExec: Pushed Filters: 
21/03/18 20:35:18 INFO CodeGenerator: Code generated in 8.933506 ms
21/03/18 20:35:18 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 284.3 KB, free 911.7 MB)
21/03/18 20:35:18 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 23.9 KB, free 911.7 MB)
21/03/18 20:35:18 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:64880 (size: 23.9 KB, free: 912.2 MB)
21/03/18 20:35:18 INFO SparkContext: Created broadcast 2 from createTable at NativeMethodAccessorImpl.java:0
21/03/18 20:35:18 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
21/03/18 20:35:18 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:18 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:18 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:18 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:18 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:18 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:18 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:18 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:18 WARN HiveExternalCatalog: Couldn't find corresponding Hive SerDe for data source provider csv. Persisting data source table `default`.`mtcars` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive.
21/03/18 20:35:18 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:18 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:19 INFO HiveMetaStore: 0: create_table: Table(tableName:mtcars, dbName:default, owner:nathan, createTime:1616096110, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col, type:array<string>, comment:from deserializer)], location:file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse/mtcars-__PLACEHOLDER__, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{path=file:/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/RtmpxYR9UR/file4c9011f87f12, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{EXTERNAL=TRUE, spark.sql.sources.schema.part.0={"type":"struct","fields":[{"name":"_c0","type":"string","nullable":true,"metadata":{}},{"name":"_c1","type":"string","nullable":true,"metadata":{}},{"name":"_c2","type":"string","nullable":true,"metadata":{}},{"name":"_c3","type":"string","nullable":true,"metadata":{}},{"name":"_c4","type":"string","nullable":true,"metadata":{}},{"name":"_c5","type":"string","nullable":true,"metadata":{}},{"name":"_c6","type":"string","nullable":true,"metadata":{}},{"name":"_c7","type":"string","nullable":true,"metadata":{}},{"name":"_c8","type":"string","nullable":true,"metadata":{}},{"name":"_c9","type":"string","nullable":true,"metadata":{}},{"name":"_c10","type":"string","nullable":true,"metadata":{}}]}, spark.sql.sources.schema.numParts=1, spark.sql.sources.provider=csv, spark.sql.create.version=2.4.3}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{}, groupPrivileges:null, rolePrivileges:null))
21/03/18 20:35:19 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=create_table: Table(tableName:mtcars, dbName:default, owner:nathan, createTime:1616096110, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col, type:array<string>, comment:from deserializer)], location:file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse/mtcars-__PLACEHOLDER__, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{path=file:/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/RtmpxYR9UR/file4c9011f87f12, serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{EXTERNAL=TRUE, spark.sql.sources.schema.part.0={"type":"struct","fields":[{"name":"_c0","type":"string","nullable":true,"metadata":{}},{"name":"_c1","type":"string","nullable":true,"metadata":{}},{"name":"_c2","type":"string","nullable":true,"metadata":{}},{"name":"_c3","type":"string","nullable":true,"metadata":{}},{"name":"_c4","type":"string","nullable":true,"metadata":{}},{"name":"_c5","type":"string","nullable":true,"metadata":{}},{"name":"_c6","type":"string","nullable":true,"metadata":{}},{"name":"_c7","type":"string","nullable":true,"metadata":{}},{"name":"_c8","type":"string","nullable":true,"metadata":{}},{"name":"_c9","type":"string","nullable":true,"metadata":{}},{"name":"_c10","type":"string","nullable":true,"metadata":{}}]}, spark.sql.sources.schema.numParts=1, spark.sql.sources.provider=csv, spark.sql.create.version=2.4.3}, viewOriginalText:null, viewExpandedText:null, tableType:EXTERNAL_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{}, groupPrivileges:null, rolePrivileges:null))	
21/03/18 20:35:19 INFO FileUtils: Creating directory if it doesn't exist: file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse/mtcars-__PLACEHOLDER__
21/03/18 20:35:19 INFO HiveMetaStore: 0: get_database: global_temp
21/03/18 20:35:19 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: global_temp	
21/03/18 20:35:19 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
21/03/18 20:35:19 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:19 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:19 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:19 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:20 INFO CodeGenerator: Code generated in 16.716845 ms
21/03/18 20:35:20 INFO CodeGenerator: Code generated in 14.333758 ms
21/03/18 20:35:20 INFO SparkContext: Starting job: collect at utils.scala:135
21/03/18 20:35:20 INFO DAGScheduler: Got job 1 (collect at utils.scala:135) with 1 output partitions
21/03/18 20:35:20 INFO DAGScheduler: Final stage: ResultStage 1 (collect at utils.scala:135)
21/03/18 20:35:20 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:35:20 INFO DAGScheduler: Missing parents: List()
21/03/18 20:35:20 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[13] at collect at utils.scala:135), which has no missing parents
21/03/18 20:35:20 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 6.5 KB, free 911.7 MB)
21/03/18 20:35:20 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 3.3 KB, free 911.7 MB)
21/03/18 20:35:20 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:64880 (size: 3.3 KB, free: 912.2 MB)
21/03/18 20:35:20 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:20 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[13] at collect at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:20 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
21/03/18 20:35:20 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, PROCESS_LOCAL, 8080 bytes)
21/03/18 20:35:20 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
21/03/18 20:35:20 INFO CodeGenerator: Code generated in 8.447566 ms
21/03/18 20:35:20 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1329 bytes result sent to driver
21/03/18 20:35:20 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 31 ms on localhost (executor driver) (1/1)
21/03/18 20:35:20 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
21/03/18 20:35:20 INFO DAGScheduler: ResultStage 1 (collect at utils.scala:135) finished in 0.043 s
21/03/18 20:35:20 INFO DAGScheduler: Job 1 finished: collect at utils.scala:135, took 0.077137 s
21/03/18 20:35:20 INFO CodeGenerator: Code generated in 19.575505 ms
21/03/18 20:35:20 INFO CodeGenerator: Code generated in 18.523899 ms
21/03/18 20:35:20 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:35:20 INFO DAGScheduler: Registering RDD 16 (count at utils.scala:135)
21/03/18 20:35:20 INFO DAGScheduler: Got job 2 (count at utils.scala:135) with 1 output partitions
21/03/18 20:35:20 INFO DAGScheduler: Final stage: ResultStage 3 (count at utils.scala:135)
21/03/18 20:35:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
21/03/18 20:35:20 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 2)
21/03/18 20:35:20 INFO DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[16] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:20 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 8.4 KB, free 911.7 MB)
21/03/18 20:35:20 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 4.5 KB, free 911.7 MB)
21/03/18 20:35:20 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:64880 (size: 4.5 KB, free: 912.2 MB)
21/03/18 20:35:20 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:20 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[16] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:20 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
21/03/18 20:35:20 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, executor driver, partition 0, PROCESS_LOCAL, 8069 bytes)
21/03/18 20:35:20 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
21/03/18 20:35:20 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1542 bytes result sent to driver
21/03/18 20:35:20 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 77 ms on localhost (executor driver) (1/1)
21/03/18 20:35:20 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 
21/03/18 20:35:20 INFO DAGScheduler: ShuffleMapStage 2 (count at utils.scala:135) finished in 0.097 s
21/03/18 20:35:20 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:35:20 INFO DAGScheduler: running: Set()
21/03/18 20:35:20 INFO DAGScheduler: waiting: Set(ResultStage 3)
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 42
21/03/18 20:35:20 INFO DAGScheduler: failed: Set()
21/03/18 20:35:20 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[19] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:20 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 7.1 KB, free 911.7 MB)
21/03/18 20:35:20 INFO BlockManagerInfo: Removed broadcast_3_piece0 on localhost:64880 in memory (size: 3.3 KB, free: 912.2 MB)
21/03/18 20:35:20 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 3.8 KB, free 911.7 MB)
21/03/18 20:35:20 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:64880 (size: 3.8 KB, free: 912.2 MB)
21/03/18 20:35:20 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:20 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[19] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:20 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 40
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 47
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 49
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 58
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 46
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 54
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 57
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 45
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 62
21/03/18 20:35:20 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:64880 in memory (size: 4.5 KB, free: 912.2 MB)
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 38
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 43
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 51
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 60
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 44
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 55
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 48
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 39
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 53
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 41
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 50
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 56
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 61
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 52
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 63
21/03/18 20:35:20 INFO ContextCleaner: Cleaned accumulator 59
21/03/18 20:35:20 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:35:20 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
21/03/18 20:35:20 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:35:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 10 ms
21/03/18 20:35:20 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1782 bytes result sent to driver
21/03/18 20:35:20 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 66 ms on localhost (executor driver) (1/1)
21/03/18 20:35:20 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 
21/03/18 20:35:20 INFO DAGScheduler: ResultStage 3 (count at utils.scala:135) finished in 0.082 s
21/03/18 20:35:20 INFO DAGScheduler: Job 2 finished: count at utils.scala:135, took 0.257305 s
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO FileSourceStrategy: Pruning directories with: 
21/03/18 20:35:21 INFO FileSourceStrategy: Post-Scan Filters: 
21/03/18 20:35:21 INFO FileSourceStrategy: Output Data Schema: struct<_c0: string, _c1: string, _c2: string, _c3: string, _c4: string ... 9 more fields>
21/03/18 20:35:21 INFO FileSourceScanExec: Pushed Filters: 
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=catalog_fake_table
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=catalog_fake_table	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=catalog_fake_table
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=catalog_fake_table	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_tables: db=default pat=*
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_tables: db=default pat=*	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:21 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:21 INFO CodeGenerator: Code generated in 29.898081 ms
21/03/18 20:35:21 INFO CodeGenerator: Code generated in 20.779913 ms
21/03/18 20:35:21 INFO CodeGenerator: Code generated in 23.084762 ms
21/03/18 20:35:22 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:35:22 INFO DAGScheduler: Registering RDD 23 (count at utils.scala:135)
21/03/18 20:35:22 INFO DAGScheduler: Got job 3 (count at utils.scala:135) with 1 output partitions
21/03/18 20:35:22 INFO DAGScheduler: Final stage: ResultStage 5 (count at utils.scala:135)
21/03/18 20:35:22 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4)
21/03/18 20:35:22 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 4)
21/03/18 20:35:22 INFO DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[23] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:22 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 7.9 KB, free 911.7 MB)
21/03/18 20:35:22 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 4.2 KB, free 911.7 MB)
21/03/18 20:35:22 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:64880 (size: 4.2 KB, free: 912.2 MB)
21/03/18 20:35:22 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:22 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[23] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:22 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
21/03/18 20:35:22 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, localhost, executor driver, partition 0, PROCESS_LOCAL, 8017 bytes)
21/03/18 20:35:22 INFO Executor: Running task 0.0 in stage 4.0 (TID 4)
21/03/18 20:35:22 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4). 1499 bytes result sent to driver
21/03/18 20:35:22 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 9 ms on localhost (executor driver) (1/1)
21/03/18 20:35:22 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 
21/03/18 20:35:22 INFO DAGScheduler: ShuffleMapStage 4 (count at utils.scala:135) finished in 0.016 s
21/03/18 20:35:22 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:35:22 INFO DAGScheduler: running: Set()
21/03/18 20:35:22 INFO DAGScheduler: waiting: Set(ResultStage 5)
21/03/18 20:35:22 INFO DAGScheduler: failed: Set()
21/03/18 20:35:22 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[26] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:22 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 7.1 KB, free 911.7 MB)
21/03/18 20:35:22 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 3.8 KB, free 911.7 MB)
21/03/18 20:35:22 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:64880 (size: 3.8 KB, free: 912.2 MB)
21/03/18 20:35:22 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:22 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[26] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:22 INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
21/03/18 20:35:22 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:35:22 INFO Executor: Running task 0.0 in stage 5.0 (TID 5)
21/03/18 20:35:22 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:35:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/03/18 20:35:22 INFO Executor: Finished task 0.0 in stage 5.0 (TID 5). 1782 bytes result sent to driver
21/03/18 20:35:22 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 6 ms on localhost (executor driver) (1/1)
21/03/18 20:35:22 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
21/03/18 20:35:22 INFO DAGScheduler: ResultStage 5 (count at utils.scala:135) finished in 0.012 s
21/03/18 20:35:22 INFO DAGScheduler: Job 3 finished: count at utils.scala:135, took 0.033538 s
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_table : db=default tbl=catalog_fake_table
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=catalog_fake_table	
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:22 INFO FileSourceStrategy: Pruning directories with: 
21/03/18 20:35:22 INFO FileSourceStrategy: Post-Scan Filters: 
21/03/18 20:35:22 INFO FileSourceStrategy: Output Data Schema: struct<_c0: string, _c1: string, _c2: string, _c3: string, _c4: string ... 9 more fields>
21/03/18 20:35:22 INFO FileSourceScanExec: Pushed Filters: 
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:22 INFO HiveMetaStore: 0: get_tables: db=default pat=*
21/03/18 20:35:22 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_tables: db=default pat=*	
21/03/18 20:35:22 INFO CodeGenerator: Code generated in 7.114877 ms
21/03/18 20:35:22 INFO SparkContext: Starting job: collect at utils.scala:61
21/03/18 20:35:22 INFO DAGScheduler: Got job 4 (collect at utils.scala:61) with 1 output partitions
21/03/18 20:35:22 INFO DAGScheduler: Final stage: ResultStage 6 (collect at utils.scala:61)
21/03/18 20:35:22 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:35:22 INFO DAGScheduler: Missing parents: List()
21/03/18 20:35:22 INFO DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[31] at map at utils.scala:54), which has no missing parents
21/03/18 20:35:22 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 6.0 KB, free 911.6 MB)
21/03/18 20:35:22 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 3.4 KB, free 911.6 MB)
21/03/18 20:35:22 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:64880 (size: 3.4 KB, free: 912.2 MB)
21/03/18 20:35:22 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:22 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 6 (MapPartitionsRDD[31] at map at utils.scala:54) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:22 INFO TaskSchedulerImpl: Adding task set 6.0 with 1 tasks
21/03/18 20:35:22 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 6, localhost, executor driver, partition 0, PROCESS_LOCAL, 8052 bytes)
21/03/18 20:35:22 INFO Executor: Running task 0.0 in stage 6.0 (TID 6)
21/03/18 20:35:22 INFO CodeGenerator: Code generated in 6.260858 ms
21/03/18 20:35:22 INFO Executor: Finished task 0.0 in stage 6.0 (TID 6). 979 bytes result sent to driver
21/03/18 20:35:22 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 6) in 14 ms on localhost (executor driver) (1/1)
21/03/18 20:35:22 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 
21/03/18 20:35:22 INFO DAGScheduler: ResultStage 6 (collect at utils.scala:61) finished in 0.020 s
21/03/18 20:35:22 INFO DAGScheduler: Job 4 finished: collect at utils.scala:61, took 0.022766 s
21/03/18 20:35:22 INFO CodeGenerator: Code generated in 5.361415 ms
21/03/18 20:35:22 INFO CodeGenerator: Code generated in 6.982878 ms
21/03/18 20:35:22 INFO SparkContext: Starting job: collect at utils.scala:135
21/03/18 20:35:22 INFO DAGScheduler: Got job 5 (collect at utils.scala:135) with 1 output partitions
21/03/18 20:35:22 INFO DAGScheduler: Final stage: ResultStage 7 (collect at utils.scala:135)
21/03/18 20:35:22 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:35:22 INFO DAGScheduler: Missing parents: List()
21/03/18 20:35:22 INFO DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[35] at collect at utils.scala:135), which has no missing parents
21/03/18 20:35:22 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 6.3 KB, free 911.6 MB)
21/03/18 20:35:22 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 3.3 KB, free 911.6 MB)
21/03/18 20:35:22 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:64880 (size: 3.3 KB, free: 912.2 MB)
21/03/18 20:35:22 INFO SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:22 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 7 (MapPartitionsRDD[35] at collect at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:22 INFO TaskSchedulerImpl: Adding task set 7.0 with 1 tasks
21/03/18 20:35:22 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 7, localhost, executor driver, partition 0, PROCESS_LOCAL, 8080 bytes)
21/03/18 20:35:22 INFO Executor: Running task 0.0 in stage 7.0 (TID 7)
21/03/18 20:35:22 INFO Executor: Finished task 0.0 in stage 7.0 (TID 7). 1286 bytes result sent to driver
21/03/18 20:35:22 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 7) in 7 ms on localhost (executor driver) (1/1)
21/03/18 20:35:22 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool 
21/03/18 20:35:22 INFO DAGScheduler: ResultStage 7 (collect at utils.scala:135) finished in 0.018 s
21/03/18 20:35:22 INFO DAGScheduler: Job 5 finished: collect at utils.scala:135, took 0.021136 s
21/03/18 20:35:22 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:35:22 INFO DAGScheduler: Registering RDD 38 (count at utils.scala:135)
21/03/18 20:35:22 INFO DAGScheduler: Got job 6 (count at utils.scala:135) with 1 output partitions
21/03/18 20:35:22 INFO DAGScheduler: Final stage: ResultStage 9 (count at utils.scala:135)
21/03/18 20:35:22 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 8)
21/03/18 20:35:22 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 8)
21/03/18 20:35:22 INFO DAGScheduler: Submitting ShuffleMapStage 8 (MapPartitionsRDD[38] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:22 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 8.4 KB, free 911.6 MB)
21/03/18 20:35:22 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 4.5 KB, free 911.6 MB)
21/03/18 20:35:22 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:64880 (size: 4.5 KB, free: 912.2 MB)
21/03/18 20:35:22 INFO SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:22 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 8 (MapPartitionsRDD[38] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:22 INFO TaskSchedulerImpl: Adding task set 8.0 with 1 tasks
21/03/18 20:35:22 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 8, localhost, executor driver, partition 0, PROCESS_LOCAL, 8069 bytes)
21/03/18 20:35:22 INFO Executor: Running task 0.0 in stage 8.0 (TID 8)
21/03/18 20:35:22 INFO Executor: Finished task 0.0 in stage 8.0 (TID 8). 1499 bytes result sent to driver
21/03/18 20:35:22 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 8) in 8 ms on localhost (executor driver) (1/1)
21/03/18 20:35:22 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool 
21/03/18 20:35:22 INFO DAGScheduler: ShuffleMapStage 8 (count at utils.scala:135) finished in 0.016 s
21/03/18 20:35:22 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:35:22 INFO DAGScheduler: running: Set()
21/03/18 20:35:22 INFO DAGScheduler: waiting: Set(ResultStage 9)
21/03/18 20:35:22 INFO DAGScheduler: failed: Set()
21/03/18 20:35:22 INFO DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[41] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:22 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 7.1 KB, free 911.6 MB)
21/03/18 20:35:22 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 3.8 KB, free 911.6 MB)
21/03/18 20:35:22 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:64880 (size: 3.8 KB, free: 912.2 MB)
21/03/18 20:35:22 INFO SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:22 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 9 (MapPartitionsRDD[41] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:22 INFO TaskSchedulerImpl: Adding task set 9.0 with 1 tasks
21/03/18 20:35:22 INFO TaskSetManager: Starting task 0.0 in stage 9.0 (TID 9, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:35:22 INFO Executor: Running task 0.0 in stage 9.0 (TID 9)
21/03/18 20:35:22 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:35:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/03/18 20:35:22 INFO Executor: Finished task 0.0 in stage 9.0 (TID 9). 1782 bytes result sent to driver
21/03/18 20:35:22 INFO TaskSetManager: Finished task 0.0 in stage 9.0 (TID 9) in 6 ms on localhost (executor driver) (1/1)
21/03/18 20:35:22 INFO TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool 
21/03/18 20:35:22 INFO DAGScheduler: ResultStage 9 (count at utils.scala:135) finished in 0.011 s
21/03/18 20:35:22 INFO DAGScheduler: Job 6 finished: count at utils.scala:135, took 0.031444 s
21/03/18 20:35:23 INFO SparkContext: Starting job: collect at utils.scala:135
21/03/18 20:35:23 INFO DAGScheduler: Got job 7 (collect at utils.scala:135) with 1 output partitions
21/03/18 20:35:23 INFO DAGScheduler: Final stage: ResultStage 10 (collect at utils.scala:135)
21/03/18 20:35:23 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:35:23 INFO DAGScheduler: Missing parents: List()
21/03/18 20:35:23 INFO DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[44] at collect at utils.scala:135), which has no missing parents
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 6.3 KB, free 911.6 MB)
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 3.3 KB, free 911.6 MB)
21/03/18 20:35:23 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:64880 (size: 3.3 KB, free: 912.2 MB)
21/03/18 20:35:23 INFO SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:23 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 10 (MapPartitionsRDD[44] at collect at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:23 INFO TaskSchedulerImpl: Adding task set 10.0 with 1 tasks
21/03/18 20:35:23 INFO TaskSetManager: Starting task 0.0 in stage 10.0 (TID 10, localhost, executor driver, partition 0, PROCESS_LOCAL, 8080 bytes)
21/03/18 20:35:23 INFO Executor: Running task 0.0 in stage 10.0 (TID 10)
21/03/18 20:35:23 INFO Executor: Finished task 0.0 in stage 10.0 (TID 10). 1286 bytes result sent to driver
21/03/18 20:35:23 INFO TaskSetManager: Finished task 0.0 in stage 10.0 (TID 10) in 4 ms on localhost (executor driver) (1/1)
21/03/18 20:35:23 INFO TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool 
21/03/18 20:35:23 INFO DAGScheduler: ResultStage 10 (collect at utils.scala:135) finished in 0.009 s
21/03/18 20:35:23 INFO DAGScheduler: Job 7 finished: collect at utils.scala:135, took 0.010818 s
21/03/18 20:35:23 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:35:23 INFO DAGScheduler: Registering RDD 47 (count at utils.scala:135)
21/03/18 20:35:23 INFO DAGScheduler: Got job 8 (count at utils.scala:135) with 1 output partitions
21/03/18 20:35:23 INFO DAGScheduler: Final stage: ResultStage 12 (count at utils.scala:135)
21/03/18 20:35:23 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 11)
21/03/18 20:35:23 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 11)
21/03/18 20:35:23 INFO DAGScheduler: Submitting ShuffleMapStage 11 (MapPartitionsRDD[47] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 8.4 KB, free 911.6 MB)
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 4.5 KB, free 911.6 MB)
21/03/18 20:35:23 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on localhost:64880 (size: 4.5 KB, free: 912.2 MB)
21/03/18 20:35:23 INFO SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:23 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 11 (MapPartitionsRDD[47] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:23 INFO TaskSchedulerImpl: Adding task set 11.0 with 1 tasks
21/03/18 20:35:23 INFO TaskSetManager: Starting task 0.0 in stage 11.0 (TID 11, localhost, executor driver, partition 0, PROCESS_LOCAL, 8069 bytes)
21/03/18 20:35:23 INFO Executor: Running task 0.0 in stage 11.0 (TID 11)
21/03/18 20:35:23 INFO Executor: Finished task 0.0 in stage 11.0 (TID 11). 1499 bytes result sent to driver
21/03/18 20:35:23 INFO TaskSetManager: Finished task 0.0 in stage 11.0 (TID 11) in 13 ms on localhost (executor driver) (1/1)
21/03/18 20:35:23 INFO TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool 
21/03/18 20:35:23 INFO DAGScheduler: ShuffleMapStage 11 (count at utils.scala:135) finished in 0.020 s
21/03/18 20:35:23 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:35:23 INFO DAGScheduler: running: Set()
21/03/18 20:35:23 INFO DAGScheduler: waiting: Set(ResultStage 12)
21/03/18 20:35:23 INFO DAGScheduler: failed: Set()
21/03/18 20:35:23 INFO DAGScheduler: Submitting ResultStage 12 (MapPartitionsRDD[50] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 7.1 KB, free 911.6 MB)
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 3.8 KB, free 911.6 MB)
21/03/18 20:35:23 INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on localhost:64880 (size: 3.8 KB, free: 912.2 MB)
21/03/18 20:35:23 INFO SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:23 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 12 (MapPartitionsRDD[50] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:23 INFO TaskSchedulerImpl: Adding task set 12.0 with 1 tasks
21/03/18 20:35:23 INFO TaskSetManager: Starting task 0.0 in stage 12.0 (TID 12, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:35:23 INFO Executor: Running task 0.0 in stage 12.0 (TID 12)
21/03/18 20:35:23 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:35:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/03/18 20:35:23 INFO Executor: Finished task 0.0 in stage 12.0 (TID 12). 1782 bytes result sent to driver
21/03/18 20:35:23 INFO TaskSetManager: Finished task 0.0 in stage 12.0 (TID 12) in 5 ms on localhost (executor driver) (1/1)
21/03/18 20:35:23 INFO TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool 
21/03/18 20:35:23 INFO DAGScheduler: ResultStage 12 (count at utils.scala:135) finished in 0.010 s
21/03/18 20:35:23 INFO DAGScheduler: Job 8 finished: count at utils.scala:135, took 0.034488 s
21/03/18 20:35:23 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:23 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:23 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:23 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:23 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:23 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:23 INFO FileSourceStrategy: Pruning directories with: 
21/03/18 20:35:23 INFO FileSourceStrategy: Post-Scan Filters: 
21/03/18 20:35:23 INFO FileSourceStrategy: Output Data Schema: struct<>
21/03/18 20:35:23 INFO FileSourceScanExec: Pushed Filters: 
21/03/18 20:35:23 INFO CodeGenerator: Code generated in 9.694592 ms
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated size 284.5 KB, free 911.3 MB)
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 23.9 KB, free 911.3 MB)
21/03/18 20:35:23 INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on localhost:64880 (size: 23.9 KB, free: 912.2 MB)
21/03/18 20:35:23 INFO SparkContext: Created broadcast 15 from count at NativeMethodAccessorImpl.java:0
21/03/18 20:35:23 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
21/03/18 20:35:23 INFO SparkContext: Starting job: count at NativeMethodAccessorImpl.java:0
21/03/18 20:35:23 INFO DAGScheduler: Registering RDD 53 (count at NativeMethodAccessorImpl.java:0)
21/03/18 20:35:23 INFO DAGScheduler: Got job 9 (count at NativeMethodAccessorImpl.java:0) with 1 output partitions
21/03/18 20:35:23 INFO DAGScheduler: Final stage: ResultStage 14 (count at NativeMethodAccessorImpl.java:0)
21/03/18 20:35:23 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 13)
21/03/18 20:35:23 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 13)
21/03/18 20:35:23 INFO DAGScheduler: Submitting ShuffleMapStage 13 (MapPartitionsRDD[53] at count at NativeMethodAccessorImpl.java:0), which has no missing parents
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_16 stored as values in memory (estimated size 14.2 KB, free 911.3 MB)
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 8.0 KB, free 911.3 MB)
21/03/18 20:35:23 INFO BlockManagerInfo: Added broadcast_16_piece0 in memory on localhost:64880 (size: 8.0 KB, free: 912.2 MB)
21/03/18 20:35:23 INFO SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:23 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 13 (MapPartitionsRDD[53] at count at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:23 INFO TaskSchedulerImpl: Adding task set 13.0 with 1 tasks
21/03/18 20:35:23 INFO TaskSetManager: Starting task 0.0 in stage 13.0 (TID 13, localhost, executor driver, partition 0, PROCESS_LOCAL, 8350 bytes)
21/03/18 20:35:23 INFO Executor: Running task 0.0 in stage 13.0 (TID 13)
21/03/18 20:35:23 INFO FileScanRDD: Reading File path: file:///var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/RtmpxYR9UR/file4c9011f87f12, range: 0-1303, partition values: [empty row]
21/03/18 20:35:23 INFO Executor: Finished task 0.0 in stage 13.0 (TID 13). 1585 bytes result sent to driver
21/03/18 20:35:23 INFO TaskSetManager: Finished task 0.0 in stage 13.0 (TID 13) in 37 ms on localhost (executor driver) (1/1)
21/03/18 20:35:23 INFO TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool 
21/03/18 20:35:23 INFO DAGScheduler: ShuffleMapStage 13 (count at NativeMethodAccessorImpl.java:0) finished in 0.050 s
21/03/18 20:35:23 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:35:23 INFO DAGScheduler: running: Set()
21/03/18 20:35:23 INFO DAGScheduler: waiting: Set(ResultStage 14)
21/03/18 20:35:23 INFO DAGScheduler: failed: Set()
21/03/18 20:35:23 INFO DAGScheduler: Submitting ResultStage 14 (MapPartitionsRDD[56] at count at NativeMethodAccessorImpl.java:0), which has no missing parents
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 7.1 KB, free 911.2 MB)
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 3.8 KB, free 911.2 MB)
21/03/18 20:35:23 INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on localhost:64880 (size: 3.8 KB, free: 912.2 MB)
21/03/18 20:35:23 INFO SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:23 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 14 (MapPartitionsRDD[56] at count at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:23 INFO TaskSchedulerImpl: Adding task set 14.0 with 1 tasks
21/03/18 20:35:23 INFO TaskSetManager: Starting task 0.0 in stage 14.0 (TID 14, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:35:23 INFO Executor: Running task 0.0 in stage 14.0 (TID 14)
21/03/18 20:35:23 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:35:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/03/18 20:35:23 INFO Executor: Finished task 0.0 in stage 14.0 (TID 14). 1782 bytes result sent to driver
21/03/18 20:35:23 INFO TaskSetManager: Finished task 0.0 in stage 14.0 (TID 14) in 5 ms on localhost (executor driver) (1/1)
21/03/18 20:35:23 INFO TaskSchedulerImpl: Removed TaskSet 14.0, whose tasks have all completed, from pool 
21/03/18 20:35:23 INFO DAGScheduler: ResultStage 14 (count at NativeMethodAccessorImpl.java:0) finished in 0.011 s
21/03/18 20:35:23 INFO DAGScheduler: Job 9 finished: count at NativeMethodAccessorImpl.java:0, took 0.065193 s
21/03/18 20:35:23 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:23 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:23 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:23 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:23 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:23 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:23 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:23 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:23 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:23 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:23 INFO FileSourceStrategy: Pruning directories with: 
21/03/18 20:35:23 INFO FileSourceStrategy: Post-Scan Filters: 
21/03/18 20:35:23 INFO FileSourceStrategy: Output Data Schema: struct<>
21/03/18 20:35:23 INFO FileSourceScanExec: Pushed Filters: 
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 284.5 KB, free 911.0 MB)
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 23.9 KB, free 910.9 MB)
21/03/18 20:35:23 INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on localhost:64880 (size: 23.9 KB, free: 912.2 MB)
21/03/18 20:35:23 INFO SparkContext: Created broadcast 18 from count at NativeMethodAccessorImpl.java:0
21/03/18 20:35:23 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
21/03/18 20:35:23 INFO SparkContext: Starting job: count at NativeMethodAccessorImpl.java:0
21/03/18 20:35:23 INFO DAGScheduler: Registering RDD 59 (count at NativeMethodAccessorImpl.java:0)
21/03/18 20:35:23 INFO DAGScheduler: Got job 10 (count at NativeMethodAccessorImpl.java:0) with 1 output partitions
21/03/18 20:35:23 INFO DAGScheduler: Final stage: ResultStage 16 (count at NativeMethodAccessorImpl.java:0)
21/03/18 20:35:23 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 15)
21/03/18 20:35:23 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 15)
21/03/18 20:35:23 INFO DAGScheduler: Submitting ShuffleMapStage 15 (MapPartitionsRDD[59] at count at NativeMethodAccessorImpl.java:0), which has no missing parents
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 14.2 KB, free 910.9 MB)
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 8.0 KB, free 910.9 MB)
21/03/18 20:35:23 INFO BlockManagerInfo: Added broadcast_19_piece0 in memory on localhost:64880 (size: 8.0 KB, free: 912.1 MB)
21/03/18 20:35:23 INFO SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:23 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 15 (MapPartitionsRDD[59] at count at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:23 INFO TaskSchedulerImpl: Adding task set 15.0 with 1 tasks
21/03/18 20:35:23 INFO TaskSetManager: Starting task 0.0 in stage 15.0 (TID 15, localhost, executor driver, partition 0, PROCESS_LOCAL, 8350 bytes)
21/03/18 20:35:23 INFO Executor: Running task 0.0 in stage 15.0 (TID 15)
21/03/18 20:35:23 INFO FileScanRDD: Reading File path: file:///var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/RtmpxYR9UR/file4c9011f87f12, range: 0-2539, partition values: [empty row]
21/03/18 20:35:23 INFO Executor: Finished task 0.0 in stage 15.0 (TID 15). 1585 bytes result sent to driver
21/03/18 20:35:23 INFO TaskSetManager: Finished task 0.0 in stage 15.0 (TID 15) in 15 ms on localhost (executor driver) (1/1)
21/03/18 20:35:23 INFO TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool 
21/03/18 20:35:23 INFO DAGScheduler: ShuffleMapStage 15 (count at NativeMethodAccessorImpl.java:0) finished in 0.028 s
21/03/18 20:35:23 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:35:23 INFO DAGScheduler: running: Set()
21/03/18 20:35:23 INFO DAGScheduler: waiting: Set(ResultStage 16)
21/03/18 20:35:23 INFO DAGScheduler: failed: Set()
21/03/18 20:35:23 INFO DAGScheduler: Submitting ResultStage 16 (MapPartitionsRDD[62] at count at NativeMethodAccessorImpl.java:0), which has no missing parents
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 7.1 KB, free 910.9 MB)
21/03/18 20:35:23 INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 3.8 KB, free 910.9 MB)
21/03/18 20:35:23 INFO BlockManagerInfo: Added broadcast_20_piece0 in memory on localhost:64880 (size: 3.8 KB, free: 912.1 MB)
21/03/18 20:35:23 INFO SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:23 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 16 (MapPartitionsRDD[62] at count at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:23 INFO TaskSchedulerImpl: Adding task set 16.0 with 1 tasks
21/03/18 20:35:23 INFO TaskSetManager: Starting task 0.0 in stage 16.0 (TID 16, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:35:23 INFO Executor: Running task 0.0 in stage 16.0 (TID 16)
21/03/18 20:35:23 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:35:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/03/18 20:35:23 INFO Executor: Finished task 0.0 in stage 16.0 (TID 16). 1782 bytes result sent to driver
21/03/18 20:35:23 INFO TaskSetManager: Finished task 0.0 in stage 16.0 (TID 16) in 6 ms on localhost (executor driver) (1/1)
21/03/18 20:35:23 INFO TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool 
21/03/18 20:35:23 INFO DAGScheduler: ResultStage 16 (count at NativeMethodAccessorImpl.java:0) finished in 0.011 s
21/03/18 20:35:23 INFO DAGScheduler: Job 10 finished: count at NativeMethodAccessorImpl.java:0, took 0.044151 s
21/03/18 20:35:23 INFO SparkContext: Invoking stop() from shutdown hook
21/03/18 20:35:23 INFO SparkUI: Stopped Spark web UI at http://localhost:4040
21/03/18 20:35:23 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/03/18 20:35:23 INFO MemoryStore: MemoryStore cleared
21/03/18 20:35:23 INFO BlockManager: BlockManager stopped
21/03/18 20:35:23 INFO BlockManagerMaster: BlockManagerMaster stopped
21/03/18 20:35:23 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/03/18 20:35:23 INFO SparkContext: Successfully stopped SparkContext
21/03/18 20:35:23 INFO ShutdownHookManager: Shutdown hook called
21/03/18 20:35:23 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-375d3165-3544-4bb2-a3b3-f92f73ee6867
21/03/18 20:35:23 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-a1b647e2-3864-437e-ac90-db2e66e7fc93
21/03/18 20:35:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/03/18 20:35:27 INFO SparkContext: Running Spark version 2.4.3
21/03/18 20:35:27 INFO SparkContext: Submitted application: sparklyr
21/03/18 20:35:27 INFO SecurityManager: Changing view acls to: nathan
21/03/18 20:35:27 INFO SecurityManager: Changing modify acls to: nathan
21/03/18 20:35:27 INFO SecurityManager: Changing view acls groups to: 
21/03/18 20:35:27 INFO SecurityManager: Changing modify acls groups to: 
21/03/18 20:35:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(nathan); groups with view permissions: Set(); users  with modify permissions: Set(nathan); groups with modify permissions: Set()
21/03/18 20:35:27 INFO Utils: Successfully started service 'sparkDriver' on port 65028.
21/03/18 20:35:28 INFO SparkEnv: Registering MapOutputTracker
21/03/18 20:35:28 INFO SparkEnv: Registering BlockManagerMaster
21/03/18 20:35:28 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/03/18 20:35:28 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/03/18 20:35:28 INFO DiskBlockManager: Created local directory at /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/blockmgr-47b2deb3-193d-4d77-943d-457960bb84d9
21/03/18 20:35:28 INFO MemoryStore: MemoryStore started with capacity 912.3 MB
21/03/18 20:35:28 INFO SparkEnv: Registering OutputCommitCoordinator
21/03/18 20:35:28 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/03/18 20:35:28 INFO SparkUI: Bound SparkUI to 127.0.0.1, and started at http://localhost:4040
21/03/18 20:35:28 INFO SparkContext: Added JAR file:/Library/Frameworks/R.framework/Versions/4.0/Resources/library/sparklyr/java/sparklyr-2.4-2.11.jar at spark://localhost:65028/jars/sparklyr-2.4-2.11.jar with timestamp 1616096128291
21/03/18 20:35:28 INFO Executor: Starting executor ID driver on host localhost
21/03/18 20:35:28 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 65032.
21/03/18 20:35:28 INFO NettyBlockTransferService: Server created on localhost:65032
21/03/18 20:35:28 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/03/18 20:35:28 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, localhost, 65032, None)
21/03/18 20:35:28 INFO BlockManagerMasterEndpoint: Registering block manager localhost:65032 with 912.3 MB RAM, BlockManagerId(driver, localhost, 65032, None)
21/03/18 20:35:28 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, localhost, 65032, None)
21/03/18 20:35:28 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, localhost, 65032, None)
21/03/18 20:35:28 INFO SharedState: loading hive config file: file:/Users/nathan/spark/spark-2.4.3-bin-hadoop2.7/conf/hive-site.xml
21/03/18 20:35:28 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse').
21/03/18 20:35:28 INFO SharedState: Warehouse path is 'file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse'.
21/03/18 20:35:29 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
21/03/18 20:35:31 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
21/03/18 20:35:32 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
21/03/18 20:35:32 INFO ObjectStore: ObjectStore, initialize called
21/03/18 20:35:32 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
21/03/18 20:35:32 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
21/03/18 20:35:34 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
21/03/18 20:35:36 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:35:36 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:35:36 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:35:36 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:35:37 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
21/03/18 20:35:37 INFO ObjectStore: Initialized ObjectStore
21/03/18 20:35:37 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
21/03/18 20:35:37 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
21/03/18 20:35:37 INFO HiveMetaStore: Added admin role in metastore
21/03/18 20:35:37 INFO HiveMetaStore: Added public role in metastore
21/03/18 20:35:37 INFO HiveMetaStore: No user is added in admin role, since config is empty
21/03/18 20:35:37 INFO HiveMetaStore: 0: get_all_databases
21/03/18 20:35:37 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_all_databases	
21/03/18 20:35:37 INFO HiveMetaStore: 0: get_functions: db=default pat=*
21/03/18 20:35:37 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_functions: db=default pat=*	
21/03/18 20:35:37 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
21/03/18 20:35:37 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/be1064df-ad83-4414-8f08-0b1503754c54_resources
21/03/18 20:35:37 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/be1064df-ad83-4414-8f08-0b1503754c54
21/03/18 20:35:37 INFO SessionState: Created local directory: /var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/nathan/be1064df-ad83-4414-8f08-0b1503754c54
21/03/18 20:35:37 INFO SessionState: Created HDFS directory: /tmp/hive/nathan/be1064df-ad83-4414-8f08-0b1503754c54/_tmp_space.db
21/03/18 20:35:37 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is file:/Users/nathan/Documents/work/NE%20Data%20Ltd/projects/catalog/inst/tinytest/spark-warehouse
21/03/18 20:35:37 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:37 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:37 INFO HiveMetaStore: 0: get_database: global_temp
21/03/18 20:35:37 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: global_temp	
21/03/18 20:35:37 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
21/03/18 20:35:37 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:37 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:37 INFO HiveMetaStore: 0: get_database: default
21/03/18 20:35:37 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_database: default	
21/03/18 20:35:37 INFO HiveMetaStore: 0: get_tables: db=default pat=*
21/03/18 20:35:37 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_tables: db=default pat=*	
21/03/18 20:35:38 INFO SparkContext: Starting job: collect at utils.scala:61
21/03/18 20:35:38 INFO DAGScheduler: Got job 0 (collect at utils.scala:61) with 1 output partitions
21/03/18 20:35:38 INFO DAGScheduler: Final stage: ResultStage 0 (collect at utils.scala:61)
21/03/18 20:35:38 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:35:38 INFO DAGScheduler: Missing parents: List()
21/03/18 20:35:38 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[4] at map at utils.scala:54), which has no missing parents
21/03/18 20:35:38 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 6.0 KB, free 912.3 MB)
21/03/18 20:35:38 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.4 KB, free 912.3 MB)
21/03/18 20:35:38 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:65032 (size: 3.4 KB, free: 912.3 MB)
21/03/18 20:35:38 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:38 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[4] at map at utils.scala:54) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:38 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
21/03/18 20:35:38 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7893 bytes)
21/03/18 20:35:38 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/03/18 20:35:38 INFO Executor: Fetching spark://localhost:65028/jars/sparklyr-2.4-2.11.jar with timestamp 1616096128291
21/03/18 20:35:38 INFO TransportClientFactory: Successfully created connection to localhost/127.0.0.1:65028 after 16 ms (0 ms spent in bootstraps)
21/03/18 20:35:38 INFO Utils: Fetching spark://localhost:65028/jars/sparklyr-2.4-2.11.jar to /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-24d5bf83-bb31-491f-abb2-832ea931a77d/userFiles-53328485-9190-4343-8f7e-3670a1765288/fetchFileTemp3457528113814610788.tmp
21/03/18 20:35:38 INFO Executor: Adding file:/private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-24d5bf83-bb31-491f-abb2-832ea931a77d/userFiles-53328485-9190-4343-8f7e-3670a1765288/sparklyr-2.4-2.11.jar to class loader
21/03/18 20:35:39 INFO CodeGenerator: Code generated in 217.480913 ms
21/03/18 20:35:39 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1013 bytes result sent to driver
21/03/18 20:35:39 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 563 ms on localhost (executor driver) (1/1)
21/03/18 20:35:39 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
21/03/18 20:35:39 INFO DAGScheduler: ResultStage 0 (collect at utils.scala:61) finished in 0.813 s
21/03/18 20:35:39 INFO DAGScheduler: Job 0 finished: collect at utils.scala:61, took 0.867103 s
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 12
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 3
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 18
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 15
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 19
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 21
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 24
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 2
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 13
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 23
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 4
21/03/18 20:35:39 INFO BlockManagerInfo: Removed broadcast_0_piece0 on localhost:65032 in memory (size: 3.4 KB, free: 912.3 MB)
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 20
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 14
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 17
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 11
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 10
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 7
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 9
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 0
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 25
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 5
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 6
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 22
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 1
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 8
21/03/18 20:35:39 INFO ContextCleaner: Cleaned accumulator 16
21/03/18 20:35:39 INFO CodeGenerator: Code generated in 17.058695 ms
21/03/18 20:35:39 INFO CodeGenerator: Code generated in 17.129166 ms
21/03/18 20:35:39 INFO SparkContext: Starting job: collect at utils.scala:135
21/03/18 20:35:39 INFO DAGScheduler: Got job 1 (collect at utils.scala:135) with 1 output partitions
21/03/18 20:35:39 INFO DAGScheduler: Final stage: ResultStage 1 (collect at utils.scala:135)
21/03/18 20:35:39 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:35:39 INFO DAGScheduler: Missing parents: List()
21/03/18 20:35:39 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[9] at collect at utils.scala:135), which has no missing parents
21/03/18 20:35:39 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 6.5 KB, free 912.3 MB)
21/03/18 20:35:39 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.3 KB, free 912.3 MB)
21/03/18 20:35:39 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:65032 (size: 3.3 KB, free: 912.3 MB)
21/03/18 20:35:39 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:39 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[9] at collect at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:39 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
21/03/18 20:35:39 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, PROCESS_LOCAL, 8080 bytes)
21/03/18 20:35:39 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
21/03/18 20:35:39 INFO CodeGenerator: Code generated in 7.475465 ms
21/03/18 20:35:39 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1329 bytes result sent to driver
21/03/18 20:35:39 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 26 ms on localhost (executor driver) (1/1)
21/03/18 20:35:39 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
21/03/18 20:35:39 INFO DAGScheduler: ResultStage 1 (collect at utils.scala:135) finished in 0.032 s
21/03/18 20:35:39 INFO DAGScheduler: Job 1 finished: collect at utils.scala:135, took 0.034884 s
21/03/18 20:35:40 INFO CodeGenerator: Code generated in 12.800769 ms
21/03/18 20:35:40 INFO CodeGenerator: Code generated in 34.456612 ms
21/03/18 20:35:40 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:35:40 INFO DAGScheduler: Registering RDD 12 (count at utils.scala:135)
21/03/18 20:35:40 INFO DAGScheduler: Got job 2 (count at utils.scala:135) with 1 output partitions
21/03/18 20:35:40 INFO DAGScheduler: Final stage: ResultStage 3 (count at utils.scala:135)
21/03/18 20:35:40 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
21/03/18 20:35:40 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 2)
21/03/18 20:35:40 INFO DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[12] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:40 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 8.4 KB, free 912.3 MB)
21/03/18 20:35:40 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 4.5 KB, free 912.3 MB)
21/03/18 20:35:40 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:65032 (size: 4.5 KB, free: 912.3 MB)
21/03/18 20:35:40 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:40 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[12] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:40 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
21/03/18 20:35:40 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, executor driver, partition 0, PROCESS_LOCAL, 8069 bytes)
21/03/18 20:35:40 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
21/03/18 20:35:40 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1542 bytes result sent to driver
21/03/18 20:35:40 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 67 ms on localhost (executor driver) (1/1)
21/03/18 20:35:40 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 
21/03/18 20:35:40 INFO DAGScheduler: ShuffleMapStage 2 (count at utils.scala:135) finished in 0.084 s
21/03/18 20:35:40 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:35:40 INFO DAGScheduler: running: Set()
21/03/18 20:35:40 INFO DAGScheduler: waiting: Set(ResultStage 3)
21/03/18 20:35:40 INFO DAGScheduler: failed: Set()
21/03/18 20:35:40 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[15] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:40 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 7.1 KB, free 912.3 MB)
21/03/18 20:35:40 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 3.8 KB, free 912.3 MB)
21/03/18 20:35:40 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:65032 (size: 3.8 KB, free: 912.3 MB)
21/03/18 20:35:40 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:40 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[15] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:40 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
21/03/18 20:35:40 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:35:40 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
21/03/18 20:35:40 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:35:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 11 ms
21/03/18 20:35:40 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1782 bytes result sent to driver
21/03/18 20:35:40 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 58 ms on localhost (executor driver) (1/1)
21/03/18 20:35:40 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 
21/03/18 20:35:40 INFO DAGScheduler: ResultStage 3 (count at utils.scala:135) finished in 0.070 s
21/03/18 20:35:40 INFO DAGScheduler: Job 2 finished: count at utils.scala:135, took 0.182037 s
21/03/18 20:35:41 INFO SparkContext: Starting job: collect at utils.scala:135
21/03/18 20:35:41 INFO DAGScheduler: Got job 3 (collect at utils.scala:135) with 1 output partitions
21/03/18 20:35:41 INFO DAGScheduler: Final stage: ResultStage 4 (collect at utils.scala:135)
21/03/18 20:35:41 INFO DAGScheduler: Parents of final stage: List()
21/03/18 20:35:41 INFO DAGScheduler: Missing parents: List()
21/03/18 20:35:41 INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[18] at collect at utils.scala:135), which has no missing parents
21/03/18 20:35:41 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 6.5 KB, free 912.3 MB)
21/03/18 20:35:41 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 3.3 KB, free 912.3 MB)
21/03/18 20:35:41 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:65032 (size: 3.3 KB, free: 912.3 MB)
21/03/18 20:35:41 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:41 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[18] at collect at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:41 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
21/03/18 20:35:41 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, localhost, executor driver, partition 0, PROCESS_LOCAL, 8080 bytes)
21/03/18 20:35:41 INFO Executor: Running task 0.0 in stage 4.0 (TID 4)
21/03/18 20:35:41 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4). 1286 bytes result sent to driver
21/03/18 20:35:41 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 13 ms on localhost (executor driver) (1/1)
21/03/18 20:35:41 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 
21/03/18 20:35:41 INFO DAGScheduler: ResultStage 4 (collect at utils.scala:135) finished in 0.019 s
21/03/18 20:35:41 INFO DAGScheduler: Job 3 finished: collect at utils.scala:135, took 0.024480 s
21/03/18 20:35:41 INFO SparkContext: Starting job: count at utils.scala:135
21/03/18 20:35:41 INFO DAGScheduler: Registering RDD 21 (count at utils.scala:135)
21/03/18 20:35:41 INFO DAGScheduler: Got job 4 (count at utils.scala:135) with 1 output partitions
21/03/18 20:35:41 INFO DAGScheduler: Final stage: ResultStage 6 (count at utils.scala:135)
21/03/18 20:35:41 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 5)
21/03/18 20:35:41 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 5)
21/03/18 20:35:41 INFO DAGScheduler: Submitting ShuffleMapStage 5 (MapPartitionsRDD[21] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:41 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 8.4 KB, free 912.2 MB)
21/03/18 20:35:41 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 4.5 KB, free 912.2 MB)
21/03/18 20:35:41 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:65032 (size: 4.5 KB, free: 912.3 MB)
21/03/18 20:35:41 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:41 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 5 (MapPartitionsRDD[21] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:41 INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
21/03/18 20:35:41 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, localhost, executor driver, partition 0, PROCESS_LOCAL, 8069 bytes)
21/03/18 20:35:41 INFO Executor: Running task 0.0 in stage 5.0 (TID 5)
21/03/18 20:35:41 INFO Executor: Finished task 0.0 in stage 5.0 (TID 5). 1499 bytes result sent to driver
21/03/18 20:35:41 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 11 ms on localhost (executor driver) (1/1)
21/03/18 20:35:41 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
21/03/18 20:35:41 INFO DAGScheduler: ShuffleMapStage 5 (count at utils.scala:135) finished in 0.019 s
21/03/18 20:35:41 INFO DAGScheduler: looking for newly runnable stages
21/03/18 20:35:41 INFO DAGScheduler: running: Set()
21/03/18 20:35:41 INFO DAGScheduler: waiting: Set(ResultStage 6)
21/03/18 20:35:41 INFO DAGScheduler: failed: Set()
21/03/18 20:35:41 INFO DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[24] at count at utils.scala:135), which has no missing parents
21/03/18 20:35:41 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 7.1 KB, free 912.2 MB)
21/03/18 20:35:41 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 3.8 KB, free 912.2 MB)
21/03/18 20:35:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:65032 (size: 3.8 KB, free: 912.3 MB)
21/03/18 20:35:41 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1161
21/03/18 20:35:41 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 6 (MapPartitionsRDD[24] at count at utils.scala:135) (first 15 tasks are for partitions Vector(0))
21/03/18 20:35:41 INFO TaskSchedulerImpl: Adding task set 6.0 with 1 tasks
21/03/18 20:35:41 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 6, localhost, executor driver, partition 0, NODE_LOCAL, 7767 bytes)
21/03/18 20:35:41 INFO Executor: Running task 0.0 in stage 6.0 (TID 6)
21/03/18 20:35:41 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
21/03/18 20:35:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/03/18 20:35:41 INFO Executor: Finished task 0.0 in stage 6.0 (TID 6). 1782 bytes result sent to driver
21/03/18 20:35:41 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 6) in 6 ms on localhost (executor driver) (1/1)
21/03/18 20:35:41 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool 
21/03/18 20:35:41 INFO DAGScheduler: ResultStage 6 (count at utils.scala:135) finished in 0.013 s
21/03/18 20:35:41 INFO DAGScheduler: Job 4 finished: count at utils.scala:135, took 0.037462 s
21/03/18 20:35:41 INFO HiveMetaStore: 0: get_table : db=default tbl=mtcars
21/03/18 20:35:41 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=mtcars	
21/03/18 20:35:41 INFO HiveMetaStore: 0: get_table : db=default tbl=catalog_fake_table
21/03/18 20:35:41 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=catalog_fake_table	
21/03/18 20:35:41 INFO HiveMetaStore: 0: get_table : db=default tbl=catalog_fake_table
21/03/18 20:35:41 INFO audit: ugi=nathan	ip=unknown-ip-addr	cmd=get_table : db=default tbl=catalog_fake_table	
21/03/18 20:35:41 INFO SparkContext: Invoking stop() from shutdown hook
21/03/18 20:35:41 INFO SparkUI: Stopped Spark web UI at http://localhost:4040
21/03/18 20:35:41 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/03/18 20:35:41 INFO MemoryStore: MemoryStore cleared
21/03/18 20:35:41 INFO BlockManager: BlockManager stopped
21/03/18 20:35:41 INFO BlockManagerMaster: BlockManagerMaster stopped
21/03/18 20:35:41 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/03/18 20:35:41 INFO SparkContext: Successfully stopped SparkContext
21/03/18 20:35:41 INFO ShutdownHookManager: Shutdown hook called
21/03/18 20:35:41 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-2f208d92-e631-45dc-ab9b-f4d7cc110972
21/03/18 20:35:41 INFO ShutdownHookManager: Deleting directory /private/var/folders/x3/lznkklbd17n2ljvwmyg8g1vc0000gn/T/spark-24d5bf83-bb31-491f-abb2-832ea931a77d
