思享工具箱

spark on yarn提交任务时报ClosedChannelException解决方案

摘要：
spark2.1出来了，想玩玩就搭了个原生的apache集群，但在standalone模式下没有任何问题，基于apachehadoop2.7.3使用sparkonyarn一直报这个错。

spark2.1出来了，想玩玩就搭了个原生的apache集群，但在standalone模式下没有任何问题，基于apache hadoop 2.7.3使用spark on yarn一直报这个错。(Java 8)

报错日志如下：

Warning: Master yarn-client is deprecated since 2.0. Please use master "yarn"with specified deploy mode instead.
17/01/31 00:12:11 INFO spark.SparkContext: Running Spark version 2.1.0
17/01/31 00:12:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/01/31 00:12:11INFO spark.SecurityManager: Changing view acls to: root
17/01/31 00:12:11INFO spark.SecurityManager: Changing modify acls to: root
17/01/31 00:12:11 INFO spark.SecurityManager: Changing view acls groupsto: 
17/01/31 00:12:11 INFO spark.SecurityManager: Changing modify acls groupsto: 
17/01/31 00:12:11 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groupswith modify permissions: Set()
17/01/31 00:12:11 INFO util.Utils: Successfully started service 'sparkDriver' on port 59775.
17/01/31 00:12:11INFO spark.SparkEnv: Registering MapOutputTracker
17/01/31 00:12:11INFO spark.SparkEnv: Registering BlockManagerMaster
17/01/31 00:12:11 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper forgetting topology information
17/01/31 00:12:11INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
17/01/31 00:12:11 INFO storage.DiskBlockManager: Created local directory at /opt/program/spark-2.1.0-bin-hadoop2.7/blockmgr-b04fc6c2-501f-4df4-ae13-f6fb0aaa6470
17/01/31 00:12:11 INFO memory.MemoryStore: MemoryStore started with capacity 366.3MB
17/01/31 00:12:11INFO spark.SparkEnv: Registering OutputCommitCoordinator
17/01/31 00:12:12INFO util.log: Logging initialized @2406ms
17/01/31 00:12:12 INFO server.Server: jetty-9.2.z-SNAPSHOT
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@64712be{/jobs,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@53499d85{/jobs/json,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@30ed9c6c{/jobs/job,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@782a4fff{/jobs/job/json,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@46c670a6{/stages,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@59fc684e{/stages/json,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5ae81e1{/stages/stage,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2fd1731c{/stages/stage/json,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5ae76500{/stages/pool,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6063d80a{/stages/pool/json,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1133ec6e{/storage,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@355e34c7{/storage/json,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@54709809{/storage/rdd,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2a2da905{/storage/rdd/json,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@24f360b2{/environment,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@60cf80e7{/environment/json,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@302fec27{/executors,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@770d0ea6{/executors/json,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@48c40605{/executors/threadDump,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@54107f42{/executors/threadDump/json,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1b11ef33{/static,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@476aac9{/,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6cea706c{/api,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3bd7f8dc{/jobs/job/kill,null,AVAILABLE}
17/01/31 00:12:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2f2bf0e2{/stages/stage/kill,null,AVAILABLE}
17/01/31 00:12:12 INFO server.ServerConnector: Started ServerConnector@780ec4a5{HTTP/1.1}{0.0.0.0:4040}
17/01/31 00:12:12INFO server.Server: Started @2508ms
17/01/31 00:12:12 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
17/01/31 00:12:12 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.56.101:4040
17/01/31 00:12:12 INFO spark.SparkContext: Added JAR file:/opt/spark/examples/jars/spark-examples_2.11-2.1.0.jar at spark://192.168.56.101:59775/jars/spark-examples_2.11-2.1.0.jar with timestamp 1485792732176
17/01/31 00:12:12 INFO client.RMProxy: Connecting to ResourceManager at node01/192.168.56.101:8032
17/01/31 00:12:13 INFO yarn.Client: Requesting a new application from cluster with 3NodeManagers
17/01/31 00:12:13 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192MB per container)
17/01/31 00:12:13 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384MB overhead
17/01/31 00:12:13 INFO yarn.Client: Setting up container launch context forour AM
17/01/31 00:12:13 INFO yarn.Client: Setting up the launch environment forour AM container
17/01/31 00:12:13 INFO yarn.Client: Preparing resources forour AM container
17/01/31 00:12:13WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
17/01/31 00:12:15 INFO yarn.Client: Uploading resource file:/opt/program/spark-2.1.0-bin-hadoop2.7/spark-6cafe3cb-9ed2-4f7f-b44f-a2bb447eaa30/__spark_libs__493282781411356296.zip -> hdfs://node01:9000/user/root/.sparkStaging/application_1485792095366_0003/__spark_libs__493282781411356296.zip
17/01/31 00:12:17 INFO yarn.Client: Uploading resource file:/opt/program/spark-2.1.0-bin-hadoop2.7/spark-6cafe3cb-9ed2-4f7f-b44f-a2bb447eaa30/__spark_conf__2188039824841197723.zip -> hdfs://node01:9000/user/root/.sparkStaging/application_1485792095366_0003/__spark_conf__.zip
17/01/31 00:12:17INFO spark.SecurityManager: Changing view acls to: root
17/01/31 00:12:17INFO spark.SecurityManager: Changing modify acls to: root
17/01/31 00:12:17 INFO spark.SecurityManager: Changing view acls groupsto: 
17/01/31 00:12:17 INFO spark.SecurityManager: Changing modify acls groupsto: 
17/01/31 00:12:17 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groupswith modify permissions: Set()
17/01/31 00:12:17INFO yarn.Client: Submitting application application_1485792095366_0003 to ResourceManager
17/01/31 00:12:18INFO impl.YarnClientImpl: Submitted application application_1485792095366_0003
17/01/31 00:12:18INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1485792095366_0003 and attemptId None
17/01/31 00:12:19 INFO yarn.Client: Application report forapplication_1485792095366_0003 (state: ACCEPTED)
17/01/31 00:12:19INFO yarn.Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1queue: default
         start time: 1485792737991final status: UNDEFINED
         tracking URL: http://node01:8088/proxy/application_1485792095366_0003/
user: root
17/01/31 00:12:20 INFO yarn.Client: Application report forapplication_1485792095366_0003 (state: ACCEPTED)
17/01/31 00:12:21 INFO yarn.Client: Application report forapplication_1485792095366_0003 (state: ACCEPTED)
17/01/31 00:12:21 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null)
17/01/31 00:12:21 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> node01, PROXY_URI_BASES -> http://node01:8088/proxy/application_1485792095366_0003), /proxy/application_1485792095366_0003
17/01/31 00:12:21INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/01/31 00:12:22 INFO yarn.Client: Application report forapplication_1485792095366_0003 (state: ACCEPTED)
17/01/31 00:12:23 INFO yarn.Client: Application report forapplication_1485792095366_0003 (state: RUNNING)
17/01/31 00:12:23INFO yarn.Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: 192.168.56.101ApplicationMaster RPC port: 0queue: default
         start time: 1485792737991final status: UNDEFINED
         tracking URL: http://node01:8088/proxy/application_1485792095366_0003/
user: root
17/01/31 00:12:23INFO cluster.YarnClientSchedulerBackend: Application application_1485792095366_0003 has started running.
17/01/31 00:12:23 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38405.
17/01/31 00:12:23 INFO netty.NettyBlockTransferService: Server created on 192.168.56.101:38405
17/01/31 00:12:23 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy forblock replication policy
17/01/31 00:12:23 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.56.101, 38405, None)
17/01/31 00:12:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.56.101:38405 with 366.3 MB RAM, BlockManagerId(driver, 192.168.56.101, 38405, None)
17/01/31 00:12:23 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.56.101, 38405, None)
17/01/31 00:12:23 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.56.101, 38405, None)
17/01/31 00:12:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@643ba1ed{/metrics/json,null,AVAILABLE}
17/01/31 00:12:24 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.56.101:43252) with ID 1
17/01/31 00:12:24 INFO storage.BlockManagerMasterEndpoint: Registering block manager node01:53420 with 366.3 MB RAM, BlockManagerId(1, node01, 53420, None)
17/01/31 00:12:25 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 1.
17/01/31 00:12:25 INFO scheduler.DAGScheduler: Executor lost: 1 (epoch 0)
17/01/31 00:12:25 ERROR client.TransportClient: Failed to send RPC 8305478367380188725 to /192.168.56.101:43246: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
17/01/31 00:12:25 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 1from BlockManagerMaster.
17/01/31 00:12:25 INFO storage.BlockManagerMasterEndpoint: Removing block manager BlockManagerId(1, node01, 53420, None)
17/01/31 00:12:25 INFO storage.BlockManagerMaster: Removed 1 successfully inremoveExecutor
17/01/31 00:12:25 INFO scheduler.DAGScheduler: Shuffle files lost for executor: 1 (epoch 0)
17/01/31 00:12:25 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to get executor loss reason for executor id 1 at RPC address 192.168.56.101:43252, but got no response. Marking as slave lost.
java.io.IOException: Failed to send RPC 8305478367380188725 to /192.168.56.101:43246: java.nio.channels.ClosedChannelException
        at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:249)
        at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:233)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:488)
        at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
        at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:438)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedChannelException
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
17/01/31 00:12:25 ERROR cluster.YarnScheduler: Lost executor 1on node01: Slave lost
17/01/31 00:12:26 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null)
17/01/31 00:12:26 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> node01, PROXY_URI_BASES -> http://node01:8088/proxy/application_1485792095366_0003), /proxy/application_1485792095366_0003
17/01/31 00:12:26INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/01/31 00:12:28 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
17/01/31 00:12:28 INFO server.ServerConnector: Stopped ServerConnector@780ec4a5{HTTP/1.1}{0.0.0.0:4040}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2f2bf0e2{/stages/stage/kill,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3bd7f8dc{/jobs/job/kill,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6cea706c{/api,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@476aac9{/,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1b11ef33{/static,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@54107f42{/executors/threadDump/json,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@48c40605{/executors/threadDump,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@770d0ea6{/executors/json,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@302fec27{/executors,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@60cf80e7{/environment/json,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@24f360b2{/environment,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2a2da905{/storage/rdd/json,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@54709809{/storage/rdd,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@355e34c7{/storage/json,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1133ec6e{/storage,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6063d80a{/stages/pool/json,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5ae76500{/stages/pool,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2fd1731c{/stages/stage/json,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5ae81e1{/stages/stage,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@59fc684e{/stages/json,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@46c670a6{/stages,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@782a4fff{/jobs/job/json,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@30ed9c6c{/jobs/job,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@53499d85{/jobs/json,null,UNAVAILABLE}
17/01/31 00:12:28 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@64712be{/jobs,null,UNAVAILABLE}
17/01/31 00:12:28 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.56.101:4040
17/01/31 00:12:28 ERROR client.TransportClient: Failed to send RPC 6600979308376699964 to /192.168.56.103:56283: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
17/01/31 00:12:28ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalStateException: Spark context stopped while waiting forbackend
        at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:614)
        at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:169)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:567)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/01/31 00:12:28 ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map()) to AM was unsuccessful
java.io.IOException: Failed to send RPC 6600979308376699964 to /192.168.56.103:56283: java.nio.channels.ClosedChannelException
        at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:249)
        at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:233)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:488)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427)
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:129)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:852)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:738)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:743)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:735)
        at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:36)
        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1072)
        at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1126)
        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1061)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedChannelException
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
17/01/31 00:12:28INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
17/01/31 00:12:28INFO spark.SparkContext: SparkContext already stopped.
Exception in thread "main" java.lang.IllegalStateException: Spark context stopped while waiting forbackend
        at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:614)
        at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:169)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:567)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
        at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
        at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/01/31 00:12:28 ERROR util.Utils: Uncaught exception inthread Yarn application state monitor
org.apache.spark.SparkException: Exception thrown inawaitResult
        at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
        at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:512)
        at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:93)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:151)
        at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:467)
        at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1588)
        at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1826)
        at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1283)
        at org.apache.spark.SparkContext.stop(SparkContext.scala:1825)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:108)
Caused by: java.io.IOException: Failed to send RPC 6600979308376699964 to /192.168.56.103:56283: java.nio.channels.ClosedChannelException
        at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:249)
        at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:233)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:488)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427)
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:129)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:852)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:738)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:743)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:735)
        at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:36)
        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1072)
        at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1126)
        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1061)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedChannelException
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
17/01/31 00:12:28 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/01/31 00:12:28INFO storage.DiskBlockManager: Shutdown hook called
17/01/31 00:12:28INFO util.ShutdownHookManager: Shutdown hook called
17/01/31 00:12:28INFO memory.MemoryStore: MemoryStore cleared
17/01/31 00:12:28INFO storage.BlockManager: BlockManager stopped
17/01/31 00:12:28 INFO util.ShutdownHookManager: Deleting directory /opt/program/spark-2.1.0-bin-hadoop2.7/spark-6cafe3cb-9ed2-4f7f-b44f-a2bb447eaa30/userFiles-4be7a61f-e6ef-4914-b896-eedb46d78dbc
17/01/31 00:12:28INFO storage.BlockManagerMaster: BlockManagerMaster stopped
17/01/31 00:12:28 INFO util.ShutdownHookManager: Deleting directory /opt/program/spark-2.1.0-bin-hadoop2.7/spark-6cafe3cb-9ed2-4f7f-b44f-a2bb447eaa30
17/01/31 00:12:28 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/01/31 00:12:28 INFO spark.SparkContext: Successfully stopped SparkContext

解决方案：

修改yarn-site.xml，添加下列property

<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>

<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

分析：

按照上述配置提供的信息，目测是我给节点分配的内存太小，yarn直接kill掉了进程，导致ClosedChannelException

文献参考：http://stackoverflow.com/questions/38988941/running-yarn-with-spark-not-working-with-java-8

实测：不修改yarn-site.xml，换成Java 7可正常执行。

标签：#yarn#info | 浏览：115 | 发布日期：

免责声明：文章转载自《spark on yarn提交任务时报ClosedChannelException解决方案》仅用于学习参考。如对内容有疑问，请及时联系本站处理。

上篇Java中关于Date等日期类的简单使用Oracle DBA 必须掌握的 查询脚本：下篇

宿迁高防，2C2G15M，22元/月；香港BGP，2C5G5M，25元/月 雨云优惠码：MjYwNzM=

相关文章

DWARF 中的 Debug Info 格式

本周花了几天的时间来研究怎么在 breakpad [1, 2] 中加入打印函数参数的功能，以期其产生的 callstack 更具可读性，方便定位崩溃原因。 现代 ELF 中的调试信息基本是以 DWARF 格式为主了，因此这几天的研究也主要将时间花在了理解 DWARF 这货是怎么工作上，感叹要把东西做到极致真是件繁琐而细致的事情。关于 DWARF，网上能找到...

Hbase 建表基本命令总结

访问hbase,以及操作hbase，命令不用使用分号hbaseshell进入hbase list查看表hbaseshell-d hbase(main):024:0>scan'.META.' =============小例子=================================================== 1.创建一个表member...

Java中的日志——Java.util.logging、log4j、commons-logging

Java中给项目程序添加log主要有三种方式，一使用JDK中的java.util.logging包，一种是log4j，一种是commons-logging。其中log4j和commons-logging都是apache软件基金会的开源项目。这三种方式的区别如下： Java.util.logging，JDK标准库中的类，是JDK 1.4 版本之后添加的日志记...

MySQL死锁

https://dev.mysql.com/doc/refman/5.7/en/innodb-deadlocks.html 什么是mysql的死锁？ A deadlock is a situation where different transactions are unable to proceed because each holds a lock t...

mysql把表(表已有数据)的某列属性由空到非空，以及常用列操作

//先把CLSF_ID为空的设置默认值 UPDATE aop_dictionaries_info SET CLSF_ID='0' WHERE CLSF_ID='' OR CLSF_ID IS NULL; //把CLSF_ID设置为非空 ALTER TABLE aop_dictionaries_info CHANGE CLSF_ID CLSF_ID varc...

CH5 ResourceManager重启

目录 概述 特性 非工作保留RM重启 工作保留RM重启 配置 Enable RM Restart 配置RM状态的保存 配置工作保留RM恢复 概述 ResourceManager是集群中绝对的资源管理工具，并且调度应用在YARN上运行。因此对于YARN来说是一个单点问题。这个文档介绍RM的重启。 有2种重启的方式： 1.非工作保留RM...

最新文章

随机推荐

思享工具箱导航