pyspark mongodb yarn

摘要:
//pyspark_管理员:admin123@192.168.2.51/pyspark.testpy“).getOrCreate()db_row=my_spark.read.format(”com.mongodb.spark.sql.DefaultSource“).load().collect()http://root@192.168.2.51:24WARNutil.Utils(警告):
from pyspark.sql import SparkSession


my_spark = SparkSession
.builder
.appName("myApp")
.config("spark.mongodb.input.uri", "mongodb://pyspark_admin:admin123@192.168.2.51/pyspark.testpy")
.config("spark.mongodb.output.uri", "mongodb://pyspark_admin:admin123@192.168.2.51/pyspark.testpy")
.getOrCreate()
db_rows = my_spark.read.format("com.mongodb.spark.sql.DefaultSource").load().collect()


 
 
from pyspark.sql import SparkSession


my_spark = SparkSession
.builder
.appName("myAppYarn")
.master('yarn')
.config("spark.mongodb.input.uri", "mongodb://pyspark_admin:admin123@192.168.2.51/pyspark.testpy")
.config("spark.mongodb.output.uri", "mongodb://pyspark_admin:admin123@192.168.2.51/pyspark.testpy")
.getOrCreate()
db_rows = my_spark.read.format("com.mongodb.spark.sql.DefaultSource").load().collect()


http://192.168.2.51:4041/executors/



pyspark mongodb yarn第1张



pyspark mongodb yarn第2张

ssh://root@192.168.2.51:22/usr/bin/python -u /root/.pycharm_helpers/pydev/pydevd.py --multiproc --qt-support=auto --client '0.0.0.0' --port 47232 --file /home/data/crontab_chk_url/pyspark/pyspark_yarn_test.py
pydev debugger: process 9892 is connecting

Connected to pydev debugger (build 172.4343.24)

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/12/03 21:40:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/12/03 21:40:24 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
17/12/03 21:40:26 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.











免责声明:文章转载自《pyspark mongodb yarn》仅用于学习参考。如对内容有疑问,请及时联系本站处理。

上篇[Go] gorm 错误处理 与 链式/Finisher方法服务器渲染和浏览器渲染的区别下篇

宿迁高防,2C2G15M,22元/月;香港BGP,2C5G5M,25元/月 雨云优惠码:MjYwNzM=

相关文章

CALayer 进阶

转载自:http://www.cofcool.net/development/2015/06/19/ios-study-note-eight-CALayer-info/ The CALayer class manages image-based content and allows you to perform animations on that con...

[转]mysql导出导入中文表解决方法

在开发过程中会经常用到mysql导出导入中文表,本文将详细介绍其如何使用,需要的朋友可以参考下。 在开发过程中会经常用到mysql导出导入中文表,本文将详细介绍其如何使用,需要的朋友可以参考下一、先针对utf8导出: (1)导出源数据库的所有表: 代码如下: mysqldump -u root -p密码 --socket=mysql.sock --def...

CentOS 7 下编译安装lnmp之PHP篇详解

一、安装环境 宿主机=> win7,虚拟机 centos => 系统版本:centos-release-7-5.1804.el7.centos.x86_64 二、PHP下载 官网 http://www.php.net/downloads.php 下载 php-7.3.0.tar.bz2 至 /usr/local/src 目录下 解压并移动至...

xinetd网络(2) 协议头解析

1:在/etc/xinetd.d下添加配置文件xhttpd 1 service xhttpd 2 { 3 socket_type = stream //每行“=”前后各最多只能有一个空格 4 protocol= tcp 5 wait = no 6 user =nobody 7 server =/home/username/xhttpd...

FI常用BAPI(转)

总帐会计:  (比较简单全部测试通过,关帐时使用)      Line item of document for ledger with summary table GL   F: BAPI_GLX_GETDOCITEMS        Closing balance of G/L account for chosen year   F: BAPI_GL_...

build.gradle文件详解<三>

 参考:http://blog.csdn.net/baidu_31093133/article/details/51860637 build.gradle配置参数详解      //声明是Android程序      apply plugin: 'com.android.application'      android {   //程序在编译的时候...