Running Spark on YARN

发表于

To launch a Spark application in cluster/yarn mode:

1
$ ./bin/spark-submit --class path.to.your.Class --master yarn --deploy-mode cluster [options] <app jar> [app options]

For example:

1
2
3
4
5
6
7
8
9
$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi \
    --master yarn \
    --deploy-mode cluster \
    --driver-memory 4g \
    --executor-memory 2g \
    --executor-cores 1 \
    --queue thequeue \
    lib/spark-examples*.jar \
    10

See the detail from:
http://spark.apache.org/docs/latest/running-on-yarn.html