INSIGHTS
6 min read
Published on 12/04/2017
Last updated on 03/21/2024
Apache Spark on Kubernetes series: Introduction to Spark on Kubernetes Scaling Spark made simple on Kubernetes The anatomy of Spark applications on Kubernetes Monitoring Apache Spark with Prometheus Spark History Server on Kubernetes Spark scheduling on Kubernetes demystified Spark Streaming Checkpointing on Kubernetes Deep dive into monitoring Spark and Zeppelin with Prometheus Apache Spark application resilience on Kubernetes
Apache Zeppelin on Kubernetes series: Running Zeppelin Spark notebooks on Kubernetes Running Zeppelin Spark notebooks on Kubernetes - deep dive
Apache Kafka on Kubernetes series: Kafka on Kubernetes - using etcdIn the last post in this series we introduced Spark on Kubernetes and demonstrated a simple approach to scaling a Spark cluster on Kubernetes, the cloud native way. We're building a platform as a service optimized to run big data workloads (and a multitude of microservices) called Pipeline, and we're adding native support for running Apache Zeppelin notebooks on Kubernetes. Apache Zeppelin is a web-based notebook that enables data-driven interactive data analytics, provides built-in integration for Apache Spark, and has about five different interpreters at its disposal to execute Scala, Python, R and SQL code on Spark. These are remotely initiated, and created and managed by
RemoteInterpreterServer
.
To run RemoteInterpreterServer, Zeppelin uses the well known Spark tool, spark-submit. This tool starts by default on every machine Zeppelin runs, consequently starting Spark in embedded
mode. Also, you can run Spark on a Yarn cluster in both client and cluster mode.
Since Kubernetes is growing more and more popular and is a de-facto standard for enterprise level orchestration, it makes more and more sense to start your Spark notebooks on Kubernetes the native way.
There's already a working version of Spark on Kubernetes, in which Spark jobs are started with spark-submit. Because that same tool is used by Zeppelin by default, all we have to do is pass the right parameters. Easy peasy.
Unfortunately - as always - it's not easy peasy.
First, let's go over our chief options insofar as Zeppelin deployment models that run on Kubernetes.
- A Zeppelin server running outside or inside a k8s cluster
- spark-submit started with deployMode = client/cluster
inside
the cluster. In client mode, the submitter launches drivers outside
of the cluster. Client mode is so far unsupported. PR-456 in-cluster client mode let's you specify your deployMode as client. However, this is not truly client mode, but the misleadingly titled in-cluster client
mode, because it only works if you're running spark-submit inside a pod.
The other mode is cluster mode, wherein Spark Driver always runs inside the k8s cluster, but in which the Zeppelin Server can run both inside and outside the cluster. In such cases, the flow of actions is identical to how it's presented in the Spark Flow blog post, with the exception that the actor initiating spark-submit
is the Zeppelin Server itself.
Let's start a notebook in both modes to see a list of running pods. Below is a list of pods for in-cluster client
mode:
<pre class="language-unknown"><code class="language-unknown">NAME READY STATUS RESTARTS AGE
spark-resource-staging-server-55d5d8744b-7r89s 1/1 Running 2 15d
zeppelin-exec-1 1/1 Running 0 2d
zeppelin-exec-2 1/1 Running 0 2d
zeppelin-server 1/1 Running 0 2d</code></pre>
Here's list of pods for cluster
mode:
NAME READY STATUS RESTARTS AGE
spark-resource-staging-server-55d5d8744b-7r89s 1/1 Running 2 15d
zeppelin-2cxx5tw2h--2a94m5j1z-1511971765863-driver 1/1 Running 0 2d
zeppelin-2cxx5tw2h--2a94m5j1z-1511971765863-exec-1 1/1 Running 0 2d
zeppelin-2cxx5tw2h--2a94m5j1z-1511971765863-exec-2 1/1 Running 0 2d
zeppelin-server 1/1 Running 0 2d
As you can see, in cluster
mode, we have a separate pod for the Spark driver, whereas in-cluster client
mode runs that driver inside the zeppelin-server pod. After experimenting with these modes, we have found that cluster
mode has a couple of advantages over in-cluster client
mode. Let's take a look:
- running Zeppelin Server and each RemoteInterpreterServer process (Spark Driver) in separate pods better suits Kubernetes best practices/patterns, instead of having one monolithic RemoteInterpreterServer. The latest Spark Driver creates a separate k8s Service to handle Executor --> Driver connections in cluster mode, which, again, better fit with Kubernetes best practices/patterns
cluster
mode works even if Zeppelin Server in not part of the K8S clusterin-cluster client
mode appears to have some problems at the moment, like executor naming, which doesn't contain any prefixes and causes problems when running multiple interpreters on the same cluster.
cluster
mode seems to be a cleaner approach that better fits with the k8s ecosystem and, at the same time, has no side-effects for those unwilling to use Kubernetes.
A couple of problems we ran into:
- communication between Zeppelin Server and RemoteInterpreterServer happens via a callback mechanism:
CallbackServer
is started by Zeppelin Server then passed to RemoteInterpreterServer. After being started by Spark, RemoteInterpreterServer connects to thisCallbackServer
and sends its connection details. Directly connecting pods to each other may cause problems in a Kubernetes cluster, the preferred method is to interconnect different processes through services. - the current mechanism that sets up dependencies for Spark interpreter in interpreter settings only works if Spark is running in embedded mode
- no separate log4j config as is the case for YARN
- if Zeppelin Server is unable to connect or loses its connection to RemoteInterpreterServer due to a k8s specific problem, the Driver pod remains there forever
Subscribe to
the Shift!
Get emerging insights on emerging technology straight to your inbox.
Unlocking Multi-Cloud Security: Panoptica's Graph-Based Approach
Discover why security teams rely on Panoptica's graph-based technology to navigate and prioritize risks across multi-cloud landscapes, enhancing accuracy and resilience in safeguarding diverse ecosystems.
Subscribe
to
the Shift
!Get on emerging technology straight to your inbox.
emerging insights
The Shift keeps you at the forefront of cloud native modern applications, application security, generative AI, quantum computing, and other groundbreaking innovations that are shaping the future of technology.