extension jupyter Apache Spark JavaScript
Need help with sparkmonitor?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.
krishnan-r

Description

Monitor Apache Spark from Jupyter Notebook

145 Stars 47 Forks Apache License 2.0 213 Commits 28 Opened issues

Services available

Need anything else?

Build Status

Spark Monitor - An extension for Jupyter Notebook

Google Summer of Code - Final Report

For the google summer of code final report of this project click here

About

+ =

SparkMonitor is an extension for Jupyter Notebook that enables the live monitoring of Apache Spark Jobs spawned from a notebook. The extension provides several features to monitor and debug a Spark job from within the notebook interface itself.


jobdisplay

Features

  • Automatically displays a live monitoring tool below cells that run Spark jobs in a Jupyter notebook
  • A table of jobs and stages with progressbars
  • A timeline which shows jobs, stages, and tasks
  • A graph showing number of active tasks & executor cores vs time
  • A notebook server extension that proxies the Spark UI and displays it in an iframe popup for more details
  • For a detailed list of features see the use case notebooks
  • How it Works

Quick Installation

pip install sparkmonitor
jupyter nbextension install sparkmonitor --py --user --symlink 
jupyter nbextension enable sparkmonitor --py --user            
jupyter serverextension enable --py --user sparkmonitor
ipython profile create && echo "c.InteractiveShellApp.extensions.append('sparkmonitor.kernelextension')" >>  $(ipython profile locate default)/ipython_kernel_config.py

For more detailed instructions click here

To do a quick test of the extension:

docker run -it -p 8888:8888 krishnanr/sparkmonitor

Integration with ROOT and SWAN

At CERN, the SparkMonitor extension would find two main use cases: * Distributed analysis with ROOT and Apache Spark using the DistROOT module. Here is an example demonstrating this use case. * Integration with SWAN, A service for web based analysis, via a modified container image for SWAN user sessions.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.