This is a Jupyter Kernel for Swift, intended to make it possible to use Jupyter with the Swift for TensorFlow project.
Operating system:
Dependencies:
python3)
python3-venv)
swift-jupyter requires a Swift toolchain with LLDB Python3 support. Currently, the only prebuilt toolchains with LLDB Python3 support are the Swift for TensorFlow Ubuntu 18.04 Nightly Builds. Alternatively, you can build a toolchain from sources (see the section below for instructions).
Extract the Swift toolchain somewhere.
Create a virtualenv, install the requirements in it, and register the kernel in it:
git clone https://github.com/google/swift-jupyter.git cd swift-jupyter python3 -m venv venv . venv/bin/activate pip install -r requirements.txt python register.py --sys-prefix --swift-toolchain
Finally, run Jupyter:
. venv/bin/activate jupyter notebook
You should be able to create Swift notebooks. Installation is done!
Operating system:
swift-jupyter requires a Swift toolchain with LLDB Python3 support. Currently, the only prebuilt toolchains with LLDB Python3 support are the Swift for TensorFlow Ubuntu 18.04 Nightly Builds. Alternatively, you can build a toolchain from sources (see the section below for instructions).
Extract the Swift toolchain somewhere.
Important note about CUDA/CUDNN: If you are using a CUDA toolchain, then you should install CUDA and CUDNN on your system without using Conda, because Conda's CUDNN is too old to work with the Swift toolchain's TensorFlow. (As of 2019-04-08, Swift for TensorFlow requires CUDNN 7.5, but Conda only has CUDNN 7.3).
Create a Conda environment and install some packages in it:
conda create -n swift-tensorflow python==3.6 conda activate swift-tensorflow conda install jupyter numpy matplotlib
Register the Swift kernel with Jupyter:
python register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs \ --swift-toolchain
Finally, run Jupyter:
jupyter notebook
You should be able to create Swift notebooks. Installation is done!
This repository also includes a dockerfile which can be used to run a Jupyter Notebook instance which includes this Swift kernel. To build the container, the following command may be used:
# from inside the directory of this repository docker build -f docker/Dockerfile -t swift-jupyter .
The resulting container comes with the latest Swift for TensorFlow toolchain installed, along with Jupyter and the Swift kernel contained in this repository.
This container can now be run with the following command:
docker run -p 8888:8888 --cap-add SYS_PTRACE -v /my/host/notebooks:/notebooks swift-jupyter
The functions of these parameters are:
-p 8888:8888exposes the port on which Jupyter is running to the host.
--cap-add SYS_PTRACEadjusts the privileges with which this container is run, which is required for the Swift REPL.
-v :/notebooksbind mounts a host directory as a volume where notebooks created in the container will be stored. If this command is omitted, any notebooks created using the container will not be persisted when the container is stopped.
To improve Docker image building, use the new Docker Buildkit system by either setting the
DOCKER_BUILDKITenvironment variable or configuring the Docker
daemon.json. The simplest way is by prepending
DOCKER_BUILDKIT=1to your
docker buildcommand:
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile -t swift-jupyter .
As of Jupyter Notebook 6.0, you can use
--gateway-url=to specify a separate Jupyter Kernel Gateway. (Or use the nb2kg server extension for pre-6.0 versions of Notebook.) This allows running the Swift for Tensorflow Jupyter kernel in a Docker container while running Jupyter Notebook somewhere else, such as your local machine.
First build the basic Swift kernel Docker image (as above), then build the kernel gateway image based on that:
# from inside the directory of this repository docker build -f docker/Dockerfile -t swift-jupyter . docker build -f kernel_gateway/Dockerfile -t swift-kg .
Using the new Docker Buildkit system is recommended, as described in the section above.
Then run the kernel gateway:
docker run -p 9999:9999 --cap-add SYS_PTRACE swift-kg
The functions of these parameters are the same as in the section above.
With the gateway running, start Jupyter Notebook in your notebook directory and pass the URL of your kernel gateway:
jupyter notebook --gateway-url 127.0.0.1:9999
Follow the Building Swift for TensorFlow instructions, with some modifications:
sudo apt-get install libpython3-dev). The LLDB build will automatically find these and build with Python 3 support.
utils/build-script, run
utils/build-toolchain-tensorflow, so that you build a whole toolchain that includes LLDB.
This will create a tar file containing the full toolchain. You can now proceed with the installation instructions from the previous section.
Install the Python 3 development headers. (For Ubuntu 20.04,
sudo apt-get install libpython3-dev).
Have a place to checkout relevant toolchains, and checkout the relevant code:
mkdir /opt/swift && cd /opt/swift git clone https://github.com/apple/llvm-project.git git clone https://github.com/apple/swift.git git clone https://github.com/apple/swift-corelibs-libdispatch.git git clone https://github.com/apple/swift-cmark.git cmark
Make sure you checked out the right branch for all dependencies. For example, the
llvm-projectshould check the branch starting with
swift, such as
swift/release/5.3. You should be able to find the correct branch with name
release/*in all these projects except the
llvm-project.
Go to
swift/utils, and run:
./build-script --release --lldb
This will build LLDB with Python3 support. Copying everything under
build/Ninja-.../lldb-...-x86_64/liband everything under
build/Ninja-.../lldb-...-x86_64/binto your Swift environment. For example:
/opt/swift-5.3/usr/.
There may be some issues with
lib/python3directory not being exactly the same as we should expect. It is safe to rename
site-packagesto
dist-packages.
With the updated LLDB toolchain, you should be able to register the Swift kernel now.
You can call Python libraries using Swift's Python interop to display rich output in your Swift notebooks. (Eventually, we'd like to support Swift libraries that produce rich output too!)
Prerequisites:
After taking care of the prerequisites, run
%include "EnableIPythonDisplay.swift"in your Swift notebook. Now you should be able to display rich output! For example:
let np = Python.import("numpy") let plt = Python.import("matplotlib.pyplot") IPythonDisplay.shell.enable_matplotlib("inline")
let time = np.arange(0, 10, 0.01) let amplitude = np.exp(-0.1 * time) let position = amplitude * np.sin(3 * time)plt.figure(figsize: [15, 10])
plt.plot(time, position) plt.plot(time, amplitude) plt.plot(time, -amplitude)
plt.xlabel("time (s)") plt.ylabel("position (m)") plt.title("Oscillations")
plt.show()
let display = Python.import("IPython.display") let pd = Python.import("pandas")
display.display(pd.DataFrame.from_records([["col 1": 3, "col 2": 5], ["col 1": 8, "col 2": 2]]))
You can display images using Swift too.
%install-swiftpm-flags -Xcc -isystem/usr/include/freetype2 -Xswiftc -lfreetype %install '.package(url: "https://github.com/IBM-Swift/BlueCryptor.git", from: "1.0.28")' Cryptor %install '.package(url: "https://github.com/KarthikRIyer/swiftplot", .branch("master"))' SwiftPlot AGGRenderer %include "EnableJupyterDisplay.swift"
Now you should be able to display images! (Currently only PNG format is supported. You also need to provide the image as a base64 String. Eventually we'd like to support other formats as well.)
For example:
import Foundation import SwiftPlot import AGGRenderer  func function(_ x: Float) -> Float { return 1.0 / x }  var aggRenderer = AGGRenderer() var lineGraph = LineGraph() lineGraph.addFunction( function, minX: -5.0, maxX: 5.0, numberOfSamples: 400, label: "1/x", color: .orange) lineGraph.plotTitle = "FUNCTION" lineGraph.drawGraph(renderer: aggRenderer) display(base64EncodedPNG: aggRenderer.base64Png())
To learn more about displaying plots using SwiftPlot take a look at the documentation here.
%installdirectives let you install SwiftPM packages so that your notebook can import them:
// Specify SwiftPM flags to use during package installation. %install-swiftpm-flags -c release// Install the DeckOfPlayingCards package from GitHub. %install '.package(url: "https://github.com/NSHipster/DeckOfPlayingCards", from: "4.0.0")' DeckOfPlayingCards
// Install the SimplePackage package that's in the kernel's working directory. %install '.package(path: "$cwd/SimplePackage")' SimplePackage
The first argument to
%installis a SwiftPM package dependency specification. The next argument(s) to
%installare the products that you want to install from the package.
%installdirectives currently have some limitations:
%install-swiftpm-flagsapply to all packages that you are installing; there is no way to specify different flags for different packages.
%install-extra-include-commandsection below.
If you get "expression failed to parse, unknown error" when you try to import a package that you installed, there is a way to get a more detailed error message.
The cell with the "%install" directives has something like "Working in: /tmp/xyzxyzxyzxyz/swift-install" in its output. There is a binary
usr/bin/swiftwhere you extracted the toolchain. Start the binary as follows:
SWIFT_IMPORT_SEARCH_PATH=/tmp/xyzxyzxyzxyz/swift-install/modules /usr/bin/swift
This gives you an interactive Swift REPL. In the REPL, do: ``` import Glibc dlopen("/tmp/xyzxyzxyzxyz/swift-install/package/.build/debug/libjupyterInstalledPackages.so", RTLD_NOW)
import TheModuleThatYouHaveTriedToInstall ```
This should give you a useful error message. If the error message says that some header files can't be found, see the section below about
%install-extra-include-command.
You can specify extra header files to be put on the header search path. Add a directive
%install-extra-include-command, followed by a shell command that prints "-I/path/to/extra/include/files". For example,
// Puts the headers in /usr/include/glib-2.0 on the header search path. %install-extra-include-command echo -I/usr/include/glib-2.0// Puts the headers returned by
pkg-config
on the header search path. %install-extra-include-command pkg-config --cflags-only-I glib-2.0
In principle, swift-jupyter should be able to infer the necessary header search paths without you needing to manually specify them, but this hasn't been implemented yet. See this forum thread for more information.
%includedirectives let you include code from files. To use them, put a line
%include ""in your cell. The kernel will preprocess your cell and replace the
%includedirective with the contents of the file before sending your cell to the Swift interpreter. must be relative to the directory containing
swift_kernel.py. We'll probably add more search paths later.
Install swift-jupyter locally using the above installation instructions. Now you can activate the virtualenv and run the tests:
. venv/bin/activate python test/fast_test.py # Fast tests, should complete in 1-2 min python test/all_test.py # Much slower, 10+ min python test/all_test.py SimpleNotebookTests.test_simple_successful # Invoke specific test method
You might also be interested in manually invoking the notebook tester on specific notebooks. See its
--helpdocumentation:
python test/notebook_tester.py --help
After building the docker image according to the instructions above,
docker run --cap-add SYS_PTRACE swift-jupyter python3 /swift-jupyter/test/all_test.py