Need help with weld?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

weld-project
2.6K Stars 236 Forks BSD 3-Clause "New" or "Revised" License 615 Commits 70 Opened issues

Description

High-performance runtime for data analytics applications

Services available

!
?

Need anything else?

Contributors list

Weld

Build Status

Documentation

Weld is a language and runtime for improving the performance of data-intensive applications. It optimizes across libraries and functions by expressing the core computations in libraries using a common intermediate representation, and optimizing across each framework.

Modern analytics applications combine multiple functions from different libraries and frameworks to build complex workflows. Even though individual functions can achieve high performance in isolation, the performance of the combined workflow is often an order of magnitude below hardware limits due to extensive data movement across the functions. Weld’s take on solving this problem is to lazily build up a computation for the entire workflow, and then optimizing and evaluating it only when a result is needed.

You can join the discussion on Weld on our Google Group or post on the Weld mailing list at [email protected].

Contents

Building

To build Weld, you need the latest stable version of Rust and LLVM/Clang++ 6.0.

To install Rust, follow the steps here. You can verify that Rust was installed correctly on your system by typing

rustc
into your shell. If you already have Rust and
rustup
installed, you can upgrade to the latest stable version with:
rustup update stable

MacOS LLVM Installation

To install LLVM on macOS, first install Homebrew. Then:

brew install [email protected]

Weld's dependencies require

llvm-config
on
$PATH
, so you may need to create a symbolic link so the correct
llvm-config
is picked up (note that you might need to add
sudo
at the start of this command):
ln -sf `brew --prefix [email protected]`/bin/llvm-config /usr/local/bin/llvm-config

To make sure this worked correctly, run

llvm-config --version
. You should see
6.0.x
.

Ubuntu LLVM Installation

To install LLVM on Ubuntu, get the LLVM 6.0 sources and then

apt-get
:

On Ubuntu 16.04 (Xenial):

bash
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
sudo apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-6.0 main"
sudo apt-get update
sudo apt-get install llvm-6.0-dev clang-6.0
On Ubuntu 14.04 (Trusty): ```bash wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add - sudo apt-add-repository "deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-6.0 main"

gcc backport is required on 14.04, for libstdc++. See https://apt.llvm.org/

sudo apt-add-repository "deb http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu trusty main" sudo apt-get update sudo apt-get install llvm-6.0-dev clang-6.0 ```

Weld's dependencies require

llvm-config
, so you may need to create a symbolic link so the correct
llvm-config
is picked up.
sudo
may be required:
ln -s /usr/bin/llvm-config-6.0 /usr/local/bin/llvm-config

To make sure this worked correctly, run

llvm-config --version
. You should see
6.0.x
or newer.

You will also need

zlib
:
sudo apt-get install zlib1g-dev

Building Weld

With LLVM and Rust installed, you can build Weld. Clone this repository, set the

WELD_HOME
environment variable, and build using
cargo
:
git clone https://www.github.com/weld-project/weld
cd weld/
export WELD_HOME=`pwd`
cargo build --release

Weld builds two dynamically linked libraries (

.so
files on Linux and
.dylib
files on Mac):
libweld
and
libweldrt
.

Finally, run the unit and integration tests:

cargo test

Documentation

The Rust Weld crate is documented here.

The

docs/
directory contains documentation for the different components of Weld.
  • language.md describes the syntax of the Weld IR.
  • api.md describes the low-level C API for interfacing with Weld.
  • python.md gives an overview of the Python API.
  • tutorial.md contains a tutorial for how to build a small vector library using Weld.

Python Bindings

Weld's Python bindings are in

python
, with examples in
examples/python
.

Grizzly

Grizzly is a subset of Pandas integrated with Weld. Details on how to use Grizzly are in

python/grizzly
. Some example workloads that make use of Grizzly are in
examples/python/grizzly
. To run Grizzly, you will also need the

WELD_HOME
environment variable to be set, because Grizzly needs to find its own native library through this variable.

Testing

cargo test
runs unit and integration tests. A test name substring filter can be used to run a subset of the tests:
   cargo test 

Tools

This repository contains a number of useful command line tools which are built automatically with the main Weld repository, including an interactive REPL for inspecting and debugging programs. More information on those tools can be found under docs/tools.md.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.