Need help with awkward-1.0?
Click the β€œchat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

scikit-hep
301 Stars 41 Forks BSD 3-Clause "New" or "Revised" License 941 Commits 69 Opened issues

Description

Manipulate JSON-like data with NumPy-like idioms.

Services available

!
?

Need anything else?

Contributors list

PyPI version Conda-Forge Python 2.7,3.5β€’3.9 BSD-3 Clause License Continuous integration tests

Scikit-HEP NSF-1836650 DOI Documentation Gitter

Awkward Array is a library for nested, variable-sized data, including arbitrary-length lists, records, mixed types, and missing data, using NumPy-like idioms.

Arrays are dynamically typed, but operations on them are compiled and fast. Their behavior coincides with NumPy when array dimensions are regular and generalizes when they're not.

Motivating example

Given an array of objects with

x
,
y
fields and variable-length nested lists like
array = ak.Array([
    [{"x": 1.1, "y": [1]}, {"x": 2.2, "y": [1, 2]}, {"x": 3.3, "y": [1, 2, 3]}],
    [],
    [{"x": 4.4, "y": [1, 2, 3, 4]}, {"x": 5.5, "y": [1, 2, 3, 4, 5]}]
])

the following slices out the

y
values, drops the first element from each inner list, and runs NumPy's
np.square
function on everything that is left:
output = np.square(array["y", ..., 1:])

The result is

[
    [[], [4], [4, 9]],
    [],
    [[4, 9, 16], [4, 9, 16, 25]]
]

The equivalent using only Python is

output = []
for sublist in array:
    tmp1 = []
    for record in sublist:
        tmp2 = []
        for number in record["y"][1:]:
            tmp2.append(np.square(number))
        tmp1.append(tmp2)
    output.append(tmp1)

Not only is the expression using Awkward Arrays more concise, using idioms familiar from NumPy, but it's much faster and uses less memory.

For a similar problem 10 million times larger than the one above (on a single-threaded 2.2 GHz processor),

  • the Awkward Array one-liner takes 4.6 seconds to run and uses 2.1 GB of memory,
  • the equivalent using Python lists and dicts takes 138 seconds to run and uses 22 GB of memory.

Speed and memory factors in the double digits are common because we're replacing Python's dynamically typed, pointer-chasing virtual machine with type-specialized, precompiled routines on contiguous data. (In other words, for the same reasons as NumPy.) Even higher speedups are possible when Awkward Array is paired with Numba.

Our presentation at SciPy 2020 provides a good introduction, showing how to use these arrays in a real analysis.

Installation

Awkward Array can be installed from PyPI using pip:

pip install awkward

You will likely get a precompiled binary (wheel), depending on your operating system and Python version. If not, pip attempts to compile from source (which requires a C++ compiler, make, and CMake).

Awkward Array is also available using conda, which always installs a binary:

bash
conda install -c conda-forge awkward

If you have already added

conda-forge
as a channel, the
-c conda-forge
is unnecessary. Adding the channel is recommended because it ensures that all of your packages use compatible versions:
conda config --add channels conda-forge
conda update --all

Getting help

How-to tutorials

Python API reference

C++ API reference

Installation for developers

Be sure to clone this repository recursively to get the header-only C++ dependencies.

git clone --recursive https://github.com/scikit-hep/awkward-1.0.git

Also be aware that the default branch is named

main
, not
master
, which could be important for pull requests from forks.

You can install it on your system with pip, which uses exactly the same procedure as deployment. This is recommended if you do not expect to change the code.

pip install .[test,dev]

Or you can build it locally for incremental development. The following reuses a local directory so that you only recompile what you've changed. This is recommended if you do expect to change the code.

python localbuild.py --pytest tests

The

--pytest tests
runs the integration tests from the
tests
directory (drop it to build only).

For more fine-grained testing, we also have tests of the low-level kernels, which can be invoked with

python dev/generate-tests.py
python -m pytest -vv -rs tests-spec
python -m pytest -vv -rs tests-cpu-kernels

Furthermore, if you have an Nvidia GPU, you can build and locally install the experimental CUDA plug-in with

pip uninstall -y awkward-cuda-kernels
python dev/generate-cuda.py
./cuda-build.sh --install

The

--install
does a local
pip install
on your system, which is the only way to use it. You can run its tests with
python dev/generate-tests.py
python -m pytest -vv -rs tests-cuda-kernels
python -m pytest -vv -rs tests-cuda

Using Awkward Array as a dependency

Python projects can simply

import awkward
.

Roadmap

Since version 0.4.0, Awkward Array has been compiled with versions of pybind11 that have ABI version 4.

The table below indicates when interface-breaking changes are planned; each is discussed in pull requests and issues. It doesn't include new additions that don't interfere with old behavior or corrections to bugs (i.e. behaviors that were never intended and fixed immediately). Each deprecated feature is announced by a

FutureWarning
that indicates the target removal version/date.

See release history for a detailed changelog of past releases. See projects for planning and prioritizing future fixes and features.

| Version number | Release date | Deprecated features removed in this version | |:--------------:|:------------:|:--------------------------------------------| | 1.0.0 | 2020-12-05 | Broadcasting NumPy ufuncs through records (#457),

lazy_cache="attach"
option in ak.from_parquet (#576). | | 1.1.0 | 2021-02-09 | Removed ak.to_arrayset/ak.from_arrayset in favor of ak.to_buffers/ak.from_buffers (#592). | | 1.2.0 | 2021-04-01 | (none) | | 1.3.0 | 2021-06-01 | (none) | | 1.4.0 | 2021-07-02 | (none) | | 1.5.0 | 2021-09-12 | (none) | | 1.6.0 | (skipped) | A deprecation was scheduled for "1.7.0, Oct 1, 2021," but we were a version number behind when October came, so the number 1.6.0 was skipped. |

| Version number | Target date | Deprecated features to remove in this version | |:--------------:|:-----------:|:----------------------------------------------| | 1.7.0 | 2021-10-01 | ak.fill_none default

axis
will be
-1
. Until then, all uses without an explicit
axis
raise warnings. | | 1.8.0 | 2021-11-01 | | | 1.9.0 | 2021-12-01 | | | 1.10.0 | 2022-01-01 | | | 1.11.0 | 2022-02-01 | |

Voting for fixes/features

You can vote for issues by adding a "thumbs up" (πŸ‘) using the "smile/pick your reaction" menu on the top-right of the issue. Issues with the most "thumbs ups" will be prioritized, though this isn't the only consideration determining when an issue gets addressed. (An estimate of how long it will take is also important.)

See the prioritized list of open issues.

Papers and talks about Awkward Array

Acknowledgements

Support for this work was provided by NSF cooperative agreement OAC-1836650 (IRIS-HEP), grant OAC-1450377 (DIANA/HEP), PHY-1520942 (US-CMS LHC Ops), and OAC-2103945 (Awkward Array).

We also thank Erez Shinan and the developers of the Lark standalone parser, which is used to parse type strings as type objects.

Thanks especially to the gracious help of Awkward Array contributors (including the original repository).


Jim Pivarski

πŸ’» πŸ“– πŸš‡ 🚧

Ianna Osborne

πŸ’»

Pratyush Das

πŸ’»

Anish Biswas

πŸ’»

glass-ships

πŸ’» ⚠️

Henry Schreiner

πŸ’» πŸš‡

Nicholas Smith

πŸ’» ⚠️

Lindsey Gray

πŸ’» ⚠️

Ellipse0934

⚠️

Dmitry Kalinkin

πŸš‡

Charles Escott

πŸ’»

Mason Proffitt

πŸ’»

Michael Hedges

πŸ’»

Jonas Rembser

πŸ’»

Jaydeep Nandi

πŸ’»

benkrikler

πŸ’»

bfis

πŸ’»

Doug Davis

πŸ’»

Joosep Pata

πŸ€”

Martin Durant

πŸ€”

Gordon Watts

πŸ€”

Nikolai Hartmann

πŸ’»

Simon Perkins

πŸ’»

.hard

πŸ’» ⚠️

HenryDayHall

πŸ’»

Angus Hollands

⚠️ πŸ’»

ioanaif

πŸ’» ⚠️

Bernhard M. Wiedemann

🚧

Matthew Feickert

🚧

Santam Roy Choudhury

⚠️

πŸ’»: code, πŸ“–: documentation, πŸš‡: infrastructure, 🚧: maintenance, ⚠: tests and feedback, πŸ€”: foundational ideas.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.