v923z/ micropython-ulab

(3.3.1) 3 months ago

Python

Shell

scipy

micropython

numpy

View on GitHub

Need help with micropython-ulab?

Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

v923z

212 Stars

56 Forks

MIT License

1.1K Commits

22 Opened issues

Description

a numpy-like fast vector module for micropython, circuitpython, and their derivatives

Services available

Need anything else?

Contributors list

Zoltán Vörös v923z

# 137,861

Jupyter...

Less

jupyter

micropy...

630 commits

Jeff Epler jepler

# 68,011

makecod...

micropy...

gzip

109 commits

Philip Howard Gadgetoid

# 61,551

Raspber...

air-qua...

micropy...

8 commits

Cal Hays CallumJHays

# 360,266

wasm-bi...

fronten...

webasse...

twitter...

6 commits

Deqing Sun DeqingSun

# 351,354

Shell

micropy...

numpy

5 commits

Taku Fukada ciscorn

# 105,785

Shell

Redis

python-...

3 commits

Jonathan Hogg jonathanhogg

# 107,355

microco...

Shell

micropy...

1 commit

黃昕暐 codemee

# 495,623

micropy...

Shell

numpy

1 commit

Nicko van Someren nickovs

# 80,714

microco...

micropy...

Tensorf...

1 commit

retsyo retsyo

# 353,530

Shell

micropy...

text-pr...

python-...

1 commit

Scott Shawcroft tannewt

# 214,495

Arduino

usb-dri...

usb-dev...

micropy...

1 commit

Readme

ulab

Name: micropython-ulab
Brand: micropython-ulab
SKU: //v923z/micropython-ulab
Rating: 2.28 (212 reviews)

ulab

is a

numpy

-like array manipulation library for micropython and CircuitPython. The module is written in C, defines compact containers for numerical data of one to four dimensions, and is fast. The library is a software-only standard

micropython

user module, i.e., it has no hardware dependencies, and can be compiled for any platform. The

float

implementation of

micropython

(

float

, or

double

) is automatically detected.

Supported functions and methods

ndarray methods

ulab

implements

numpy

ndarray

with the

==

!=

<=

>=

**

+=

-=

*=

/=

**=

binary operators, and the

len

abs

unary operators that operate element-wise. Type-aware

ndarray

s can be initialised from any

micropython

iterable, lists of iterables via the

array

constructor, or by means of the

arange

concatenate

diag

eye

frombuffer

full

linspace

logspace

ones

, or

zeros

functions.

ndarray

s can be sliced, and iterated on, and have a number of their own methods, and properties, such as

flatten()

itemsize

reshape()

shape

size

strides

tobytes()

, and

transpose()

and

numpy

and

scipy

functions

In addition,

ulab

includes universal functions, many

numpy

functions, and functions from the

numpy.fft

numpy.linalg

scipy.linalg

scipy.optimize

scipy.signal

, and

scipy.special

modules. A complete list of available routines can be found under micropython-ulab.

ulab

utilities

The

utils

module contains functions for interfacing with peripheral devices supporting the buffer protocol.

user

module

User-defined functions operating on numerical data can easily be added via the

user

module. This allows for transparent extensions, without having to change anything in the core. Hints as to how to work with

ndarray

s at the C level can be found in the programming manual.

Customising the firmware

If flash space is a concern, unnecessary functions can be excluded from the compiled firmware with pre-processor switches. In addition,

ulab

also has options for trading execution speed for firmware size. A thorough discussion on how the firmware can be customised can be found in the corresponding section of the user manual.

Usage

ulab

sports a

numpy/scipy

-compatible interface, which makes porting of

CPython

code straightforward. The following snippet should run equally well in

micropython

, or on a PC.

try:
    from ulab import numpy
    from ulab import scipy
except ImportError:
    import numpy
    import scipy.special

x = numpy.array([1, 2, 3])
scipy.special.erf(x)

Finding help

Documentation can be found on readthedocs under micropython-ulab, as well as at circuitpython-ulab. A number of practical examples are listed in Jeff Epler's excellent circuitpython-ulab overview.

Benchmarks

Representative numbers on performance can be found under ulab samples.

Firmware

Compiled

Compiled firmware for many hardware platforms can be downloaded from Roberto Colistete's gitlab repository: for the pyboard, and for ESP8266. Since a number of features can be set in the firmware (threading, support for SD card, LEDs, user switch etc.), and it is impossible to create something that suits everyone, these releases should only be used for quick testing of

ulab

. Otherwise, compilation from the source is required with the appropriate settings, which are usually defined in the

mpconfigboard.h

file of the port in question.

ulab

is also included in the following compiled

micropython

variants and derivatives:

```
CircuitPython
```
for SAMD51 and nRF microcontrollers https://github.com/adafruit/circuitpython
```
MicroPython for K210
```
https://github.com/loboris/MicroPythonK210LoBo
```
MaixPy
```
https://github.com/sipeed/MaixPy
```
OpenMV
```
https://github.com/openmv/openmv
```
pimoroni-pico
```
https://github.com/pimoroni/pimoroni-pico
```
pycom
```
https://pycom.io/

Compiling

If you want to try the latest version of

ulab

micropython

or one of its forks, the firmware can be compiled from the source by following these steps:

UNIX port

Simply clone the

ulab

repository with

git clone https://github.com/v923z/micropython-ulab.git ulab

and then run

./build.sh

This command will clone

micropython

, and build the

unix

port automatically, as well as run the test scripts. If you want an interactive

unix

session, you can launch it in

ulab/micropython/ports/unix

STM-based boards

First, you have to clone the

micropython

repository by running

git clone https://github.com/micropython/micropython.git

on the command line. This will create a new repository with the name

micropython

. Staying there, clone the

ulab

repository with

git clone https://github.com/v923z/micropython-ulab.git ulab

If you don't have the cross-compiler installed, your might want to do that now, for instance on Linux by executing

sudo apt-get install gcc-arm-none-eabi

If this step was successful, you can try to run the

make

command in the port's directory as

make BOARD=PYBV11 USER_C_MODULES=../../../ulab all

which will prepare the firmware for pyboard.v.11. Similarly,

make BOARD=PYBD_SF6 USER_C_MODULES=../../../ulab all

will compile for the SF6 member of the PYBD series. If your target is

unix

, you don't need to specify the

BOARD

parameter.

Provided that you managed to compile the firmware, you would upload that by running either

dfu-util --alt 0 -D firmware.dfu

python pydfu.py -u firmware.dfu

In case you got stuck somewhere in the process, a bit more detailed instructions can be found under https://github.com/micropython/micropython/wiki/Getting-Started, and https://github.com/micropython/micropython/wiki/Pyboard-Firmware-Update.

ESP32-based boards

Firmware for

Espressif

boards can be built in two different ways. These are discussed in the next two paragraphs. A solution for issues with the firmware size is outlined in the last paragraph in this section.

Compiling with cmake

Beginning with version 1.15,

micropython

switched to

cmake

on the ESP32 port. If your operating system supports

CMake > 3.12

, you can either simply download, and run the single build script, or follow the step in this section. Otherwise, you should skip to the next one, where the old,

make

-based approach is discussed.

In case you encounter difficulties during the build process, you can consult the (general instructions for the ESP32)[https://github.com/micropython/micropython/tree/master/ports/esp32#micropython-port-to-the-esp32].

First, clone the

ulab

, the

micropython

, as well as the

espressif

repositories:

export BUILD_DIR=$(pwd)

git clone https://github.com/v923z/micropython-ulab.git ulab git clone https://github.com/micropython/micropython.git

cd $BUILD_DIR/micropython/

git clone -b v4.0.2 --recursive https://github.com/espressif/esp-idf.git

Also later releases of

esp-idf

are possible (e.g.

v4.2.1

Then install the

ESP-IDF

tools:

cd esp-idf
./install.sh
. ./export.sh

Next, build the

micropython

cross-compiler, and the

ESP

sub-modules:

cd $BUILD_DIR/micropython/mpy-cross
make
cd $BUILD_DIR/micropython/ports/esp32
make submodules

At this point, all requirements are installed and built. We can now compile the firmware with

ulab

. In

$BUILD_DIR/micropython/ports/esp32

create a

makefile

with the following content:

BOARD = GENERIC
USER_C_MODULES = $(BUILD_DIR)/ulab/code/micropython.cmake

include Makefile

You specify with the

BOARD

variable, what you want to compile for, a generic board, or

TINYPICO

(for

micropython

version >1.1.5, use

UM_TINYPICO

), etc. Still in

$BUILD_DIR/micropython/ports/esp32

, you can now run

make

Compiling with make

If your operating system does not support a recent enough version of

CMake

, you have to stay with

micropython

version 1.14. The firmware can be compiled either by downloading and running the build script, or following the steps below:

First, clone

ulab

with

git clone https://github.com/v923z/micropython-ulab.git ulab

and then, in the same directory,

micropython

git clone https://github.com/micropython/micropython.git

At this point, you should have

ulab

, and

micropython

side by side.

With version 1.14,

micropython

switched to

cmake

on the

ESP32

port, thus breaking compatibility with user modules.

ulab

can, however, still be compiled with version 1.14. You can check out a particular version by pinning the release tag as

cd ./micropython/ git checkout tags/v1.14

Next, update the submodules,

git submodule update --init
cd ./mpy-cross && make # build cross-compiler (required)

and find the ESP commit hash

cd ./micropython/ports/esp32
make ESPIDF= # will display supported ESP-IDF commit hashes
# output should look like: """
# ...
# Supported git hash (v3.3): 9e70825d1e1cbf7988cf36981774300066580ea7
# Supported git hash (v4.0) (experimental): 4c81978a3e2220674a432a588292a4c860eef27b

Choose an ESPIDF version from one of the options printed by the previous command:

ESPIDF_VER=9e70825d1e1cbf7988cf36981774300066580ea7

In the

micropython

directory, create a new directory with

bash
mkdir esp32

Your

micropython

directory should now look like

ls
ACKNOWLEDGEMENTS    CONTRIBUTING.md  esp32     lib      mpy-cross  README.md
CODECONVENTIONS.md  docs             examples  LICENSE  ports      tests
CODEOFCONDUCT.md    drivers          extmod    logo     py         tools

./micropython/esp32

, download the software development kit with

git clone https://github.com/espressif/esp-idf.git esp-idf
cd ./esp-idf
git checkout $ESPIDF_VER
git submodule update --init --recursive # get idf submodules
pip install -r ./requirements.txt # install python reqs

Next, still staying in

./micropython/eps32/esd-idf/

, install the ESP32 compiler. If using an ESP-IDF version >= 4.x (chosen by

$ESPIDF_VER

above), this can be done by running

. $BUILD_DIR/esp-idf/install.sh

. Otherwise, for version 3.x, run the following commands in in

.micropython/esp32/esp-idf

# for 64 bit linux curl https://dl.espressif.com/dl/xtensa-esp32-elf-linux64-1.22.0-80-g6c4433a-5.2.0.tar.gz | tar xvz

for 32 bit

curl https://dl.espressif.com/dl/xtensa-esp32-elf-linux32-1.22.0-80-g6c4433a-5.2.0.tar.gz | tar xvz

don't worry about adding to path; we'll specify that later

also, see https://docs.espressif.com/projects/esp-idf/en/v3.3.2/get-started for more info

Finally, build the firmware:

cd ./micropython/ports/esp32 # temporarily add esp32 compiler to path export PATH=../../esp32/esp-idf/xtensa-esp32-elf/bin:$PATH export ESPIDF=../../esp32/esp-idf # req'd by Makefile export BOARD=GENERIC # options are dirs in ./boards export USER_C_MODULES=../../../ulab # include ulab in firmware

make submodules & make all

If it compiles without error, you can plug in your ESP32 via USB and then flash it with:

make erase && make deploy

What to do, if the firmware is too large?

When selecting

BOARD=TINYPICO

, the firmware is built but fails to deploy, because it is too large for the standard partitions. We can rectify the problem by creating a new partition table. In order to do so, in

$BUILD_DIR/micropython/ports/esp32/

, copy the following 8 lines to a file named

partitions_ulab.cvs

# Notes: the offset of the partition table itself is set in
# $ESPIDF/components/partition_table/Kconfig.projbuild and the
# offset of the factory/ota_0 partition is set in makeimg.py
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x200000,
vfs,      data, fat,     0x220000, 0x180000,

This expands the

factory

partition by 128 kB, and reduces the size of

vfs

by the same amount. Having defined the new partition table, we should extend

sdkconfig.board

by adding the following two lines:

CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions_ulab.csv"

This file can be found in

$BUILD_DIR/micropython/ports/esp32/boards/TINYPICO/

. Finally, run

make clean

, and

make

. The new firmware contains the modified partition table, and should fit on the microcontroller.

RP2-based boards

RP2 firmware can be compiled either by downloading and running the single build script, or executing the commands below.

First, clone

micropython

git clone https://github.com/micropython/micropython.git

Then, setup the required submodules:

cd micropython
git submodule update --init lib/tinyusb
git submodule update --init lib/pico-sdk
cd lib/pico-sdk
git submodule update --init lib/tinyusb

You'll also need to compile

mpy-cross

cd ../../mpy-cross
make

That's all you need to do for the

micropython

repository. Now, let us clone

ulab

(in a directory outside the micropython repository):

cd ../../
git clone https://github.com/v923z/micropython-ulab ulab

With this setup, we can now build the firmware. Back in the

micropython

repository, use these commands:

cd ports/rp2
make USER_C_MODULES=/path/to/ulab/code/micropython.cmake

micropython

and

ulab

were in the same folder on the computer, you can set

USER_C_MODULES=../../../ulab/code/micropython.cmake

. The compiled firmware will be placed in

micropython/ports/rp2/build

Compiling for CircuitPython

Adafruit Industries always include a relatively recent version of

ulab

in their nightly builds. However, if you really need the bleeding edge, you can easily compile the firmware from the source. Simply clone

circuitpython

, and move the commit pointer to the latest version of

ulab

(

ulab

will automatically be cloned with

circuitpython

git clone https://github.com/adafruit/circuitpython.git

cd circuitpyton/extmod/ulab

update ulab here

git checkout master git pull

You might have to check, whether the

CIRCUITPY_ULAB

variable is set to

for the port that you want to compile for. You find this piece of information in the

make

fragment:

circuitpython/ports/port_of_your_choice/mpconfigport.mk

After this, you would run

make

with the single

BOARD

argument, e.g.:

make BOARD=mini_sam_m4

Issues, contributing, and testing

If you find a problem with the code, please, raise an issue! An issue should address a single problem, and should contain a minimal code snippet that demonstrates the difference from the expected behaviour. Reducing a problem to the bare minimum significantly increases the chances of a quick fix.

Feature requests (porting a particular function from

numpy

scipy

) should also be posted at ulab issue.

Contributions of any kind are always welcome. If you feel like adding to the code, you can simply issue a pull request. If you do so, please, try to adhere to

micropython

's coding conventions.

However, you can also contribute to the documentation (preferably via the jupyter notebooks, or improve the tests.

Testing

If you decide to lend a hand with testing, here are the steps:

Write a test script that checks a particular function, or a set of related functions!
Drop this script in one of the folders in ulab tests!
Run the ./build.sh script in the root directory of
```
ulab
```
! This will clone the latest
```
micropython
```
, compile the firmware for
```
unix
```
, execute all scripts in the
```
ulab/tests
```
, and compare the results to those in the expected results files, which are also in
```
ulab/tests
```
, and have an extension
```
.exp
```
. In case you have a new snippet, i.e., you have no expected results file, or if the results differ from those in the expected file, a new expected file will be generated in the root directory. You should inspect the contents of this file, and if they are satisfactory, then the file can be moved to the
```
ulab/tests
```
folder, alongside your snippet.

About the developer

Description

Services available

Need anything else?

Contributors list

ulab

Supported functions and methods

ndarray methods

numpy
and
scipy
functions

ulab
utilities

user
module

Customising the firmware

Usage

Finding help

Benchmarks

Firmware

Compiled

Compiling

UNIX port

STM-based boards

ESP32-based boards

Compiling with cmake

Compiling with make

for 32 bit

curl https://dl.espressif.com/dl/xtensa-esp32-elf-linux32-1.22.0-80-g6c4433a-5.2.0.tar.gz | tar xvz

don't worry about adding to path; we'll specify that later

also, see https://docs.espressif.com/projects/esp-idf/en/v3.3.2/get-started for more info

What to do, if the firmware is too large?

RP2-based boards

Compiling for CircuitPython

update ulab here

Issues, contributing, and testing

Testing