necessary build tools for the archiconda distribution
This project has been superseded by
miniforge, a project by conda-forge.
See: - https://github.com/conda-forge/miniforge - https://conda-forge.org/status/
This repository holds many configuration scripts for the Archiconda3 distribution. Focused on porting conda-forge's work to 64 bit Arm processors.
The goal of this repository is develop the groundwork needed to compile conda-forge on aarch64. Once that is layed out, we will be working toward backporting much of this work to conda-forge
anaconda-clientusing archiconda + conda-forge.
For users to install Archiconda3, they need to be able to install the
anaconda-clientdo not need to be in that package, they can be obtained from
What this means, is that every package that
condadepends on, to run, needs to be an arch specific package. The packages used to build those package do not need to be arch specific packages.
For now, these are the necessary packages:
We are basically compute bound at this point. shippable gives us 1 CI to use at a time (for one organization).
1 build, even for a nearly empty pure python package, takes about 3 minutes.
See for example the package setuptools: https://app.shippable.com/github/Archiconda/setuptools-feedstock/dashboard
The 3 minutes (180 seconds) are broken up as follows:
We can maybe cut 50 seconds of this by uploading our own container.
But ultimately, we only have 1 CI, with 1 parallel job at a time, so we cannot run too many feedstocks at once.
It does have 96 cores, so maybe we can find a different way to parallelize things? I really feel like that might be over complicating things.
I haven't had many problems compiling standard software.
For example, python compiled on the first shot, and the issues were primarely due to the fact that some software hardcodes binutil dependencies.
Therefore, we can potentially not have shippable automatically get triggered.
Users would have to wait until a regular linux 64 bit build passes, before triggering the shippable build.
python conda conda constructor selenium chromedriver-binary python-chromedriver-binary
He will build the following critial packages:
The compilers and libstdc
These include the following:
binutils_impl_linux-aarch64-2.29.1-hc862510_0.tar.bz2 binutils_linux-aarch64-2.29.1-h1dbaa89_0.tar.bz2 crosstool-ng-188.8.131.521_g5888cf1-5.tar.bz2 gcc_impl_linux-aarch64-7.3.0-h68995b2_0.tar.bz2 gcc_linux-aarch64-7.3.0-h98564e2_0.tar.bz2 gdb_linux-aarch64-7.12.1-h6bc79d0_0.tar.bz2 gfortran_impl_linux-aarch64-7.3.0-h5c90dd9_0.tar.bz2 gfortran_linux-aarch64-7.3.0-h98564e2_0.tar.bz2 gxx_impl_linux-aarch64-7.3.0-h5c90dd9_0.tar.bz2 gxx_linux-aarch64-7.3.0-h98564e2_0.tar.bz2 libgcc-ng-7.3.0-h5c90dd9_0.tar.bz2 libgfortran-ng-7.3.0-h6bc79d0_0.tar.bz2 libstdcxx-ng-7.3.0-h5c90dd9_0.tar.bz2
Technically you don't need make (listed below), but i've had so much trouble building it, I'm just going to take jjhelmus'
These packages are available from his c4aarch64 anaconda channel.
He also created a tag on anaconda.org, I assume through cross compilation, where you can bootstrap a conda for aarch64.
Create a docker image that uses Centos7, and has Archiconda installed.
There is a bug in the version of conda in the bootstrap, that doesn't allow you to have multiple channels. The maximum number of channels is 2. This is why this next step is important.
Use that anaconda channel to create an anaconda installer. See the
installerdirectory in this repositiory.
Start building pacakges.
You can try and build make, but after you do that, try and build m4 with your newly cut version of make. I couldn't get it to work. I think jjhelmus did something special. Or I did something wrong :/
The build order is documented https://github.com/Archiconda/build-tools/issues/4
Most of the heavy lifting is done by
fork_conda_forge.py, a python script that does:
fork_conda_forge.pyto believe that I'm not stealing your credentials.
conda smithy rerender
Archicondato rerender things correctly.
Because the repository has been enabled, pushing to that branch will trigger a build on shippable.
The Docker image should:
# Set the locale ENV LANG en_US.UTF-8 ENV LANGUAGE en_US:en ENV LC_ALL en_US.UTF-8
Install a few basic pacakges.
wgetare necessary to help bootstrap the process.
Add the archiconda3 to the path.
If you are creating the docker image while running on your personal laptop, it is useful to use
qemu-staticto run an
Put the line ```
COPY qemu-aarch64-static /usr/bin/ ```
and you should be gravy.
I've found that it can be helpful to build and test using qemu and docker. It might be as simple as iterating quickly to find what dependencies were missing from the original packages.
Often, binutils are hardcoded, so you have to pass compilation flags to tell the build system what
arcommand to use.
It might be useful to mount a local directory to use with the docker build system.
For example, the following command mounts the registration directory which contains a bunch of feedstocks and runs the
archiconda/centos7image. The first command it runs is
bashallows you to interact with the system.
docker run -v /home/mark2/git/aarch64/build-tools/registration:/feedstocks -i -t archiconda/centos7 bash
Be warned, CPU emulation is SLOW and will make your computer crawl.
I started compiling
cmakeat the same time it did on shippable, and didn't even get through the bootstrap when it has finished compiling.