An open-source toolkit for large-scale genomic analysis
Glow is an open-source toolkit to enable bioinformatics at biobank-scale and beyond.
The toolkit includes the building blocks that you need to perform the most common analyses right away:
Glow makes genomic data work with Spark, the leading engine for working with large structured datasets. It fits natively into the ecosystem of tools that have enabled thousands of organizations to scale their workflows to petabytes of data. Glow bridges the gap between bioinformatics and the Spark ecosystem.
Glow works with datasets in common file formats like VCF, BGEN, and Plink as well as high-performance big data standards. You can write queries using the native Spark SQL APIs in Python, SQL, R, Java, and Scala. The same APIs allow you to bring your genomic data together with other datasets such as electronic health records, real world evidence, and medical images. Glow makes it easy to parallelize existing tools and libraries implemented as command line tools or Pandas functions.
This project is built using sbt and Java 8.
To build and run Glow, you must install conda and activate the environment in
conda env create -f python/environment.yml conda activate glow
When the environment file changes, you must update the environment:
conda env update -f python/environment.yml
Start an sbt shell using the
FYI: The following SBT projects are built on Spark 3.1.2/Scala 2.12.8 by default. To change the Spark version and Scala version, set the environment variablesSPARK_VERSIONandSCALA_VERSION.
To compile the main code:
To run all Scala tests:
To test a specific suite:
To run all Python tests:
python/testThese tests will run with the same Spark classpath as the Scala tests.
To test a specific Python test file:
When using the
pytestkey, all arguments are passed directly to the pytest runner.
To run documentation tests:
To run the Scala, Python and documentation tests:
To run Scala tests against the staged Maven artifact with the current stable version:
To test your changes on a Databricks cluster, you'll need to build and install the Python and Scala artifacts.
To build an uber jar (Glow + dependencies) with your changes:
The uber jar will be at a path like
To build a wheel with the Python code:
conda activate glow)
python setup.py bdist_wheel
The wheel file will be at a path like
You can then install these libraries on a Databricks cluster.
To run Python unit tests from inside IntelliJ, you must: - Open the "Terminal" tab in IntelliJ - Activate the glow conda environment (
conda activate glow) - Start an sbt shell from inside the terminal (
The "sbt shell" tab in IntelliJ will NOT work since it does not use the glow conda environment.
To test or testOnly in remote debug mode with IntelliJ IDEA set the remote debug configuration in IntelliJ to 'Attach to remote JVM' mode and a specific port number (here the default port number 5005 is used) and then modify the definition of options in groupByHash function in build.sbt to
val options = ForkOptions().withRunJVMOptions(Vector("-Xmx1024m")).withRunJVMOptions(Vector("-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005"))