dhallj

by travisbrown

travisbrown / dhallj

Dhall for Java

146 Stars 4 Forks Last release: 4 months ago (v0.4.0) BSD 3-Clause "New" or "Revised" License 505 Commits 9 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Dhall for Java

Build status Gitter Maven Central

This project is an implementation of the Dhall configuration language for the Java Virtual Machine.

Our goal for this project is to make it as easy as possible to integrate Dhall into JVM build systems (see the dhall-kubernetes demonstration below for a concrete example of why you might want to do this).

The core modules have no external dependencies, are Java 7-compatible, and are fairly minimal:

$ du -h modules/core/target/dhall-core-0.4.1-SNAPSHOT.jar
148K    modules/core/target/dhall-core-0.4.1-SNAPSHOT.jar

$ du -h modules/parser/target/dhall-parser-0.4.1-SNAPSHOT.jar 104K modules/parser/target/dhall-parser-0.4.1-SNAPSHOT.jar

There are also several Scala modules that are published for Scala 2.12 and 2.13. While most of the examples in this README are focused on Scala, you shouldn't need to know or care about Scala to use the core DhallJ modules.

This project has been supported in part by Permutive. Please see our monthly reports for updates on the work of the Permutive Community Engineering team.

Table of contents

Status

The current release of this project supports Dhall 17.0.0. We're running the Dhall acceptance test suites for parsing, normalization, CBOR encoding and decoding, hashing, and type inference, and currently all tests are passing.

There are several known issues:

While we think the project is reasonably well-tested, it's very new, is sure to be full of bugs, and nothing about the API should be considered stable at the moment. Please use responsibly.

Getting started

The easiest way to try things out is to add the Scala wrapper module to your build. If you're using sbt that would look like this:

libraryDependencies += "org.dhallj" %% "dhall-scala" % "0.4.0"

This dependency includes two packages:

org.dhallj.syntax
and
org.dhallj.ast
.

The

syntax
package provides some extension methods, including a
parseExpr
method for strings (note that this method returns an
Either[ParsingFailure, Expr]
, which we unwrap here with
Right
):
scala> import org.dhallj.syntax._
import org.dhallj.syntax._

scala> val Right(expr) = "\(n: Natural) -> [n + 0, n + 1, 1 + 1]".parseExpr expr: org.dhallj.core.Expr = λ(n : Natural) → [n + 0, n + 1, 1 + 1]

Now that we have a Dhall expression, we can type-check it:

scala> val Right(exprType) = expr.typeCheck
exprType: org.dhallj.core.Expr = ∀(n : Natural) → List Natural

We can "reduce" (or β-normalize) it:

scala> val normalized = expr.normalize
normalized: org.dhallj.core.Expr = λ(n : Natural) → [n, n + 1, 2]

We can also α-normalize it, which replaces all named variables with indexed underscores:

scala> val alphaNormalized = normalized.alphaNormalize
alphaNormalized: org.dhallj.core.Expr = λ(_ : Natural) → [_, _ + 1, 2]

We can encode it as a CBOR byte array:

scala> alphaNormalized.getEncodedBytes
res0: Array[Byte] = Array(-125, 1, 103, 78, 97, 116, 117, 114, 97, 108, -123, 4, -10, 0, -124, 3, 4, 0, -126, 15, 1, -126, 15, 2)

And we can compute its semantic hash:

scala> alphaNormalized.hash
res1: String = c57cdcdae92638503f954e63c0b3ae8de00a59bc5e05b4dd24e49f42aca90054

If we have the official

dhall
CLI installed, we can confirm that this hash is correct:
$ dhall hash <<< '\(n: Natural) -> [n + 0, n + 1, 1 + 1]'
sha256:c57cdcdae92638503f954e63c0b3ae8de00a59bc5e05b4dd24e49f42aca90054

We can also compare expressions:

scala> val Right(other) = "\\(n: Natural) -> [n, n + 1, 3]".parseExpr
other: org.dhallj.core.Expr = λ(n : Natural) → [n, n + 1, 3]

scala> normalized == other res2: Boolean = false

scala> val Some(diff) = normalized.diff(other) diff: (Option[org.dhallj.core.Expr], Option[org.dhallj.core.Expr]) = (Some(2),Some(3))

And apply them to other expressions:

scala> val Right(arg) = "10".parseExpr
arg: org.dhallj.core.Expr = 10

scala> expr(arg) res3: org.dhallj.core.Expr = (λ(n : Natural) → [n + 0, n + 1, 1 + 1]) 10

scala> expr(arg).normalize res4: org.dhallj.core.Expr = [10, 11, 2]

We can also resolve expressions containing imports (although at the moment dhall-scala doesn't support remote imports or caching; please see the section on import resolution below for details about how to set up remote import resolution if you need it):

val Right(enumerate) =
     |   "./dhall-lang/Prelude/Natural/enumerate".parseExpr.flatMap(_.resolve)
enumerate: org.dhallj.core.Expr = let enumerate : Natural → List Natural = ...

scala> enumerate(arg).normalize res5: org.dhallj.core.Expr = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Note that we're working with values of type

Expr
, which comes from dhall-core, which is a Java module. The
Expr
class includes static methods for creating
Expr
values:
scala> import org.dhallj.core.Expr
import org.dhallj.core.Expr

scala> Expr.makeTextLiteral("foo") res6: org.dhallj.core.Expr = "foo"

scala> Expr.makeEmptyListLiteral(Expr.Constants.BOOL) res7: org.dhallj.core.Expr = [] : Bool

If you're working from Scala, though, you're generally better off using the constructors included in the

org.dhallj.ast
package, which provide more type-safety:
scala> TextLiteral("foo")
res8: org.dhallj.core.Expr = "foo"

scala> NonEmptyListLiteral(BoolLiteral(true), Vector()) res9: org.dhallj.core.Expr = [True]

The

ast
package also includes extractors that let you pattern match on
Expr
values:
scala> expr match {
     |   case Lambda(name, _, NonEmptyListLiteral(first +: _)) => (name, first)
     | }
res10: (String, org.dhallj.core.Expr) = (n,n + 0)

Note that we don't have exhaustivity checking for these extractors, although we might be able to add that in an eventual Dotty version.

In addition to dhall-scala, there's a (more experimental) dhall-scala-codec module, which supports encoding and decoding Scala types to and from Dhall expressions. If you add it to your build, you can write the following:

scala> import org.dhallj.codec.syntax._
import org.dhallj.codec.syntax._

scala> List(List(1, 2), Nil, List(3, -4)).asExpr res0: org.dhallj.core.Expr = [[+1, +2], [] : List Integer, [+3, -4]]

You can even decode Dhall functions into Scala functions (assuming you have the appropriate codecs for the input and output types):

val Right(f) = """

let enumerate = ./dhall-lang/Prelude/Natural/enumerate

let map = ./dhall-lang/Prelude/List/map

in (n: Natural) -> map Natural Integer Natural/toInteger (enumerate n)

""".parseExpr.flatMap(_.resolve)

And then:

scala> val Right(scalaEnumerate) = f.as[BigInt => List[BigInt]]
scalaEnumerate: BigInt => List[BigInt] = org.dhallj.codec.Decoder$$anon$11$$Lambda$15614/[email protected]

scala> scalaEnumerate(BigInt(3)) res1: List[BigInt] = List(0, 1, 2)

Eventually we'll probably support generic derivation for encoding Dhall expressions to and from algebraic data types in Scala, but we haven't implemented this yet.

Converting to other formats

DhallJ currently includes several ways to export Dhall expressions to other formats. The core module includes very basic support for printing Dhall expressions as JSON:

scala> import org.dhallj.core.converters.JsonConverter
import org.dhallj.core.converters.JsonConverter

scala> import org.dhallj.parser.DhallParser.parse import org.dhallj.parser.DhallParser.parse

scala> val expr = parse("(λ(n: Natural) → [n, n + 1, n + 2]) 100") expr: org.dhallj.core.Expr.Parsed = (λ(n : Natural) → [n, n + 1, n + 2]) 100

scala> JsonConverter.toCompactString(expr.normalize) res0: String = [100,101,102]

This conversion supports the same subset of Dhall expressions as

dhall-to-json
(e.g. it can't produce JSON representation of functions, which means the normalization in the example above is necessary—if we hadn't normalized the conversion would fail).

There's also a module that provides integration with Circe, allowing you to convert Dhall expressions directly to (and from)

io.circe.Json
values without intermediate serialization to strings:
scala> import org.dhallj.circe.Converter
import org.dhallj.circe.Converter

scala> import io.circe.syntax._ import io.circe.syntax._

scala> Converter(expr.normalize) res0: Option[io.circe.Json] = Some([ 100, 101, 102 ])

scala> Converter(List(true, false).asJson) res1: org.dhallj.core.Expr = [True, False]

Another module supports converting to any JSON representation for which you have a Jawn facade. For example, the following build configuration would allow you to export spray-json values:

libraryDependencies ++= Seq(
  "org.dhallj"    %% "dhall-jawn" % "0.4.0",
  "org.typelevel" %% "jawn-spray" % "1.0.0"
)

And then:

scala> import org.dhallj.jawn.JawnConverter
import org.dhallj.jawn.JawnConverter

scala> import org.typelevel.jawn.support.spray.Parser import org.typelevel.jawn.support.spray.Parser

scala> val toSpray = new JawnConverter(Parser.facade) toSpray: org.dhallj.jawn.JawnConverter[spray.json.JsValue] = [email protected]

scala> toSpray(expr.normalize) res0: Option[spray.json.JsValue] = Some([100,101,102])

Note that unlike the dhall-circe module, the integration provided by dhall-jawn is only one way (you can convert Dhall expressions to JSON values, but not the other way around).

We also support YAML export via SnakeYAML (which doesn't require a Scala dependency):

scala> import org.dhallj.parser.DhallParser.parse
import org.dhallj.parser.DhallParser.parse

scala> import org.dhallj.yaml.YamlConverter import org.dhallj.yaml.YamlConverter

scala> val expr = parse("{foo = [1, 2, 3], bar = [4, 5]}") expr: org.dhallj.core.Expr.Parsed = {foo = [1, 2, 3], bar = [4, 5]}

scala> println(YamlConverter.toYamlString(expr)) foo:

  • 1
  • 2
  • 3 bar:
  • 4
  • 5

You can use the YAML exporter with dhall-kubernetes, for example. Instead of maintaining a lot of verbose and repetitive and error-prone YAML files, you can keep your configuration in well-typed Dhall files (like this example) and have your build system export them to YAML:

import org.dhallj.syntax._, org.dhallj.yaml.YamlConverter



val kubernetesExamplePath = "../dhall-kubernetes/1.17/examples/deploymentSimple.dhall" val Right(kubernetesExample) = kubernetesExamplePath.parseExpr.flatMap(_.resolve)

And then:

scala> println(YamlConverter.toYamlString(kubernetesExample.normalize))
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      name: nginx
  template:
    metadata:
      name: nginx
    spec:
      containers:
      - image: nginx:1.15.3
        name: nginx
        ports:
        - containerPort: 80

It's not currently possible to convert to YAML without the SnakeYAML dependency, although we may support a simplified version of this in the future (something similar to what we have for JSON in the core module).

Import resolution

There are currently two modules that implement import resolution (to different degrees).

dhall-imports

The first is dhall-imports, which is a Scala library built on cats-effect that uses http4s for its HTTP client. This module is intended to be a complete implementation of the import resolution and caching specification.

It requires a bit of ceremony to set up:

import cats.effect.{ContextShift, IO, Resource}
import org.dhallj.core.Expr
import org.dhallj.imports.syntax._
import org.dhallj.parser.DhallParser
import org.http4s.client.Client
import org.http4s.client.blaze.BlazeClientBuilder
import scala.concurrent.ExecutionContext

implicit val cs: ContextShift[IO] = IO.contextShift(ExecutionContext.global)

val client: Resource[IO, Client[IO]] = BlazeClientBuilderIO.resource

And then if we have some definitions like this:

val concatSepImport = DhallParser.parse("https://prelude.dhall-lang.org/Text/concatSep")

val parts = DhallParser.parse("""["foo", "bar", "baz"]""") val delimiter = Expr.makeTextLiteral("-")

We can use them with a function from the Dhall Prelude like this:

scala> val resolved = client.use { implicit c =>
     |   concatSepImport.resolveImports[IO]()
     | }
resolved: cats.effect.IO[org.dhallj.core.Expr] = IO$-633277477

scala> val result = resolved.map { concatSep => | Expr.makeApplication(concatSep, Array(delimiter, parts)).normalize | } result: cats.effect.IO[org.dhallj.core.Expr] =

scala> result.unsafeRunSync res0: org.dhallj.core.Expr = "foo-bar-baz"

(Note that we could use dhall-scala to avoid the use of

Array
above.)

Classpath imports

We support an extension of the spec which allows you to also import expressions from the classpath using the syntax

let e = classpath:/absolute/path/to/file in e
. The semantics are subject to change as we get more experience with it but currently it should generally have the same behaviour as an absolute path import of a local file (but files on the classpath can import each other using relative paths). This includes it being protected by the referential sanity check so that remote imports cannot exfiltrate information from the classpath.

Also note that classpath imports as location are currently not supported as the spec requires that an import as Location must return an expression of type

.

dhall-imports-mini

The other implementation is dhall-imports-mini, which is a Java library that depends only on the core and parser modules, but that doesn't support remote imports or caching.

The previous example could be rewritten as follows using dhall-imports-mini and a local copy of the Prelude:

import org.dhallj.core.Expr
import org.dhallj.imports.mini.Resolver
import org.dhallj.parser.DhallParser

val concatSep = Resolver.resolve(DhallParser.parse("./dhall-lang/Prelude/Text/concatSep"), false)

val parts = DhallParser.parse("""["foo", "bar", "baz"]""") val delimiter = Expr.makeTextLiteral("-")

And then:

scala> Expr.makeApplication(concatSep, Array(delimiter, parts)).normalize
res0: org.dhallj.core.Expr = "foo-bar-baz"

It's likely that eventually we'll provide a complete pure-Java implementation of import resolution, but this isn't currently a high priority for us.

Command-line interface

We include a command-line interface that supports some common operations. It's currently similar to the official

dhall
and
dhall-to-json
binaries, but with many fewer options.

If GraalVM Native Image is available on your system, you can build the CLI as a native binary (thanks to sbt-native-packager).

$ sbt cli/graalvm-native-image:packageBin

$ cd cli/target/graalvm-native-image/

$ du -h dhall-cli 8.2M dhall-cli

$ time ./dhall-cli hash --normalize --alpha <<< "λ(n: Natural) → [n, n + 1]" sha256:a8d9326812aaabeed29412e7b780dc733b1e633c5556c9ea588e8212d9dc48f3

real 0m0.009s user 0m0.000s sys 0m0.009s

$ time ./dhall-cli type <<< "{foo = [1, 2, 3]}" {foo : List Natural}

real 0m0.003s user 0m0.000s sys 0m0.003s

$ time ./dhall-cli json <<< "{foo = [1, 2, 3]}" {"foo":[1,2,3]}

real 0m0.005s user 0m0.004s sys 0m0.001s

Even on the JVM it's close to usable, although you can definitely feel the slow startup:

$ cd ..

$ time java -jar ./cli-assembly-0.4.0-SNAPSHOT.jar hash --normalize --alpha <<< "λ(n: Natural) → [n, n + 1]" sha256:a8d9326812aaabeed29412e7b780dc733b1e633c5556c9ea588e8212d9dc48f3

real 0m0.104s user 0m0.106s sys 0m0.018s

There's probably not really any reason you'd want to use

dhall-cli
right now, but I think it's a pretty neat demonstration of how Graal can make Java (or Scala) a viable language for building native CLI applications.

Other stuff

dhall-testing

The dhall-testing module provides support for property-based testing with ScalaCheck in the form of

Arbitrary
(and
Shrink
) instances:
scala> import org.dhallj.core.Expr
import org.dhallj.core.Expr

scala> import org.dhallj.testing.instances._ import org.dhallj.testing.instances._

scala> import org.scalacheck.Arbitrary import org.scalacheck.Arbitrary

scala> Arbitrary.arbitrary[Expr].sample res0: Option[org.dhallj.core.Expr] = Some(Optional (Optional (List Double)))

scala> Arbitrary.arbitrary[Expr].sample res1: Option[org.dhallj.core.Expr] = Some(Optional (List

It includes (fairly basic) support for producing both well-typed and probably-not-well-typed expressions, and for generating arbitrary values of specified Dhall types:

scala> import org.dhallj.testing.WellTypedExpr
import org.dhallj.testing.WellTypedExpr

scala> Arbitrary.arbitrary[WellTypedExpr].sample res2: Option[org.dhallj.testing.WellTypedExpr] = Some(WellTypedExpr(8436008296256993755))

scala> genForType(Expr.Constants.BOOL).flatMap(_.sample) res3: Option[org.dhallj.core.Expr] = Some(True)

scala> genForType(Expr.Constants.BOOL).flatMap(_.sample) res4: Option[org.dhallj.core.Expr] = Some(False)

scala> genForType(Expr.makeApplication(Expr.Constants.LIST, Expr.Constants.INTEGER)).flatMap(_.sample) res5: Option[org.dhallj.core.Expr] = Some([+1522471910085416508, -9223372036854775809, ...

This module is currently fairly minimal, and is likely to change substantially in future releases.

dhall-javagen and dhall-prelude

The dhall-javagen module lets you take a DhallJ representation of a Dhall expression and use it to generate Java code that will build the DhallJ representation of that expression.

This is mostly a toy, but it allows us for example to distribute a "pre-compiled" jar containing the Dhall Prelude:

scala> import java.math.BigInteger
import java.math.BigInteger

scala> import org.dhallj.core.Expr import org.dhallj.core.Expr

scala> val ten = Expr.makeNaturalLiteral(new BigInteger("10")) ten: org.dhallj.core.Expr = 10

scala> val Prelude = org.dhallj.prelude.Prelude.instance Prelude: org.dhallj.core.Expr = ...

scala> val Natural = Expr.makeFieldAccess(Prelude, "Natural") Natural: org.dhallj.core.Expr = ...

scala> val enumerate = Expr.makeFieldAccess(Natural, "enumerate") enumerate: org.dhallj.core.Expr = ...

scala> Expr.makeApplication(enumerate, ten).normalize res0: org.dhallj.core.Expr = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Note that the resulting jar (which is available from Maven Central as dhall-prelude) is many times smaller than either the Prelude source or the Prelude serialized as CBOR.

Developing

The project includes the currently-supported version of the Dhall language repository as a submodule, so if you want to run the acceptance test suites, you'll need to clone recursively:

git clone --recurse-submodules [email protected]:travisbrown/dhallj.git

Or if you're like me and always forget to do this, you can initialize the submodule after cloning:

git submodule update --init

This project is built with sbt, and you'll need to have sbt installed on your machine.

We're using the JavaCC parser generator for the parsing module, and we have our own sbt plugin for integrating JavaCC into our build. This plugin is open source and published to Maven Central, so you don't need to do anything to get it, but you will need to run it manually the first time you build the project (or any time you update the JavaCC grammar):

sbt:root> javacc
Java Compiler Compiler Version 7.0.5 (Parser Generator)
File "Provider.java" does not exist.  Will create one.
File "StringProvider.java" does not exist.  Will create one.
File "StreamProvider.java" does not exist.  Will create one.
File "TokenMgrException.java" does not exist.  Will create one.
File "ParseException.java" does not exist.  Will create one.
File "Token.java" does not exist.  Will create one.
File "SimpleCharStream.java" does not exist.  Will create one.
Parser generated with 0 errors and 1 warnings.
[success] Total time: 0 s, completed 12-Apr-2020 08:48:53

After this is done, you can run the tests:

sbt:root> test
...
[info] Passed: Total 1319, Failed 0, Errors 0, Passed 1314, Skipped 5
[success] Total time: 36 s, completed 12-Apr-2020 08:51:07

Note that a few tests require the dhall-haskell

dhall
CLI. If you don't have it installed on your machine, these tests will be skipped.

There are also a few additional slow tests that must be run manually:

sbt:root> slow:test
...
[info] Passed: Total 4, Failed 0, Errors 0, Passed 4
[success] Total time: 79 s (01:19), completed 12-Apr-2020 08:52:41

Community

This project supports the Scala code of conduct and wants all of its channels (Gitter, GitHub, etc.) to be inclusive environments.

Copyright and license

All code in this repository is available under the 3-Clause BSD License.

Copyright Travis Brown and Tim Spence, 2020.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.