Parser Technology Analysis

As the Enso parser has some fairly unique requirements placed upon it, the choice of implementation technology is of paramount importance. Choosing the correct technology ensures that we can meet all of the requirements placed upon the parser.

<!-- MarkdownTOC levels="2,3" autolink="true" -->

• Technology Requirements for the Parser
• Issues With the Previous Implementation
• Choosing Rust
  • Downsides of Rust

<!-- /MarkdownTOC -->

Technology Requirements for the Parser

As the parser has to work both for the Engine and for the IDE, it has a strange set of requirements:

• The implementation language must be able to run on native platforms, as well as in the browser via WASM (not JavaScript due to the marshalling overhead).
• The implementation language should permit excellent native performance on both native and web platforms, by giving implementers fine-grained control over memory usage.
• The implementation language must be able to target all primary platforms: macOS, Linux and Windows.

Issues With the Previous Implementation

The previous implementation of the parser was implemented in Scala, and had some serious issues that have necessitated this rewrite:

• Performance: The structures used to implement the parser proved inherently difficult for a JIT to optimise, making performance far worse than expected on the JVM.
• ScalaJS Sub-Optimal Code Generation: The JavaScript generated by ScalaJS was very suboptimal for these structures, making the parser even slower when run in the browser.
• JS as a Browser Target: To transfer textual data between WASM and JS incurs a significant marshalling overhead. As the IDE primarily works with textual operations under the hood, this proved to be a significant slowdown.

Choosing Rust

Rust, then, is an obvious choice for the following reasons:

• It can be compiled natively into the IDE binary, providing them with excellent performance.
• As a native language it can use JNI to directly create JVM objects on the JVM heap, for use by the compiler.
• As a native language it can be called directly via JNI.
• There is potential in the future for employing Graal's LLVM bitcode interpreter to execute the parser safely in a non-native context.

Downsides of Rust

This is not to say that choosing rust doesn't come with some compromises:

• It significantly complicates the CI pipeline for the engine, as we will have to build native artefacts for use by the runtime itself.
• As a non-JVM language, the complexity of working with it from Scala and Java is increased. We will need to maintain a full definition of the AST in Scala to permit the compiler to work properly with it.