docs/compiler/README.md
Hello! Welcome to the documentation for the Gleam compiler. I hope you have fun with the project!
<!-- vscode-markdown-toc --> <!-- vscode-markdown-toc-config numbering=false autoSave=true /vscode-markdown-toc-config --> <!-- /vscode-markdown-toc -->The project is made up of several Rust crates (projects):
compiler-core: This project parses, analyses, and compiles Gleam projects.
It is entirely pure and has no IO so that is provided by the other Rust crates
that wrap this one.language-server: This project contains code for the Gleam language server,
including autocomplete, code actions, hover and other features.compiler-cli: A command line interface that wraps the core compiler and
language server, providing IO to files and to the console.compiler-wasm: A JavaScript interface to the core compiler via web assembly.
Suitable for running in a web browser.In addition to the Rust code there are these components:
Makefile: A makefile that defines shortcut commands for common tasks when
working in the Gleam codebase. Run make help to view them.test: A collection of (mostly) Gleam projects that serve as integration tests for the
compiler.deny.toml: Configuration for the cargo-deny tool which is used to ensure
that the Rust libraries the compiler depends on adhere to our expectations.containers: A collection of docker-files used to produce OCI containers for
each Gleam release..github/workflows: GitHub Actions workflow definitions, used to build, test,
and release new versions of the project.docs: You're looking at it pal.The process for compiling Gleam modules within the compiler looks roughly like this:
Gleam source code .cache binaries
▼ ▼
┌────────────────────┐ ┌───────────────────────┐
│ Parser │ │ Metadata deserializer │
└────────────────────┘ └───────────────────────┘
│ │
Untyped AST Module metadata
└─────────┐ ┌────────┘ │
▼ ▼ │
┌─────────────────────┐ │
│ Dependency sorter │ │
└─────────────────────┘ │
│ │
Untyped AST │
(sorted by deps) │
▼ │
┌───────────────────┐ │
│ Type checker │◄─────┘
└───────────────────┘
│
┌────── Typed AST ──────┐
▼ ▼
┌────────────────────┐ ┌─────────────────────┐
│ Code generator │ │ Metadata serializer │
└────────────────────┘ └─────────────────────┘
│ │
│ ▼
Erlang or JavaScript .cache binaries
printing algebra
▼
┌────────────────────┐
│ Pretty printer │
└────────────────────┘
│
▼
Erlang or JavaScript
source code
We like automated tests! They're a great way to verify that the compiler is doing what we expect it do as we make changes.
make test: Run all the tests. Worth doing before committing any changes.make test-watch: Run the Rust unit tests when files are saved.make language-test: Run the cross-platform language integration tests in
test/language.make language-test-watch: Run said tests when files are saved.make javascript-prelude-test: Run the unit tests for the JavaScript code
that implements the Gleam prelude when compiling to JavaScript.make javascript-prelude-test-watch: Run said tests when files are saved.The *-watch commands require the watchexec program to be
installed.
The compiler makes heavy use of snapshot testing using the
cargo-insta tool.
If you're not familiar with snapshot testing instead of writing an input and an expected output (as in normal example based tests) you write only the input. The snapshot testing tool can then be used to mark any new outputs as accepted and saved into the repository as snap files. If an output for one of the tests changes then it is considered a failed test and the programmer has the option to either reject the new version (as it is an incorrect result) or accept it as the new correct output.
This style of testing saves us huge amounts of time as manually updating all the expected output when we make changes to the output format of the compiler or error messaging is time-consuming and very dull. With snapshot testing it takes seconds.
# Run the tests
make test
# Interactively verify changes to the snapshots
cargo insta review