waspc/README.md
This directory contains source code of the wasp CLI/compiler (aka waspc), and this README is aimed at the contributors to the project.
If you are a Wasp user and not a contributor (yet 😉), you might want to look into the following resources instead (Project page, Docs).
If you would like to make your first contribution, here is a handy checklist we made for you:
kitchen-sink example running (follow Basics).main and have it accepted! Check Typical workflow and Branching and merging strategy for guidance, and consult Codebase overview for more details on how Wasp compiler works internally.The core of waspc is implemented in Haskell, but you will also see a lot of Javascript/TypeScript and other web technologies which apps generated with Wasp use.
You don't have to be an expert in Haskell to contribute or understand the code since we don't use complicated Haskell features much. Most of the code is relatively simple and straight-forward, and we are happy to help with anything you get stuck with.
The main result of building the project is wasp executable (also referred to as CLI), which is at the same time Wasp compiler, CLI, and Wasp project runner in one - one main tool for everything Wasp-related.
Note that while waspc is the name of this Haskell project, the executable's name is wasp when released, and wasp-cli during development.
Clone this repo (or its fork) to your machine.
Position yourself in this directory (waspc/) and make sure that you are on the main branch.
run script./run script captures the most common commands/workflows for the development of waspc (similar to the role that scripts in package.json play in npm projects), e.g. ./run build, ./run test, ... .
It serves both as a convenient way to easily run common development tasks and also as a sort of documentation on how we develop waspc. You will see us mentioning it often in the rest of this README.
Running ./run without any arguments will print help/usage, which is a good way to discover the typical waspc development workflows.
[!TIP] To make it easy to run the
./runscript from any place in your waspc codebase, you can create a bash alias that points to it, e.g.:shalias wrun="/home/martin/git/wasp-lang/wasp/waspc/run"
The best way to install the Haskell toolchain is via ghcup.
Once you have ghcup installed, run the ghcup tui command to install and select the correct versions of cabal (package manager), hls (language server), and ghc (compiler).
To find out what the correct versions of these tools are for the waspc:
./run ghcup-set to determine the specific versions of ghc and hls you should use. While the primary purpose of this command is to set them to the correct versions, it will also print those versions as a side effect.cabal, just go with the latest one.[!NOTE] On Mac, we recommend using the official ghcup installer over Homebrew, as it works out of the box.
You need to have the correct version of node (and npm) installed and set.
Check .nvmrc file to learn which version of node that should be.
./run build
to build the whole waspc project.
This might take a while (e.g. 10 mins) if you are doing it for the very first time, due to having to download all the dependencies (which will later be cached). If that is the case, relax and feel free to get yourself a cup of coffee! When somebody asks what you are doing, you can finally rightfully say "compiling!" 😀.
./run test
to ensure all the tests are passing.
[!NOTE] On Windows, the powershell's active code page should be changed to utf-8 to support special letters in some test suits (ref. PR3027).
pschcp 65001
wasp CLI./run wasp-cli
to run the wasp-cli executable (that you previously built with ./run build)!
Since you provided no arguments, you should see help/usage.
Note that the executable during development is named wasp-cli, unlike wasp which is how it is named when you install a public release of Wasp via official installation method.
This is to make it easier to differentiate between the development version of wasp CLI and the released version.
kitchen-sink exampleFirst, position yourself in the examples/kitchen-sink/ dir.
Then, run the dev database:
./run wasp-cli db start
Leave it running, open a new terminal (tab), and run
./run wasp-cli db migrate-dev
to update the database schema.
Run
cp .env.server.example .env.server
to set up some mock env vars for the server, just to get the things running.
Finally, run
./run wasp-cli start
to run the example app in the development mode.
If you are doing this for the very first time, it might take a minute or so to download and install npm dependencies.
When done, new tab in your browser should open and you will see the Kitchen Sink App!
[!NOTE] You will notice that some functionality in the
kitchen-sinkis not working. That is because the env vars in.env.serverfiles are just mock values. Checkkitchen-sink's README for more details on how to set up env vars for development.
main../run ghcid from the root of the waspc project: this will run a process that watches the Haskell project and reports any Haskell compiler errors. Leave it running.
ghcid globally first. You can do it with cabal install ghcid.src/ or cli/src/ or data/), and update tests if that makes sense (see Test).
Fix any errors shown by HLS/ghcid.
Rinse and repeat. If you're an internal team member, postpone updating waspc e2e tests tests until approval (see here)../run build:hs to build the Haskell/cabal project, and ./run wasp-cli to both build and run it. If you changed code in data/packages/, you will also need to run ./run build:packages (check TypeScript Packages section for more details). Alternatively, you can also run slower ./run build to at the same time build Haskell, TS packages, and any other piece of the project in one command.kitchen-sink app, which we always keep updated. Also, if you added a new feature, add it to this app (+ tests) if needed. Check its README for more details (including how to run it)../run test to confirm that all the tests are passing. If needed, accept changes in the waspc e2e tests with ./run test:waspc:e2e:accept-all. Check "Tests" for more info.Changelog.md. Also, bump version in waspc.cabal and ChangeLog.md if needed. If you are not sure how to decide which version to go with, check out how we determine the next version./web/docs.main. Yay!Do not update waspc e2e tests until your PR is approved.
Accepting waspc e2e tests prematurely:
Carefully review waspc e2e tests diffs before accepting them (see Waspc e2e tests for more details).
If the feature you are implementing is complex, be it due to its design or technical implementation, we recommend creating a design doc (aka RFC). It is a great way to share the idea you have with others while also getting help and feedback.
To create one, make a PR that adds a markdown document under wasp/docs/design-docs, and in that markdown document explain the thinking behind and choices made when deciding how to implement a feature.
Others will comment on your design doc, and once it has gone through needed iterations and is approved, you can start with the implementation of the feature (in a separate PR).
Wasp is implemented in Haskell.
Codebase is split into library (src/) and CLI (which itself has a library cli/src/ and thin executable wrapper cli/exe/).
CLI is actually wasp executable, and it uses the library, where most of the logic is.
Wasp compiler takes .wasp files + everything in the src/ dir (JS, HTML, ...) and generates a web app that consists of a client, server, and database.
Wasp compiler code is split into 2 basic layers: Analyzer (frontend) and Generator (backend).
Wasp file(s) are analyzed by Analyzer, where they are first parsed, then typechecked, and then evaluated into a central IR (Intermediate Representation), which is AppSpec (src/Wasp/AppSpec.hs).
Check src/Wasp/Analyzer.hs for more details.
AppSpec is passed to the Generator, which based on it decides how to generate a web app.
Output of Generator is a list of FileDrafts, where each FileDraft explains how to create a file on the disk.
Therefore, Generator doesn't generate anything itself, instead it provides instructions (FileDrafts) on how to generate the web app.
FileDrafts are using mustache templates a lot (they can be found in data/Generator/templates). Alongside mustache templates,
generated apps use internal npm packages (WaspLibs) that contain core logic and are unit tested. WaspLibs are bundled with Wasp
and installed into generated apps (see WaspLibs for more details).
Generator is split into three generators, for the three main parts of the web app: WebAppGenerator, ServerGenerator and DbGenerator.
After Generator provides us with FileDrafts, web app is generated by writing the FileDrafts to the disk.
Generated web app consists of a client, server and database.
Client is written with React and react-query. Server is written in NodeJS and uses ExpressJs. Database is abstracted via Prisma.
We can run Wasp project with wasp start.
This will first compile the app, generate JS code in the .wasp/out/ dir, and then run npm start for the client, npm start for the server, and also run the database.
On any changes you do to the source code of Wasp, Wasp project gets recompiled, and then changes in the generated app are picked up by the npm start of the client/server, therefore updating the web app.
src/ -> main source code, librarycli/src/ -> rest of the source code, cli, uses librarycli/exe/ -> thin executable wrapper around cli library codetests/, e2e-tests/, cli/tests/, waspls/tests/, starters-e2e-tests -> testsdata/Generator/templates/ -> mustache templates for the generated client/server.libs/ -> internal npm packages (WaspLibs) that are bundled with Wasp and copied into generated apps (see WaspLibs for more details)packages/ -> TypeScript packages used by Wasp compiler (see TypeScript Packages for more details)data/Cli/starters/ -> starter templates for new projectswaspc, while implemented in Haskell, relies on TypeScript for some of its functionality (e.g. for parsing TS code, or for deployment scripts). In these cases, the Haskell code runs these TS packages as separate processes and communicates through input/output streams. These packages are located in data/packages/ and are normal npm projects. See data/packages/README.md for how the projects are expected to be set up.
In order for waspc's Haskell code to correctly use these TS packages (and to also have them correctly bundled when generating the release tarball), they need to be correctly installed/built in the waspc_datadir dir.
To do so in development, run ./run build:packages when any changes are made to these packages. We also run it in CI when building the release.
WaspLibs are internal npm packages located under libs/ and are bundled with Wasp and installed into generated apps. A more detailed description of WaspLibs can be found in the libs/README.md.
For tests in Haskell we are using Tasty testing framework. Tasty lets us combine different types of tests into a single test suite.
In Tasty, there is a main test file that is run when test suite is run. In that file we need to manually compose test tree out of tests that we wrote. We organize tests in test groups, which are then recursively grouped resulting in a test tree. Cool thing is that we can organize tests this way however we want and also mix different type of tests (hspec, quickcheck, and whatever else we want).
Tests are normally split in files of course, so we need to import those all the way up to the main test file, however we organize our test groups/trees.
In order to avoid need for manual organization and importing of test files described above, we are using tasty-discover which does this for us. It automatically detects files containing tests and organizes them for us into a test tree (and also takes care of importing). This means we only need to create a file, write tests in it and that is it. Test functions however do need to be prefixed with special prefix to indicate which type of test are they: spec* for Hspec, prop* for QuickCheck and similar. Check docs for more details. We can however still organize tests manually if we want in Tasty test trees, and then we just prefix them with test_ and tasty-discover will pick them up from there.
Additionally, currently we limited tasty-discover to auto-detect only files ending with Test.hs (*Test.hs glob). We might remove that requirement in the future if it proves to have no benefit.
To summarize: If you are writing new tests, just put them in a file that ends with Test.hs in tests/ dir and that is it.
For unit testing, we use Hspec.
For property testing, we use Quickcheck.
We additionally use doctest for testing code examples in documentation.
All tests go into tests/ directory.
Haskell build tools don't have a good support for mixing them with source files.
To run tests:
./run testwaspc tests only, you can do ./run test:waspc.waspc unit tests only, you can do ./run test:waspc:unit../run test:waspc:unit "Some test description to match".waspc e2e tests only, you can do ./run test:waspc:e2e../run test:cli../run test:waspls.kitchen-sink e2e tests, you can do ./run test:kitchen-sink../run test:examples../run test:starters.Inside of waspc e2e tests we have snapshot tests that run the wasp-cli on a couple of prepared projects, check that they successfully run, and also compare generated app with the expected generated app (golden output).
This means that when you make a change in your code that modifies the generated app, snapshot tests will fail while showing a diff between the new generated app and the previous (golden) one.
This gives you an opportunity to observe these differences and ensure that they are intentional and that you are satisfied with them.
It is the PR author's (or the reviewers for outside contributions) responsibility to carefully review these diffs.
Do not blindly accept changes, ensure they align with your intended modifications.
If you notice something unexpected or weird, you have an opportunity to fix it.
Once you are indeed happy with the changes in the generated app, you will want to update the golden output to the new (current) output, so that tests pass.
Basically, you want to say "I am ok with the changes and I accept them as the new state of things.".
Easiest way to do this is to use the convenient command from the ./run script:
./run test:waspc:e2e:accept-all
To run the Haskell code analysis, run:
./run code-check
This will check if code is correctly formatted, if it satisfies linter, and if it passes static analysis.
These same checks are required to pass the CI, so make sure this is passing before making a PR.
TODO: For now we check only the code formatting during the CI. In the future, once we make sure all the warnings are passing, we will also check linter and static analysis during the CI, but that is not happening yet.
For formatting Haskell code we use Ormolu.
Normally we set it up in our editors to run on file save.
You can also run it manually with
./run check:ormolu
to see if there is any formatting that needs to be fixed, or with
./run format:ormolu
to have Ormolu actually format (in-place) all files that need formatting.
NOTE: When you run it for the first time it might take a while (~10 minutes) for all the dependencies to get installed. The subsequent runs will be much faster.
We use hlint for linting our Haskell code.
You can use
./run hlint
to run the hlint on Wasp codebase.
NOTE: When you run it for the first time it might take a while (~10 minutes) for all the dependencies to get installed. The subsequent runs will be much faster.
We use stan to statically analyze our codebase.
The easiest way to run it is to use
./run stan
This will build the codebase, run stan on it (while installing it first, if needed, with the correct version of GHC) and then write results to the CLI and also generate report in the stan.html.
NOTE: When you run it for the first time it might take a while (~10 minutes) for all the dependencies to get installed. The subsequent runs will be much faster.
This repo contains both the source code that makes up a Wasp release (under waspc), as well as our website containing documentation and blog posts (under web), and also Mage web app (under mage). In order to facilitate the development of Wasp code while still allowing for website / Mage updates or hotfixes of the current release, we have decided on the following minimal branching strategy.
All Wasp development should be done on feature branches. They form the basis of PRs that will target one of the two following branches:
main: this branch contains all the actively developed new features and corresponding documentation updates. Some of these things may not yet be released, but anything merged into main should be in a release-ready state.
release: this branch contains the source code of current/latest Wasp release, as well as the documentation and blog posts currently published and therefore visible on the website, and also currently published version of Mage.
main, we do the following:
main branch by merging release into it. There might be conflicts but they shouldn't be too hard to fix. Once main is updated, you can create a new waspc release from it, as well as deploy the website from it.release branch to this new main by merging main into it. There will be no conflicts since we already resolved all of them in the previous step.How do I know where I want to target my PR, to release or main?
main is ready for publishing, then you want to target release. This could be website content update, new blog post, documentation (hot)fix, compiler hotfix that we need to release quickly via a new patch version, update for Mage that needs to go out now, ... .main. These are new features, refactorings, docs accompanying new features, ... .release (docs, Mage) uses/references version of wasp that was last released (so one that is also on release).release represents the present, and is for changes to the already published stuff.main represents near future, and is for changes to the to-be-published stuff.[!IMPORTANT] If you merge a PR or push any changes to
release, a new PR will automatically be created to updatemainwith the changes fromrelease. You will be assigned it, and it's your responsibility to merge it.
We use Github Actions for CI.
CI runs for any commits on main branch, for pull requests, and for any commits tagged with tag that starts with v.
During CI, we build and test Wasp code on Linux, MacOS and Windows.
If commit is tagged with tag starting with v, github draft release is created from it containing binary packages.
We also have a workflow for deploying example apps to Fly.io (release-examples-deploy.yaml). This workflow can be run manually from the GitHub UI and should typically be run from the release branch after publishing a new release to ensure the deployed examples are using the latest stable version of Wasp.
You can run this workflow:
From GitHub UI: https://github.com/wasp-lang/wasp/actions/workflows/release-examples-deploy.yaml (click "Run workflow" and select the release branch)
From GitHub CLI:
# Deploy with latest Wasp version
gh workflow run release-examples-deploy --ref release
# Deploy with specific Wasp version
gh workflow run release-examples-deploy --ref release -f version=0.13.2
If you put [skip ci] in commit message, that commit will be ignored by Github Actions.
We also wrote a new-release script which you can use to help you with creating new release: you need to provide it with new version (./new-release 0.3.0) and it will check that everything is all right, create appropriate tag and push it, therefore triggering CI to create new release on Github.
NOTE: If building of your commit is suddenly taking much longer time, it might be connected with cache on Github Actions. If it happens just once every so it is probably nothing to worry about. If it happens consistently, we should look into it.
Do the steps marked with 👉 for every release of waspc.
Do the non-bold steps when necessary (decide for each step depending on the changes, e.g. some can be skipped if there were no breaking changes).
merge-before-release.main (go to https://github.com/wasp-lang/open-saas/actions/workflows/e2e-tests.yml and manually run the workflow on the main branch).release branch into main. You can see the latest PR at https://github.com/wasp-lang/wasp/pull/release.git fetch) and ensure local main is up to date.main) into a new RC branch called rc-<version> (e.g., rc-0.19.1) and do the rest of the steps from there.waspc.cabal should already be correct, but double check and update it if needed.
waspc.cabal: create a PR, wait for approval and all the checks (CI) to pass. Then squash and merge the PR into the RC branch.ChangeLog.md and migration guides.npm run docusaurus docs:version {version} in the web dir. Check the README in the web dir for more details. Commit this change to the RC branch and push it.release branch and fast-forward it to the RC branch by running git merge --ff rc-<version>.release and then run ./new-release 0.x.y.
npm dist-tag add @wasp.sh/wasp-cli@<version> latest for users to get the newest version when they install through npm.release branch to remote.release back to main (you can also find it here). Make sure to merge that PR (create a merge commit, don't squash or rebase). This ensures that main is ahead of release and we won't have merge conflicts in future releases.release branch.release branch.waspc follows typical SemVer versioning scheme, so major.minor.patch.
There is one slightly peculiar thing though: waspc, besides being a wasp compiler and CLI, also contains wasp language server (waspls) inside it, under the subcommand wasp waspls.
So how do changes to waspls affect the version of waspc, since they are packaged together as one exe? We have decided, for practical reasons, to have them affect the patch number, possibly maybe minor, but not major.
Making a test release, especially "Release Candidate" (RC) release is useful when you want to test the release without it being published to the normal users. If doing this, steps are the following:
Create a new branch called rc-<version> (e.g., rc-0.19.0) by branching out of the last commit you want to release (probably latest main).
Locally execute the new-release script. Append -rc.N to the version number to make it obvious that this release is a pre-release used for testing (e.g., ./new-release 0.19.1-rc.1).
The script will throw some warnings which you should accept.
Once the draft release is created on Github:
rc-<version> branch to remote.Since npm installs the latest release by default, it will skip this pre-release (which is what we wanted). You can install it by pasing an explicit version! That way user's don't get in touch with it, but we can install and use it normally:
npm i -g @wasp.sh/[email protected]
Create a new checklist in Notion and go through the "Before the release" section. If you find problems, fix them on the rc branch and create a new RC following the same process (e.g., 0.19.0-rc.2, see step 2).
External documentation, for users of Wasp, is hosted at https://wasp.sh/docs, and its source is available at web/docs, next to the website and blog.
Wasp's magic GPT web app generator aka Wasp AI aka Mage is hosted at https://usemage.ai and its source is available at mage.
Make sure to update it when changes modify how Wasp works.
We are documenting best practices related to Haskell in our Haskell Handbook.
Why don't we use a cabal freeze file to lock our dependencies?
In order to better support a wider range of developer operating systems, we have decided against using a cabal freeze file and instead use cabal's index-state feature to get package version pinning from hackage. See this question for more: https://github.com/haskell/cabal/issues/8059 .
When writing a comment, we prefer starting it with a capital letter.
If it starts with a capital letter, it must end with a punctuation.
If it doesn't start with a capital letter, it shouldn't end with a punctuation.
When writing a TODO or NOTE, use all capital letters, like this:
-- TODO: Wash the car.
-- NOTE: This piece of code is slow.
If you wish, you can add your name to TODO / NOTE. This is useful if you think there is a fair chance that reader of that TODO / NOTE might want to consult with its author. You can do it like this:
-- TODO(martin): Doesn't work on my machine in some unusual use cases.
For detailed reasoning/discussion on all listed rules, check Issue #487.
When defining top-level named functions, we prefer to use statements:
// good
function foo(param) {
// ...
}
// bad
const foo = (param) => {
// ...
};
// bad
const foo = function (param) {
// ...
};
When defining inline function expressions, we prefer the arrow syntax:
// good
const squares = arr.map((x) => x * x);
// bad
const squares = arr.map(function (x) {
return x * x;
});