SDK Size Benchmark

To avoid future issues like #2747, it's important to validate the v3 SDK packages' sizes are reasonable. This package provides a command line tool to benchmark and validate the SDK size metrics, including:

• publish size: The size of the package's source code published to npm.
• publish files: The count of individual files fo the package's source code published to npm.
• Webpack compatibility: Whether users can run Webpack command successfully with minimal configuration, and version of Webpack tool.
• Rollup compatibility: Whether users can run Rollup command successfully with minimal configuration, and version of Rollup tool.
• Esbuild compatibility: Whether users can run Esbuild command successfully with minimal configuration, and version of Esbuild tool.

By default, the report Markdown file is generated under report.md

How to use it

By default, the tool doesn't require any options, but it offers some options in case you need to customize the behavior. You can see the full descriptions by running:

# from project root
yarn test:size --help

here's the options description:

Options:
  --version           Show version number                              [boolean]
  --since             Run the size benchmark on changed package since last
                      release or main branch
              [string] [choices: "all", "last_release", "main"] [default: "all"]
  --scopeConfigPath
  [string] [default: "/path/to/project/root/scripts/benchmark-size/scope.json"]
  --limitConfigPath
  [string] [default: "/path/to/project/root/scripts/benchmark-size/limit.json"]
  --skipLocalPublish  Skip publishing the packages locally. You can skip it if
                      you didn't change any packages since last execution
                                                      [boolean] [default: false]
  --rawOutputPath [string] [default: "/path/to/project/root/benchmark/size/raw"]
  --reportPath                                                [string] [default:
        "/path/to/project/root/benchmark/size/report.md"]
  --skipRawOutput     Whether to generate the raw benchmark data to configured
                      path                            [boolean] [default: false]
  --help              Show help                                        [boolean]

Important Configuration Files

The maintainers only need to test the packages that is public for customer to consume publicly like all service clients and some utility packages like presigner. The other internal packages are tested as dependencies of those public packages. There are 2 config files that maintainers need to update to make sure the tool runs on all packages it suppose to run and validate them with reasonable limit:

• scope.json: Define what packages are considered as public, so they will exist as an entry in the report. You can use wild card in the package entry. For packages that requires peer dependencies, you can specify dependencies list. You can skip running the browser bundlers test if the package is Node.js runtime only by setting the skipBundlerTests boolean entry.
• limit.json: Define how to validate the packages' size. For each package, you can validate either of the metrics the tool generates: publishSize and publishFiles. For each metrics, you can validate them by limit -- the absolute value, or hike -- the percentage of increase since recent value.

How does it work

The command works in following process:

1. Detect the packages that changed since last release, thus requires validating their size metrics before next release.
2. Validate they have been built locally and release to a local NPM registry
3. For each changed package, creating a temporary project with the project templates The templates generation context is exactly defined in scope.json.
4. Run NPM install from the local registry and calculate the npm size metrics.
5. Run Webpack, Rollup, and Esbuild bundlers on the temporary project and calculate the bundle size for each of them.
6. Clear temporary projects.
7. Update the report.md. Validating them according to limit.json

Prior arts referred during the implementation.