BENCHMARKS.md
All benchmarks were computed on macOS (Apple M3 Max 16, 128 GB) with the following tool versions:
Benchmark performance may vary across operating systems, and from project to project. This document includes benchmarks from a variety of projects to provide a representative example of real-world usage: Black, discord.py, Home Assistant, isort, Jinja, pandas, pandas-stubs, Prefect, and PyTorch.
For instructions on running the benchmarks, see
ty_benchmark/README.md.
black
-----
Benchmark 1: ty
Time (mean ± σ): 75.9 ms ± 1.8 ms [User: 601.3 ms, System: 37.9 ms]
Range (min … max): 72.5 ms … 81.8 ms 36 runs
Warning: Ignoring non-zero exit code.
Benchmark 2: Pyrefly
Time (mean ± σ): 125.9 ms ± 2.3 ms [User: 472.6 ms, System: 52.1 ms]
Range (min … max): 121.8 ms … 131.5 ms 22 runs
Warning: Ignoring non-zero exit code.
Benchmark 3: mypy
Time (mean ± σ): 1.301 s ± 0.021 s [User: 1.204 s, System: 0.090 s]
Range (min … max): 1.283 s … 1.355 s 10 runs
Benchmark 4: Pyright
Time (mean ± σ): 1.410 s ± 0.011 s [User: 18.056 s, System: 1.033 s]
Range (min … max): 1.395 s … 1.429 s 10 runs
Warning: Ignoring non-zero exit code.
Summary
ty ran
1.66 ± 0.05 times faster than Pyrefly
17.15 ± 0.50 times faster than mypy
18.59 ± 0.47 times faster than Pyright
-------------------------------------------------------------------------------
discord.py
----------
Benchmark 1: ty
Time (mean ± σ): 155.1 ms ± 2.3 ms [User: 1726.5 ms, System: 73.1 ms]
Range (min … max): 152.0 ms … 161.7 ms 18 runs
Warning: Ignoring non-zero exit code.
Benchmark 2: Pyrefly
Time (mean ± σ): 196.4 ms ± 5.4 ms [User: 1210.5 ms, System: 114.7 ms]
Range (min … max): 189.9 ms … 207.8 ms 14 runs
Warning: Ignoring non-zero exit code.
Benchmark 3: mypy
Time (mean ± σ): 2.492 s ± 0.009 s [User: 2.387 s, System: 0.100 s]
Range (min … max): 2.481 s … 2.505 s 10 runs
Warning: Ignoring non-zero exit code.
Benchmark 4: Pyright
Time (mean ± σ): 3.467 s ± 0.067 s [User: 48.685 s, System: 1.909 s]
Range (min … max): 3.344 s … 3.574 s 10 runs
Warning: Ignoring non-zero exit code.
Summary
ty ran
1.27 ± 0.04 times faster than Pyrefly
16.06 ± 0.24 times faster than mypy
22.35 ± 0.54 times faster than Pyright
-------------------------------------------------------------------------------
homeassistant
-------------
Benchmark 1: ty
Time (mean ± σ): 1.981 s ± 0.020 s [User: 26.755 s, System: 2.283 s]
Range (min … max): 1.956 s … 2.013 s 10 runs
Warning: Ignoring non-zero exit code.
Benchmark 2: Pyrefly
Time (mean ± σ): 2.565 s ± 0.031 s [User: 23.667 s, System: 4.246 s]
Range (min … max): 2.522 s … 2.623 s 10 runs
Warning: Ignoring non-zero exit code.
Benchmark 3: mypy
Time (mean ± σ): 24.341 s ± 0.120 s [User: 44.400 s, System: 4.381 s]
Range (min … max): 24.167 s … 24.590 s 10 runs
Benchmark 4: Pyright
Time (mean ± σ): 23.703 s ± 0.132 s [User: 325.588 s, System: 23.749 s]
Range (min … max): 23.534 s … 23.927 s 10 runs
Warning: Ignoring non-zero exit code.
Summary
ty ran
1.30 ± 0.02 times faster than Pyrefly
11.97 ± 0.14 times faster than Pyright
12.29 ± 0.14 times faster than mypy
-------------------------------------------------------------------------------
isort
-----
Benchmark 1: ty
Time (mean ± σ): 252.9 ms ± 2.8 ms [User: 497.3 ms, System: 25.4 ms]
Range (min … max): 249.9 ms … 259.6 ms 11 runs
Warning: Ignoring non-zero exit code.
Benchmark 2: Pyrefly
Time (mean ± σ): 109.1 ms ± 1.2 ms [User: 291.1 ms, System: 42.2 ms]
Range (min … max): 106.6 ms … 112.1 ms 25 runs
Warning: Ignoring non-zero exit code.
Benchmark 3: mypy
Time (mean ± σ): 666.4 ms ± 7.4 ms [User: 596.3 ms, System: 64.6 ms]
Range (min … max): 661.6 ms … 686.9 ms 10 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs.
Benchmark 4: Pyright
Time (mean ± σ): 2.224 s ± 0.020 s [User: 16.910 s, System: 0.951 s]
Range (min … max): 2.188 s … 2.255 s 10 runs
Warning: Ignoring non-zero exit code.
Summary
Pyrefly ran
2.32 ± 0.04 times faster than ty
6.11 ± 0.10 times faster than mypy
20.38 ± 0.29 times faster than Pyright
-------------------------------------------------------------------------------
jinja
-----
Benchmark 1: ty
Time (mean ± σ): 77.4 ms ± 1.2 ms [User: 399.4 ms, System: 23.1 ms]
Range (min … max): 74.8 ms … 80.4 ms 35 runs
Warning: Ignoring non-zero exit code.
Benchmark 2: Pyrefly
Time (mean ± σ): 104.0 ms ± 1.6 ms [User: 273.1 ms, System: 34.6 ms]
Range (min … max): 101.3 ms … 108.0 ms 27 runs
Warning: Ignoring non-zero exit code.
Benchmark 3: mypy
Time (mean ± σ): 752.8 ms ± 2.5 ms [User: 691.2 ms, System: 58.1 ms]
Range (min … max): 749.6 ms … 757.4 ms 10 runs
Warning: Ignoring non-zero exit code.
Benchmark 4: Pyright
Time (mean ± σ): 1.311 s ± 0.019 s [User: 16.258 s, System: 0.940 s]
Range (min … max): 1.283 s … 1.341 s 10 runs
Warning: Ignoring non-zero exit code.
Summary
ty ran
1.34 ± 0.03 times faster than Pyrefly
9.73 ± 0.15 times faster than mypy
16.95 ± 0.35 times faster than Pyright
-------------------------------------------------------------------------------
pandas
------
Benchmark 1: ty
Time (mean ± σ): 468.1 ms ± 14.3 ms [User: 5237.0 ms, System: 183.0 ms]
Range (min … max): 448.3 ms … 492.4 ms 10 runs
Warning: Ignoring non-zero exit code.
Benchmark 2: Pyrefly
Time (mean ± σ): 753.0 ms ± 26.3 ms [User: 7074.4 ms, System: 283.7 ms]
Range (min … max): 710.2 ms … 794.1 ms 10 runs
Warning: Ignoring non-zero exit code.
Benchmark 3: mypy
Time (mean ± σ): 12.685 s ± 0.058 s [User: 12.319 s, System: 0.348 s]
Range (min … max): 12.603 s … 12.776 s 10 runs
Benchmark 4: Pyright
Time (mean ± σ): 7.225 s ± 0.076 s [User: 104.063 s, System: 3.411 s]
Range (min … max): 7.118 s … 7.323 s 10 runs
Warning: Ignoring non-zero exit code.
Summary
ty ran
1.61 ± 0.07 times faster than Pyrefly
15.43 ± 0.50 times faster than Pyright
27.10 ± 0.84 times faster than mypy
-------------------------------------------------------------------------------
pandas-stubs
------------
Benchmark 1: ty
Time (mean ± σ): 101.5 ms ± 2.5 ms [User: 522.3 ms, System: 45.0 ms]
Range (min … max): 95.4 ms … 107.9 ms 28 runs
Benchmark 2: Pyrefly
Time (mean ± σ): 201.8 ms ± 2.2 ms [User: 662.1 ms, System: 108.7 ms]
Range (min … max): 198.0 ms … 205.7 ms 14 runs
Benchmark 3: mypy
Time (mean ± σ): 6.925 s ± 0.054 s [User: 6.640 s, System: 0.270 s]
Range (min … max): 6.842 s … 7.016 s 10 runs
Benchmark 4: Pyright
Time (mean ± σ): 2.202 s ± 0.023 s [User: 24.999 s, System: 1.526 s]
Range (min … max): 2.182 s … 2.248 s 10 runs
Summary
ty ran
1.99 ± 0.05 times faster than Pyrefly
21.69 ± 0.59 times faster than Pyright
68.20 ± 1.79 times faster than mypy
-------------------------------------------------------------------------------
prefect
-------
Benchmark 1: ty
Time (mean ± σ): 124.9 ms ± 6.7 ms [User: 814.5 ms, System: 65.0 ms]
Range (min … max): 117.7 ms … 144.7 ms 21 runs
Warning: Ignoring non-zero exit code.
Benchmark 2: Pyrefly
Time (mean ± σ): 239.9 ms ± 8.9 ms [User: 1376.5 ms, System: 174.1 ms]
Range (min … max): 231.3 ms … 261.3 ms 11 runs
Warning: Ignoring non-zero exit code.
Benchmark 3: mypy
Time (mean ± σ): 688.0 ms ± 2.1 ms [User: 620.1 ms, System: 64.4 ms]
Range (min … max): 684.6 ms … 690.9 ms 10 runs
Benchmark 4: Pyright
Time (mean ± σ): 4.163 s ± 0.047 s [User: 56.799 s, System: 2.423 s]
Range (min … max): 4.055 s … 4.220 s 10 runs
Warning: Ignoring non-zero exit code.
Summary
ty ran
1.92 ± 0.12 times faster than Pyrefly
5.51 ± 0.29 times faster than mypy
33.34 ± 1.82 times faster than Pyright
-------------------------------------------------------------------------------
pytorch
-------
Benchmark 1: ty
Time (mean ± σ): 1.425 s ± 0.070 s [User: 17.265 s, System: 0.996 s]
Range (min … max): 1.328 s … 1.550 s 10 runs
Warning: Ignoring non-zero exit code.
Benchmark 2: Pyrefly
Time (mean ± σ): 1.495 s ± 0.023 s [User: 12.595 s, System: 1.090 s]
Range (min … max): 1.460 s … 1.523 s 10 runs
Warning: Ignoring non-zero exit code.
Benchmark 3: mypy
Time (mean ± σ): 28.003 s ± 0.059 s [User: 27.363 s, System: 0.623 s]
Range (min … max): 27.865 s … 28.083 s 10 runs
Warning: Ignoring non-zero exit code.
Benchmark 4: Pyright
Time (mean ± σ): 16.782 s ± 0.354 s [User: 229.931 s, System: 9.369 s]
Range (min … max): 16.010 s … 17.246 s 10 runs
Warning: Ignoring non-zero exit code.
Summary
ty ran
1.05 ± 0.05 times faster than Pyrefly
11.78 ± 0.63 times faster than Pyright
19.65 ± 0.97 times faster than mypy
---------------------------------------------------------------------------------------- benchmark 'black': 3 tests ----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_incremental_edit[black-ty] 16.3074 (1.0) 17.6650 (1.0) 16.8509 (1.0) 0.4333 (1.0) 16.8048 (1.0) 0.6540 (1.0) 2;0 59.3441 (1.0) 10 1
test_incremental_edit[black-pyrefly] 64.9062 (3.98) 88.6891 (5.02) 80.3565 (4.77) 6.7338 (15.54) 82.4764 (4.91) 6.6577 (10.18) 2;1 12.4445 (0.21) 10 1
test_incremental_edit[black-pyright] 471.0820 (28.89) 483.3831 (27.36) 476.7415 (28.29) 4.5999 (10.62) 475.3801 (28.29) 7.6845 (11.75) 4;0 2.0976 (0.04) 10 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------- benchmark 'discord.py': 3 tests ----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_incremental_edit[discord.py-ty] 20.7345 (1.0) 21.4773 (1.0) 21.0356 (1.0) 0.2497 (1.0) 20.9884 (1.0) 0.2250 (1.0) 4;1 47.5384 (1.0) 10 1
test_incremental_edit[discord.py-pyrefly] 75.3165 (3.63) 84.3949 (3.93) 79.6597 (3.79) 3.0452 (12.20) 79.0860 (3.77) 5.7342 (25.49) 5;0 12.5534 (0.26) 10 1
test_incremental_edit[discord.py-pyright] 494.5313 (23.85) 510.1415 (23.75) 501.1908 (23.83) 5.1140 (20.48) 500.8561 (23.86) 6.0744 (27.00) 4;0 1.9952 (0.04) 10 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------- benchmark 'homeassistant': 3 tests ----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_incremental_edit[homeassistant-ty] 31.1909 (1.0) 32.4904 (1.0) 31.6418 (1.0) 0.3980 (1.0) 31.5861 (1.0) 0.6218 (1.0) 2;0 31.6038 (1.0) 10 1
test_incremental_edit[homeassistant-pyrefly] 62.3380 (2.00) 80.1021 (2.47) 74.3572 (2.35) 6.5629 (16.49) 77.2817 (2.45) 6.0889 (9.79) 2;2 13.4486 (0.43) 10 1
test_incremental_edit[homeassistant-pyright] 549.7410 (17.63) 559.2366 (17.21) 555.8262 (17.57) 2.8730 (7.22) 555.6597 (17.59) 3.3599 (5.40) 3;0 1.7991 (0.06) 10 1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------- benchmark 'isort': 3 tests ----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_incremental_edit[isort-ty] 22.7207 (1.0) 23.8097 (1.0) 23.2448 (1.0) 0.2987 (1.0) 23.2919 (1.0) 0.3664 (1.0) 3;0 43.0203 (1.0) 10 1
test_incremental_edit[isort-pyrefly] 41.9209 (1.85) 47.4118 (1.99) 44.2379 (1.90) 1.8795 (6.29) 44.1403 (1.90) 2.2086 (6.03) 4;0 22.6051 (0.53) 10 1
test_incremental_edit[isort-pyright] 402.2271 (17.70) 424.7920 (17.84) 407.4667 (17.53) 6.4873 (21.72) 405.7113 (17.42) 4.4778 (12.22) 1;1 2.4542 (0.06) 10 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------- benchmark 'jinja': 3 tests ----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_incremental_edit[jinja-ty] 14.2233 (1.0) 22.3435 (1.0) 16.0890 (1.0) 3.1011 (1.02) 14.8336 (1.0) 0.6692 (1.0) 2;2 62.1541 (1.0) 10 1
test_incremental_edit[jinja-pyrefly] 53.5632 (3.77) 64.2786 (2.88) 60.3105 (3.75) 3.0269 (1.0) 60.9064 (4.11) 2.7117 (4.05) 3;1 16.5809 (0.27) 10 1
test_incremental_edit[jinja-pyright] 473.3500 (33.28) 495.9946 (22.20) 486.1107 (30.21) 6.5302 (2.16) 485.8079 (32.75) 7.6950 (11.50) 3;0 2.0571 (0.03) 10 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------- benchmark 'pandas': 3 tests -----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_incremental_edit[pandas-ty] 116.1185 (1.0) 128.8929 (1.0) 123.5417 (1.0) 3.9921 (1.26) 124.1933 (1.0) 4.7554 (1.31) 4;0 8.0944 (1.0) 10 1
test_incremental_edit[pandas-pyrefly] 217.3018 (1.87) 227.4668 (1.76) 223.2796 (1.81) 3.1775 (1.0) 223.1632 (1.80) 3.6275 (1.0) 4;0 4.4787 (0.55) 10 1
test_incremental_edit[pandas-pyright] 614.2513 (5.29) 697.1629 (5.41) 629.0611 (5.09) 24.6550 (7.76) 621.8455 (5.01) 11.1711 (3.08) 1;1 1.5897 (0.20) 10 1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------- benchmark 'prefect': 3 tests ----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_incremental_edit[prefect-ty] 7.2370 (1.0) 7.9504 (1.0) 7.4966 (1.0) 0.2191 (1.0) 7.4831 (1.0) 0.3126 (1.0) 3;0 133.3938 (1.0) 10 1
test_incremental_edit[prefect-pyrefly] 38.4028 (5.31) 58.4893 (7.36) 50.0226 (6.67) 5.4995 (25.10) 51.3131 (6.86) 2.6352 (8.43) 3;3 19.9909 (0.15) 10 1
test_incremental_edit[prefect-pyright] 629.7112 (87.01) 641.8833 (80.74) 634.1208 (84.59) 3.4590 (15.79) 633.7089 (84.69) 4.0681 (13.01) 2;0 1.5770 (0.01) 10 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------- benchmark 'pytorch': 3 tests ----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_incremental_edit[pytorch-ty] 4.7391 (1.0) 5.3335 (1.0) 5.0034 (1.0) 0.1866 (1.0) 4.9951 (1.0) 0.2898 (1.0) 3;0 199.8644 (1.0) 10 1
test_incremental_edit[pytorch-pyrefly] 18.2195 (3.84) 23.2889 (4.37) 20.8633 (4.17) 1.9483 (10.44) 20.0028 (4.00) 3.7130 (12.81) 5;0 47.9310 (0.24) 10 1
test_incremental_edit[pytorch-pyright] 384.9073 (81.22) 398.1545 (74.65) 393.6819 (78.68) 4.7748 (25.59) 395.3731 (79.15) 5.1413 (17.74) 2;0 2.5401 (0.01) 10 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------- benchmark 'black': 3 tests ----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_fetch_diagnostics[black-ty] 53.3337 (1.0) 54.9496 (1.0) 54.2475 (1.0) 0.5337 (1.0) 54.3359 (1.0) 0.6449 (1.0) 4;0 18.4340 (1.0) 10 1
test_fetch_diagnostics[black-pyrefly] 156.6517 (2.94) 158.9951 (2.89) 157.8805 (2.91) 0.7690 (1.44) 157.9544 (2.91) 1.3770 (2.14) 4;0 6.3339 (0.34) 10 1
test_fetch_diagnostics[black-pyright] 308.8407 (5.79) 318.3376 (5.79) 312.8436 (5.77) 3.0956 (5.80) 312.6159 (5.75) 4.8106 (7.46) 4;0 3.1965 (0.17) 10 1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------- benchmark 'discord.py': 3 tests ----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_fetch_diagnostics[discord.py-ty] 102.6882 (1.0) 106.1560 (1.0) 104.4634 (1.0) 1.0352 (1.0) 104.3930 (1.0) 0.6182 (1.0) 3;3 9.5727 (1.0) 10 1
test_fetch_diagnostics[discord.py-pyrefly] 198.0532 (1.93) 214.1243 (2.02) 203.3859 (1.95) 5.9123 (5.71) 200.4584 (1.92) 8.7875 (14.21) 2;0 4.9168 (0.51) 10 1
test_fetch_diagnostics[discord.py-pyright] 622.5617 (6.06) 630.4986 (5.94) 627.1560 (6.00) 2.6559 (2.57) 628.1141 (6.02) 4.4541 (7.20) 3;0 1.5945 (0.17) 10 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------- benchmark 'homeassistant': 3 tests --------------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_fetch_diagnostics[homeassistant-ty] 91.3597 (1.0) 93.5181 (1.0) 92.4661 (1.0) 0.6728 (1.55) 92.4869 (1.0) 0.9634 (1.40) 3;0 10.8148 (1.0) 10 1
test_fetch_diagnostics[homeassistant-pyrefly] 196.0575 (2.15) 197.3119 (2.11) 196.8354 (2.13) 0.4332 (1.0) 196.9403 (2.13) 0.6865 (1.0) 4;0 5.0804 (0.47) 10 1
test_fetch_diagnostics[homeassistant-pyright] 1,230.1069 (13.46) 1,243.7687 (13.30) 1,236.8218 (13.38) 3.9046 (9.01) 1,236.5184 (13.37) 3.2753 (4.77) 3;1 0.8085 (0.07) 10 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------- benchmark 'isort': 3 tests ----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_fetch_diagnostics[isort-ty] 50.0815 (1.0) 50.9842 (1.0) 50.5261 (1.0) 0.2854 (1.0) 50.5538 (1.0) 0.2660 (1.0) 4;0 19.7918 (1.0) 10 1
test_fetch_diagnostics[isort-pyrefly] 98.0248 (1.96) 108.0117 (2.12) 102.6928 (2.03) 4.4972 (15.76) 100.4502 (1.99) 9.0160 (33.89) 5;0 9.7378 (0.49) 10 1
test_fetch_diagnostics[isort-pyright] 386.7665 (7.72) 412.8930 (8.10) 394.7164 (7.81) 7.3152 (25.63) 392.5132 (7.76) 4.6696 (17.55) 2;1 2.5335 (0.13) 10 1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------- benchmark 'jinja': 3 tests ----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_fetch_diagnostics[jinja-ty] 61.2241 (1.0) 62.2987 (1.0) 61.7260 (1.0) 0.4157 (1.0) 61.5479 (1.0) 0.7797 (1.0) 4;0 16.2006 (1.0) 10 1
test_fetch_diagnostics[jinja-pyrefly] 127.6552 (2.09) 130.5587 (2.10) 129.4125 (2.10) 0.8431 (2.03) 129.3920 (2.10) 0.8771 (1.12) 3;1 7.7272 (0.48) 10 1
test_fetch_diagnostics[jinja-pyright] 385.6590 (6.30) 396.4412 (6.36) 392.3081 (6.36) 3.5613 (8.57) 392.7521 (6.38) 5.8229 (7.47) 3;0 2.5490 (0.16) 10 1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------- benchmark 'pandas': 3 tests ---------------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_fetch_diagnostics[pandas-ty] 352.5637 (1.0) 356.3200 (1.0) 354.1020 (1.0) 1.3537 (1.38) 353.5966 (1.0) 1.9687 (1.64) 3;0 2.8240 (1.0) 10 1
test_fetch_diagnostics[pandas-pyrefly] 550.4653 (1.56) 553.6238 (1.55) 552.6125 (1.56) 0.9803 (1.0) 552.6247 (1.56) 1.2040 (1.0) 2;0 1.8096 (0.64) 10 1
test_fetch_diagnostics[pandas-pyright] 1,925.1942 (5.46) 1,971.0623 (5.53) 1,938.8082 (5.48) 13.3665 (13.64) 1,934.1315 (5.47) 14.8707 (12.35) 2;1 0.5158 (0.18) 10 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------- benchmark 'prefect': 3 tests --------------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_fetch_diagnostics[prefect-ty] 149.9016 (1.0) 159.7440 (1.0) 151.8716 (1.0) 2.8373 (1.0) 151.0643 (1.0) 1.0242 (1.0) 1;1 6.5845 (1.0) 10 1
test_fetch_diagnostics[prefect-pyrefly] 431.7601 (2.88) 442.3384 (2.77) 434.3835 (2.86) 3.0814 (1.09) 433.3456 (2.87) 2.5392 (2.48) 1;1 2.3021 (0.35) 10 1
test_fetch_diagnostics[prefect-pyright] 1,085.5778 (7.24) 1,103.1126 (6.91) 1,093.8162 (7.20) 4.9728 (1.75) 1,093.9088 (7.24) 5.1468 (5.03) 3;0 0.9142 (0.14) 10 1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------ benchmark 'pytorch': 3 tests -----------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_fetch_diagnostics[pytorch-ty] 45.9327 (1.0) 47.0847 (1.0) 46.4518 (1.0) 0.3982 (1.0) 46.3758 (1.0) 0.7203 (1.0) 3;0 21.5277 (1.0) 10 1
test_fetch_diagnostics[pytorch-pyrefly] 115.0990 (2.51) 148.5506 (3.15) 131.3375 (2.83) 15.8990 (39.92) 131.2013 (2.83) 29.4464 (40.88) 2;0 7.6140 (0.35) 10 1
test_fetch_diagnostics[pytorch-pyright] 687.8946 (14.98) 705.4540 (14.98) 696.1597 (14.99) 6.0340 (15.15) 696.7016 (15.02) 8.3710 (11.62) 4;0 1.4365 (0.07) 10 1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------