.agents/skills/cuda-cutlass-fmha-incremental-rebuild/SKILL.md
The general false-green principles (stale binary, wrong-artifact mtime) are summarised in the
ort-testskill's "False-green taxonomy". This skill is the CUDA/CUTLASS-specific detail.
nvcc-generated depfiles do not track the CUTLASS fused-MHA headers under
onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/ (e.g. kernel_forward.h,
fmha_launch_template.h). These headers are #included by the fmha_sm*.cu
translation units, but the build system does not record that dependency.
Consequence: after you edit one of those headers, an incremental build.sh:
fmha_sm*.cu,[100%] Built target ... and exits 0,fmha_sm*.cu.o objects and the
libonnxruntime_providers_cuda.so they link into — unchanged (same mtime as
the pre-edit build).(Do not use the gtest test-exe mtime as the stale symptom: in the shared-provider
build the exe dlopens the .so and is not relinked, so its mtime stays old even
after a correct rebuild — see "How to confirm" below. The reliable diagnostic signal
is the fmha_sm*.cu.o / .so mtime.)
So your "successful" rebuild is running the old kernel. Tests that should now pass (or fail) reflect the previous code, not your edit. This silently invalidates any FAIL→PASS / PASS→FAIL verification.
Before rebuilding after editing any cutlass_fmha/*.h header:
touch onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/*.cu
Then run the normal build command. This forces the fmha_sm*.cu translation units
(and downstream binaries) to recompile against your header change.
Confirm that the artifact which actually links the recompiled fmha_sm*.cu.o
is newer than your header edit.
⚠️ Do NOT just check the test EXE mtime — it can falsely flag a good build as
stale. In the shared-provider build configuration (the default here), the CUDA
execution provider is a shared module: the recompiled fmha_sm*.cu.o link into
libonnxruntime_providers_cuda.so, and the onnxruntime_provider_test executable
dlopens that .so — it is not relinked. So after a correct rebuild the
test exe mtime stays old while the .so advances. Checking the exe alone
would wrongly conclude the build was stale.
Check the right artifact for your link mode:
.so that links the recompiled .o —
build/<dir>/<cfg>/libonnxruntime_providers_cuda.soonnxruntime_provider_test)Safest check — stat both the recompiled object and the .so, and confirm BOTH
are newer than the header edit:
stat -c '%y %n' onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/kernel_forward.h
# in your build dir, e.g. build/Debug_quickbuild/Debug/:
stat -c '%y %n' libonnxruntime_providers_cuda.so
# and the actual recompiled object (path varies by build dir):
find . -name 'fmha_sm80.cu.o' -exec stat -c '%y %n' {} +
If the .so (and the fmha_sm*.cu.o) timestamps are older than (or equal to) the
header edit, the build was stale — touch the .cu files and rebuild. The most
reliable signal of all is behavioral: a test that was failing now passes (a stale
binary cannot flip its result).
This is the CUDA/CUTLASS instance of false-green mode 1 (zero-match / wrong binary) —
see the ort-test skill's "False-green taxonomy" for the general principle. In short:
attention/MEA/Flash boundary gtests (e.g. FlashStructuralEmptyRows*,
Attention_Causal_NonPadKVSeqLen_MEA_*) live in onnxruntime_provider_test, which CI
runs; onnxruntime_test_all does not contain them and gives a false green. Verify the
MEA/Flash boundary fix against onnxruntime_provider_test.
Full ORT CUDA builds are large (test binaries ~1 GB each; a build dir can reach
tens of GB). On a shared box, /home filling to 100% makes builds fail in
non-obvious places — e.g. git submodule sync reporting No space left on device
or a config.lock error, not an obvious "disk full" at the compile step.
Before a big rebuild, check free space and clean only clearly-stale, regenerable build directories (old dated experiment dirs). Never delete another agent's active build dir or anything ambiguous:
df -h /home
du -sh build/* | sort -h