skills/dnanexus-integration/references/operations-and-troubleshooting.md
Start with:
dx --version
dx whoami
dx pwd
Do not run dx env, dx env --bash, or ua --env in captured output; they
display credentials.
Record:
Do not record tokens, whole environments, signed URLs, or unredacted PHI.
AuthErrorPermissionDenieddx logindx whoami
dx pwd
Shell variables override ~/.dnanexus_config/environment.json. If the saved
CLI state should win:
source "$HOME/.dnanexus_config/unsetenv"
dx whoami
dx pwd
If saved state should be discarded:
dx clearenv
Then authenticate again. Do not compare token values.
Also check:
Do not solve a context error by broadening permissions automatically.
ResourceNotFounddx ls "project-xxxx:/folder"
dx find data \
--path "project-xxxx:/folder" \
--name "sample.bam" \
--name-mode exact \
--json
DNAnexus permits duplicate object names. For mutation, select by ID after reviewing all matches.
Use project-qualified links:
dxpy.dxlink("file-xxxx", "project-xxxx")
Possible causes:
For Upload Agent, repeat the same command to resume. Use --wait-on-close for
automation that needs a closed file.
Inspect state by ID. A dependent job can be submitted while an input is open
or closing, but it remains waiting_on_input; do not expect compute to start
until the file closes.
Abandoned files can remain billable until platform cleanup or explicit removal. Confirm the exact incomplete object before deleting it.
.gzThis is expected for uncompressed inputs. Upload Agent compresses by default.
Use --do-not-compress when preserving original bytes or naming is required.
Upload Agent identifies resumable files from a signature that includes path, size, modification time, compression mode, and chunk size. Changing those inputs can create a new object.
Review incomplete/closed matches before using --do-not-resume.
Archived files cannot be downloaded until unarchive completes. Confirm retrieval cost and timing before requesting unarchive.
Current dx-toolkit warns when a file or generated download URL has been flagged as malicious. Treat this as a stop condition:
Use Download Agent with a reviewed manifest. Check free space, run progress,
and use inspect after completion. Rerun the same manifest to repair missing
or checksum-mismatched parts.
waiting_on_inputCheck:
depends_on jobs reached done.runnable for a long timeCheck:
Do not immediately switch to a larger/on-demand instance without estimating cost.
waiting_on_outputCheck:
InputErrorOutputErroroutputSpec.job_output.json contains {} when there are no outputs.AppErrorExpected app-recognized problem. Follow the actionable message and fix input or configuration. Do not retry unchanged.
AppInternalErrorUnexpected application failure or nonzero process exit:
stdout/stderr.AppInsufficientResourceErrorReview detailed metrics and determine whether memory or storage was exhausted.
Options:
Automatic instance upgrade requires the organization policy and a matching restart count. It upgrades within the same family; it is not a substitute for profiling.
ExecutionError / dependency installation failureCheck:
execDepends package manager/name/version is valid.Prefer a pinned asset, virtual environment, or saved container image over runtime installation.
SpotInstanceInterruptionThis can be transient. Confirm the app is restartable/idempotent. Use a bounded retry policy. For critical runs, consider high priority/on-demand after cost approval.
JobTimeoutExceededCheck configured timeoutPolicy and platform 30-day limit. Splitting a
workflow is often safer than simply increasing the timeout.
CostLimitExceededThe configured cost limit terminated the execution. Review:
Do not raise the limit until the cause and new maximum are approved.
SpendingLimitExceeded / OrgExpiredThese are account/organization controls. Report the exact billing entity and stop. Do not attempt to route charges to another account or project without explicit authorization.
AuthError in a long jobJob credentials inherit root authorization and have a limited lifetime. A revoked/expired token can terminate the tree. Re-running requires a new authorized launch; it does not resurrect the old job.
For partially_failed:
Example:
dx find jobs --root-execution "analysis-xxxx" --all-jobs
dx watch "job-failed" --get-streams
For a rerun:
dx run --clone "analysis-xxxx" \
--rerun-stage "stage-name" \
--destination "project-xxxx:/reruns/run-001"
Review reuse and output-folder behavior before launch.
The head job supervises subjobs. Find the failed child:
dx find jobs --root-execution "job-head" --all-jobs
dx watch "job-child" --get-streams
Check:
cpus, memory, and diskCurrent guidance allows at most 20 preserved sessions per project. Confirm the target session exists and no other job is writing it. Clean old work/cache folders only after explicit review.
Check per-domain token configuration without printing
DXCOMPILER_WDL_IMPORT_BEARER_TOKENS. Confirm:
DNAnexus supports approved SSH/debug-hold workflows. These settings can expose live job data and extend workspace lifetime.
Before enabling:
--clone does not preserve SSH/debug flags. This is a safety property; do not
automatically add them back.
Current documented defaults include:
Limits can vary by region/account and change. Bound searches, paginate, use batch APIs, and back off on throttling.
Distinguish three independent controls:
Increasing one limit does not bypass another. Project monthly behavior can also depend on the billing organization's policy.
Before rerun or scale-up, report:
Failed and terminated user jobs can still be billed. Platform-internal failure billing differs by reason and customer contract.
Require exact IDs and confirmation for:
dx rm / object removalNever recursively remove /. Removal can also delete orphaned hidden linked
objects and cannot be undone.
protected and destroyProtected serve different purposes:
protected raises project-data deletion from CONTRIBUTE to ADMINISTER.destroyProtected prevents whole-project destruction until an authorized
administrator explicitly clears it.Project destruction removes all objects and refuses to proceed with active
jobs unless terminateJobs=true. Never clear deletion protection or force
job termination as part of a generic cleanup request.
Provide only:
Exclude: