hack/docs/Adding_Detectors_external.md
Secret Detectors have these two major functions:
The purpose of Secret Detectors is to discover secrets with exceptionally high signal. High rates of false positives are not accepted.
We are interested in detectors for services that meet at least one of these criteria
If you think that something should be included outside of these guidelines, please let us know.
net/http library to make requests instead of bringing in another library.common.SaneHttpClient for the http.Client whenever possible.In some instances, services will update their token format, requiring a new regex to properly detect secrets in addition to supporting the previous token format. Accommodating this can be done without adding a net-new detector. We provide a Versioner interface that can be implemented.
v1 and v2. Move the existing detector and tests into v1, and add new files to v2.
Ex: <packagename>/<old_files> -> <packagename>/v1/<old_files>, <packagename>/v2/<new_files>Note: Be sure to update the tests to reference the new secret values in GSM, or the tests will fail.
Implement the Versioner interface. GitHub example implementation.)
Add a 'version' field in ExtraData for both existing and new detector versions.
Update the existing detector in DefaultDetectors in /pkg/engine/defaults/defaults.go
Proceed from step 3 of Creating a new Secret Scanner
Add a new Secret Detector enum to the DetectorType list here.
Run make protos to update the .pb files.
Generate the Secret Detector
go run hack/generate/generate.go detector <DetectorType enum name>
example: go run hack/generate/generate.go detector SampleAPI
Add the Secret Detector to TruffleHog's Default Detectors
Add the secret scanner to the pkg/engine/defaults/defaults.go file like github.com/trufflesecurity/trufflehog/v3/pkg/detectors/<detector_name> and
<detector_name>.Scanner{}
Complete the Secret Detector.
The previous step templated a boilerplate + some example code as a package in the pkg/detectors folder for you to work on.
The Secret Detector can be completed with these general steps:
SecretParts on every Result your detector emits. See Populating SecretParts./pkg/engine/defaults/defaults.go.To ensure the quality of your PR, make sure your tests are passing with verified credentials.
Create a file called .env with this env file format:
SECRET_TYPE_ONE=value
SECRET_TYPE_ONE_INACTIVE=v@lue
Export the TEST_SECRET_FILE variable, pointing to the env file:
export TEST_SECRET_FILE=".env"
The .env file should be in the new detector's directory like this:
├── tailscale
│ ├── .env
│ ├── tailscale.go
│ └── tailscale_test.go
Now that a .env file is present, the test file can load secrets locally.
Next, update the tests as necessary. A test file has already been generated by the go run hack/generate/generate.go command from earlier. There are 5 cases that have been generated:
Make any necessary updates to the tests. Note there might not be any changes required as the tests generated by the go run hack/generate/generate.go command are pretty good.
Here is an exemplary test file for a detector which covers all 5 test cases.
Now run the tests and check to make sure they are passing ✔️!
go test ./pkg/detectors/<detector> -tags=detectors
If the tests are passing, feel free to open a PR!
SecretParts is the structured source of truth for the credential components a detector found. It is a map[string]string on detectors.Result that stores each part of the credential under a descriptive key. Downstream consumers (analyzers, Secret Storage, and — in future work — the Raw/RawV2 mapping layer and dedup hashing) rely on it.
Every Result your detector emits must populate SecretParts. Populate it whether or not the secret is verified.
Most detectors find a single opaque token. Use one entry keyed by "key" — this is the established convention in the codebase:
s1 := detectors.Result{
DetectorType: detector_typepb.DetectorType_Example,
Raw: []byte(match),
SecretParts: map[string]string{"key": match},
}
When the credential has more than one component (e.g. AWS access-key + secret-access-key, OAuth client-id + client-secret, or a token bound to an endpoint/host), use one entry per part with descriptive keys:
s1 := detectors.Result{
DetectorType: detector_typepb.DetectorType_Example,
Raw: []byte(accessKeyID),
RawV2: []byte(accessKeyID + secretAccessKey),
SecretParts: map[string]string{
"access_key_id": accessKeyID,
"secret_access_key": secretAccessKey,
},
}
"key". This matches the bulk of existing detectors in pkg/detectors/.snake_case keys that name each part (e.g. client_id, client_secret, access_key_id, secret_access_key, username, password, domain, host, endpoint). If the detector also has a corresponding analyzer in pkg/analyzer/analyzers/, the keys must match what the analyzer expects, since analyzers read directly from this map.SecretParts. Unrelated metadata belongs in ExtraData.There are two types of reasons that secret verification can fail:
In TruffleHog parlance, the first type of verification response is called determinate and the second type is called indeterminate. Verification code should distinguish between the two by returning an error object in the result struct only for indeterminate failures. In general, a verifier should return an error (indicating an indeterminate failure) in all cases that haven't been explicitly identified as determinate failure states.
For example, consider a hypothetical authentication endpoint that returns 200 OK for valid credentials and 403 Forbidden for invalid credentials. The verifier for this endpoint could make an HTTP request and use the response status code to decide what to return:
200 response would indicate that verification succeeded. (Or maybe any 2xx response.)403 response would indicate that verification failed determinately and no error object should be returned.dos2unix.
sudo apt install dos2unix
trufflehog local directory and convert scripts/gen_proto.sh file in Unix format.
dos2unix ./scripts/gen_proto.sh
make protos