Custom Rules for Kingfisher¶

Kingfisher's candidate detector catalog is sourced from the Betterleaks catalog and selected Veles detectors. New generally useful detectors and validation improvements should still be contributed to Betterleaks first. When the Betterleaks catalog covers a Veles detector, the Betterleaks detector is preferred and the overlapping Veles detector is not bundled.

The current built-in catalog contains 488 rules, including selected Veles rules and hidden component/helper rules. See the built-in rules listing for the per-rule catalog and capability counts.

The Betterleaks source is pinned to v2.0.0-rc.1. It includes Bitbucket Data Center HTTP access tokens (betterleaks.bitbucket-data-center-token), Cloudflare cfut_ user tokens and cfat_ account tokens (betterleaks.cloudflare-api-key.2), and Tableau personal access tokens beginning with + or /. Cloudflare account-token validation requires a nearby account ID; the hidden betterleaks.cloudflare-account-id.1 component is optional for detection and searched within 5L (the match line and up to four lines before or after). Without it, a cfat_ token remains unvalidated. The former betterleaks.cloudflare-api-key rule is now betterleaks.cloudflare-api-key.1; scan --rule cloudflare-api-key continues to select both formats. Direct validation requires an unambiguous selector: use validate --rule cloudflare-api-key.2 for the new token formats.

Kingfisher fully supports loading both the Kingfisher rule format (.yml/.yaml) and Betterleaks TOML (.toml) for custom rules. Either format can be used for shared or private detections.

The Betterleaks TOML format is supported for custom rules as well as for Kingfisher's built-in catalog. Use it for generally useful detectors that should be developed and shared through Betterleaks. Custom Betterleaks TOML rules are automatically placed in the custom. namespace.

The YAML schema documented below is the Kingfisher rule format. It supports detection, validation, revocation, components, and checksum requirements for custom rules. The built-in catalog is maintained separately: the maintainer tool converts upstream rules to an internal YAML representation inside the compressed bundle and readable betterleaks.yml and veles.yml files. These generated files preserve the same rules and collection metadata; they are not maintained separately. Each file cites its sources, and the provenance manifest records its hash and per-rule origins, including Kingfisher-authored helpers. See the crate documentation for loading, licensing, and regeneration details.

A rule in the Kingfisher rule format is a YAML document that describes how to detect and optionally validate or revoke secrets. With custom rules you can:

• Extend Kingfisher without touching Rust code
• Tune sensitivity via entropy and confidence
• Plug in live checks against external services

Load .toml, .yml, or .yaml custom rules with --rules-path. They are additive to the Betterleaks and Veles built-ins; pass --load-builtins=false for a custom-only scan. A rules directory can contain both formats, or you can repeat --rules-path to load separate files:

kingfisher scan --rules-path ./company.yml --rules-path ./team.toml ./src/
kingfisher rules check --rules-path ./custom-rules/ --load-builtins=false --no-update-check

Kingfisher's imported-rules-capabilities.yml is not another rule format. Its source-specific sections may bind existing imported detectors to operational validation, access-map, revocation, confidence, authority, narrow filter behavior, or derive bare token patterns with bare: true, but must not contain detector regexes. See the overlay reference.

Veles support is built-in-only. crates/kingfisher-rules/data/veles-rules.yml pins an OSV-SCALIBR commit and selects upstream Veles plugin IDs that have explicit import adapters; --rules-path does not accept Veles source or configuration.

1. Rule Schema¶

Each Kingfisher YAML rule file defines one or more entries under a top‑level rules: list. Every entry supports the following fields:

rules:
  - name:           # (string) Human-friendly rule name
    id:             # (string) Unique identifier (e.g. custom.aws.secret-key)

    pattern: |      # (multi-line regex) Detection pattern
      (?xi)
      aws
      (?:.|[\n\r]){0,32}?
      \b([[:alnum:]/+=]{40})\b

    min_entropy: 3.5                # (float) Minimum Shannon entropy
    confidence:  medium             # (enum: low | medium | high)

    examples:                       # (list) strings that must match
      - AWS_SECRET="AKIA…"

    references:                     # (optional list) context URLs
      - https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html

    visible: true                   # (bool) hide helper matches when false

    depends_on_rule:                # (optional) capture chaining
      - rule_id: custom.aws.access-key-id
        variable: AKID              # referenced as {{ AKID }}

    pattern_requirements:         # (optional) character/word requirements
      min_digits: 1                 # require at least 1 digit
      min_uppercase: 1              # require at least 1 uppercase letter
      min_lowercase: 1              # require at least 1 lowercase letter
      min_special_chars: 1          # require at least 1 special character
      special_chars: "!@#$%^&*()"   # optional: custom special character set
      ignore_if_contains:                # optional: drop matches containing these words
        - test

    validation:                     # (optional) live validation
      type: Http
      content:
        request:
          method: GET
          url: https://api.example.com/v1/check
          headers:
            X-Secret: "{{ TOKEN }}"
            X-Id:     "{{ AKID }}"
          response_is_html: true # by default, validation responses containing HTML or considered invalid. Set to `true` if you expect HTML returned from a validation response
          response_matcher:
            - report_response: true   # always include raw payload
            - type: StatusMatch
              status: [200]           # positive check
            - type: StatusMatch
              status: [401,403]
              negative: true          # negative check → must NOT match
            - type: HeaderMatch
              header: content-type
              expected: ["application/json"]
            - type: JsonValid

    # NOTE: Some providers are gRPC-only (no REST endpoint). For those, use Grpc validation.
    validation:
      type: Grpc
      content:
        request:
          url: https://api.example.com/<package>.<Service>/<Method>
          headers:
            content-type: application/grpc
            te: trailers
            Authorization: "Bearer {{ TOKEN }}"
          # Raw bytes are allowed (YAML \u0000 escapes become NUL bytes).
          body: "\u0000\u0000\u0000\u0000\u0000"
          response_matcher:
            - report_response: true
            - type: HeaderMatch
              header: grpc-status
              expected: ["0"]

    revocation:                     # (optional) revoke a secret
      type: Http
      content:
        request:
          method: POST
          url: https://api.example.com/v1/revoke
          headers:
            Authorization: "Bearer {{ TOKEN }}"
          response_matcher:
            - report_response: true
            - type: StatusMatch
              status: [200, 202]

AWS access key revocation can use:

revocation:
  type: AWS

GCP service account key revocation can use:

revocation:
  type: GCP

Multi-Step Revocation¶

Some services require a 2-step revocation process: 1. Lookup Step: Make a request to retrieve an ID or token 2. Delete Step: Use that ID to perform the actual revocation

For these cases, use HttpMultiStep:

revocation:
  type: HttpMultiStep
  content:
    steps:
      - name: lookup_token_id                    # Step 1: Get the token ID
        request:
          method: GET
          url: https://api.example.com/v1/tokens/current
          headers:
            Authorization: "Bearer {{ TOKEN }}"
          response_matcher:
            - type: StatusMatch
              status: [200]
        extract:                                  # Extract values from response
          TOKEN_ID:                               # Variable name (uppercase)
            type: JsonPath                        # Extraction method
            path: "$.data.id"                     # JSONPath to the value

      - name: revoke_token                        # Step 2: Delete using the ID
        request:
          method: DELETE
          url: https://api.example.com/v1/tokens/{{ TOKEN_ID }}
          headers:
            Authorization: "Bearer {{ TOKEN }}"
          response_matcher:
            - report_response: true
            - type: StatusMatch
              status: [204]

Validation Types¶

Kingfisher supports these validation types:

1. Assumed: a rule-level marker for secret material accepted with high confidence without live validation. It produces the Assumed Valid (Not Live-Validated) finding status and is included by the actionable filter, but not by the strict active filter. In colorized pretty output it uses the same bright color as an active credential but with a locked icon. It counts as skipped validation by default and as successful validation with the actionable filter.
2. Http and Grpc: YAML-native validation flows. Prefer these first.
3. Typed validators: schema-level validation families already modeled in the rule schema, such as AWS, AzureStorage, Coinbase, CredentialUri, Ethereum, GCP, MongoDB, MySQL, Postgres, Jdbc, and JWT. CredentialUri dispatches a named URI capture (falling back to TOKEN) to HTTPS Basic Auth or a supported database validator while leaving other unsupported schemes unvalidated. Ethereum is deterministic and network-free; it validates key material and derives an address without claiming the address is active.
4. Raw validators: provider-specific or protocol-specific exception paths dispatched through validation: type: Raw.

Overlapping rules and validation deduplication¶

A token can match multiple rules. Kingfisher keeps those findings separate and does not choose a winning provider. Each matching rule with a validator is validated independently; for example, a token matching both OpenAI and DeepSeek rules can produce one request and result for each provider. Repeated occurrences of the same rule and validation input are grouped into one request, while dependency values are included when relevant. Rules without validation, Assumed rules, and non-authoritative rules make no live validation request. Report deduplication also includes the rule ID, so different rules remain separate. --no-dedup controls reported occurrences and does not disable this internal validation grouping.

Rules without a validation block produce Not Attempted findings. They are not treated as active, inactive, or skipped because no validation result was produced.

An assumed validation is configured as:

validation:
  type: Assumed

A Kingfisher custom private-key rule can use this marker when its high-signal format can be accepted without a provider request; its findings are reported as Assumed Valid (Not Live-Validated).

Raw validation looks like this:

validation:
  type: Raw
  content: kraken

Use Raw only when the provider check cannot be expressed reliably with Http or Grpc and does not justify a new reusable validator family. Raw validator implementations live in crates/kingfisher-scanner/src/validation/raw.rs.

Typed validators are safer and more reusable because the validator kind is part of the schema. Raw validators are string-dispatched; an unknown content name is reported as an unimplemented, unverified validation result. If you need a Rust-backed exception path for one provider, prefer Raw; reserve new typed validators for stable validation families that can be reused across rules.

The Ethereum family uses content: private_key, public_key, or mnemonic. Mnemonics use the standard m/44'/60'/0'/0/0 path and explicitly assume an empty BIP-39 passphrase, so the reported address is a candidate. Validation responses contain only allowlisted derivation metadata and never the private key or seed phrase.

gRPC Validation (Grpc)¶

Some services (notably CLI/SDK control planes) are gRPC-only. For these, validation: type: Http is not sufficient because gRPC status is typically returned via HTTP/2 trailers (grpc-status, grpc-message). Kingfisher’s Grpc validator performs an HTTP/2 request and evaluates matchers against the merged headers+trailers.

Grpc is currently intended for unary requests and expects you to provide a fully-qualified method URL:

validation:
  type: Grpc
  content:
    request:
      url: https://api.modal.com/modal.client.ModalClient/ClientHello
      headers:
        content-type: application/grpc
        te: trailers
        x-modal-token-id: "{{ TOKEN_ID }}"
        x-modal-token-secret: "{{ TOKEN }}"
        x-modal-client-type: "1"
        x-modal-client-version: "1.0.0"
      body: "\u0000\u0000\u0000\u0000\u0000"  # Empty protobuf frame
      response_matcher:
        - report_response: true
        - type: HeaderMatch
          header: grpc-status
          expected: ["0"]

responser_matcher variants. Multiple can be used

2. Multi-Step Revocation¶

Some APIs require a two-step revocation process:

1. Step 1 (Lookup): Query the API to retrieve an internal ID, token identifier, or other metadata
2. Step 2 (Delete): Use the extracted value(s) to perform the actual revocation/deletion

Kingfisher supports up to 2 sequential steps in a revocation workflow. Each step can extract values from its response, making them available as variables in subsequent steps.

Response Extractors¶

Values can be extracted from HTTP responses using the following methods:

Multi-Step Revocation Schema¶

revocation:
  type: HttpMultiStep
  content:
    steps:
      - name: <step_name>              # Optional: human-readable step name
        request:                       # Standard HTTP request configuration
          method: GET|POST|DELETE|...
          url: https://api.example.com/...
          headers:
            Header-Name: "value"
          body: "optional request body"
          response_matcher:            # Required for final step only
            - type: StatusMatch
              status: [200]
        extract:                       # Optional: extract variables from response
          VARIABLE_NAME:               # Variable name (uppercase recommended)
            type: JsonPath|Regex|Header|Body|StatusCode
            path: "$.path.to.value"    # For JsonPath
            pattern: "regex pattern"   # For Regex (use first capture group)
            name: "header-name"        # For Header

      - name: <next_step>              # Subsequent steps can use extracted variables
        request:
          method: DELETE
          url: https://api.example.com/tokens/{{ VARIABLE_NAME }}
          response_matcher:
            - type: StatusMatch
              status: [204]

Multi-Step Revocation Requirements¶

• Minimum 1, Maximum 2 steps: You must define at least 1 step and no more than 2 steps
• Final step requires response_matcher: The last step must include a response_matcher to determine success/failure
• Intermediate steps are optional: Earlier steps don't require response matchers but can have them for validation
• Variables flow forward: Variables extracted in step 1 are available in step 2 via Liquid templates (e.g., {{ TOKEN_ID }})
• All standard Liquid filters apply: You can use filters on extracted variables just like with {{ TOKEN }}

Example 1: Basic Two-Step Revocation¶

This example shows a service that requires looking up a token's ID before deletion:

rules:
  - name: Example Service Token
    id: custom.example.token
    pattern: |
      (?xi)
      example_token_
      [A-Za-z0-9]{32}
    min_entropy: 3.5
    examples:
      - example_token_abc123def456ghi789jkl012mno345
    validation:
      type: Http
      content:
        request:
          method: GET
          url: https://api.example.com/v1/auth/verify
          headers:
            Authorization: "Bearer {{ TOKEN }}"
          response_matcher:
            - type: StatusMatch
              status: [200]
    revocation:
      type: HttpMultiStep
      content:
        steps:
          # Step 1: Look up the token ID
          - name: lookup_token_id
            request:
              method: GET
              url: https://api.example.com/v1/tokens/current
              headers:
                Authorization: "Bearer {{ TOKEN }}"
              response_matcher:
                - type: StatusMatch
                  status: [200]
            extract:
              TOKEN_ID:
                type: JsonPath
                path: "$.data.token_id"

          # Step 2: Delete the token using the ID
          - name: delete_token
            request:
              method: DELETE
              url: https://api.example.com/v1/tokens/{{ TOKEN_ID }}
              headers:
                Authorization: "Bearer {{ TOKEN }}"
              response_matcher:
                - report_response: true
                - type: StatusMatch
                  status: [204]

Example 2: Using Multiple Extraction Methods¶

This example demonstrates extracting values using different methods:

revocation:
  type: HttpMultiStep
  content:
    steps:
      # Step 1: Get metadata from multiple sources
      - name: get_token_metadata
        request:
          method: GET
          url: https://api.service.com/tokens/info
          headers:
            Authorization: "Bearer {{ TOKEN }}"
          response_matcher:
            - type: StatusMatch
              status: [200]
        extract:
          # Extract from JSON body
          TOKEN_ID:
            type: JsonPath
            path: "$.id"

          # Extract from response header
          ACCOUNT_ID:
            type: Header
            name: X-Account-ID

          # Extract using regex
          TOKEN_TYPE:
            type: Regex
            pattern: '"type":\s*"([^"]+)"'

      # Step 2: Use all extracted values
      - name: revoke_token
        request:
          method: POST
          url: https://api.service.com/accounts/{{ ACCOUNT_ID }}/tokens/{{ TOKEN_ID }}/revoke
          headers:
            Authorization: "Bearer {{ TOKEN }}"
            Content-Type: application/json
          body: '{"token_type":"{{ TOKEN_TYPE }}"}'
          response_matcher:
            - type: StatusMatch
              status: [200, 204]

Example 3: Complex JSONPath Extraction¶

JSONPath supports nested objects and array indexing:

extract:
  # Extract from nested object
  USER_ID:
    type: JsonPath
    path: "$.data.user.id"

  # Extract from array (first element)
  FIRST_TOKEN_ID:
    type: JsonPath
    path: "$.tokens[0].id"

  # Extract from nested array
  SESSION_ID:
    type: JsonPath
    path: "$.data.sessions[0].session_id"

Example 4: Single-Step Migration Path¶

Existing single-step revocations remain unchanged and continue to work:

# This continues to work as before
revocation:
  type: Http
  content:
    request:
      method: DELETE
      url: https://api.service.com/tokens/revoke
      headers:
        Authorization: "Bearer {{ TOKEN }}"
      response_matcher:
        - type: StatusMatch
          status: [204]

When to Use Multi-Step Revocation¶

Use multi-step revocation when:

• The API requires looking up an ID first: Some services don't accept the token directly for revocation
• You need metadata from the token: The revocation endpoint requires additional information only available via a separate API call
• The service uses indirect revocation: The token must be associated with another resource (session, key, credential) that needs to be identified first

Do NOT use multi-step revocation when:

• The API accepts the token directly: Use the simpler single-step Http revocation
• You need more than 2 steps: Kingfisher supports a maximum of 2 steps
• The service provides a native revocation method: Use AWS or GCP types when applicable

3. Templating with Liquid¶

Kingfisher leverages the Liquid template engine for dynamic parts of HTTP request bodies, headers, query parameters, and multipart payloads. The engine supports both built-in and custom filters to manipulate the captured secret (TOKEN) or other named captures ({{ NAME }}).

Using Liquid Filters in Validation and Revocation¶

• Capture Injection: The unnamed capture from your regex becomes {{ TOKEN }}. Named captures are made available as uppercase variables (e.g. {{ RDMVAL }}).
• Filter Pipeline: You can chain filters using the pipe (|) syntax:

{{ TOKEN | b64enc | url_encode }}

{{ TOKEN | hmac_sha256: "my-secret-key" }}

Built-in & Custom Liquid Filters¶

Below is the complete list of Liquid filters available in Kingfisher, along with their usage patterns and examples.

Chaining & Composition: Filters can be stacked; e.g.:

Authorization: Basic {{ "api:" | append: TOKEN | b64enc }}

Runtime Values: Filters like unix_timestamp and uuid are evaluated at runtime, enabling nonces, timestamps, and unique IDs in your requests.

Stable Request Values: HTTP and gRPC validation requests also expose stable per-request template variables. Use these when the same generated value must appear in multiple places within one request. Currently: - REQUEST_RFC1123_DATE - REQUEST_UNIX_MILLIS

How depends_on_rule Works¶

• Dependency Declaration:
  In your YAML rule definition, you add a depends_on_rule section. Here you specify:
• rule_id: The identifier of the rule whose output is required.

• variable: The name (typically in uppercase) that will be used to reference the captured value from the dependency rule.

• Chaining Captures:
  After matching a blob, Kingfisher resolves each dependency against matches in that same blob. One distinct captured value (for example, an access key ID) is made available to the dependent rule. Repeated occurrences of the same value count as one candidate.

• Using the Captured Value:
  This captured value can then be used during the validation phase. For instance, if you have a rule for an Algolia Admin API Key that depends on an Algolia Application ID (captured as APPID), the validation logic can incorporate the APPID value to confirm that the secret matches the expected pattern or format for that specific account.

• Detection vs validation:
  depends_on_rule supplies capture chaining and validation context. A required dependency with within also gates detection, as described below. Dependencies do not require parser verification.

Dependency windows and ambiguous pairs¶

depends_on_rule:
  - rule_id: custom.service.url
    variable: BASEURL
    within: "2L"
    optional: false

within limits candidates using full regex-match spans in the same blob. Line amounts include the primary line: 2L allows one line before and one after a single-line primary match; it is not a radius of two lines.

C measures byte offsets, not Unicode characters. Directions also apply to C. Comma-separated limits such as 2L,20C combine line and column constraints. 0 or an empty window allows the whole blob.

A required dependency with within and no candidate removes the primary finding. Without within, a missing required dependency retains the finding and skips validation with status 428. optional: true permits an absent dependency.

By default, multiple distinct values in the eligible window retain the finding but skip validation with status 428 and an ambiguous dependency explanation. Kingfisher does not send a credential to a guessed endpoint. Narrow the window or use direct validation with explicit variables to resolve the association. Comment and README matches can participate; dependency matching does not interpret imports or exclude comments. A lone comment candidate is therefore still eligible.

Opt-in candidate verification¶

A dependency that supplies a credential component can opt into bounded try-and-verify pairing:

depends_on_rule:
  - rule_id: custom.service.secret
    variable: CLIENT_SECRET
    within: "5L"
    verify_candidates: true

Built-in opt-ins cover AWS access keys and session tokens, BrowserStack, ClickHouse Cloud, MongoDB Atlas service accounts, PlanetScale, Razorpay, and Wiz. AWS session tokens may try both access-key IDs and secrets within the same total combination budget.

Private YAML rules can enable it for typed AWS validation or Http validation with a literal, fixed HTTP(S) URL, no Host header override, and no multipart uploads. Supported Betterleaks expressions may call aws.validate, or http.get and http.post with a provably fixed HTTP(S) origin and literal header names without a Host override. URLs may concatenate dynamic values after a literal prefix containing the complete origin and a slash; dynamic hosts and indirect URL variables remain unsupported. Pure helper calls are limited to validate.unknown, bytes, base64.encode, and strings.urlQueryEscape. HTTP validation disables redirects for any rule with a verify_candidates dependency, including findings with an unambiguous component. This keeps redirect policy consistent when ordinary validation and candidate searches share cached results. Discovered endpoint dependencies and unsupported validators retain strict ambiguity handling, even if opted in. Every ambiguous dependency must be eligible before a search starts. Variables or helper-rule IDs naming a URL, URI, host, domain, endpoint, address, port, or server are excluded from candidate verification even when opted in. Typed AWS validation additionally accepts only AKID and AWS_SECRET_ACCESS_KEY, the dependency variables that it consumes.

Candidates remain restricted to the dependency window and are deduplicated by value. Matching assignment-name prefixes rank first (for example, kms_aws_key / kms_aws_secret or production_client_id / production_client_secret), then shared bracketed-object or INI-section context, then byte distance. These are ordering hints, not proof of association. Failed name matches fall back to other candidates. Multiple ambiguous components are combined in ranked order without materializing their full Cartesian product.

A search tries at most 16 distinct combinations, with a total time budget of 30 seconds or --validation-timeout, whichever is shorter (10 seconds by default). At most four searches run concurrently across repositories, one combination at a time per search. Existing request retries and provider rate limits still apply; retries can make more than one request per combination. The time budget includes waiting for a search slot. Each complete credential combination uses the normal validation cache. A successful secret remains available for other access-key IDs.

Only authoritative successful validation selects a pair and clears its ambiguity. Exhausted budgets and searches with no successful combination remain unresolved (status 428); timeouts, throttling, and unavailable verification remain inconclusive. They do not establish that the primary credential is inactive. --no-validate leaves multi-candidate findings ambiguous, even when assignment names match. Unselected candidate values are internal and never included in reports or generated validation commands; the selected values obey --redact.

Association happens before deduplication. Identical secrets with different resolved contexts remain separate findings, so a bare definition cannot hide a paired occurrence. Context is not inferred across files. JSON/JSONL findings expose dependent_captures (including BASEURL when captured) and ambiguous_dependencies (variable-to-candidate-count). Resolved values are omitted with --redact because components can themselves contain secrets.

For self-hosted validation, see provider endpoint overrides. These global overrides apply to scanning as well as direct validation. CredentialUri HTTP validation honors --tls-mode off; scan-time lax requires the rule's tls_mode: lax opt-in, just like HTTP rules. Direct validate applies lax globally.

Use depends_on_rule to require one rule before another runs:¶

depends_on_rule:
  - rule_id: custom.algolia.app-id   # helper rule ID
    variable: APPID                     # captured as {{ APPID }}

• Capture flow: First rule captures APPID → second rule injects {{ APPID }} into validation HTTP request or pattern
• Visible control: set visible: false on the supporting rule so it doesn’t clutter your report for non-secret matches
• Primary secret rule: leave the secret rule visible unless it is also only a helper; helper rules should usually be the ones marked visible: false

Algolia Example¶

Consider this example rule for an Algolia Application ID and Admin Key combination. To validate that this is an active credential, both must be matched:

rules:
  - name: Algolia Admin API Key
    id: custom.algolia.admin-key
    pattern: |
      (?xi)
      algolia
      (?:.|[\n\r]){0,32}?
      \b
      (
        [a-z0-9]{32}
      )
      \b
    min_entropy: 3.5
    confidence: medium
    examples:
      - algolia_api_key = "ij1mut5oe606wlrf5z4u8u31264z3gag"
    validation:
      type: Http
      content:
        request:
          headers:
            X-Algolia-API-Key: '{{ TOKEN }}'
            X-Algolia-Application-Id: '{{ APPID }}'
          method: GET
          response_matcher:
            - report_response: true
            - status:
                - 200
              type: StatusMatch
          url: https://{{ APPID }}-dsn.algolia.net/1/keys
    depends_on_rule:
      - rule_id: "custom.algolia.app-id"
        variable: APPID

  - name: Algolia Application ID
    id: custom.algolia.app-id
    pattern: |
      (?xi)
      algolia
      (?:.|[\n\r]){0,16}?
      \b
      (
        [A-Z0-9]{10}
      )
      \b               
    min_entropy: 3.5
    visible: false
    confidence: medium
    examples:
      - algolia_app_id = "WRB8YLFW7Y"

How It Works:¶

• Algolia Application ID Rule (custom.algolia.app-id):

This rule scans for an Algolia Application ID—a 10-character alphanumeric string. It is marked with visible: false so that even if it matches, the finding is not directly reported. Its primary role is to provide a supporting value for other rules rather than to be flagged as a secret by itself.

• Algolia Admin API Key Rule (custom.algolia.admin-key): This rule detects the Algolia Admin API Key using a regex pattern. It includes a depends_on_rule property that specifies a dependency on the Algolia Application ID rule.

• The dependency declares that the rule requires the output of the Algolia Application ID rule, and the captured value is assigned to the variable APPID.

• In the validation section, this captured APPID is used dynamically in the HTTP request (for example, in the header X-Algolia-Application-Id and in the URL).

The dependency mechanism (depends_on_rule) ensures that:

• Non-secret data (like an application ID) is captured without cluttering the scan report (thanks to visible: false).
• The secret (the API key) is validated in context, with the necessary supporting information automatically injected.
• Rules remain modular and extensible; you can update the dependent rule or its pattern independently, and the change will automatically be reflected where the value is used.

The visible: false Property¶

The visible: false property tells Kingfisher to hide the finding from the final scan report. This is particularly useful for rules that capture data not meant to be reported as a secret, but rather to serve as supporting context for another rule.

For example, a rule might match a username, an email address, an AWS Access Key ID, or an Application ID. While these pieces of information are captured during scanning, they are not secrets on their own. Instead, they are used by other rules—via the depends_on_rule mechanism—to validate an associated secret. By marking such rules as visible: false, you prevent these non-secret findings from cluttering your report, yet their values remain available for dependent rules.

visible: false helps keep the scan output focused on actual secrets while still capturing important contextual data needed for comprehensive validation.

Regex Authoring¶

Kingfisher uses Hyperscan/Vectorscan-compatible regular expressions. Write patterns for the credential format documented by the provider, then add only enough surrounding context to avoid generic matches.

• Start multi-line patterns with (?x) for free-spacing mode. Use (?xi) only when the entire expression is intentionally case-insensitive; otherwise scope it to contextual text, for example (?i:api[_-]?key), and keep the token format case-sensitive.
• Prefer POSIX character classes for token alphabets: use [[:alnum:]] for letters and digits, and [[:xdigit:]] for hexadecimal values. Add only documented extra characters, such as [[:alnum:]_=-]; do not use an alphanumeric class when the provider specifies a narrower format.
• With (?i) or (?xi), character ranges are already case-insensitive. Do not write redundant ranges such as [a-zA-Z] or [a-fA-F]; use a semantic POSIX class where applicable, or a single range such as [A-Z] when the token format genuinely requires letters only.
• Put the secret in one unnamed capture, which becomes {{ TOKEN }}. Use (?:...) for structural groups, and add named captures only when validation or a checksum needs their values.
• Hyperscan/Vectorscan does not support lookaround. Use bounded context, min_entropy, and pattern_requirements to reduce false positives instead of lookahead or lookbehind.
• Bound flexible separators and cross-line context (for example, (?:.|[\n\r]){0,32}?) so a rule cannot consume an unbounded region of a file. Prefer a provider prefix, field name, or assignment delimiter over a broad keyword alone.
• Include positive examples for every rule. When a nearby format is easy to confuse with a real credential, test it against a fixture or a targeted scan. Check an individual file with kingfisher rules check before running the rule-crate tests.

POSIX Character Classes¶

Hyperscan/Vectorscan supports POSIX character classes inside bracket expressions, written as [[:name:]]. These are the classes relevant to rule authoring:

Notes:

• [:alnum:] is the workhorse for token alphabets; it covers both letter cases and digits.
• [:lower:] and [:upper:] match a single case by default. Under (?i) / (?xi) they become case-insensitive, so [:lower:] will also match uppercase letters and vice versa.
• [:digit:] matches only 0-9; it does not include letters. For letters and digits use [:alnum:]; for lowercase letters and digits combine them as [[:lower:][:digit:]].
• [:xdigit:] matches 0-9, a-f, and A-F regardless of the (?i) flag.
• Classes can be combined inside one bracket expression, for example [[:alnum:]_=-] adds underscore, equals, and hyphen to the alphanumeric set.

Checksum Intelligence¶

Modern API tokens increasingly include built-in checksums, short internal digests that make each credential self-verifiable. For background, see GitHub's write-up on newer token formats and why checksums reduce false positives.

The Kingfisher rule format supports checksum-aware matching, enabling offline structural verification of credentials without calling third-party APIs.

By validating each token's internal checksum, when supported by the token format, Kingfisher filters structurally invalid or fake tokens before validation runs.

Why this matters

• Offline verification: no API call is required.
• Industry-aligned: compatible with prefix and checksum token designs such as modern PATs.
• Lower false positives: invalid tokens are filtered out by structure alone.

The checksum block below documents the Liquid templating and capture requirements for implementing this behavior in a custom rule.

Character Requirements¶

The pattern_requirements field allows you to specify data type requirements for matched secrets. This is particularly useful when:

• Your regex pattern must be permissive (due to Vectorscan limitations)
• You want to enforce password complexity requirements
• You need to filter out low-quality matches that lack certain character types

Kingfisher's regex engine (Hyperscan) does not support lookahead assertions like (?=.*\d) to require specific character types. Instead, use the pattern_requirements field to filter matches post-detection.

Available Requirements¶

pattern_requirements:
  min_digits: 1              # Require at least 1 digit (0-9)
  min_uppercase: 1           # Require at least 1 uppercase letter (A-Z)
  min_lowercase: 1           # Require at least 1 lowercase letter (a-z)
  min_special_chars: 1       # Require at least 1 special character
  special_chars: "!@#$%^&*"  # Optional: define which characters are "special"
  ignore_if_contains:             # Optional: reject matches containing any of these (case-insensitive)
    - test
    - demo
  checksum:                      # Optional: compare rendered values to drop mismatched formats
    actual:
      template: "{{ MATCH | suffix: 6 }}"   # Liquid template for the observed checksum
      requires_capture: checksum            # (optional) skip unless this capture is present
    expected: "{{ BODY | crc32 | base62: 6 }}"  # Liquid template to render the expected checksum
    skip_if_missing: true                   # (optional) treat missing captures as legacy tokens

All fields are optional. If special_chars is not specified, the default set includes: !@#$%^&*()_+-=[]{}|;:'",.<>?/\~`

ignore_if_contains performs a case-insensitive substring check. If any entry (after trimming whitespace) appears within the match, the match is discarded. This is helpful for dropping known dummy tokens such as "test" or "demo" that otherwise satisfy the regex.

The optional checksum block renders Liquid templates against the match to determine whether the captured checksum matches your expectation. Both templates gain access to {{ MATCH }}, {{ FULL_MATCH }}, and every named capture in two forms: the original capture name and its uppercase alias (e.g. {{ body }} and {{ BODY }}). Use helper filters like suffix, crc32, and base62 to mirror provider-specific checksum pipelines. If a required capture is missing or the rendered values differ, Kingfisher skips the finding—logging the reason, including checksum lengths, at the DEBUG level. Set skip_if_missing to true to treat absent captures as legacy matches.

When any of these filters remove a match it is logged at the DEBUG level so you can see exactly why the skip occurred. If you need to keep every match even when one of these substrings appears, pass --no-ignore-if-contains to kingfisher scan. The flag disables this post-processing step without changing the rule definitions.

Are requires_capture and skip_if_missing equivalent?¶

requires_capture - Optional field that names a specific regex capture that must be present before the checksum templates are evaluated. - In the engine, Kingfisher checks whether that capture exists in the match context. If it’s missing, the behavior falls back to whatever skip_if_missing dictates (fail or treat as a legacy match).

skip_if_missing - Boolean switch that controls what happens when Kingfisher can’t render the checksum—because there’s no match context or a required capture is absent. - true: silently skip (pass) the match so legacy, non-checksum tokens are still accepted. - false: treat the situation as a validation failure.

In short, requires_capture identifies which capture must exist, while skip_if_missing determines whether missing data is a hard failure or an allowed legacy case.

Example: Secure API Key¶

rules:
  - name: Secure API Key
    id: custom.secure_api.1
    pattern: |
      (?xi)
      api[_-]?key
      (?:.|[\n\r]){0,32}?
      \b
      ([[:alnum:]!@#$%^&*]{20,})
      \b
    min_entropy: 4.0
    confidence: high
    pattern_requirements:
      min_digits: 1           # Must contain at least 1 digit
      min_uppercase: 1        # Must contain at least 1 uppercase letter
      min_lowercase: 1        # Must contain at least 1 lowercase letter
      min_special_chars: 1    # Must contain at least 1 special character
      ignore_if_contains:
        - test
    examples:
      - api_key = "MyS3cur3K3y!2024"
      - 'api-key: "Abc123!@#Token"'

In this example: - The regex pattern is permissive: [[:alnum:]!@#$%^&*]{20,} matches any combination of those characters - The pattern_requirements filters out matches that don't have at least one of each required type - A match like "abcdefghijklmnopqrst" would be rejected (no uppercase, no digit, no special) - A match like "Abc123!SecureToken" would be accepted (has all required types) - A match like "Test123!SecureToken" would be rejected because it contains the ignore_if_contains term test

Example: Excluding Dummy Values¶

rules:
  - name: Token without placeholders
    id: custom.token.2
    pattern: |-
      (?i)token[:=]\s*([[:alnum:]]{12,})
    pattern_requirements:
      ignore_if_contains:
        - placeholder
        - sample
    examples:
      - token: "REALVALUE1234"

Example: Custom Special Characters¶

rules:
  - name: Token with Custom Special Chars
    id: custom.token.1
    pattern: |
      (?xi)
      token
      (?:.|[\n\r]){0,16}?
      \b([[:alnum:]$%^]{16,})\b
    min_entropy: 3.5
    confidence: medium
    pattern_requirements:
      min_special_chars: 2
      special_chars: "$%^"    # Only these characters count as "special"
    examples:
      - token = "abc$%defgh123456"

How It Works¶

1. Vectorscan regex matches a pattern in the input
2. Entropy check filters low-complexity matches (if min_entropy is set)
3. Character requirements check filters matches that don't meet the criteria
4. Validation checks verify the secret is live (if validation is configured)

Matches that fail the character requirements check are silently dropped with a debug log message.

Writing Custom Rules¶

When writing custom rules, consider the following best practices:

1. Multi-line Regex: Write your regex patterns over multiple lines for clarity. Use the (?x) flag to enable free-spacing mode.
2. Optimize for Performance: Structure your regex to minimize backtracking. Use non-capturing groups where possible and keep the pattern as concise as possible.
3. Validation Integration: Define a validation section if you want to verify the detected secret. Prefer Http or Grpc; use an existing typed validator when the rule matches a supported validator family; use Raw only for rare provider-specific exception paths. You can use Liquid templating to insert dynamic values where supported. Use the unnamed capture as TOKEN and any named captures in uppercase.
4. Revocation Integration: Define a revocation section if you want to revoke a detected secret. It uses the same HTTP request format and template variables as validation.
5. Test with Examples: Always include examples that should match, and run kingfisher rules check for the changed rule file.

Examples¶

Below are some examples to guide you in writing custom rules

Anthropic API Key¶

rules:
  - name: Anthropic API Key
    id: custom.anthropic.api-key
    pattern: |
      (?xi)                    
      \b                       
      (                        
        sk-ant-api
        \d{2,4}
        -
        [\w\-]{93}
        AA
      )                        
      \b                       
    min_entropy: 3.3
    confidence: medium
    examples:
      - sk-ant-api668-Clm512odot9WDD7itfUU9R880nefA1EtYZDbpE-C9b0XQEWpqFKf9DQUo03vOfXl16oSmyar1CLF1SzV3YzpZJ6bahcpLAA
    references:
      - https://docs.anthropic.com/claude/reference/authentication
    validation:
      type: Http
      content:
        request:
          body: |
            {
              "model": "claude-3-haiku-20240307",
              "max_tokens": 1024,
              "messages": [
                {"role": "user", "content": "respond only with 'success'"}
              ]
            }
          headers:
            Content-Type: application/json
            anthropic-version: "2023-06-01"
            x-api-key: '{{ TOKEN }}'
          method: POST
          response_matcher:
            - report_response: true
            - status:
                - 200
              type: StatusMatch
            - type: WordMatch
              words:
                - '"type":"message"'
                - "credit balance is too low"
          url: https://api.anthropic.com/v1/messages

FileIO Secret Key¶

rules:
  - name: FileIO Secret Key
    id: custom.fileio.api-key
    pattern: |
      (?xi)
      \b
      fileio
      (?:.|[\n\r]){0,32}?
      (?:SECRET|PRIVATE|ACCESS|KEY|TOKEN)
      (?:.|[\n\r]){0,16}?
      \b
      (
        [A-Z0-9]{16}
        (?:\.[A-Z0-9]{7}){2}
        \.[A-Z0-9]{8}
      )
      \b
    min_entropy: 3.3
    confidence: medium
    examples:
      - fileio SECRETKEY = Z9Y8X7W6V5U4T3S2R1Q0.P9O8N7M6L5K4J3H2G1F
      - fileio.PRIVATE.TOKEN = F0E1D2C3B4A596877869.5E4D3C2B1A0Z9Y8X7W6V
      - fileio_key = M8N6B4V2C0X9Z7L5K3J1.H2G4F6D8S0A9P7O5I3U1
    validation:
      type: Http
      content:
        request:
          method: GET
          url: https://file.io/api/v2/account
          headers:
            Authorization: "Bearer {{ TOKEN }}"
          response_matcher:
            - report_response: true
            - type: StatusMatch
              status: [200]
            - type: HeaderMatch
              header: content-type
              expected: ["application/json"]
            - type: JsonValid

Advanced Example¶

This advanced example uses the liquid-rs filters included with Kingfisher to sign requests that validate Alibaba Cloud long-lived and STS temporary credential pairs:

rules:
  - name: Alibaba Access Key ID
    id: custom.alibabacloud.access-key-id
    pattern: |
      (?x)
      \b
      (
        LTAI[A-Za-z0-9]{17,21}
      )
      \b
    pattern_requirements:
      min_digits: 2
      min_uppercase: 1
      min_lowercase: 1
    min_entropy: 4.0
    confidence: medium
    visible: false
    examples:
      - LTAI8x2NiGqfyJGx7eLDhp12
      - LTAI5GqyJGhp12ad31L5hpix
  - name: Alibaba Access Key Secret
    id: custom.alibabacloud.secret-key
    pattern: |
      (?x)
      \b
      (?:
        (?i:alibaba|alibaba[\s_-]*cloud|aliyun)
        |
        LTAI[A-Za-z0-9]{17,21}
      )
      (?:.|[\n\r]){0,80}?
      (?i:access[\s_-]*key[\s_-]*secret|access[\s_-]*secret|secret|token|key)
      (?:.|[\n\r]){0,16}?
      (?:
        [=:]
        |
        ["']\s*:\s*["']
      )
      \s*
      ["']?
      (
        [A-Za-z0-9]{30}
      )
      \b
      ["']?
    min_entropy: 4.2
    confidence: medium
    examples:
      - alibaba_secret = 7jkWdTjKLnSlGddwPR5gBn65PHcZG6
      - alibaba-token = aJHKLnSlGddwPR5g7jkWdTBn65PHc5
      - AccessKeyId=LTAI8x2NiGqfyJGx7eLDhp12 AccessKeySecret=7jkWdTjKLnSlGddwPR5gBn65PHcZG6
    validation:
      type: Http
      content:
        request:
          method: GET
          url: >
            {%- assign nonce = "" | uuid | upcase -%}
            {%- assign raw_timestamp = "" | iso_timestamp_no_frac -%}
            {%- assign timestamp = raw_timestamp | replace: ":", "%3A" -%}

            {%- capture params -%}
            AccessKeyId={{ AKID | url_encode }}&Action=GetCallerIdentity&Format=JSON&SignatureMethod=HMAC-SHA1&SignatureNonce={{ nonce }}&SignatureVersion=1.0&Timestamp={{ timestamp }}&Version=2015-04-01
            {%- endcapture -%}
            {%- assign encoded_params = params | replace: "+", "%20" | replace: "*", "%2A" | replace: "%7E", "~" -%}
            {%- assign query_string = encoded_params | url_encode | replace: "%2D", "-" | replace: "%2E", "." -%}

            {%- assign signature_base_string = "GET&%2F&" | append: query_string -%}
            {%- assign token_amp = TOKEN | append: "&" -%}

            {%- assign hmacsignature = signature_base_string | hmac_sha1: token_amp | url_encode -%}

            https://sts.aliyuncs.com/?{{ params }}&Signature={{ hmacsignature }}
          headers:
            Accept: application/json
          response_matcher:
            - report_response: true
            - type: StatusMatch
              status: [200]
            - type: WordMatch
              words: ['"Arn"']
    depends_on_rule:
      - rule_id: custom.alibabacloud.access-key-id
        variable: AKID
  - name: Alibaba STS Access Key ID
    id: custom.alibabacloud.session-access-key-id
    pattern: |
      (?x)
      \b
      (
        STS\.[A-Za-z0-9]{16,64}
      )
      \b
    min_entropy: 3.0
    confidence: medium
    visible: false
    examples:
      - STS.NTKaenSkmLhG4HpM576UV
      - STS.FJ6EMcS1JLZgAcBJSTDG1Z4CE
  - name: Alibaba STS Security Token
    id: custom.alibabacloud.session-secret-key
    pattern: |
      (?xi)
      \b
      (?:security[\s_-]*token|sts[\s_-]*token|x[\s_-]*oss[\s_-]*security[\s_-]*token|alibaba[\s_-]*cloud[\s_-]*security[\s_-]*token|aliyun[\s_-]*security[\s_-]*token)
      (?:.|[\n\r]){0,16}?
      (?:
        [=:]
        |
        ["']\s*:\s*["']
      )
      \s*
      ["']?
      (
        CAIS[A-Za-z0-9+/_=-]{20,1024}
      )
      (?:["'\s,;}&\]]|$)
    min_entropy: 4.0
    confidence: medium
    visible: false
    examples:
      - securityToken = "CAISuwJ1q6Ft5B2yu9Kiaa5E0VnVJ8q2o3P4r5S6t7U8v9W0xYz"
      - ALIBABA_CLOUD_SECURITY_TOKEN=CAIS/gF1q6Ft5B2yfSjIr5eDA9xjJCcl57eKC7A3ThnJA
  - name: Alibaba STS Access Key Secret
    id: custom.alibabacloud.session-token
    pattern: |
      (?x)
      \b
      (?:
        (?i:alibaba|alibaba[\s_-]*cloud|aliyun|sts)
        |
        STS\.[A-Za-z0-9]{16,64}
      )
      (?:.|[\n\r]){0,120}?
      (?i:access[\s_-]*key[\s_-]*secret|access[\s_-]*secret)
      (?:.|[\n\r]){0,16}?
      (?:
        [=:]
        |
        ["']\s*:\s*["']
      )
      \s*
      ["']?
      (
        [A-Za-z0-9]{30,64}
      )
      \b
      ["']?
    min_entropy: 4.2
    confidence: medium
    examples:
      - STS.NTKaenSkmLhG4HpM576UV AccessKeySecret=wyLTSmsyPGP1ohvvw8xYgB29dlGI8KMiH2pK
      - "aliyun sts access_key_secret: 6itECZnhbG2RU6ktTSBSd6JxeLHKPWyBtSS62"
    validation:
      type: Http
      content:
        request:
          method: GET
          url: >
            {%- assign nonce = "" | uuid | upcase -%}
            {%- assign raw_timestamp = "" | iso_timestamp_no_frac -%}
            {%- assign timestamp = raw_timestamp | replace: ":", "%3A" -%}

            {%- capture params -%}
            AccessKeyId={{ STS_AKID | url_encode }}&Action=GetCallerIdentity&Format=JSON&SecurityToken={{ SECURITY_TOKEN | url_encode }}&SignatureMethod=HMAC-SHA1&SignatureNonce={{ nonce }}&SignatureVersion=1.0&Timestamp={{ timestamp }}&Version=2015-04-01
            {%- endcapture -%}
            {%- assign encoded_params = params | replace: "+", "%20" | replace: "*", "%2A" | replace: "%7E", "~" -%}
            {%- assign query_string = encoded_params | url_encode | replace: "%2D", "-" | replace: "%2E", "." -%}

            {%- assign signature_base_string = "GET&%2F&" | append: query_string -%}
            {%- assign token_amp = TOKEN | append: "&" -%}

            {%- assign hmacsignature = signature_base_string | hmac_sha1: token_amp | url_encode -%}

            https://sts.aliyuncs.com/?{{ params }}&Signature={{ hmacsignature }}
          headers:
            Accept: application/json
          response_matcher:
            - report_response: true
            - type: StatusMatch
              status: [200]
            - type: WordMatch
              words: ['"Arn"']
    depends_on_rule:
      - rule_id: custom.alibabacloud.session-access-key-id
        variable: STS_AKID
      - rule_id: custom.alibabacloud.session-secret-key
        variable: SECURITY_TOKEN

Provider formats and context requirements¶

Built-in DeepSeek, Kimi/Moonshot, ZAI, and Voyage AI rules use bare: true in the Betterleaks capability overlay to derive medium-confidence patterns without provider assignments. MiniMax already matches bare tokens upstream. Token formats, entropy filters, and validation remain sourced from upstream. Both detection-only and validating scans use these patterns; live validation runs only after detection. Other contextual rules may still require a provider assignment before the token; a trailing comment does not satisfy it.

For extracted credential corpora, preserve provider assignments and required components or use validate with an explicit rule and endpoint. The pinned AWS access-token rule already recognizes ASIA IDs. AWS temporary credentials may require an access key ID, secret access key, and session token; a lone ASIA fragment does not establish a complete validatable credential.

New provider formats and length variants belong in the upstream Betterleaks catalog. Redacted shape masks alone do not establish reliable token boundaries, length ranges, or provider-specific validation behavior. Do not widen shared sk- patterns across providers solely to increase recall: that can send a token to the wrong provider. Use documented formats and synthetic positive/negative fixtures when proposing upstream variants.