Upgrading¶
Upgrading tartufo
from release 2 to release 3 introduces some behavioral and
interface changes. Current users of release 2 should review this summary to
understand how to transition to release 3 as painlessly as possible.
General Behavioral Changes¶
tartufo
release 3 is generally more accurate than previous releases. It may
detect problems that were not recognized by release 2 scans (especially earlier
2.x releases). A scan of your code base prior to upgrading will simplify the
process of identifying new findings that are attributable to these behavior
changes so they can be remediated or suppressed.
Remote Repository Scanning¶
tartufo
releases between 2.2.0 and 2.9.0 (inclusive) mishandled remote
repositories. Only the repository’s default branch was scanned; secrets
present only on other branches would not be discovered.
Additionally, the --branch branch-name
option did not operate correctly.
Some versions scanned nothing and reported no errors, and other versions aborted
immediately after reporting the branch did not exist (even when it did).
tartufo
release 3 scans all remote repository branches by default, and
correctly scans only a single branch if one is specified using --branch
. As
a consequence, it may discover secrets that were not reported by earlier versions.
These fixes were backported to tartufo
release 2.10.0.
Live Output¶
tartufo
release 3 reports findings incrementally as a scan progresses; previous
releases did not perform any reporting until the entire scan was completed.
Entropy Scanning¶
Beginning with release 3, tartufo
recognizes base64url-encoded strings in
addition to base64-encoded strings.
If your code contains base64url encodings (or strings that look like base64url encodings), these strings now will be checked for high entropy and may produce new findings.
Additionally, strings that contain combinations of base64 and base64url character sets (whether they are actual encodings or not) will be scanned differently by release 3. Previously, base64 substrings would be extracted and scanned independently, but now the larger string will be scanned (once) in its entirety. This can result in signature changes (because the new suspect string is larger than the string recognized by release 2.x) and possibly fewer findings (because one longer string will be flagged instead of multiple substrings). Real-life files do not typically contain sequences that will exhibit this behavior.
Shallow Repositories¶
When tartufo
release 2 scanned a shallow repository (a repository with no
refs or branches found locally), it did not actually scan anything.
In the same situation, tartufo
release 3 scans the repository HEAD as a single
commit, effectively scanning the entire existing codebase (but none of its history)
at once.
This scenario is commonly encountered in GitHub actions, which perform shallow checkouts.
Nonfunctional Options¶
tartufo
release 3 uses pygit2 instead of GitPython to access git repositories.
While this provides vastly improved performance with
generally equivalent functionality, some less-frequently used options require
reimplementation and currently are nonfunctional. We plan to provide either
replacements or reimplementations in the future.
The --since-commit
option is intended to restrict scans to a subset of
repository history; the --max-depth
option provides roughly the same
functionality specified differently. Both options are ignored by tartufo
release 3. Refer to #267 for more information about this topic.
Changes to Default Behavior¶
Some defaults have changed for the new release. If you wish to retain the previous behavior, adjust your configuration options to request it explicitly.
Regex Scanning¶
Previously, tartufo
did not perform regex scanning for sensitive strings by
default. Release 3 does perform regex scanning by default.
Explicitly disable regex scanning to preserve the old behavior:
[tool.tartufo]
regex = false
Alternatively, add --no-regex
to your tartufo
command line.
Retired Options¶
Some options that were deprecated in later 2.x releases no longer are supported by version 3. You will need to alter your command line and/or configuration options to specify the required information in a release 3-compatible manner.
Fetch Before Local Scans¶
tartufo
release 2 supported command option --fetch
for local repository
scans, in order to force an update of the repository before scanning it. tartufo
release 3 no longer recognizes this option.
Instead of using --fetch
, perform an explicit git fetch
command prior to
executing tartufo
.
If you were using --no-fetch
, simply remove the option. tartufo
release 3
never performs fetches prior to scanning local repositories.
Output Formatting¶
tartufo
release 2 supported command options --json
and --compact
to
control output formatting. tartufo
release 3 no longer recognizes these options.
Replace --json
with --output-format json
, and replace --compact
with
--output-format compact
.
Path Scoping¶
tartufo
release 2 supported command options --include-paths
and
--exclude-paths
in order to control which files were (or were not) scanned.
In either case, the option accepted a filename which was expected to contain path
patterns to include or exclude, respectively. tartufo
release 3 no longer
recognizes these options.
It is recommended that these path expressions be migrated from the external file
to your pyproject.toml
file and converted to TOML array of tables format.
The supported formats are described in Limiting Scans by Path.
Deprecated Options¶
tartufo
release 3 deprecates some release 2 options. Although no action is
required at this time, replacing these options with their newer equivalents will
reduce future disruptions when they are retired.
Updating Signatures¶
tartufo
release 3.2.0 deprecated a number of signatures that were generated
with the leading +/- from the git diff erroneously. These signatures will no
longer work in release 4. An additional command tartufo update-signatures
was
added which scans a local repository, automatically updates the deprecated
exclude-signatures in your tartufo config file, and removes any resulting duplicates.
Use --no-update-configuration
to prevent tartufo
from overwriting your config.
Use --no-remove-duplicates
to prevent tartufo
from removing duplicate signatures.
When removing duplicate signatures tartufo
will keep the first signature it finds
and discard the rest.
External Rules Files¶
The --rules
command option accepts a filename that is expected to contain
one or more rule patterns. tartufo
release 3 deprecates this option.
It is recommended that these patterns be migrated from the external file to your
pyproject.toml
file and converted to TOML array of tables format.
The supported formats are described in Rule Patterns.
Entropy Scan Sensitivity¶
The new --entropy-sensitivity
option is intended to replace both
--b64-entropy-score
and --hex-entropy-score
. The new option adjusts
sensitivity for both encodings consistently, using a scale of 0-100. To convert:
Users of
--b64-entropy-score
should divide the provided value by 0.06 to obtain the equivalent--entropy-sensitivity
settingUsers of
--hex-entropy-score
should divide the provided value by 0.04 to obtain the equivalent--entropy-sensitivity
setting
Users who require different base64 and hexadecimal sensitivities should open an issue that explains their use case.