Cache Troubleshooting

If you've followed the setup instructions and still can't seem to make it work, this guide is for you.

Enable -cacheverbose

You can use the -cacheverbose command line option to emit additional information about cache operations, and in particular, the Cache Key generated for each Object File being compiled.

Combining -cacheverbose with -cache, -cacheread, or -cachewrite will output additional information in the following format:

The Cache Key is a unique identifier representing an entry in the cache. Cache Keys are generated by combining hashes of various inputs that feed into compilation of an Object File. Cache Keys are in comprised as follows:

For an object to be retrieved from the cache, the Cache Key used during storage and retrieval must match exactly. Differences in components of the Cache Key imply compiled output results will differ, inhibiting caching.

If cache hits are expected but not realized, checking for differences between Cache Keys will highlight the component resulting in said cache misses. For each component, you can troubleshoot further using the information below.

The source code used for compilation must precisely match for caching to be possible. i.e. Two machines must be compiling the same source code, resulting in the same preprocessed output.

You can view the preprocessed source yourself by directly invoking the compiler with an appropriate command line. You can first invoke FASTBuild with -showcmds to see what command line is being using:

You can then invoke the preprocessor command line yourself, piping the output to a file as per the following example:

If you do this on two different machines, you can compare the results to look for discrepancies.

• NOTE: If you are using LightCache, you will need to manually append the preprocessor option (usually /E or -E)

Common causes of Source Hash discrepancies are detailed below:

Source Files

Machines must be compiling the same code to benefit from caching and the source files must match. A single byte in a single file can result in different compiler output, and thus can prevent caching. Depending on the compiler, line endings may also matter.

Command Line Defines

The defines provided to the compiler on the command line will alter the generated code. Machines must be using the same compiler defines to benefit from caching. (See Command Line Hash below.)

__DATE__ and __TIME__ Macros

Particular care should be taken to avoid __TIME__ and __DATE__ macros which cause cache mismatches due to their continually changing nature (every second and every day respectively).

These macros are usually best avoided anyway, as they can be misleading since they reflect the time an Object File is generated, rather than when an executable or dll was generated. A post-link "stamping" process should usually be preferred for injection of date/time strings into final build artifacts.

If complete removal of these macros is not possible, limitting their use to only certain Object Files will still allow other Object Files to generate consistent Cache Keys.

The command line used to invoke the compiler is hashed. This includes the full path to the files being compiled. As such, the source code and working directory must match between multiple machines.

You can use the -showcmds option to view the command line being used to check for differences as follows:

On one machine, perform a write-only operation:

On another machine, perform a read-only operation:

If you do this on two different machines, you can compare the results to look for discrepancies.

• NOTE: If you are using LightCache, the command line will not be emitted. You make like to disable the LightCache while troubleshooting.

• NOTE: The .UseRelativePaths_Experimental feature of Compiler() may adequately relax this limitation on full paths in the future.

The Toolchain Hash represents the compiler binaries and additional files as specified by the Compiler() in your FASTBuild config. As different compilers and compiler versions can emit different results, the toolchain must match for caching to be successful. Ensure that your machines are using the same bff files, specifying the same binaries and that they are identical.

• NOTE: Versioning of compiler binaries along with source files in source control is recommended.

If using Precompiled Headers when compiling an Object File, the hash of the precompiled header associated with an Object File is calculated. When not using Precompiled Headers, this component is always 00000000.

For Precompiled Headers (which can also be cached), all Cache Key components are treated normally and the Precompiled Header component is 00000000.

To troubleshoot discrepancies in the Cache Key for a Precompiled Header itself, use the same steps as for a normal Object File.

The Cache Version is used by FASTBuild to allow for invalidation of existing caches when breaking changes to the caching format are made. These changes are rare. If you have a different Cache Version in your Cache Key, you are using incompatible FASTBuild versions.

The LightCache feature replaces the use of the Compiler's preprocessor with FASTBuild's own internal processor. This accelerates Source Hash generation by an order of magnitude, but may add some additional considerations when troubleshooting.

No Preprocessor

Since there is no preprocessor invocation, there is no preprocessor command line that you can examine with -showcmds. As a work-around, you can disable use of the LightCache until you have caching working.

Different Source Hash

The LightCache uses a different mechanism for hashing the source files and as such the Source Hash component of the Cache Key differs compared to using the Compiler's preprocessor. IF debugging caching, be sure to consistently use (or not) the LightCache to ensure you are comparing comparable Cache Keys.