Incremental Build and Build Cache

Since the first release, Booster has supported incremental builds for Transform. When some source files or dependencies change, it does not build all inputs, but only selects the changed portion for building. Moreover, BoosterTransform also supports build cache.

In addition to BoosterTransform, the following also support build cache:

How Incremental Build Works

Before executing a task for the first time, Gradle calculates the inputs. The calculation includes the paths of input files and the hash of each file's content. After the task completes successfully, Gradle calculates the outputs. The calculation includes a set of output files and the hash value of each file's content. Gradle retains both calculation results before the next task execution.

Before each subsequent task execution, Gradle calculates the current task's inputs and outputs. If the old and new calculations are the same, Gradle assumes the output is up-to-date (UP-TO-DATE) and skips that task. If they are different, Gradle executes the task. Similarly, Gradle retains both calculation results before the next task execution.

If the file statistics have not changed, Gradle will reuse the calculation results from the last run. This means that when the file statistics have not changed, Gradle will not detect changes. Taking Gradle 4.6 as an example, the key code is as follows:

@NonNullApi
public class DefaultTaskUpToDateState implements TaskUpToDateState {

     private final TaskStateChanges inputFileChanges;
     private final OutputFileTaskStateChanges outputFileChanges;
     private final TaskStateChanges allTaskChanges;
     private final TaskStateChanges rebuildChanges;
     private final TaskStateChanges outputFilePropertyChanges;

     /**
      * @param lastExecution Historical execution record from last run, containing previous calculation results
      * @param thisExecution Current execution record, containing latest calculation results
      * @param task          The task to be executed
      */
     public DefaultTaskUpToDateState(TaskExecution lastExecution, TaskExecution thisExecution, TaskInternal task) {
         // Some code omitted
         this.allTaskChanges = new ErrorHandlingTaskStateChanges(task, new SummaryTaskStateChanges(
                 MAX_OUT_OF_DATE_MESSAGES,
                 previousSuccessState,
                 taskTypeState,
                 inputPropertyChanges,
                 inputPropertyValueChanges,
                 outputFilePropertyChanges,
                 outputFileChanges,
                 inputFilePropertyChanges,
                 inputFileChanges,
                 discoveredInputFileChanges));
         this.rebuildChanges = new ErrorHandlingTaskStateChanges(task, new SummaryTaskStateChanges(
                 1,
                 previousSuccessState,
                 taskTypeState,
                 inputPropertyChanges,
                 inputPropertyValueChanges,
                 inputFilePropertyChanges,
                 outputFilePropertyChanges,
                 outputFileChanges));
     }

     /* This method is called for UP-TO-DATE check */
     @Override
     public TaskStateChanges getAllTaskChanges() {
         return allTaskChanges;
     }

     /* This method is called to check if incremental build can be executed */
     @Override
     public TaskStateChanges getRebuildChanges() {
         return rebuildChanges;
     }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

Task Inputs and Outputs

We know that incremental implementation is based on calculating the Task's inputs and outputs to determine whether re-execution is needed. So what exactly are the Task's inputs and outputs? Taking the familiar JavaCompile Task as an example, in most people's understanding, its output is class files and its input is java source files. Is this really the case?

In Gradle, the inputs and outputs of the JavaCompile task are shown in the following diagram:

Task inputs and outputs

For a Gradle Task, anything that affects the output result is an input. For example: input files, build environment, target version number, etc.

As part of incremental builds, Gradle checks whether inputs or outputs have changed since the last build. If there are no changes, Gradle considers the task up-to-date and skips executing its actions.

WARNING

Incremental builds will not work unless a task has at least one task output

To simplify custom Task support for incremental builds, Gradle provides a mechanism that uses Annotations to annotate Task properties and automatically completes Task incremental checking and calculation during the build process. The following table lists the Annotations provided by Gradle:

Annotation	Type	Description
`@Input`	`Serializable`	Serializable input object
`@InputFile`	`File`	Single input file (not including directories)
`@InputDirectory`	`File`	Single input directory (not including files)
`@InputFiles`	`Iterable<File>`	Collection of input files or directories
`@Classpath`	`Iterable<File>`	Collection of input files and directories in classpath
`@CompileClasspath`	`Iterable<File>`	Collection of input files and directories in compile classpath
`@OutputFile`	`File`	Single output file
`@OutputDirectory`	`File`	Single output directory
`@OutputFiles`	`Map<String, File>` or `Iterable<File>`	Collection or mapping of output files
`@OutputDirectories`	`Map<String, File>` or `Iterable<File>`	Collection or mapping of output directories
`@Destroys`	`File` or `Iterable<File>`	Specifies one or more files destroyed by this task. Note that a task can define inputs/outputs or destroyables, but not both
`@LocalState`	`File` or `Iterable<File>`	Local task state, these files will be removed when the task is restored from cache
`@Nested`	Any custom type	Nested property
`@Console`	Any type	Helper property, indicates the annotated property is neither an input nor an output property
`@Internal`	Any type	Internal use property, indicates the annotated property is neither an input nor an output property
`@ReplacedBy`	Any type	Indicates this property has been replaced by another, ignored as input or output
`@SkipWhenEmpty`	`File`	Used with `@InputFiles` / `@InputDirectory`, tells Gradle to skip the task if the corresponding file or directory is empty and all other input files declared with this annotation are empty
`@Incremental`	`Provider<FileSystemLocation>` or `FileCollection`	Used with `@InputFiles` / `@InputDirectory`, tells Gradle to track changes to file properties. Changes can be queried via `InputChanges.getFileChanges()`
`@Optional`	Any type	Optional property
`@PathSensitive`	`File`	Type of file property

When Incremental Build Does Not Take Effect

Incremental build will not take effect in the following situations:

No historical build records
Gradle version has changed
The upToDateWhen condition implemented by the Task returns false
Input properties, non-incremental file properties, or output files have changed since the last build.

How to Write Incremental Build Tasks

If Gradle has a history of previous task executions, and the only change to the task execution context since execution is the input files, then Gradle can determine which input files the task needs to reprocess. In this case, the IncrementalTaskInputs.outOfDate() operation is executed for any added or modified input files. The IncrementalTaskInputs.removed() operation is executed for all deleted input files. Use the InputChanges.isIncremental() method to check if the task execution is incremental. Here is the sample code:

class IncrementalTask : DefaultTask() {

    @get:Input
    private int fileCount = 10

    @get:OutputDirectory
    private val outputDir: File
        get() = project.file("${project.buildDir}/booster/test/")

    @TaskAction
    void perform(inputs: IncrementalTaskInputs) {
        if (!inputs.isIncremental()) {
            project.delete(outputDir.listFiles())
        }

        inputs.outOfDate { change ->
            println("out of date: ${change.file.name}")
        }

        inputs.removed { change ->
            println("removed: ${change.file.name}")
        }
    }

}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

Android Gradle Plugin Incremental Build

Android Gradle Plugin (hereinafter referred to as AGP) is based on Gradle for builds, and added support for incremental compilation in version 2.0.0. This feature has been gradually improved in subsequent versions.

Incremental Build in Transform

In the Bytecode Processing Pipeline chapter, we have introduced the concept of the Transform API. Transform is not a Task, but the TransformTask it belongs to supports incremental builds. The Transform API itself provides an isIncremental() method. Only when this method returns true will TransformTask pass incremental build information to the context TransformInvocation for use by the corresponding Transform.

Stability of TransformTask Incremental Builds

In some cases, developers may encounter incremental compilation failures, or even build failures. There are many reasons for this, and we cannot list all cases, but we can offer some suggestions:

For App developers
- You can upgrade Gradle and the plugins used in the build process to the latest stable versions provided officially
- Reduce task generation through the build environment
For plugin developers
- Try not to modify the inputs of other tasks
- In the Transform phase, set the isIncremental() method implementation to return true

What is Build Cache

Gradle build cache is a caching mechanism at the granularity of Gradle tasks. If a task can be cached (CacheableTask), its output results can be cached; if a build later finds that the task already has a matching build cache (Build-Cache), it can directly reuse the build artifacts, thereby reducing full build time.

Gradle build cache is divided into two types: local build cache (Local Build Cache) and remote build cache (Remote Build Cache), which are suitable for different scenarios:

Local Build-Cache is mainly used for developers' own development. For example: building after switching branches;
Remote Build-Cache is more suitable for remote CI packaging scenarios. By reusing build cache between machines, overall packaging efficiency can be improved.

Build Cache Calculation Rules

The official documentation's chapter on Build-Cache Cacheable Tasksopen in new window briefly lists the factors that affect Build-Cache calculation, such as task properties and classpath, input/output property names, etc. In fact, Gradle calculates these properties and finally obtains a unique identifier, which is used to find and cache file output results. The following is the key code for Cache-Key calculation in Gradle 4.6:

public TaskOutputCachingBuildCacheKey calculate(TaskInternal task, TaskExecution execution) {

    TaskOutputCachingBuildCacheKeyBuilder builder = new DefaultTaskOutputCachingBuildCacheKeyBuilder(task.getIdentityPath());
    if (buildCacheDebugLogging) {
        builder = new DebuggingTaskOutputCachingBuildCacheKeyBuilder(builder);
    }
    builder.appendTaskImplementation(execution.getTaskImplementation());
    builder.appendTaskActionImplementations(execution.getTaskActionImplementations());

    SortedMap<String, ValueSnapshot> inputProperties = execution.getInputProperties();
    for (Map.Entry<String, ValueSnapshot> entry : inputProperties.entrySet()) {
        DefaultBuildCacheHasher newHasher = new DefaultBuildCacheHasher();
        entry.getValue().appendToHasher(newHasher);
        if (newHasher.isValid()) {
            HashCode hash = newHasher.hash();
            builder.appendInputPropertyHash(entry.getKey(), hash);
        } else {
            builder.inputPropertyLoadedByUnknownClassLoader(entry.getKey());
        }
    }

    SortedMap<String, FileCollectionSnapshot> inputFilesSnapshots = execution.getInputFilesSnapshot();
    for (Map.Entry<String, FileCollectionSnapshot> entry : inputFilesSnapshots.entrySet()) {
        FileCollectionSnapshot snapshot = entry.getValue();
        builder.appendInputPropertyHash(entry.getKey(), snapshot.getHash());
    }

    SortedSet<String> outputPropertyNamesForCacheKey = execution.getOutputPropertyNamesForCacheKey();
    for (String cacheableOutputPropertyName : outputPropertyNamesForCacheKey) {
        builder.appendOutputPropertyName(cacheableOutputPropertyName);
    }

    return builder.build();
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

How to Enable Build Cache

Command Line

When running builds from the command line, add the --build-cache parameter:

$ ./gradlew assembleDebug --build-cache

gradle.properties

Add the property org.gradle.caching in the gradle.properties file as shown below:

org.gradle.caching=true

Gradle Built-in Cacheable Tasks

Gradle has many built-in Cacheable Tasks, for example:

Java toolchain: JavaCompileopen in new window, Javadocopen in new window
Groovy toolchain: GroovyCompileopen in new window, Groovydocopen in new window
Scala toolchain: ScalaCompileopen in new window, PlatformScalaCompileopen in new window, ScalaDocopen in new window
Native toolchain: CppCompileopen in new window, CCompileopen in new window, SwiftCompileopen in new window
Testing: Testopen in new window
Code quality tasks: Checkstyleopen in new window, CodeNarcopen in new window, Pmdopen in new window
JaCoCo: JacocoMergeopen in new window, JacocoReportopen in new window
Other tasks: AntlrTaskopen in new window, ValidatePluginsopen in new window, WritePropertiesopen in new window

How to Write Cacheable Tasks

Using the `@CacheableTask` Annotation

Taking booster-task-compression-cwebpopen in new window and booster-task-compression-pngquantopen in new window as examples, all subclasses of CompressImages implement task caching through @CacheableTask.

Using the Runtime API

In addition to @CacheableTask, Gradle also provides cache-related APIs. To enable task caching, you need to call the TaskOutputs.cacheIf() method, for example:

tasks.register<NpmTask>("bundle") {
    args.set(listOf("run", "bundle"))

    outputs.cacheIf { true }

    inputs.dir(file("scripts"))
        .withPropertyName("scripts")
        .withPathSensitivity(PathSensitivity.RELATIVE)

    inputs.files("package.json", "package-lock.json")
        .withPropertyName("configFiles")
        .withPathSensitivity(PathSensitivity.RELATIVE)

    outputs.file("$buildDir/bundle.js")
        .withPropertyName("bundle")
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

For more details about Cacheable Task, please refer to: https://docs.gradle.org/current/userguide/build_cache.html#enable_caching_of_non_cacheable_tasksopen in new window

WARNING

Not all tasks are suitable for caching. Cache files are essentially compressed packages. If a task has a short execution time, or if its output changes every time, configuring that task as cacheable is meaningless and may even reduce build efficiency. For such tasks, we can simply configure them as incremental tasks

Configuring Build Cache

Configure the cache through the settings.gradle configuration file, as shown below:

buildCache {
  local {
      directory = File(rootDir, "build-cache")
      removeUnusedEntriesAfterDays = 30
  }
  remote<HttpBuildCache> {
      url = uri("https://booster.johnson.io:8123/cache/")
      credentials {
          username = "$USERNAME"
          password = "$PASSWORD"
      }
  }
}

1
2
3
4
5
6
7
8
9
10
11
12
13

For more about Gradle Build Cache, please refer to: https://docs.gradle.org/current/userguide/build_cache.htmlopen in new window

Deploying Build Cache Service

Gradle officially provides a Docker image for Build Cache gradle/build-cache-nodeopen in new window. For how to use Build Cache Node, please refer to: https://docs.gradle.com/build-cache-node/open in new window.

Android Gradle Plugin Build Cache

After AGP 2.3.0, build tasks enable their own build cache by default. This cache is different from the Gradle cache, and the cache is stored by default in the ~/.android/build-cache/ path. For more details, please refer to: Using the Build Cacheopen in new window.

# Incremental Build and Build Cache

# How Incremental Build Works

# Task Inputs and Outputs

# Incremental Build Related Annotations

# When Incremental Build Does Not Take Effect

# How to Write Incremental Build Tasks

# Android Gradle Plugin Incremental Build

# Incremental Build in Transform

# Stability of TransformTask Incremental Builds

# What is Build Cache

# Build Cache Calculation Rules

# How to Enable Build Cache

# Command Line

# gradle.properties

# Gradle Built-in Cacheable Tasks

# How to Write Cacheable Tasks

# Using the @CacheableTask Annotation

# Using the Runtime API

# Configuring Build Cache

# Deploying Build Cache Service

# Android Gradle Plugin Build Cache