Docker layer caching
When a layer changes in your Docker image, that layer, and all subsequent layers, are rebuilt. Docker layer caching can cache unchanged layers and reuse them across pipeline runs, which can dramatically reduce build times.
To maximize savings, consider modifying your Dockerfile to use the cache efficiently.
Docker Layer Caching, an Intelligence Feature
With Docker Layer Caching (DLC) , a Harness CI Intelligence feature, Harness seamlessly caches Docker layers between builds to accelerate the time it takes to build Docker images.
You can use DLC with any build infrastructure. When you use DLC with Harness CI Cloud, the cache is stored in the Harness-managed environment.
Docker Layer Caching for self-managed build infrastructure is an early access feature and is behind the feature flag CI_ENABLE_DLC_SELF_HOSTED
. Contact Harness Support to enable the feature.
Cache storage
When you use Docker Layer Caching with Harness CI Cloud, the cache is stored in the Harness-managed environment. When running builds on self-managed infrastructure, you will need to provide your own storage.
- Harness Cloud
- Self-managed build infrastructures
When you use Docker Layer Caching with Harness CI Cloud, you don't need to bring your own storage, because the cache is stored in Harness-managed Harness Cloud storage.
All pipelines in the account use the same cache storage, and each build tool has a unique cache key that is used to restore the appropriate cache data at runtime.
The cache storage limit depends on your subscription plan type:
- Free: 2 GB
- Team: 5 GB
- Enterprise: 10 GB
Harness doesn't limit the number of caches you can store, but, once you reach your storage limit, Harness continues to save new caches by automatically evicting old caches.
The cache retention window is 15 days, which resets whenever a cache is updated.
When running builds in self-managed infrastructures, configure S3-compatible default object storage that Harness can use to seamlessly store and manage the cache.
If your storage isn't S3-compatible or your don't want to use access key and secret key for authentication, consider using remote cache image instead.
We suggest that you consider setting bucket level retention policy for efficient cache management.
Enable Docker Layer Caching
-
If you're not using Harness Cloud build infrastructure, you must configure S3-compatible global object storage that Harness can use to store and manage caches.
This is not required for Harness Cloud build infrastructure. For more information, go to Cache storage.
-
To enable Docker layer caching with Harness CI cloud, select Enable Docker layer caching in your Build and Push steps.
Here is a YAML example of a Build and Push an image to Docker Registry step that uses DLC.
- step:
type: BuildAndPushDockerRegistry
name: Build and push to Docker
identifier: Build_and_push_to_Docker
spec:
connectorRef: YOUR_DOCKER_CONNECTOR_ID
repo: DOCKER_USERNAME/DOCKER_REPO_NAME
caching: true
tags:
- <+pipeline.sequenceId>
Enabling DLC when running on Kubernetes requires privileged mode on the cluster where the builds run.
Remote cache image
Remote cache image is an alternative to Harness CI Intelligence Docker layer caching.
Remote caching leverages your existing Docker registry to pull previously built layers. Each Docker layer is uploaded as an image to a Docker repo you identify. If the same layer is used in subsequent builds, Harness downloads the layer from the Docker repo. You can also specify the same Docker repo for multiple Build and Push steps, enabling them to share the same remote cache.
If you enable both Harness Intelligence Docker layer caching and set remote cache image on the same Build and Push step, Harness will use the remote cache image for caching.
To enable remote cache image in your Build and Push steps:
-
Select the Remote Cache Image option.
-
Enter the name of the remote cache repository/registry and image, depending on the step you are using. For example:
- Build and Push to Docker:
NAMESPACE/IMAGE
orREGISTRY/IMAGE
- Build and Push to GCR:
gcr.io/PROJECT_ID/IMAGE
- Build and Push to GAR:
LOCATION-docker.pkg.dev/PROJECT_ID/REPO/IMAGE
- Build and Push to ECR:
APP/IMAGE
- Build and Push to ACR:
CONTAINER_REGISTRY.azurecr.io/IMAGE
- Build and Push to Docker:
-
For the best performance, make sure:
- The remote cache repo/registry exists in the same host/account and project/organization as the build image.
- The specified repo/registry and image already exist. For some providers, Harness can automatically create the repository if it doesn't already exist, but this is not guaranteed. For any provider, Harness needs an existing image to overwrite.
Optimize Docker images and Dockerfiles
In addition to enabling Docker layer caching, the following practices can reduce Docker image build times.
Pre-build images that include all required dependencies
If most of the build time is spent downloading dependencies, you should pre-build an image with all required dependencies in a separate pipeline. Then, set up a periodic pipeline that builds the image with all the latest dependencies and pushes it to your Docker registry. Use this image in all of your build pipelines.
Pre-building images with all required dependencies is more efficient than downloading them to a baseline image as part of the Build setup. This is especially true if you update your images often to ensure that they include all the latest updates.
Exclude unnecessary files and packages from your images
In addition to reducing build times, excluding unnecessary files and packages makes the resulting images smaller, simpler, and more portable. You can use dockerignore files to exclude unnecessary files and folders from your images.
Optimize Dockerfiles
- Sort multi-line arguments in your Dockerfile alphabetically. This makes it easier to update and avoid duplicate packages.
- Review Docker's best practices for writing Dockerfiles.
- Write your Dockerfiles to use Docker layer caching efficiently.