Skip to main content

Tutorial: Create a build-scan-push pipeline (STO and CI)

In this tutorial, you'll create an end-to-end pipeline that uses STO and CI steps to build an image and pushes it to Docker Hub only if the codebase and image contain no critical vulnerabilties. This pipeline uses two popular open-source tools:

You can copy/paste the YAML pipeline example below into Harness and update it with your own infrastructure, connectors, and access tokens.

The following steps describe the workflow:

  1. An STO Bandit step scans the codebase and ingests the scan results.

  2. If the code has no critical vulnerabilities, a CI Build and Push step builds a test image and pushes it to Docker Hub.

  3. An Aqua Trivy step scans the image and ingests the results.

  4. If the image has no critical vulnerabilities, another Build and Push step pushes a prod image to Docker Hub.

scan-build-scan-push tutorial pipeline

Prerequisites
  • This tutorial has the following prerequisites:

    • Harness STO and CI module licenses.

    • You must have a Security Testing Developer or SecOps role assigned.

    • A basic understanding of key STO concepts and good practices is recommended. Your first STO pipeline is a good introduction.

    • GitHub requirements:

      • A GitHub account and access token.

      • A GitHub connector that specifies your account (http://github.com/my-account) but not a specific repository (http://github.com/my-account/my-repository).

      • Your GitHub account should include a repository with Python code. The repo should also include a Dockerfile for creating an image.

        This tutorial uses the dvpwa repository as an example. The simplest setup is to fork this repository into your GitHub account.

    • Docker requirements:

      • A Docker Hub account and access token.
      • A Docker connector is required to push the image.
    • Your GitHub and Docker Hub access tokens must be stored as Harness secrets.

Set up your codebase

This tutorial uses Bandit to scan the target repository https://github.com/williamwissemann/dvpwa (specified in the Codebase for this pipeline).

  1. Fork the following example repository into your GitHub account. This is a Python repo with known vulnerabilities: https://github.com/williamwissemann/dvpwa.

  2. If you don't have a GitHub connector, do the following:

    1. In your Harness project, select Project Setup > Connectors.
    2. Select New Connector, then select Code Repositories > GitHub.
    3. Set the GitHub connector settings as appropriate.
      • Use Account for the URL type.
      • This tutorial uses Harness Cloud, so select Connect through Harness Platform when prompted for the connectivity mode.

Set up your pipeline

Do the following:

  1. Select Security Testing Orchestration (left menu, top) > Pipelines > Create a Pipeline. Enter a name and click Start.

  2. In the new pipeline, select Add stage > Build.

  3. Set up your stage as follows:

    1. Enter a Stage Name.

    2. Select Third-party Git provider and then select your GitHub connector.

    3. For Repository Name, click the type selector ("tack" button on the right) and then select Runtime Expression.

  4. Expand Overview > Advanced and add the following stage variables.

    You'll be specifying runtime inputs for some of these variables. This enables you to specify the code repo, branch, image label, and image tag, and other variables at runtime.

    • GITHUB_USERNAME — Select Secret as the type and enter your GitHub login name.

    • GITHUB_REPO — Select String for the type and Runtime Input for the value (click the "tack button" to the right of the value field).

    • GITHUB_BRANCH — Select String and Runtime Input.

    • DOCKERHUB_USERNAME — Select String as the type and enter your DockerHub login name.

    • DOCKER_IMAGE_LABEL — Select String and Runtime Input.

    • DOCKER_IMAGE_TAG — Select String and Runtime Input.

  5. In the Pipeline Editor, go to Infrastructure and select Cloud, Linux, and AMD64 for the infrastructure, OS, and architecture.

    You can also use a Kubernetes or Docker build infrastructure, but these require additional work to set up. For more information, go to Set up a build infrastructure for STO.

note

The following step is required for Kubernetes or Docker infrastructures only. If you're using Harness Cloud, go to Scan the code.

Add a Docker-in-Docker background step

The following use cases require a Docker-in-Docker background step in your pipeline:

  • Container image scans on Kubernetes and Docker build infrastructures
  • Custom Scan steps on Kubernetes and Docker build infrastructures
    • Required for all target types and Orchestration/DataLoad modes

The following use cases do not require a Docker-in-Docker background step:

  • Harness Cloud AMD64 build infrastructures
  • SAST/DAST/configuration scans that use a scanner-specific step and not a Custom Scan step.
  • Ingestion scans where the data file has already been generated
Set up a Docker-in-Docker background step
  1. Go to the stage where you want to run the scan.

  2. In Overview, add the shared path /var/run.

  3. In Execution, do the following:

    1. Click Add Step and then choose Background.
    2. Configure the Background step as follows:
      1. Dependency Name = dind

      2. Container Registry = The Docker connector to download the DinD image. If you don't have one defined, go to Docker connector settings reference.

      3. Image = docker:dind

      4. Under Entry Point, add the following: dockerd

        In most cases, using dockerd is a faster and more secure way to set up the background step. For more information, go to the TLS section in the Docker quick reference.

        If the DinD service doesn't start with dockerd, clear the Entry Point field and then run the pipeline again. This starts the service with the default entry point.

      5. Under Additional Configuration, select the Privileged checkbox.

Configure the background step

Scan the code

  1. In the Pipeline Studio, go to Execution and add a Bandit step to your pipeline.

  2. Configure the step as follows:

    1. Scan Mode = Orchestration

    2. Target name — Click the value-type selector (tack button to the right of the input field) and select Expression. Then enter the following expression: <+stage.variables.GITHUB_REPO>

    3. Target variant — Select Expression as the value type and enter the following: <+stage.variables.GITHUB_BRANCH>

      When scanning a code repo, you generally want to specify the repo name and branch for the target.

info

In most cases, you want to set the Fail on Severity when you add a scan step to your pipeline. In this tutorial, you're setting the first of two scan steps. In this case, you can keep Fail on Severity set to None and set it after you finish setting up and testing your pipeline.

Analyze the results

At this point, you might want to run a scan and view the detected issues.

  1. Select Save, and then select Run.

  2. In Run Pipeline, enter the repository and target settings you want to use. If you're using the dvpwa repository, enter the following:

    1. Under Codebase:
      • Repository name : dvpwa
      • Branch name : master
    2. Under Stage:
      • Target name : dvpwa (= the repo name)
      • Target variant : master (= the branch name)
  3. Run the pipeline. When the execution finishes, select Security Tests to view the scan results.

Build and push a test image

Harness CI includes a set of Build and Push steps that take a code repo with a Dockerfile, build a container image, and push it to an artifact repository.

You'll now add one of these steps to build and push to your Docker Hub account.

  1. Add a Build and Push to Docker Registry step after the Bandit step.

  2. Configure the step as follows:

    1. Name = build_push_test_image

    2. Docker Connector — Select your Docker Hub connector.

    3. Docker Repository — Select Expression for the value type, then enter the following:

      <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>

    4. Tags— Select Expression for the value type, then enter the following:

      <+stage.variables.DOCKER_IMAGE_TAG>-scantest-DONOTUSE

Scan the test image

Add an Aqua Trivy step to your pipeline after the build step and configure it as follows:

  1. Scan Mode = Orchestration In orchestrated mode, the step runs the scan and ingests the results in one step.

  2. Target name — Click the "tack" button on the right side of the input field and select Expression. Then enter the following expression: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>

  3. Target variant — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKER_IMAGE_TAG>-scantest-DONOTUSE

  4. Container image Type = V2

  5. Domain = docker.io

  6. Container image name — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>

  7. Container image tag — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKER_IMAGE_TAG>

  8. Fail on Severity = Critical

Run the pipeline and verify your results

This is a good time to run your pipeline and verify that it can scan the image.

  1. Click Run and set the GitHub and Docker variables.

    If you forked the dvpwa repository repo into your GitHub account and want to use that, set the fields like this:

    • GITHUB_REPO = dvpwa
    • GITHUB_BRANCH= master
    • DOCKER_IMAGE_LABEL = dvpwa
    • DOCKER_IMAGE_TAG = 1.

    With this setup, you'll build and push an image with a tag that looks like this: 1.x-scantest-DONOTUSE

    tip

    Input sets enable you to reuse a single pipeline for multiple scenarios. You can define each scenario in an input set and then select the relevant input set at runtime. To save these inputs, click Save as New Input Set.

  2. Click Run Pipeline and view the results in Security Tests.

Build and push the prod image

Assuming that the Trivy scan detected no critical vulnerabilities, you can now build and push a prod version of your image to Docker Hub. This step is identical to the previous test image step, except for the image tag: the test image tag is 1.x-scantest-DONOTUSE and the prod image tag is 1.x.

  1. Add a Build and Push to Docker Registry step after the Semgrep ingest step.

  2. Configure the step as follows:

    1. Name = build_push_test_image

    2. Docker Connector — Select your Docker Hub connector.

    3. Container image name — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>

    4. Container image tag — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKER_IMAGE_TAG><+pipeline.sequenceID>

YAML pipeline example

Here's an example of the pipeline you created in this tutorial. If you copy this example, replace the placeholder values with appropriate values for your project, organization, connectors, and access token.

pipeline:
name: sto-ci-tutorial-test
identifier: stocitutorialtest
projectIdentifier: YOUR_HARNESS_PROJECT_ID
orgIdentifier: YOUR_HARNESS_ORGANIZATION_ID
tags: {}
properties:
ci:
codebase:
connectorRef: YOUR_CODEBASE_CONNECTOR_ID
repoName: <+input>
build: <+input>
stages:
- stage:
name: build_scan_push_with_ci
identifier: build_scan_push_with_ci
description: ""
type: CI
spec:
cloneCodebase: true
platform:
os: Linux
arch: Amd64
runtime:
type: Cloud
spec: {}
execution:
steps:
- step:
type: Bandit
name: scan_python_code
identifier: scan_python_code
spec:
mode: orchestration
config: default
target:
name: <+stage.variables.GITHUB_REPO>
type: repository
variant: <+stage.variables.GITHUB_BRANCH>
advanced:
log:
level: info
- step:
type: BuildAndPushDockerRegistry
name: build_push_test_image
identifier: build_push_test_image
spec:
connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR
repo: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>
tags:
- <+stage.variables.DOCKER_IMAGE_TAG><+pipeline.sequenceId>-scantest-DONOTUSE
- step:
type: AquaTrivy
name: scan_test_image
identifier: scan_test_image
spec:
mode: orchestration
config: default
target:
name: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>
type: container
variant: <+stage.variables.DOCKER_IMAGE_TAG><+pipeline.sequenceId>
advanced:
log:
level: info
privileged: true
image:
type: docker_v2
name: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>
domain: docker.io
access_token: <+secrets.getValue("YOUR_IMAGE_REGISTRY_ACCESS_TOKEN")
tag: <+stage.variables.DOCKER_IMAGE_TAG><+pipeline.sequenceId>-scantest-DONOTUSE
sbom:
format: spdx-json
- step:
type: BuildAndPushDockerRegistry
name: build_push_prod_image
identifier: build_push_prod_image
spec:
connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR
repo: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>
tags:
- <+stage.variables.DOCKER_IMAGE_TAG><+pipeline.sequenceId>
timeout: 30m
variables:
- name: GITHUB_REPO
type: String
description: ""
required: true
value: <+input>
- name: GITHUB_BRANCH
type: String
description: ""
required: false
value: <+input>
- name: DOCKERHUB_USERNAME
type: String
description: ""
required: false
value: myusername
- name: _IMAGE_LABEL
type: String
description: ""
required: false
value: <+input>
- name: DOCKER_IMAGE_TAG
type: String
description: ""
required: false
value: <+input>