Skip to main content

Performance test reports

This topic includes a series of test reports focusing on the performance and scalability of Harness Self-Managed Enterprise Edition. The reports provide detailed insights into the test environment, resource configurations, test scenarios, and their outcomes.

The test reports offer valuable insights into the performance, scalability, and reliability of Harness Self-Managed Enterprise Edition. They demonstrate the platform's capability to handle concurrent CI/CD executions effectively across different scenarios and workload conditions. The detailed results and observations serve as a reference for evaluating the platform's suitability for various deployment environments and project requirements.

Test environment

The test reports document the environment in which the performance tests were conducted. This includes details such as the Kubernetes version used (GKE), the database configuration (Mongo Atlas M60), and the Redis setup (GCP Memory Store).

Harness services

The reports outline the various services utilized within the Harness Self-Managed Enterprise Edition, deployed using Helm charts. Each service's configuration, including the number of replicas and resource allocations per replica (CPU and memory), is provided for a comprehensive understanding of the setup.

Test scenarios

Concurrent CI executions

Concurrent CI executions are initiated with specific steps, including initializing Kubernetes pods, cloning repositories, and executing parallel steps with sleep intervals. These tests evaluate the platform's ability to handle high loads of CI executions efficiently.

Concurrent CD executions

Concurrent CD executions involve fetching Docker artifacts from external repositories and executing sequential steps, including Canary deploy, Canary delete, Rolling deploy, and Kubernetes Delete. These tests focus on the platform's ability to orchestrate complex deployment workflows under heavy loads.

Performance test reports

Harness publishes performance test reports with each release. Select a report below to view report details.

2024 reports

June 5, 2024

This report details information about the following:

  1. Test environment and resource configuration
  2. Test scenario and results

Environment

  • GKE (Kubernetes Version): 1.27.x

Database

  • Mongo Atlas M60

Redis

  • GCP Memory Store (11 GB)

Harness services

Helm chart

Service NameReplicasCPU (per replica)Memory (per replica)Version
access-control415harness-0.15.0
ci-manager436harness-0.15.0
pipeline-service7410harness-0.15.0
manager7312harness-0.15.0
log-service3312harness-0.15.0
ng-manager626harness-0.15.0
scm20.51harness-0.15.0
gateway514harness-0.15.0
default-backend10.10.2harness-0.15.0
nginx-ingress-controller1510harness-0.15.0
change-data-capture146harness-0.15.0
next-gen-ui20.50.5harness-0.15.0
ng-auth-ui20.10.1harness-0.15.0
platform-service20.53harness-0.15.0
template-service218harness-0.15.0
sto-core40.51.5harness-0.15.0
sto-manager236harness-0.15.0
ui30.10.5harness-0.15.0
policy-mgmt30.31harness-0.15.0
timescaledb212harness-0.15.0
ng-dashboard-aggregator20.252harness-0.15.0

Override file

override-perf.yaml

Test scenarios

> 2500 concurrent CI Executions INLINE

Each CI pipeline does the following:

  • Initializes a Kubernetes pod and Git clone repo
  • Runs 5 parallel steps (100 sec sleep)
  • Runs template with 2 parallel steps (140sec sleep)

Projects: 1 Pipelines: 2500 Stages per pipeline: 1 Delegates: 15 (1cpu/4gi) Trigger type: webhook Test class: CI_PIPELINE_WEBHOOK_RUN

Result: PASS Avg Execution Time: 6.34min

> 2000 concurrent CD Executions INLINE

Each CD pipeline does the following:

  • Fetches a Docker artifact from an AWS ECR repo
  • Runs the following steps in order:
    • Canary deploy
    • Canary delete
    • Rolling deploy
    • K8s Delete

Projects: 1 Pipelines: 2000 Stages per pipeline: 1 Delegates: 50 (1cpu/4gi) Test class: CD_PIPELINE_RUN

Result: PASS Avg Execution Time: 5.46min

April 30, 2024

This report details information about the following:

  1. Test environment and resource configuration
  2. Test scenario and results

Environment

  • GKE (Kubernetes Version): 1.26.x

Database

  • Mongo Atlas M60

Redis

  • GCP Memory Store (11 GB)

Harness services

Helm chart

Service NameReplicasCPU (per replica)Memory (per replica)Version
access-control415harness-0.15.0
ci-manager436harness-0.15.0
pipeline-service7410harness-0.15.0
manager7312harness-0.15.0
log-service3312harness-0.15.0
ng-manager626harness-0.15.0
scm20.51harness-0.15.0
gateway514harness-0.15.0
default-backend10.10.2harness-0.15.0
nginx-ingress-controller1510harness-0.15.0
change-data-capture146harness-0.15.0
next-gen-ui20.50.5harness-0.15.0
ng-auth-ui20.10.1harness-0.15.0
platform-service20.53harness-0.15.0
template-service218harness-0.15.0
sto-core40.51.5harness-0.15.0
sto-manager236harness-0.15.0
ui30.10.5harness-0.15.0
policy-mgmt30.31harness-0.15.0
timescaledb212harness-0.15.0
ng-dashboard-aggregator20.252harness-0.15.0

Override file

override-perf.yaml

Test scenarios

> 2500 concurrent CI Executions INLINE

Each CI pipeline does the following:

  • Initializes a Kubernetes pod and Git clone repo
  • Runs 5 parallel steps (100 sec sleep)
  • Runs template with 2 parallel steps (140sec sleep)

Projects: 1 Pipelines: 2500 Stages per pipeline: 1 Delegates: 15 (1cpu/4gi) Trigger type: webhook Test class: CI_PIPELINE_WEBHOOK_RUN

Result: PASS Avg Execution Time: 6.45min

> 2000 concurrent CD Executions INLINE

Each CD pipeline does the following:

  • Fetches a Docker artifact from an AWS ECR repo
  • Runs the following steps in order:
    • Canary deploy
    • Canary delete
    • Rolling deploy
    • K8s Delete

Projects: 1 Pipelines: 2000 Stages per pipeline: 1 Delegates: 50 (1cpu/4gi) Test class: CD_PIPELINE_RUN

Result: PASS Avg Execution Time: 5.20min

April 1, 2024

This report details information about the following:

  1. Test environment and resource configuration
  2. Test scenario and results

Environment

  • GKE (Kubernetes Version): 1.26.x

Database

  • Mongo Atlas M60

Redis

  • GCP Memory Store (11 GB)

Harness services

Helm chart

Service NameReplicasCPU (per replica)Memory (per replica)Version
access-control415harness-0.14.6
ci-manager436harness-0.14.6
pipeline-service7410harness-0.14.6
manager7312harness-0.14.6
log-service3312harness-0.14.6
ng-manager526harness-0.14.6
scm20.51harness-0.14.6
gateway514harness-0.14.6
default-backend10.10.2harness-0.14.6
nginx-ingress-controller1510harness-0.14.6
change-data-capture146harness-0.14.6
next-gen-ui20.50.5harness-0.14.6
ng-auth-ui20.10.1harness-0.14.6
platform-service20.53harness-0.14.6
template-service218harness-0.14.6
ti-service216harness-0.14.6
sto-core40.51.5harness-0.14.6
sto-manager236harness-0.14.6
gitops222harness-0.14.6
ui30.10.5harness-0.14.6
policy-mgmt30.31harness-0.14.6
timescaledb212harness-0.14.6
verification-svc20.34harness-0.14.6
ng-dashboard-aggregator20.252harness-0.14.6

Override file

override-perf.yaml

Test scenarios

> 2000 concurrent CI Executions INLINE

Each CI pipeline does the following:

  • Initializes a Kubernetes pod and Git clone repo
  • Runs 5 parallel steps (100 sec sleep)
  • Runs template with 2 parallel steps (140sec sleep)

Projects: 1 Pipelines: 2000 Stages per pipeline: 1 Delegates: 15 (1cpu/4gi) Trigger type: webhook Test class: CI_PIPELINE_WEBHOOK_RUN

Result: PASS Avg Execution Time: 6.45min

> 2000 concurrent CD Executions INLINE

Each CD pipeline does the following:

  • Fetches a Docker artifact from an AWS ECR repo
  • Runs the following steps in order:
    • Canary deploy
    • Canary delete
    • Rolling deploy
    • K8s Delete

Projects: 1 Pipelines: 2000 Stages per pipeline: 1 Delegates: 47 (1cpu/4gi) Test class: CD_PIPELINE_RUN

Result: PASS Avg Execution Time: 5.20min

February 29, 2024

This report details information about the following:

  1. Test environment and resource configuration
  2. Test scenario and results

Environment

  • GKE (Kubernetes Version): 1.26.x

Database

  • Mongo Atlas M60

Redis

  • GCP Memory Store (11 GB)

Harness services

Helm chart : https://github.com/harness/helm-charts/releases/tag/harness-0.13.4

Service NameReplicasCPU (per replica)Memory (per replica)Version
access-control415harness-0.13.4
ci-manager436harness-0.13.4
pipeline-service7410harness-0.13.4
manager7312harness-0.13.4
log-service3312harness-0.13.4
ng-manager526harness-0.13.4
scm20.51harness-0.13.4
gateway514harness-0.13.4
default-backend10.10.2harness-0.13.4
nginx-ingress-controller1510harness-0.13.4
change-data-capture146harness-0.13.4
next-gen-ui20.50.5harness-0.13.4
ng-auth-ui20.10.1harness-0.13.4
platform-service20.53harness-0.13.4
template-service218harness-0.13.4
ti-service216harness-0.13.4
sto-core40.51.5harness-0.13.4
sto-manager236harness-0.13.4
gitops222harness-0.13.4
ui30.10.5harness-0.13.4
policy-mgmt30.31harness-0.13.4
timescaledb212harness-0.13.4
verification-svc20.34harness-0.13.4
ng-dashboard-aggregator20.252harness-0.13.4

Override file

override-perf.yaml

Test scenarios

> 2000 concurrent CI Executions INLINE

Each CI pipeline does the following:

  • Initializes a Kubernetes pod and git clone repo
  • Runs 5 parallel steps (100 sec sleep)
  • Runs template with 2 parallel steps (140sec sleep)

Projects: 1 Pipelines: 2000 Stages per pipeline: 1 Delegates: 15 (1cpu/4gi) Trigger type: webhook Test class: CI_PIPELINE_WEBHOOK_RUN

Result : PASS Avg Execution Time: 6.5min

> 1500 concurrent CD Executions INLINE

Each CD pipeline does the following:

  • Fetches a Docker artifact from an AWS ECR repo
  • Runs the following steps in order:
    • Canary deploy
    • Canary delete
    • Rolling deploy
    • K8s Delete

Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 40 (1cpu/4gi) Test class: CD_PIPELINE_RUN

Result: PASS Avg Execution Time: 5.1min

January 31, 2024

This report details information about the following:

  1. Test environment and resource configuration
  2. Test scenario and results

Environment

  • GKE (Kubernetes Version): 1.25.x

Database

  • Mongo Atlas M60

Redis

  • GCP Memory Store (11 GB)

Harness services

Helm chart : https://github.com/harness/helm-charts/releases/tag/harness-0.13.0

Service NameReplicasCPU (per replica)Memory (per replica)Version
access-control415harness-0.13.0
ci-manager436harness-0.13.0
pipeline-service7410harness-0.13.0
manager7312harness-0.13.0
log-service3312harness-0.13.0
ng-manager526harness-0.13.0
scm20.51harness-0.13.0
gateway514harness-0.13.0
default-backend10.10.2harness-0.13.0
nginx-ingress-controller1510harness-0.13.0
change-data-capture146harness-0.13.0
next-gen-ui20.50.5harness-0.13.0
ng-auth-ui20.10.1harness-0.13.0
platform-service20.53harness-0.13.0
template-service218harness-0.13.0
ti-service216harness-0.13.0
sto-core40.51.5harness-0.13.0
sto-manager236harness-0.13.0
gitops222harness-0.13.0
ui30.10.5harness-0.13.0
policy-mgmt30.31harness-0.13.0
timescaledb212harness-0.13.0
verification-svc20.34harness-0.13.0
ng-dashboard-aggregator20.252harness-0.13.0

Override file

override-perf.yaml

Test scenarios

> 2000 concurrent CI Executions INLINE

Each CI pipeline does the following:

  • Initializes a Kubernetes pod and Git clone repo
  • Runs 5 parallel steps (100 sec sleep)
  • Runs template with 2 parallel steps (140sec sleep)

Projects: 1 Pipelines: 2000 Stages per pipeline: 1 Delegates: 15 (1cpu/4gi) Trigger type: webhook Test class: CI_PIPELINE_WEBHOOK_RUN

Result : PASS Avg Execution Time: 6.5min

> 1500 concurrent CI Executions GitX

Each CI pipeline does the following:

  • Initializes a Kubernetes pod and git clone repo
  • Runs 5 parallel steps (360 sec sleep) and echo statements

Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 15 (1cpu/2gi) Trigger type: webhook Test class: CI_PIPELINE_REMOTE_RUN

Result: PASS Avg Execution Time: 8.5min

> 1000 concurrent CD Executions INLINE

Each CD pipeline does the following:

  • Fetches a Docker artifact from an AWS ECR repo
  • Runs the following steps in order:
    • Canary deploy
    • Canary delete
    • Rolling deploy
    • K8s Delete

Projects: 1 Pipelines: 1000 Stages per pipeline: 1 Delegates: 26 (1cpu/4gi) Test class: CD_PIPELINE_RUN

Result: PASS Avg Execution Time: 4.5min

2023 reports

December 28, 2023

This report details information about the following:

  1. Test environment and resource configuration
  2. Test scenario and results

Environment

  • GKE (Kubernetes Version): 1.25.x

Database

  • Mongo Atlas M60

Redis

  • GCP Memory Store (5GB)

Harness services

Helm chart

Service NameReplicasCPU (per replica)Memory (per replica)Version
access-control515harness-0.11.2
ci-manager636harness-0.11.2
pipeline-service10412harness-0.11.2
manager6312harness-0.11.2
log-service3312harness-0.11.2
ng-manager438harness-0.11.2
scm20.51harness-0.11.2
gateway226harness-0.11.2
default-backend10.10.2harness-0.11.2
nginx-ingress-controller2510harness-0.11.2
change-data-capture145harness-0.11.2
next-gen-ui20.50.5harness-0.11.2
ng-auth-ui20.10.1harness-0.11.2
platform-service20.53harness-0.11.2
template-service218harness-0.11.2
ti-service136harness-0.11.2
sto-core10.50.75harness-0.11.2
sto-manager136harness-0.11.2
gitops122harness-0.11.2
ui10.50.5harness-0.11.2
policy-mgmt10.50.5harness-0.11.2
timescaledb212harness-0.11.2

Override file

override-perf-ci-cd-ff.yaml

Manager config : update LOG_STREAMING_SERVICE_EXTERNAL_URL = <smp host url>/log-service/

Test scenarios

> 1800 concurrent CI Executions INLINE

Each CI pipeline does the following:

  • Initializes a Kubernetes pod and Git clone repo
  • Runs 5 parallel steps (100 sec sleep)
  • Runs template with 2 parallel steps (140sec sleep)

Projects: 1 Pipelines: 1800 Stages per pipeline: 1 Delegates: 15 (1cpu/2gi) trigger: webhook

Result: PASS Total Execution Time: 7.2min

> 1500 concurrent CI Executions GitX

Each CI pipeline does the following:

  • Initializes a Kubernetes pod and Git clone repo
  • Runs 5 parallel steps (360 sec sleep) and echo statements

Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 15 (1cpu/2gi) trigger: webhook

Result: PASS Total Execution Time: 10.3min

> 1000 concurrent CD Executions INLINE

Each CD pipeline:

  • Fetches a Docker artifact from AWS ECR repo
  • Runs the following steps in order:
    • Canary deploy
    • Canary delete
    • Rolling deploy
    • K8s Delete

Projects: 1 Pipelines: 1000 Stages per pipeline: 1 Delegates: 27 (1cpu/4gi)

Result: PASS Total Execution Time: 4.5min

October 27, 2023

This report details information about the following:

  1. Test environment and resource configuration
  2. Test scenario and results

Environment

  • GKE (Kubernetes Version): 1.26.x

Database

  • Mongo Atlas M60

Redis

  • GCP Memory Store (5GB)

Harness services

Helm chart

Service NameReplicasCPU (per replica)Memory (per replica)Version
access-control515harness-0.9.2
ci-manager636harness-0.9.2
pipeline-service10412harness-0.9.2 + fixes
(to be released in 0.11)
manager6312harness-0.9.2 + fixes
(to be released in 0.10)
log-service11024harness-0.9.2
ng-manager438harness-0.9.2 + fixes
(to be released in 0.10)
scm20.51harness-0.9.2
gateway226harness-0.9.2
default-backend10.10.2harness-0.9.2
nginx-ingress-controller2510harness-0.9.2
change-data-capture145harness-0.9.2
next-gen-ui20.50.5harness-0.9.2
ng-auth-ui20.10.1harness-0.9.2
platform-service22.53harness-0.9.2
template-service218harness-0.9.2 + fixes
(to be released in 0.11)
ti-service136harness-0.9.2
sto-core10.50.75harness-0.9.2
sto-manager136harness-0.9.2
gitops122harness-0.9.2
ui10.50.5harness-0.9.2
policy-mgmt10.50.5harness-0.9.2
timescaledb212harness-0.9.2

Override file

override-perf-ci-cd-ff.yaml

Manager config

update LOG_STREAMING_SERVICE_EXTERNAL_URL = <smp host url>/log-service/

Test scenarios

> 1500 concurrent CI executions INLINE

Each CI pipeline does the following:

  • Initializes a Kubernetes pod and Git clone repo
  • Runs 5 parallel steps (70 sec sleep)
  • Runs template with 2 parallel steps (140sec sleep)

Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 10 (1cpu/2gi)

Result: PASS Total Execution Time: 6min

> 1500 concurrent CI executions GitX

Each CI pipeline does the following:

  • Initializes a Kubernetes pod and Git clone repo
  • Runs 5 parallel steps (140 sec sleep) and echo statements

Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 10 (1cpu/2gi)

Result: PASS Total Execution Time: 5min

> 500 concurrent CD executions INLINE

Each CD pipeline does the following:

  • Fetches a Docker artifact from an AWS ECR repo
  • Runs the following steps in order:
    • Canary deploy
    • Canary delete
    • Rolling deploy
    • K8s Delete

Projects: 1 Pipelines: 500 Stages per pipeline: 1 Delegates: 18 (1cpu/4gi)

Result: PASS Total Execution Time: 4.5min

October 6, 2023

This report details information about the following:

  1. Test environment and resource configuration
  2. Test scenario and results

Environment

  • GKE (Kubernetes Version): 1.26.x

Database

  • Mongo Atlas M60

Redis

  • GCP Memory Store (5GB)

Harness services

Helm chart

Service NameReplicasCPU (per replica)Memory (per replica)Version
access-control515harness-0.8.2
ci-manager636harness-0.8.2
pipeline-service10412harness-0.8.2 + fixes
(to be released in 0.11)
manager6312harness-0.8.2
log-service11024harness-0.8.2
ng-manager438harness-0.8.2
scm20.51harness-0.8.2
gateway226harness-0.8.2
default-backend10.10.2harness-0.8.2
nginx-ingress-controller2510harness-0.8.2
change-data-capture145harness-0.8.2
next-gen-ui20.50.5harness-0.8.2
ng-auth-ui20.10.1harness-0.8.2
platform-service22.53harness-0.8.2
template-service218harness-0.8.2 + fixes
(to be released in 0.11)
ti-service136harness-0.8.2
sto-core10.50.75harness-0.8.2
sto-manager136harness-0.8.2
gitops122harness-0.8.2
ui10.50.5harness-0.8.2
policy-mgmt10.50.5harness-0.8.2
timescaledb212harness-0.8.2
Override file

override-perf-ci-cd-ff.yaml

Test scenarios

> 1500 concurrent CI executions INLINE

Each CI pipeline does the following:

  • Initializes a Kubernetes pod and Git clone repo
  • Runs 4 parallel steps (70 sec sleep)
  • Runs template with 2 parallel steps (140sec sleep)

Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 10 (1cpu/2gi)

Result: PASS Total Execution Time: 6min

> 1500 concurrent CI executions GitX

Each CI pipeline does the following:

  • Initializes a K8s pod and Git clone repo
  • Runs 5 parallel steps (140 sec sleep) and echo statements

Projects: 1 Pipelines: 1500 Stages per pipeline: 1 Delegates: 10 (1cpu/2gi)

Result: PASS Total Execution Time: 4.2min

> 500 concurrent CD executions INLINE

Each CD pipeline does the following:

  • Fetches a Docker artifact from an AWS ECR repo
  • Runs the following steps in order:
    • Canary deploy
    • Canary delete
    • Rolling deploy
    • K8s Delete

Projects: 1 Pipelines: 500 Stages per pipeline: 1 Delegates: 18 (1cpu/4gi)

Result: PASS Total Execution Time: 4.2min