ECS task stop

ECS task stop is an AWS fault that injects chaos to stop the ECS tasks based on the services or task replica ID and checks the task availability.

• This fault results in the unavailability of the application running on the tasks.
• This experiment is applicable to both serverless ECS tasks and those backed by EC2 instances. These experiments generally involve task-level chaos or access restrictions without causing direct in-container or in-VM disruptions.

Use cases

This fault determines the resilience of an application when ECS tasks unexpectedly stop due to task being unavailable.

Prerequisites

• Kubernetes >= 1.17
• Sufficient AWS access to stop the ECS tasks.
• The target ECS tasks should be in a healthy state.
• Kubernetes secret that has the AWS access configuration (key) in the CHAOS_NAMESPACE. Below is a sample secret file:

apiVersion: v1
kind: Secret
metadata:
  name: cloud-secret
type: Opaque
stringData:
  cloud_config.yml: |-
    # Add the cloud AWS credentials respectively
    [default]
    aws_access_key_id = XXXXXXXXXXXXXXXXXXX
    aws_secret_access_key = XXXXXXXXXXXXXXX

tip

HCE recommends that you use the same secret name, that is, cloud-secret. Otherwise, you will need to update the AWS_SHARED_CREDENTIALS_FILE environment variable in the fault template with the new secret name and you won't be able to use the default health check probes.

Below is an example AWS policy to help execute the fault.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ecs:ListServices",
                "ecs:ListTasks",
                "ecs:StopTask",
                "ecs:DescribeServices",
                "ecs:DescribeTasks"
            ],
            "Resource": "*"
        }
    ]
}

note

• Refer to AWS Named Profile For Chaos to know how to use a different profile for AWS faults.
• Refer to the superset permission (or policy) to execute all AWS faults.
• Refer to the common attributes and AWS-specific tunables to tune the common tunables for all faults and aws specific tunables.

Mandatory tunables

Tunable	Description	Notes
CLUSTER_NAME	Name of the target ECS cluster.	For example, cluster-1.
REGION	Region name of the target ECS cluster.	For example, us-east-1.
SERVICE_NAME	Target ECS service name.	For example, app-svc. For more information, go to ECS service name.
TASK_REPLICA_ID	Comma-separated target task replica IDs.	SERVICE_NAME and TASK_REPLICA_ID are mutually exclusive. If both the values are provided, SERVICE_NAME takes precedence. For more information, go to ECS task replica ID.

Optional tunables

Tunable	Description	Notes
TOTAL_CHAOS_DURATION	Duration to insert chaos (in seconds).	Defaults: 30s. For more information, go to duration of the chaos.
CHAOS_INTERVAL	Time interval between two successive instance terminations (in seconds).	Default: 30s. For more information, go to chaos interval.
TASK_REPLICA_AFFECTED_PERC	Percentage of total tasks that are targeted.	Default: 100. For more information, go to ECS task replica affected percentage.
SEQUENCE	Sequence of chaos execution for multiple instances.	Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution.
RAMP_TIME	Period to wait before and after injecting chaos (in seconds).	For example, 30 s. For more information, go to ramp time.
AWS_SHARED_CREDENTIALS_FILE	Path to the AWS secret credentials.	Defaults to /tmp/cloud_config.yml.

ECS service name

Service name whose tasks are stopped. Tune it by using the SERVICE_NAME environment variable.

The following YAML snippet illustrates the use of this environment variable:

# stop the tasks of an ECS cluster
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ecs-task-stop
    spec:
      components:
        env:
        # provide the name of ECS cluster
        - name: CLUSTER_NAME
          value: 'demo'
        - name: SERVICE_NAME
          vale: 'test-svc'
        - name: REGION
          value: 'us-east-1'
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'

ECS task replica IDs

Task replicas that have a specific ID which are to be stoppee. Tune it by using the TASK_REPLICA_ID environment variable.

The following YAML snippet illustrates the use of this environment variable:

# stop the tasks of an ECS cluster
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ecs-task-stop
    spec:
      components:
        env:
        # provide the name of ECS cluster
        - name: CLUSTER_NAME
          value: 'demo'
        - name: TASK_REPLICA_ID
          vale: '1b751cf956e34e54b9d83b6a5c067f60,20d5041c044941dfb2126f1722d10558'
        - name: REGION
          value: 'us-east-1'
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'

ECS task replica affected percentage

Number of tasks to target (in percentage). Tune it by using the TASK_REPLICA_AFFECTED_PERC environment variable.

The following YAML snippet illustrates the use of this environment variable:

# stop the tasks of an ECS cluster
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ecs-task-stop
    spec:
      components:
        env:
        # provide the name of ECS cluster
        - name: CLUSTER_NAME
          value: 'demo'
        - name: SERVICE_NAME
          vale: 'test-svc'
        - name: TASK_REPLICA_AFFECTED_PERC
          vale: '100'
        - name: REGION
          value: 'us-east-1'
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'