Argo Workflows

https://argo-workflows.readthedocs.io/en/

Argo Workflows is a kubernetes-native engine for orchestrating parallel jobs on Kubernetes. You can define complex multi-step workflows as a sequence of tasks with each step running as a pod in the cluster.

Installation Optinos

https://argo-workflows.readthedocs.io/en/latest/installation/
There are 3 installation options for Argo Workflows:
- A cluster install will watch and execute workflows in all namespaces. This is the default installation option when installing using the official release manifests.
  - the controller manages all workflows across the entire cluster
- A namespace install only executes workflows in the namespace it is installed in (typically argo). Look for namespace-install.yaml in the release assets.
- A managed namespace install: only executes workflows in a separate namespace from the one it is installed in
Argo Workflows can be installed with helm

Architecture

Argo namespace contains the argo api server, etc.
The workflow controller watches workflow resources
The user namespace is where the actual workflow pods run

Argo CLI

Install: https://argo-workflows.readthedocs.io/en/latest/walk-through/argo-cli/

Core Concepts

A Workflow is both the execution definition and the live record of that execution’s state.
spec.entrypoint selects the template that starts the workflow; spec.templates define the work and how it is orchestrated.
Templates are broadly either:
- Work templates: container, script, resource, suspend, plugin, containerSet, and http.
- Invocators: steps and dag, which call other templates and control execution order.
A steps template uses nested lists: each outer list runs after the previous one, while steps in the same inner list run in parallel.
A dag template expresses dependencies between tasks. Tasks with no dependencies can start immediately; dependent tasks wait for their prerequisites. For example, tasks B and C can run in parallel after A, and D can wait for both B and C.
Container and script standard output is exposed as a workflow result variable, such as {{steps.<NAME>.outputs.result}} or {{tasks.<NAME>.outputs.result}}, depending on whether it was invoked by steps or a DAG.
See the Workflow Concepts documentation for examples and details.

Argo Workflow Specification

https://argo-workflows.readthedocs.io/en/latest/default-workflow-specs/
Default Workflow spec values can be set on the controller config map that will apply to all Workflows executed from said controller. Default values are most useful for config-related fields that you want to repeat across all Workflows, such as garbage collection. If a Workflow has a value that also has a default value set in the config map, the Workflow’s value will take precedence.

Workflows are installed as CRDs in Kubernetes

  apiVersion: argoproj.io/v1alpha1
  kind: Workflow
  metadata:
    generateName: test-workflow- # prefix for generated name
    annotations:
      argo: workflows
    labels:
      foo: bar
  spec:
    entryPoint: cowsay
    templates:
    - name: cowsay
      container: 
        image: rancher/cowsay:latest
        command: [cowsay]
        args: ["Argo Workflows!!!"]
    ttlStrategy:
      secondsAfterSuccess: 5     # Time to live after workflow is successful
    parallelism: 3

Templates

Templates can be thought of as functions. They contain code that is to be executed.

Template Types

There are 9 types of templates divided into 2 categories: Template Definitions and Template Invocators
9 types:
- Template Definitions:
  - Container - The most commonly used type. Defines a container to be scheduled. Conforms to the Kubernetes container spec
  - Script - A convenience wrapper around a container. You can define a script to execute
  - Resource - Define a kubernetes resource to be managed by Argo Workflow (for example, a configmap)
  - Suspend - A suspend template used to suspend execution of the workflow
  - ContainerSet - A set of containers
  - HTTP - Can execute HTTP requests
  - Plugin - random plugins (3rd party, etc.)
- Template Invocators
  - Steps
  - DAG
    - ‘Directed Acyclic Graph’
    - DAGs flow forward only, there are no cycles
    - Think of a fan-out/fan-in workflow

Outputs

Output parameters provide the mechanism to use the output of step in another step

  apiVersion: argoproj.io/v1alpha1
  kind: Workflow
  metadata:
    generateName: outputs-
    namespace: argo
  spec:
    entrypoint: main
    templates:
      - name: main
        steps:
          - - name: generate
              template: generate
          - - name: consume
              template: consume
              arguments:
                parameters:
                  - name: value
                    value: "{{steps.generate.outputs.parameters.result}}"

Workflow Templates

Workflow templates are like a reusable recipe for workflows.
You must first define the template:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: greeter
  namespace: argo
spec:
  templates:
    - name: greet
      inputs:
        parameters:
          - name: name
            value: world
      container:
        image: alpine:3.20
        command: [echo]
        args: ["hello {{inputs.parameters.name}}"]

Then use the template:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: use-template-
  namespace: argo
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: greet
            templateRef:
              name: greeter
              template: greet
            arguments:
              parameters: [{name: name, value: ryan}]

Artifacts

When running workflows, it is very common to have steps that generate or consume artifacts. Often, the output artifacts of one step may be used as input artifacts to a subsequent step.
https://argo-workflows.readthedocs.io/en/latest/walk-through/artifacts/
example:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-passing-
spec:
  entrypoint: artifact-example
  templates:
  - name: artifact-example
    steps:
    - - name: generate-artifact
        template: hello-world-to-file
    - - name: consume-artifact
        template: print-message-from-file
        arguments:
          artifacts:
          # bind message to the hello-art artifact
          # generated by the generate-artifact step
          - name: message
            from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"

  - name: hello-world-to-file
    container:
      image: busybox
      command: [sh, -c]
      args: ["echo hello world | tee /tmp/hello_world.txt"]
    outputs:
      artifacts:
      # generate hello-art artifact from /tmp/hello_world.txt
      # artifacts can be directories as well as files
      - name: hello-art
        path: /tmp/hello_world.txt

  - name: print-message-from-file
    inputs:
      artifacts:
      # unpack the message input artifact
      # and put it at /tmp/message
      - name: message
        path: /tmp/message
    container:
      image: alpine:3.23
      command: [sh, -c]
      args: ["cat /tmp/message"]

Tell argo to use the artifact repo by updating this configmap:

data:
  artifactRepository: |
    s3:
      bucket: my-bucket
      keyFormat: prefix/in/bucket     #optional
      endpoint: my-minio-endpoint.default:9000        #AWS => s3.amazonaws.com; GCS => storage.googleapis.com
      insecure: true                  #omit for S3/GCS. Needed when minio runs without TLS
      accessKeySecret:                #omit if accessing via AWS IAM
        name: my-minio-cred
        key: accessKey
      secretKeySecret:                #omit if accessing via AWS IAM
        name: my-minio-cred
        key: secretKey
      useSDKCreds: true               #tells argo to use AWS SDK's default provider chain, enable for things like IRSA support
      addressingStyle: "virtual-hosted"

WithItems

This iterates over a list of items with withItems, substituting a string for each instantiated template:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-
spec:
  entrypoint: loop-example
  templates:
  - name: loop-example
    steps:
    - - name: print-message-loop
        template: print-message
        arguments:
          parameters:
          - name: message
            value: "{{item}}"
        withItems:              # invoke print-message once for each item in parallel
        - hello world           # item 1
        - goodbye world         # item 2

  - name: print-message
    inputs:
      parameters:
      - name: message
    container:
      image: busybox
      command: [echo]
      args: ["{{inputs.parameters.message}}"]

Keyboard shortcuts

notebook