bomonike

temporal.png

Temporal Technologies (at Temporal.io) provides a scalable “orchestrator” that manages application states within distributed mission-critical workflows, especially a set of long-running asynchronous</stong> services working together.

Temporal’s job is to keep each service to a consistent state.

Temporal’s application state tracking enables automatic retries, timeouts, rollbacks due to process failures.

The system is a part of overall efforts toward organizations assessing their resilience maturity.

Temporal is open-sourced under MIT licensing at https://github.com/temporalio/temporal so a self-hosted local environment can be setup locally instead of running Temporal Services in the Temporal-managed Temporal Cloud. See https://docs.temporal.io/self-hosted-guide

Cloud costs

Temporal makes money by providing a managed AWS cloud service (through AWS PrivateLink networking from 12+ regions) at https://temporal.io/cloud (documentation at https://docs.temporal.io/cloud) VIDEO: building it.

Billing can be AWS Marketplace

Temporal people

Moscow-born and now Bellevue, Washington resident Maxim Fateev, Temporal Co-Founder and CEO, worked at Google and Uber after working on the Amazon Queue Service, where he realized that VIDEO: “queues are not actually not a good way to link services together, if you have complex transactions”.

He built the Temporal team partly with people who created Uber’s Cadence system, such as CTO Sama Abbas, who worked on Microsoft’s Azure Durable Functions.

Head of Product is Ryland Goldstein.

Customers

VIDEO: “Temporal in 7 Minutes - the TL;DR Intro” (in 2022) mentions that Netflix uses Temporal for its Spinniker product. Other users are Stripe, Snap, etc. Customer presentations at Temporal’s 2023 conference include JPMC, Yum! Brands, Twilio, HashiCorp, Instacart. VIDEO: DataDog is “one of the most aggressive users of Temporal”.

Socials

Architecture

These two diagrams summarize the relationship among the concepts (vocabulary) used when working with Temporal:

VIDEO:
temporal-arch-935x713.png

Application code to present GUI and process back-end Business Logic (shown at the upper-left) adopt Temporal by adding among its servers Temporal Client that communicates commands to the Temporal Service (on the upper-right) working in the Cloud or locally.

The term Temporal Service was previously Temporal Cluster”. In productive mode, the Temporal Service still runs in a cluster of several machines for redundancy.

The Temporal Service obtains status information by making Queries to Temporal Clients. A Query is a synchronous operation used to report the state of a Workflow Execution.

Temporal Worker Processes (at the lower-left) execute Activity and Workflow functions, then communicate the results back to the Temporal Server. So each Worker is a wrapper around compiled Workflow and Activity code.

The Temporal Service configuration is the setup and configuration details of a Temporal Service, defined using YAML.

The Temporal Service consists of Persistence and Visibility stores used to track the progress of Workflow function execution.

Similarities

Temporal is like Istio for Kubernetes, which provides a proxy sidecar to handle communications so app developers don’t need to write code in every program to detect and handle failures, and fail gracefully from server crashes or network outages. Temporal allows developers to “write code as if failures don’t even exist” and focus more on business logic.

Unlike Kubernetes, Temporal does not concern itself with containers but with REST/RPC services of any kind (which can be run within a container).

Unlike Apache Airflow, which also offers Workflow as Code, Temporal doesn’t run code but works as external observers. Also Temporal is multi-language while Airflow is Python only. Temporal uses “durable execution” abstraction to specify orchestration while airflow uses a DAG. As durable execution is just code in a top level programming language it is much more powerful, user friendly and supports many more use cases than any DAG based system. BTW you can implement Airflow DAG on top of Temporal if needed. There are dozens of DSLs already implemented on top of it. Also Temporal scales many orders of magnitude better than Airflow.

Concepts and Glossary

VIDEO temporal-svcs-986x818.png</a>

Temporal’s Glossary defines terms such as: Activity Definition, Activity Execution, Advanced Visibility, Child Workflow Execution, Custom Data Converter, List Filters, Search Attributes, Side Effect, Task Queue, Temporal Client, Temporal Web UI, Worker, Workflow Definition, Workflow Execution, Workflow Id, Workflow Id Reuse Policy, Workflow Type

Tutorials

Temporal provides tutorial to explore core terminology and concepts.

In this project, the file worker/main.go contains the code for the Worker.

Workers can only execute Workflows and Activities registered to it. Workers listen only to the Task Queue that it is registered to knows which piece of code to execute from Tasks that it gets from the Task Queue.

Like the program that started the Workflow, it connects to the Temporal Cluster and specifies the Task Queue to use. It also registers the Workflow and the three Activities.

Observability

Temporal makes use of open-source observability platforms Prometheus and dashboarding with Grafana. Temporal senses and logs delays, back-pressure, failures, retries, time-outs, etc. into an Elastic database.

Rate of processing

APS (Actions Per Second) measures how many high-level operations (Actions) are performed each second within a Temporal Cloud. APS is the basis for throttling Actions such as starting or signaling a Workflow within the limits set for each Namespace.

RPS (Requests Per Second) measures how many low-level gRPC requests are issued in the Temporal Service, both in self-hosted Temporal and Temporal Cloud. RPS is used to manage rates at the service level, such as the Frontend, History, or Matching Services.

Latency

The Temporal Platform has built-in Visibility APIs to get the state of a Workflow Execution at any time, as well as metric endpoints to monitor the health of the platform.

“Temporal Cloud has a p99 latency SLO of 200ms per region.”

Temporal reports that, over a week-long period In March 2024, p90 and p99 latancy for starting and signaling Workflow Executions for specific Operations: * StartWorkflowExecution 24ms 54ms * SignalWorkflowExecution 14ms 40ms * SignalWithStartWorkflowExecution 24ms 61ms

Components

PROTIP: See the ITV series (on US PBS) about the UK Postal Service (between 1999 and 2015) wrongly sent 900 workers to prison for stealing when the problem was the system’s failure to properly handle failed transactions, and the government covered it up.

Of course, developers still need to code for idempotent execution.

Temporal and Durable Execution encapsulate most of the complexities of event-driven architecture. This allow developers to focus on what matters: business logic.

temporal-work-svc.png</a>

PROTIP: As of this writing, sessions that requires both activity and workflow worker support is only available in Go and not in Python. Use host specific task queue as a workaround.

Saga Pattern

A very unique feature of the Temporal Platform is that its Timers support business processes that could span decades.

temporal-saga.png

The project mimics a “money transfer” application with a single Workflow function orchestrating the execution of Withdraw() and Deposit() functions. Temporal calls these functions Activity functions.

The word Temporal refers to a scalable and reliable runtime for Reentrant Processes called Temporal Workflow Executions.

A “Temporal Application” is a set of Temporal Workflow Executions, which are reliable, durable function executions. Workflow Executions orchestrate the execution of Activities, which execute a single, well-defined action, such as calling another service, transcoding a media file, or sending an email message.

An Activity Task contains the context needed to make an Activity Task Execution.

An Activity Task Execution occurs when a Worker uses the context provided from the Activity Task and executes the Activity Definition.

An Activity Heartbeat is a ping from the Worker that is executing the Activity to the Temporal Service. Each ping informs the Temporal Service that the Activity Execution is making progress and the Worker has not crashed. The Temporal Server is a grouping of four horizontally scalable services.

Archival is a feature specific to a Self-hosted Temporal Service that automatically backs up Event Histories from Temporal Service persistence to a custom blob store after the Closed Workflow Execution retention period is reached.

To start the money transfer, a message is sent to the Temporal Cluster.

Install CLI

PROTIP: Don’t install static versions of programs by using download links such as: * https://temporal.download/cli/archive/v0.11.0?platform=darwin&arch=arm64
for Intel Macs (with AMD processors) * https://temporal.download/cli/archive/v0.11.0?platform=darwin&arch=amd64
for Apple Silicon Macs (with ARM processors) It’s because we want a easy way to update to the latest version with the latest security patches.

  1. Temporal’s documentation doesn’t mention this, but there is a Homebrew package for Temporal which automatically detects Intel Macs vs Apple Silicon Macs. 
    brew info Temporal

    ==> temporal: stable 1.0.0 (bottled), HEAD
Command-line interface for running and interacting with Temporal Server and UI
https://temporal.io/
Not installed
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/t/temporal.rb
License: MIT
==> Dependencies
Build: go ✔
==> Options
--HEAD
    Install HEAD version
==> Analytics
install: 1,319 (30 days), 3,403 (90 days), 9,568 (365 days)
install-on-request: 1,320 (30 days), 3,391 (90 days), 9,526 (365 days)
build-error: 0 (30 days)
  2. Install: 
     brew install temporal

     ==> Downloading https://formulae.brew.sh/api/formula.jws.json
 ########################################################################################### 100.0%
 ==> Downloading https://formulae.brew.sh/api/cask.jws.json
 ########################################################################################### 100.0%
 ==> Downloading https://ghcr.io/v2/homebrew/core/temporal/manifests/1.0.0
 ########################################################################################### 100.0%
 ==> Fetching temporal
 ==> Downloading https://ghcr.io/v2/homebrew/core/temporal/blobs/sha256:e7e5a6383d9aff3b052c4770293
 ########################################################################################### 100.0%
 ==> Pouring temporal--1.0.0.arm64_sonoma.bottle.tar.gz
 ==> Caveats
 Bash completion has been installed to:
 /opt/homebrew/etc/bash_completion.d
 ==> Summary
 🍺  /opt/homebrew/Cellar/temporal/1.0.0: 9 files, 71MB
 ==> Running `brew cleanup temporal`...
 Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
 Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

    PROTIP: Brew installs onto folder “/opt/homebrew/Cellar/” on Apple Silicon, unlike folder “/usr/local/bin” with manual installs in Temporal docs.

  3. Start-dev sets up the Web UI, creates the default Namespace, and establishes an in-memory database: 
    temporal server start-dev --ui-port 8233 --db-filename temporal.db

    time=2024-08-12T04:38:12.684 level=ERROR msg="service failures" operation=GetSystemInfo error="Frontend is not healthy yet"
time=2024-08-12T04:38:12.685 level=WARN msg="error creating sdk client" service=worker error="failed reaching server: Frontend is not healthy yet"
CLI 1.0.0 (Server 1.24.2, UI 2.28.0)
&nbsp;
Server:  localhost:7233
UI:      http://localhost:8233
Metrics: http://localhost:53787/metrics

  4. Switch to a brower to view the Temporal Web UI at its default port: http://localhost:8233

    temporal-ui-first.png

  5. View the Temporal Service at: http://localhost:7233

  6. View the log stream at: http://localhost:53787/metrics

  7. To stop it, press CTRL+C at the Terminal.

Temporal SDKs

Temporal SDKs (as VIDEO: as career)

The basic building blocks of a Temporal Application are Workflows, Activities execution, and Worker processess.

The Temporal Client connect to a Temporal Service (Cloud) and start a Workflow Execution.

  1. Create/navigate to the folder where you download from GitHub: 
    mkdir -p ~/my-github

  2. View the repo at:

    https://github.com/temporalio/money-transfer-project-template-go

  3. If you want to fork the repo to your own account, click that “Fork” button. But also specify your account name for commands in the remainder of this document: 
    GH_ACCT="temporalio"
  4. VIDEO: To download the Go example: 
    git clone "https://github.com/$GH_ACCT/money-transfer-project-template-go"
cd money-transfer-project-template-go

    INFO: The sample application in this tutorial models a money transfer between two accounts. Money comes out of one account and goes into another. However, there are a few things that can go wrong with this process. If the withdrawal fails, then there’s no need to attempt a deposit. But if the withdrawal works but the deposit fails, then the money needs to go back to the original account.

  5. Let’s view the repo: 
    ls

    The .go programs:

    • activity.go
    • banking-client.go

    • shared.go

    • workflow.go

    • workflow_test.go

    • start/main.go

    • worker/main.go

    • go.mod
    • go.sum

  6. edit go.mod

  7. View each of the files. NOTE: The first time you open a .go program in Visual Studio Code, accept the Install.

    In workflow.go, the Workflow Definition defines the Workflow Execution’s constraints. A Workflow Definition in Go is a regular Go function that accepts a Workflow Context and some input values.

    Workflow Executions orchestrate the execution of Activities, which execute a single, well-defined action, such as calling another service, transcoding a media file, or sending an email message.

    A Temporal Application is a set of Temporal Workflow Executions, which are reliable, durable function executions.

  8. Create a new Terminal window and navigate to your project folder.

    This tutorial uses a separate start/main.go program to start the Workflow. But most real applications start a Workflow as part of another program. For example, you might start the Workflow in response to a button press or an API call.

    You can make the call synchronously or asynchronously. Here we do it synchronously by fetching the return value of the Workflow execution with we.Get. This call waits for the Workflow execution to complete before continuing.

  9. Start the app for the first time: 
    go run start/main.go

    Go might download some dependencies initially, but after those downloads complete, you’ll see output that looks like the following:

    2022/11/14 10:52:20 INFO  No logger configured for temporal client. Created default one.
2022/11/14 10:52:20 Starting transfer from account 85-150 to account 43-812 for 250
2022/11/14 10:52:20 WorkflowID: pay-invoice-701 RunID: 3312715c-9fea-4dc3-8040-cf8f270eb53c

    The RunID will be different each time.

  10. To see the Workflow, switch to the browser view of the Temporal Web UI at its default port and press Refresh: http://localhost:8233

  11. Click the Workflow ID for a Workflow to see its input values, timeout configurations, scheduled retries, number of attempts, stack traceable errors, and more.

  12. There is a “Start Workflow” button at the upper-right.

    There is a tctl command-line tool to start a Temporal Workflow. In this tutorial the SDK starts the Workflow, which is how most Workflows get started in a live environment.

    To start a Workflow Execution, you connect to the Temporal Cluster, specify the Task Queue the Workflow should use, and start the Workflow with the input parameters it expects.

    In a real application, you may invoke this code when someone submits a form, presses a button, or visits a certain URL. In this tutorial, you’ll create a small command-line program that starts the Workflow Execution.

    The Task Queue is where Temporal Workers look for Workflows and Activities to execute. You define Task Queues by assigning a name as a string. You’ll use this Task Queue name when you start a Workflow Execution, and you’ll use it again when you define your Workers. To ensure your Task Queue names are consistent, place the Task Queue name in a variable you can share across your project.

    In this project, the Task Queue name is defined in a shared location: file shared.go which contains:

    const MoneyTransferTaskQueueName = “TRANSFER_MONEY_TASK_QUEUE”

    type PaymentDetails struct {
    SourceAccount string
    TargetAccount string
    Amount        int
    ReferenceID   string
}

    https://github.com/temporalio/money-transfer-project-template-go

Java:

  1. VIDEO: Java example:

    https://learn.temporal.io/getting_started/java/

    https://github.com/tsurdilo/temporal-java-workshop

  2. The Python example: https://www.youtube.com/watch?v=jxqPWS_LRzw&t=1166s&pp=ygUMVGVtcG9yYWwgMTAx

    https://learn.temporal.io/getting_started/python/

  3. The PHP example:

    https://learn.temporal.io/getting_started/php/

  4. The .NET example:

    https://learn.temporal.io/getting_started/dotnet/

  5. VIDEO: The TypeScript example: (JavaScript) VIDEO: complete intro

    https://learn.temporal.io/getting_started/typescript/

Namespace and recovery

A Namespace is a unit of isolation within the Temporal Platform

VIDEO: Cloud Multi-region Namespaces.

A Global Namespace is a Namespace that duplicates data from an active Temporal Service to a standby Service using the replication to keep both Namespaces in sync. Global Namespaces are designed to respond to service issues like network congestion. When service to the primary Cluster is compromised, a Failover transfers control from the active to the standby cluster.

A multi-region Namespace (MRN) is a Temporal Cloud Namespace that is configured to work across an active region and a standby region. Each region corresponds to a dedicated Temporal Cloud Service. Temporal Cloud automatically replicates Workflow Executions and metadata from the active to the standby region. MRNs are designed to respond to service issues as they arise. In the event that the Namespace’s performance is compromised, a Failover transfers control from the active to the standby region.

VIDEO: Availability and Disaster Recvoery in Temporal Cloud Building

Multi-Cluster Replication asynchronously replicates Workflow Executions from active Clusters to other passive Clusters, for backup and state reconstruction.

Demo

  1. VIDEO: demo show starting the local server VIDEO 
    temporal server start-dev

    or on macOS:

    brew install temporal

  2. Temporal provides Dev tools, including its tcld (Temporal Cloud command-Line interface) CLI commands tcld version, login, logout, user, account, feature, apikey, generate-certificates, namespace, request.

    https://docs.temporal.io/cli/tcld

  3. Web UI/tctl

  4. “Temporal Go SDK” is the framework for authoring workflows and activities using Temporal was developed using Go language

    https://github.com/temporalio/go-sdk

    https://docs.temporal.io/workflows

  5. Sample application: https://github.com/temporalio/background-checks, at https://learn.temporal.io/examples/

    https://learn.temporal.io/examples/go/background-checks/

    a long-running human-driven workflow, described in VIDEO: Learning Paths

Debugging

Temporal has built-in debugging and troubleshooting tools, such as “replays” that allow developers to step through an execution to see what happened.

VIDEO: Debug with the Temporal VS Code extension

Workflow Diagram Service

Give it SW JSON/YAML, get back diagram SVG. By Tihomir Surdilovic. https://github.com/tsurdilo/workflow-diagram-service

https://github.com/tsurdilo/my-temporal-dockercompose

https://github.com/tsurdilo/temporal-patient-onboarding