Kafka: GitHub Actions & GitLab CI

Spin up a real Kafka service from your GitHub Actions or GitLab CI pipeline, run your tests against it, and tear it down automatically

👋 Welcome to the Stackhero documentation!

Stackhero makes it easy to get started with a fully managed Kafka cloud solution. Here is what you can expect:

  • Unlimited message sizes and transfers to handle your growing data needs.
  • Effortless updates, apply the latest improvements with a single click.
  • Top-tier performance and strong security, all powered by your own private and dedicated VM.

If you are looking to save time and streamline your workflow, you might want to give Stackhero's Kafka cloud hosting a try. It only takes about 5 minutes to get up and running!

This guide will walk you through running a real, dedicated Kafka service inside your CI pipeline using either GitHub Actions or GitLab CI. By following these steps, you can test your code against a live Kafka instance under production-like conditions, so you no longer need to rely on mocks or simulations. Apache Kafka is a distributed event streaming platform built to carry huge streams of events through your systems in real time.

Each time your pipeline runs, you will get a brand new Kafka instance. This means your tests interact with the same kind of service your users will see in production. The workflow automatically creates a temporary stack, adds Kafka, waits for it to be ready, retrieves the generated credentials, runs a smoke test using netcat (a TCP check), and then always cleans everything up once the run is complete.

You will use the Stackhero CLI, a standalone command-line tool that makes launching and managing Stackhero services quick and straightforward.

1. Create an access token and store it as a CI secret

To enable the CLI to work non-interactively, you will need an access token (format: usr-xxxxxx:tokenId). You only need to create this token once and then add it to your CI pipeline as a secure, encrypted secret.

  1. Create the token: In your Stackhero dashboard, click your profile picture at the top right, go to Your account, then Access tokens, and click Create token.
  2. For GitHub Actions: In your repository, go to Settings > Secrets and variables > Actions > New repository secret, and enter the token as STACKHERO_TOKEN.
  3. For GitLab CI: In your project, go to Settings > CI/CD > Variables > Add variable, set the key to STACKHERO_TOKEN, and check Masked (and Protected if your CI runs only on protected branches).

Do not include your access token directly in your pipeline YAML file. If it is present in YAML, it could be exposed to anyone with repository access and may appear in build logs. Storing it as a CI secret keeps it encrypted and masked, ensuring your token stays secure.

2. The lifecycle script

Here is a complete example script that manages the full service lifecycle. It demonstrates how little setup is required, just a few commands. You can copy the ready-made YAML for your platform from the sections below.

#!/bin/bash
set -euo pipefail

# STACKHERO_TOKEN is supplied by the CI secret, the CLI picks it up automatically.

stackName="ci-kafka-$$"          # Unique stack name for each run
serviceStore="kafka"             # The Kafka service store
instance="10G"        # Change this as needed (see step 3)
region="europe"                         # Region name (see stackhero regions-list)

# 1. Install the CLI on the runner
curl -fsSL https://www.stackhero.io/install.sh | sh

# 2. Install the client required for smoke testing (jq + curl are the baseline)
apt-get update && apt-get install -y --no-install-recommends jq curl netcat-openbsd

# 3. Create a dedicated stack for this run
stackId=$(stackhero --format=script stack-create --name="$stackName")
echo "Stack created: $stackId"

# 4. Add Kafka and capture its service id
serviceId=$(stackhero --format=script service-add \
  --stack="$stackId" \
  --service-store="$serviceStore" \
  --instance="$instance" \
  --region="$region")
echo "Service added: $serviceId"

# 5. Wait until the service is running (this may take a couple of minutes)
stackhero service-wait-for --service="$serviceId"

# 6. Read the configuration (contains the generated credentials)
config=$(stackhero service-configuration-get --service="$serviceId" --format=json)

# 7. Extract the credentials you need
host=$(echo "$config" | jq -r '.configuration.domain')

# 8. Smoke test: Open a TCP connection to the broker port.
nc -z -w 5 "$host" 9092

echo "✅ Kafka is reachable from CI."

The teardown step, which deletes the service, waits for its removal, and then deletes the stack, is handled in the platform-specific sections below. This approach makes sure cleanup always happens, even if a smoke test fails.

A stack can only be deleted once it is empty. Always delete the service first and wait for its removal, then delete the stack. The service-wait-for command ensures the service is either running or deleted before proceeding, making it the right tool for deletion waits too.

3. Pick the instance size

The examples below use the entry-level instance 10G for Kafka by default. This is a solid choice for most workloads, but you are welcome to adjust it as needed. To view all available instance types for Kafka, you can run:

# The NAME column shows the value to pass to --instance
stackhero instances-store-list --service-store=kafka

4. GitHub Actions

To get started, you can save the following as .github/workflows/ci.yml. From now on, every push and pull request will run tests against a real Kafka instance.

name: CI with Kafka

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      STACKHERO_TOKEN: ${{ secrets.STACKHERO_TOKEN }}
      STACK_NAME: ci-kafka-${{ github.run_id }}-${{ github.run_attempt }}
      INSTANCE: "10G"   # Change this as needed (see step 3)
      REGION: europe
    steps:
      - uses: actions/checkout@v4

      - name: Install the Stackhero CLI and the client
        run: |
          curl -fsSL https://www.stackhero.io/install.sh | sh
          apt-get update && apt-get install -y --no-install-recommends jq curl netcat-openbsd

      - name: Create the Kafka service
        run: |
          set -euo pipefail
          STACK_ID=$(stackhero --format=script stack-create --name="$STACK_NAME")
          echo "STACK_ID=$STACK_ID" >> "$GITHUB_ENV"
          SERVICE_ID=$(stackhero --format=script service-add \
            --stack="$STACK_ID" \
            --service-store="kafka" \
            --instance="$INSTANCE" \
            --region="$REGION")
          echo "SERVICE_ID=$SERVICE_ID" >> "$GITHUB_ENV"
          stackhero service-wait-for --service="$SERVICE_ID"

      - name: Run tests against Kafka
        run: |
          set -euo pipefail
          config=$(stackhero service-configuration-get --service="$SERVICE_ID" --format=json)
          host=$(echo "$config" | jq -r '.configuration.domain')
          # Open a TCP connection to the broker port.
          nc -z -w 5 "$host" 9092
          echo "✅ Kafka is reachable from CI."
          # You can run your own test suite here using the credentials above ...

      - name: Tear down (always, even on failure)
        if: always()
        run: |
          if [ -n "${SERVICE_ID:-}" ]; then
            stackhero service-delete --service="$SERVICE_ID" --confirm
            stackhero service-wait-for --service="$SERVICE_ID"
          fi
          if [ -n "${STACK_ID:-}" ]; then
            stackhero stack-delete --stack="$STACK_ID" --confirm
          fi

The teardown step is configured with if: always() so it runs no matter what, making sure your Kafka instance is deleted and you are not billed for unused resources.

5. GitLab CI

You can save this configuration as .gitlab-ci.yml. With this setup, every pipeline run spins up a fresh real Kafka for your tests.

test:
  image: ubuntu:24.04
  variables:
    STACK_NAME: "ci-kafka-$CI_PIPELINE_ID-$CI_JOB_ID"
    INSTANCE: "10G"   # Change this as needed (see step 3)
    REGION: "europe"
    SERVICE_STORE: "kafka"
  # STACKHERO_TOKEN comes from the CI/CD variable you created in step 1.
  script:
    - set -euo pipefail
    - curl -fsSL https://www.stackhero.io/install.sh | sh
    - apt-get update && apt-get install -y --no-install-recommends jq curl netcat-openbsd
    - STACK_ID=$(stackhero --format=script stack-create --name="$STACK_NAME")
    - echo "STACK_ID=$STACK_ID" >> deploy.env
    - SERVICE_ID=$(stackhero --format=script service-add --stack="$STACK_ID" --service-store="$SERVICE_STORE" --instance="$INSTANCE" --region="$REGION")
    - echo "SERVICE_ID=$SERVICE_ID" >> deploy.env
    - stackhero service-wait-for --service="$SERVICE_ID"
    - config=$(stackhero service-configuration-get --service="$SERVICE_ID" --format=json)
    - host=$(echo "$config" | jq -r '.configuration.domain')
    # Open a TCP connection to the broker port.
    - nc -z -w 5 "$host" 9092
    - echo "✅ Kafka is reachable from CI."
    # You can run your own test suite here using the credentials above ...
  after_script:
    - test -f deploy.env && . ./deploy.env || true
    - >
      if [ -n "${SERVICE_ID:-}" ]; then
        stackhero service-delete --service="$SERVICE_ID" --confirm
        stackhero service-wait-for --service="$SERVICE_ID"
      fi
    - >
      if [ -n "${STACK_ID:-}" ]; then
        stackhero stack-delete --stack="$STACK_ID" --confirm
      fi

In GitLab, cleanup happens inside after_script. This section is always executed, even if the job fails, so your Kafka resources are removed and you are not charged for resources you are not using.

In GitLab, after_script runs in a fresh shell. To handle this, the script writes the service and stack IDs to deploy.env during the job and reloads them before teardown. This makes sure that even if something fails mid-job, your resources are still cleaned up.

That is the complete CI lifecycle for Kafka: create a stack, add the service, wait, retrieve credentials, smoke-test, and always tear down. Each pipeline run gets a real, isolated service, nothing left running when you are done. For more information about available commands and non-interactive STACKHERO_TOKEN authentication, you may want to explore the full CLI documentation.

Other articles for Kafka

  1. Automate with the CLI
    Start Kafka, retrieve its credentials, and change its configuration programmatically with the Stackhero CLI