Kafka: 5. GitLab CI

This documentation is part of the GitHub Actions & GitLab CI guide. You can view the complete guide here: Launch a real Kafka service from your GitHub Actions or GitLab CI pipeline, run your tests against it, and automatically tear it down.

👋 Welcome to the Stackhero documentation!

Stackhero makes it easy to set up a fully managed Kafka cloud solution. Here’s what you can expect:

Unlimited message size and transfers to support your data growth.

Effortless updates—apply the latest improvements with a single click.

High-level performance and enhanced security, all powered by your own private, dedicated VM.

If you want to save time and streamline your operations, try Stackhero’s Kafka cloud hosting. You can get started in about 5 minutes!

You can save this configuration as .gitlab-ci.yml. With this setup, each pipeline run spins up a fresh real Kafka for your tests.

test:
  image: ubuntu:24.04
  variables:
    STACK_NAME: "ci-kafka-$CI_PIPELINE_ID-$CI_JOB_ID"
    INSTANCE: "10G"   # Change this as needed (see step 3)
    REGION: "europe"
    SERVICE_STORE: "kafka"
  # STACKHERO_TOKEN comes from the CI/CD variable you created in step 1.
  script:
    - set -euo pipefail
    - curl -fsSL https://www.stackhero.io/install.sh | sh
    - apt-get update && apt-get install -y --no-install-recommends jq curl netcat-openbsd
    - STACK_ID=$(stackhero --format=script stack-create --name="$STACK_NAME")
    - echo "STACK_ID=$STACK_ID" >> deploy.env
    - SERVICE_ID=$(stackhero --format=script service-add --stack="$STACK_ID" --service-store="$SERVICE_STORE" --instance="$INSTANCE" --region="$REGION")
    - echo "SERVICE_ID=$SERVICE_ID" >> deploy.env
    - stackhero service-wait-for --service="$SERVICE_ID"
    - config=$(stackhero service-configuration-get --service="$SERVICE_ID" --format=json)
    - host=$(echo "$config" | jq -r '.configuration.domain')
    # Open a TCP connection to the broker port.
    - nc -z -w 5 "$host" 9092
    - echo "✅ Kafka is reachable from CI."
    # You can run your own test suite here using the credentials above ...
  after_script:
    - test -f deploy.env && . ./deploy.env || true
    - >
      if [ -n "${SERVICE_ID:-}" ]; then
        stackhero service-delete --service="$SERVICE_ID" --confirm
        stackhero service-wait-for --service="$SERVICE_ID"
      fi
    - >
      if [ -n "${STACK_ID:-}" ]; then
        stackhero stack-delete --stack="$STACK_ID" --confirm
      fi

On GitLab, cleanup is handled in after_script. This section is always executed, even if the job fails, ensuring your Kafka resources are deleted and you are not billed for unused resources.

In GitLab, after_script runs in a fresh shell. To handle this, the script writes the service and stack IDs to deploy.env during the job and reloads them before cleanup. This ensures that even if something fails mid-job, your resources are still deleted.

That is the complete CI lifecycle for Kafka: create a stack, add the service, wait, retrieve credentials, smoke-test, and always tear down. Each pipeline run gets a real, isolated service, with nothing left running when you are done. For more information about available commands and non-interactive STACKHERO_TOKEN authentication, you can refer to the full CLI documentation.

Kafka: 5. GitLab CI

👋 Welcome to the Stackhero documentation!

Continue reading this guide

Other articles for Kafka