Kafka: 5. GitLab CI

This documentation is part of the GitHub Actions & GitLab CI guide. View the full guide here: Spin up a real Kafka service from your GitHub Actions or GitLab CI pipeline, run your tests against it, and tear it down automatically.

👋 Welcome to the Stackhero documentation!

Stackhero gives you a fully managed Kafka cloud environment, ready in minutes. Here is what you can expect:

• No limits on message size or transfer: scale your data without worrying about caps.
• Simple, one-click updates keep your platform current without downtime or manual work.
• Consistent high performance and robust security on your own private, dedicated infrastructure.

If you want to move faster and reduce operational overhead, Stackhero provides Kafka cloud hosting to help you. You can have a production-ready cluster in about 5 minutes.

You can save this configuration as .gitlab-ci.yml. With this setup, every pipeline run spins up a fresh real Kafka for your tests.

test:
  image: ubuntu:24.04
  variables:
    STACK_NAME: "ci-kafka-$CI_PIPELINE_ID-$CI_JOB_ID"
    INSTANCE: "10G"   # Change this as needed (see step 3)
    REGION: "europe"
    SERVICE_STORE: "kafka"
  # STACKHERO_TOKEN comes from the CI/CD variable you created in step 1.
  script:
    - set -euo pipefail
    - curl -fsSL https://www.stackhero.io/install.sh | sh
    - apt-get update && apt-get install -y --no-install-recommends jq curl netcat-openbsd
    - STACK_ID=$(stackhero --format=script stack-create --name="$STACK_NAME")
    - echo "STACK_ID=$STACK_ID" >> deploy.env
    - SERVICE_ID=$(stackhero --format=script service-add --stack="$STACK_ID" --service-store="$SERVICE_STORE" --instance="$INSTANCE" --region="$REGION")
    - echo "SERVICE_ID=$SERVICE_ID" >> deploy.env
    - stackhero service-wait-for --service="$SERVICE_ID"
    - config=$(stackhero service-configuration-get --service="$SERVICE_ID" --format=json)
    - host=$(echo "$config" | jq -r '.configuration.domain')
    # Open a TCP connection to the broker port.
    - nc -z -w 5 "$host" 9092
    - echo "✅ Kafka is reachable from CI."
    # You can run your own test suite here using the credentials above ...
  after_script:
    - test -f deploy.env && . ./deploy.env || true
    - >
      if [ -n "${SERVICE_ID:-}" ]; then
        stackhero service-delete --service="$SERVICE_ID" --confirm
        stackhero service-wait-for --service="$SERVICE_ID"
      fi
    - >
      if [ -n "${STACK_ID:-}" ]; then
        stackhero stack-delete --stack="$STACK_ID" --confirm
      fi

In GitLab, cleanup happens inside after_script. This section is always executed, even if the job fails, so your Kafka resources are removed and you are not charged for resources you are not using.

In GitLab, after_script runs in a fresh shell. To handle this, the script writes the service and stack IDs to deploy.env during the job and reloads them before teardown. This makes sure that even if something fails mid-job, your resources are still cleaned up.

That is the complete CI lifecycle for Kafka: create a stack, add the service, wait, retrieve credentials, smoke-test, and always tear down. Each pipeline run gets a real, isolated service, nothing left running when you are done. For more information about available commands and non-interactive STACKHERO_TOKEN authentication, you may want to explore the full CLI documentation.