Kafka Setup for 4ALLPORTAL

This guide explains how to set up a Kafka broker and configure your 4ALLPORTAL instance to use it.

WARNING

Kafka is a mandatory component of every 4ALLPORTAL installation.
All current and future 4ALLPORTAL features that rely on event-driven processing require a running Kafka broker. Installations without Kafka will not be able to use these features and will not receive support for the affected functionality.

Overview

Kafka is the event backbone of the 4ALLPORTAL. The 4ALLPORTAL emits change events to a Kafka broker, which are processed by the 4ALLPORTAL infrastructure to power features such as cloud services, integrations, and event-driven workflows. When the feature flag kafka-on-premise has been activated for your installation by 4ALLPORTAL GmbH, these events are additionally forwarded to the 4ALLPORTAL cloud infrastructure — enabling you to use the cloud services provisioned for your account.

Prerequisites

Before you start, make sure you have:

• A running Kafka broker accessible by your 4ALLPORTAL instance (either via the Docker-internal network or within your Kubernetes cluster)
• Outbound HTTPS access from your 4ALLPORTAL instance to https://api.4allportal.cloud (required for cloud services)
• The feature flag kafka-on-premise activated for your installation by 4ALLPORTAL GmbH (required for cloud service forwarding)

Configuration

To connect your 4ALLPORTAL to your Kafka broker, add the following environment variables to the 4allportal service in your docker-compose.yml.

Required Environment Variables

Variable	Example value	Description
SPRING_KAFKA_BOOTSTRAP-SERVERS	kafka:9092	Address of the Kafka broker. Use the internal Docker service name or the Kubernetes-internal service address of your broker.
FOURALLPORTAL_KAFKA_FORWARDING	true	Enables forwarding of Kafka events to the 4ALLPORTAL cloud infrastructure. Must be set to true for on-premise installations.

These variables must be added to the environment section of the 4ALLPORTAL service in your docker-compose.yml:

services:
  4allportal:
    image: registry.4allportal.net/4allportal:<version>
    environment:
      SPRING_KAFKA_BOOTSTRAP-SERVERS: kafka:9092
      FOURALLPORTAL_KAFKA_FORWARDING: "true"

Docker Compose Setup with Kafka

To run Kafka alongside 4ALLPORTAL in Docker Compose, add a kafka service to your docker-compose.yml. We recommend the apache/kafka image — the official Apache Kafka image (available since Kafka 3.7), with built-in KRaft mode (no ZooKeeper required) and no licensing restrictions.

Full Example docker-compose.yml

version: "3"

services:
  4allportal:
    image: registry.4allportal.net/4allportal:<version>
    ports:
      - "8181:8181"
    environment:
      GENERAL_EXTERNAL_URL: https://your-4allportal-url.com
      SPRING_KAFKA_BOOTSTRAP-SERVERS: kafka:9092
      FOURALLPORTAL_KAFKA_FORWARDING: "true"
    volumes:
      - ./assets:/4allportal/data/data
      - ./data:/4allportal/data
    depends_on:
      kafka:
        condition: service_healthy

  kafka:
    image: apache/kafka:latest
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - ./kafka-data:/var/lib/kafka/data
    healthcheck:
      test: ["CMD-SHELL", "/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list"]
      interval: 10s
      timeout: 5s
      retries: 10

  db:
    image: mariadb
    environment:
      MYSQL_ROOT_PASSWORD: toor
    ports:
      - "3306:3306"
    volumes:
      - ./db:/var/lib/mysql

When running Kafka and 4ALLPORTAL in the same Docker Compose network, use the Kafka service name as the hostname (e.g. kafka:9092). The depends_on with condition: service_healthy ensures that 4ALLPORTAL only starts after Kafka is ready.

Required Kafka Topic

The 4ALLPORTAL requires the topic fourallportal-change-event to be present in your Kafka broker before the application starts. The example Compose file above disables Kafka's automatic topic creation (KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false") — topics must be created explicitly.

There are two ways to do this: via a dedicated init container in Docker Compose, or manually using docker exec.

Option 1: Init Container (Recommended)

Add a short-lived kafka-init service to your docker-compose.yml. It runs once after Kafka is healthy, creates the topic, and then exits.

  kafka-init:
    image: apache/kafka:latest
    depends_on:
      kafka:
        condition: service_healthy
    command: >
      /opt/kafka/bin/kafka-topics.sh
      --bootstrap-server kafka:9092
      --create
      --if-not-exists
      --topic fourallportal-change-event
      --partitions 1
      --replication-factor 1
    restart: on-failure

The --if-not-exists flag makes the command idempotent — safe to run on every docker compose up without failing when the topic already exists.

Option 2: Manual Creation via docker exec

WARNING

Topics created this way are stored on the Kafka volume (./kafka-data) and survive container restarts and recreation. However, if the volume is deleted (e.g. via docker compose down -v), the topic is gone and must be created again manually. The init container approach (Option 1) handles this automatically on every docker compose up.

To create the topic manually after your Kafka container is running:

docker exec -it <kafka-container-name> /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create \
  --topic fourallportal-change-event \
  --partitions 1 \
  --replication-factor 1

Replace <kafka-container-name> with the actual name of your Kafka container (find it with docker ps). With the example Compose file above the container is typically named <project>-kafka-1.

To verify that the topic was created successfully:

docker exec -it <kafka-container-name> /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --topic fourallportal-change-event

Gateway Connectivity

Your 4ALLPORTAL instance must be able to reach the 4ALLPORTAL Gateway:

• Gateway URL: https://api.4allportal.cloud

Ensure that outbound HTTPS (port 443) traffic to this address is permitted by your firewall or network policies. Without this connectivity, event forwarding to the cloud services will not work even after the feature flag has been activated.

The 4ALLPORTAL Gateway (https://api.4allportal.cloud) is also required for other services such as in-app support. See our system requirements for a full list of required endpoints.

Activating Cloud Services

Once your Kafka broker is running and the environment variables are set, contact 4ALLPORTAL GmbH to request activation of the feature flag kafka-on-premise for your installation.

After activation:

1. Your 4ALLPORTAL starts emitting events to your local Kafka broker.
2. These events are forwarded via the 4ALLPORTAL Gateway to the 4ALLPORTAL cloud infrastructure.
3. The cloud services provisioned for your account become available.

If you have any questions about the available cloud services or need assistance with the setup, please reach out to our support team.