All writing
kafka python fastapi async

Kafka Python Producer Example for FastAPI Production Systems

Ayush Kaushik 6 min read
Kafka Python Producer Example for FastAPI Production Systems

I run Kafka producers in FastAPI services that handle millions of events per day. The pattern below is what I use in production: aiokafka for async I/O, idempotent transactions for exactly-once semantics, and FastAPI lifespan hooks for clean startup and shutdown. This Kafka Python producer example skips the hello-world fluff and goes straight to the configuration that survives network partitions and broker failures.

How do I integrate a Kafka producer with FastAPI lifespan?

Use the lifespan async context manager to start the producer on application startup and flush it on shutdown. This guarantees the event loop is running before the producer connects and that in-flight messages drain before the process exits.

app/kafka/producer.py
from contextlib import asynccontextmanager
from aiokafka import AIOKafkaProducer
from aiokafka.errors import KafkaError
from fastapi import FastAPI
import json
import logging
logger = logging.getLogger(__name__)
producer: AIOKafkaProducer | None = None
@asynccontextmanager
async def kafka_lifespan(app: FastAPI):
global producer
producer = AIOKafkaProducer(
bootstrap_servers="kafka:9092",
client_id="orders-service",
acks="all",
enable_idempotence=True,
transactional_id="orders-txn-1",
max_in_flight_requests_per_connection=5,
retries=5,
retry_backoff_ms=1000,
value_serializer=lambda v: json.dumps(v).encode("utf-8"),
key_serializer=lambda k: k.encode("utf-8") if k else None,
)
await producer.start()
await producer.init_transactions()
logger.info("Kafka producer started and transactions initialized")
try:
yield
finally:
await producer.flush()
await producer.stop()
logger.info("Kafka producer stopped cleanly")
app = FastAPI(lifespan=kafka_lifespan)

The transactional_id must be stable across restarts for the same logical producer instance. I bind it to the service name and a fixed partition identifier. If you run multiple replicas behind a load balancer each needs a unique transactional_id or you will hit producer fence errors.

What async producer patterns work with aiokafka?

Wrap the send operation in a transaction when you need exactly-once semantics across multiple topics or when you combine Kafka writes with database updates. The pattern is begin, send, commit. Any exception triggers an abort.

app/services/order_service.py
from app.kafka.producer import producer
from aiokafka.errors import KafkaError
import uuid
async def create_order(order_data: dict) -> str:
order_id = str(uuid.uuid4())
event = {"order_id": order_id, **order_data, "status": "created"}
await producer.begin_transaction()
try:
await producer.send_and_wait(
topic="orders.created",
key=order_id,
value=event,
headers=[("source", b"orders-api"), ("version", b"1")],
)
await producer.send_and_wait(
topic="inventory.reserve",
key=order_id,
value={"order_id": order_id, "items": order_data["items"]},
)
await producer.commit_transaction()
except KafkaError:
await producer.abort_transaction()
raise
return order_id

send_and_wait blocks until the broker acknowledges the write according to acks=all. For fire-and-forget throughput you can use send and collect futures, but you lose ordering guarantees per partition. I only use fire-and-forget for high-volume telemetry where occasional loss is acceptable.

How do I configure idempotent transactions for exactly-once semantics?

Exactly-once requires three settings working together. Set enable_idempotence=True, acks="all", and max_in_flight_requests_per_connection=5 (the default when idempotence is on). The broker deduplicates by producer_id and sequence number per partition. The transactional_id maps to that producer_id and survives restarts.

Failure mode I have seen: a network blip causes the producer to retry a batch. Without idempotence the broker writes duplicates. With idempotence but without transactions a partial batch can commit. Transactions make the multi-topic write atomic. The cost is latency. Each transaction adds a round trip to the transaction coordinator. On my clusters that adds 2-5 ms per batch. Worth it for orders. Not worth it for clickstream.

Which serialization strategy should I use with Schema Registry?

JSON is fine for internal tools and low-volume topics. Avro or Protobuf pay off when you have many consumers, schema evolution requirements, or need to enforce contracts across teams. I use fastavro with Confluent Schema Registry.

app/kafka/serializers.py
import io
import fastavro
import requests
from functools import lru_cache
SCHEMA_REGISTRY_URL = "http://schema-registry:8081"
@lru_cache(maxsize=128)
def get_schema(subject: str) -> dict:
resp = requests.get(f"{SCHEMA_REGISTRY_URL}/subjects/{subject}/versions/latest")
resp.raise_for_status()
return resp.json()["schema"]
def avro_serializer(schema_subject: str):
schema = fastavro.parse_schema(get_schema(schema_subject))
def _serialize(record: dict) -> bytes:
buf = io.BytesIO()
fastavro.schemaless_writer(buf, schema, record)
return buf.getvalue()
return _serialize
# Usage in producer config
value_serializer=avro_serializer("orders.created-value")

Register schemas in CI. Break the build if compatibility checks fail. I learned this the hard way when a consumer crashed because a producer added a required field without a default. The Schema Registry BACKWARD compatibility mode would have caught it.

How do delivery guarantees work with acks, retries, and error handling?

acks=all waits for the leader and all in-sync replicas. This is the only setting that prevents data loss when a broker fails. Pair it with retries=5 and retry_backoff_ms=1000. The producer will retry on transient errors like NotEnoughReplicas or LeaderNotAvailable.

app/kafka/error_handling.py
from aiokafka.errors import (
ProducerFenced,
OutOfOrderSequenceNumber,
UnknownProducerId,
KafkaTimeoutError,
)
import logging
logger = logging.getLogger(__name__)
async def send_with_dlq(topic: str, key: str, value: dict, max_retries: int = 3):
for attempt in range(max_retries):
try:
future = await producer.send(topic, key=key, value=value)
await future
return
except (ProducerFenced, OutOfOrderSequenceNumber, UnknownProducerId) as e:
logger.error("Fatal producer error, requires restart: %s", e)
raise
except KafkaTimeoutError:
logger.warning("Timeout on attempt %d/%d", attempt + 1, max_retries)
if attempt == max_retries - 1:
await write_to_dead_letter_queue(topic, key, value)
raise

Fatal errors like ProducerFenced mean another producer instance stole the transactional_id. This happens when you scale replicas without unique IDs. The fix is either unique IDs per replica or a singleton producer per transactional_id. I prefer the singleton pattern using a sidecar or a dedicated producer service for transactional workloads.

Docker Compose for local development

Spin up Kafka, Schema Registry, and a UI with one file. This matches the producer config above.

docker-compose.yml
version: "3.8"
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.5.0
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
image: confluentinc/cp-kafka:7.5.0
depends_on: [zookeeper]
ports: ["9092:9092"]
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_LOG_RETENTION_HOURS: 168
schema-registry:
image: confluentinc/cp-schema-registry:7.5.0
depends_on: [kafka]
ports: ["8081:8081"]
environment:
SCHEMA_REGISTRY_HOST_NAME: schema-registry
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092
kafka-ui:
image: provectuslabs/kafka-ui:latest
depends_on: [kafka, schema-registry]
ports: ["8080:8080"]
environment:
KAFKA_CLUSTERS_0_NAME: local
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
KAFKA_CLUSTERS_0_SCHEMAREGISTRY: http://schema-registry:8081

Run docker compose up -d. The UI at localhost:8080 lets you inspect topics, schemas, and consumer lag. I keep this file in every repo that touches Kafka. It eliminates the “works on my machine” gap between local and staging.

When not to use this pattern

Avoid transactions if you publish to a single topic and can tolerate at-least-once with idempotent consumers. Avoid aiokafka if your team has no async experience; the Confluent Python client is synchronous and easier to debug. Avoid Schema Registry for throwaway topics or prototypes. The operational overhead is real. I covered the client trade-offs in Kafka Python vs Confluent Kafka: Production Benchmarks if you are deciding between libraries.

For a full walkthrough from Docker to FastAPI integration see Kafka Python Tutorial: From Docker to FastAPI. That post covers consumer groups and health checks which I skipped here to focus on the producer path.

FAQ

Does aiokafka support synchronous Flask or Django apps? No. aiokafka requires an async event loop. Use the Confluent Python client for synchronous frameworks or run the producer in a separate async service.

What happens if the transaction coordinator is unavailable? The producer throws TransactionCoordinatorNotAvailable on begin_transaction. Implement a circuit breaker and alert. Do not retry indefinitely; the coordinator outage usually means a broker issue that needs human intervention.

Can I share one producer instance across FastAPI workers? No. Each worker process needs its own producer instance with a unique transactional_id or you will hit fencing errors. Run a dedicated producer service if you need a single logical producer.

How do I test exactly-once semantics locally? Kill the producer process mid-transaction. Verify the consumer never sees the partial batch. Use the Kafka UI to check topic offsets and transaction markers.

Key Takeaways

  • Use FastAPI lifespan for producer lifecycle management
  • Enable idempotence and transactions together for exactly-once
  • Serialize with Avro and Schema Registry for contract safety
  • Set acks=all and configure retries for durability
  • Unique transactional_id per producer instance prevents fencing
  • Docker Compose with Schema Registry mirrors production locally

Working on something similar?

If you're building backend or AI systems and want a second set of senior eyes, let's talk.

Keep reading

Related articles