kafkakn

Why it exists

A service compiled to a single Kotlin/Native binary has no way to talk to Kafka: the clients are JVM libraries, and the one native client ever published for Kotlin (com.icemachined:kafka-client 0.2.0, October 2022) ships three native artefacts and no JVM variant — so it has no way to check itself except by believing what a broker tells it. kafkakn is built the other way round. The jvm target is not there for portability; it is there as the differential oracle: one commonTest suite runs on both arms against one broker, and the native arm is correct when it agrees with the reference implementation.

That asymmetry has already paid twice. The two clients put the same key on different partitions — librdkafka hashes with CRC32, the Java producer with murmur2 — and each arm was consistent with the broker on its own, so neither could have noticed alone. And the first published klib carried the bindings and not the C: this repository's own binaries linked because of options on the build machine, and a stranger's link failed with fourteen undefined symbols. It took a build that was not this one to find it.

What you get

• A producer whose send suspends until the broker has acknowledged. Holding a RecordMetadata is the acknowledgement; there is no third outcome to inspect. This follows from one measured fact: rd_kafka_produce only enqueues, a record that was never queued produces no delivery report, and a binding that counts delivery reports can lose 264 826 records of a million while flushing successfully. No public API here exposes a delivery-report count, and that is a build gate rather than a habit.
• A consumer that assigns and seeks, joins groups with manual commits, offers a Flow over poll, cooperative rebalancing, static membership, and the KIP-848 group protocol as a value the caller chooses.
• An admin client: create, delete and describe topics; describe the cluster; list and describe consumer groups and read their committed offsets and lag; move and delete group offsets; change a topic's configuration incrementally; add partitions; delete records.
• TLS, client certificates, SASL PLAIN, SCRAM and OAUTHBEARER with a token the caller supplies. Certificate trust cannot be turned off: the key that would do it is refused on both arms, because it exists only on the one without an oracle.
• librdkafka and its TLS stack inside the klib. A downstream link needs no configuration of its own, and ldd on the linked binary is the same set as on the same binary built without kafkakn — measured on a downstream build, not on this repository's test binary.

Install

Published to reposilite only, never to Maven Central, with no compatibility promise. Every publish is a version of its own, 0.1.0.<n>, tagged v0.1.0.<n> at the commit it was built from and never overwritten; the newest is the highest tag.

plugins {
    kotlin("multiplatform") version "2.4.20"
}

repositories {
    maven("https://reposilite.kotlin.website/snapshots") {
        content { includeGroupAndSubgroups("io.github.youndie") }
    }
    mavenCentral()
}

kotlin {
    linuxX64("native") {
        binaries.executable { entryPoint = "main" }
    }
    sourceSets.commonMain.dependencies {
        implementation("io.github.youndie.kafkakn:kafkakn-core:0.1.0.12")
    }
}

The Kotlin version is part of the instructions. A klib carries metadata that a build on another compiler refuses, so 2.4.20 is the version this is known to work with rather than a placeholder. Measured from an empty machine — a JDK, Gradle and nothing else — that build file gets to a record on a topic in 106–109 s, of which 74–78 s is the Kotlin/Native toolchain downloading.

What the native artefact requires

The same shared libraries as a binary built without it: libc, libgcc_s, libm, libpthread, libresolv and the rest of the loader's usual set — linking kafkakn adds none and removes none. The one thing it does raise is the glibc floor, to 2.17, which is manylinux2014's, because that is where the C bundle is built. The bundle carries a one-line local patch to librdkafka 2.13.0 for that glibc, kept as a file in the repository and re-applied on every bump; nothing from this project is filed upstream.

Limitations

• jvm and linuxX64 are the published targets. linuxArm64 and macosArm64 build for contributors and pass the same suite against the same broker, but are not published and not run in CI, by decision. No macosX64, no Windows, no musl.
• Reposilite only; nothing on Maven Central, and no compatibility promise between versions. What would change that is a condition, not a date: somebody outside this portfolio wanting the library.
• Topic metadata is planned and not yet in. OIDC token fetching is not planned — the native bundle has no curl, so OAUTHBEARER takes a token the caller already has.
• Schema Registry and Streams are in neither client underneath, so they are not a gap between kafkakn and what it wraps — and they are not here either.
• send does not batch for you: one call is one record and one acknowledgement. Throughput comes from calling it concurrently; both clients batch internally once records are in flight together.