Need help with kafka-streams-dotnet?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

LGouellec
218 Stars 32 Forks MIT License 521 Commits 22 Opened issues

Description

.NET Stream Processing Library for Apache Kafka 🚀

Services available

!
?

Need anything else?

Contributors list

.NET Stream Processing Library for Apache Kafka TM · GitHub license · Join the chat at https://discord.gg/J7Jtxum build Nuget Nuget (with prereleases)

Quality Statistics

Sonar Cloud Quality Gate Sonar Cloud Quality Gate Sonar Cloud Reliability Rate Sonar Cloud Security Rate Sonar Cloud Maintainability Rate Sonar Cloud Duplicated Code

Project Statistics

GitHub issues GitHub pull requests



Streamiz Kafka .NET is .NET stream processing library for Apache Kafka.

It's allowed to develop .NET applications that transform input Kafka topics into output Kafka topics. It's supported .NET Standard 2.1.

So this library can be used by >= .NET Core 3.0 and/or .NET 5.0 applications.

It's a rewriting inspired by Kafka Streams. Finally it will provide the same functionality as Kafka Streams.

This project is being written. Thanks for you contribution !

Timeline

  • June/July 2021 - 1.2.0 - Persistent state store (eg: RocksDB Store), Repartition and Changelog topics
  • End 2021 - 1.3.0 - Processor API, Metrics, Interactive Queries, Standby Replica

Documentation

Read the full documentation on https://lgouellec.github.io/kafka-streams-dotnet/

Installation

Nuget packages are listed to nuget.org

Install the last version with :

shell
dotnet add package Streamiz.Kafka.Net

Usage

There, a sample streamiz application :

static async System.Threading.Tasks.Task Main(string[] args)
{ 
    var config = new StreamConfig();
    config.ApplicationId = "test-app";
    config.BootstrapServers = "192.168.56.1:9092";
    config.SaslMechanism = SaslMechanism.Plain;
    config.SaslUsername = "admin";
    config.SaslPassword = "admin";
    config.SecurityProtocol = SecurityProtocol.SaslPlaintext;
    config.AutoOffsetReset = AutoOffsetReset.Earliest;
    config.NumStreamThreads = 2;

StreamBuilder builder = new StreamBuilder();

var kstream = builder.Stream<string string>("stream");
var ktable = builder.Table("table", InMemory<string string>.As("table-store"));

kstream.Join<string string stringserdes>(ktable, (v, v1) =&gt; $"{v}-{v1}")
       .To("join-topic");

Topology t = builder.Build();
KafkaStream stream = new KafkaStream(t, config);

Console.CancelKeyPress += (o, e) =&gt; {
    stream.Dispose();
};

await stream.StartAsync();

}

TODO implementation

  • [X] Global state store
  • [X] Refactor Subtopology & StreamTask assignment
  • [X] Refactor Topology description for corresponding to java implementation
  • [X] Refactor Partition grouper + record queue
  • [X] Statefull processors impl
  • [ ] Transform and Through Processor
  • [ ] Supress Processor (.suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded())))
  • [X] Rocks DB state implementation
  • [X] Changelog Topic
  • [ ] Task restoring
  • [ ] Repartition topic
  • [ ] Repartition Processor KAFKA-8611 | PR #7170
  • [ ] Processor API
  • [ ] Sample projects (Micro-services, console sample, topology implementation, etc ..) which use Streamiz package (see)
  • [X] Protobuf SerDes which interact Confluent Schema Registry
  • [ ] Json SerDes which interact Confluent Schema Registry
  • [ ] Optimizing Kafka Streams Topologies
  • [ ] Standby Replica
  • [ ] Interactive Queries
  • [ ] Metrics

Contributing

Owners:

Maintainers:

Streamiz Kafka .Net is a community project. We invite your participation through issues and pull requests! You can peruse the contributing guidelines.

When adding or changing a service please add tests and documentations.

Support

You can found support here

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.