Cloudflare has been using Kafka in production since 2014. We have come a long way since then, and currently run 14 distinct Kafka clusters, across multiple data centers, with roughly 330 nodes. Between them, over a trillion messages have been processed over the last eight years.
Cloudflare uses Kafka to decouple microservices and communicate the creation, change or deletion of various resources via a common data format in a fault-tolerant manner. This decoupling is one of many factors that enables Cloudflare engineering teams to work on multiple features and products concurrently.We learnt a lot about Kafka on the way to one trillion messages, and built some interesting internal tools to ease adoption that will be explored in this blog post. The focus in this blog post is on inter-application communication use cases alone and not logging (we have other Kafka clusters that power the dashboards where customers view statistics that handle more than one trillion messages each day). I am an engineer on the Application Services team and our team has a charter to provide tools/services to product teams, so they can focus on their core competency which is delivering value to our customers.In this blog Iād like to recount some of our experiences in the hope that it helps other engineering teams who are on a similar journey of adopting Kafka widely.ToolingOne of our Kafka clusters is creatively named Messagebus. It is the most general purpose cluster we run, and was created to:Prevent data silos;Enable services to communicate more clearly with basically zero integration cost (more on how we achieved this below);Encourage the use of a self-documenting communication format and therefore removing the problem of out of date documentation.To make it as easy to use as possible and to encourage adoption