The Failure of Traditional Ownership Models
Scaling event collection across microservices often fails because organizations misalign responsibility.
- Decentralized Ownership (The "Wild West"): When every service team manages its own publishing, transport, and schema logic, inconsistency is inevitable. The author observed fragmented schemas (e.g., mixed camelCase and snake_case), redundant event flooding, and a lack of discoverability. This approach fails because it forces domain teams to become experts in transport infrastructure, leading to technical debt.
- Centralized Ownership (The "Bottleneck"): Creating a dedicated event platform team to gatekeep schemas and libraries creates a velocity trap. The platform team lacks the domain context to validate business logic, yet they become the mandatory approval layer for every change. This creates a disconnect where the people with the knowledge (service teams) are blocked by the people with the infrastructure authority (platform team).
The Sidecar Collector Pattern
The successful approach involves decoupling domain logic from transport logic using a sidecar agent (e.g., Vector).
- The Workflow: Services emit structured logs rather than handling Kafka publishing directly. A sidecar agent runs alongside each service, reading these logs and validating them against a central schema registry.
- Schema Governance: The schema registry is managed as a shared Git repository. Changes are proposed via pull requests and reviewed by the relevant domain stakeholders, not a central platform team.
- Validation and Feedback: If a log fails schema validation, the agent rejects it immediately. This forces the service team to catch errors in their own CI/CD pipeline rather than discovering them as downstream failures later.
Aligning Knowledge with Responsibility
The core insight is that organizational boundaries must match knowledge boundaries.
- Domain Teams own the meaning of events (schemas and business logic). They are best positioned to define what an event represents.
- Infrastructure Teams own the transport of events (Kafka configuration and agent maintenance). They are best positioned to ensure reliability and performance.
The sidecar agent acts as the technical seam between these two groups, allowing each to operate within their area of expertise without interfering with the other's velocity.