Lead Kafka Engineer

Cloud Operations, Apache Kafka, Kubernetes, Python, UNIX shell scripting

Hyderabad, Bangalore, Pune, Chennai, Gurgaon

We are looking to hire an experienced Lead Kafka Engineer who will be responsible for installing, monitoring, troubleshooting, and maintaining Kafka platform, ensuring optimal performance, security, developing new features/automation/integration.

Responsibilities

Install and provision new Kafka clusters and supporting infrastructure
Continuously monitor the health and performance of Kafka platforms and data pipelines
Troubleshoot and resolve issues related to data pipelines, network disruptions, and other infrastructure failures
Perform regular performance tuning and cost optimization for Kafka components
Manage the upgrade process for Kafka platforms, including planning, testing, and implementation
Enforce and manage security protocols including access control, encryption, and conduct regular security audits
Implement disaster recovery procedures and conduct regular platform backups
Oversee capacity management and scaling projections
Document technical procedures, configurations, issue resolutions and share knowledge across teams
Collaborate with internal and vendor support teams for escalated issue resolutions
Maintain and enhance Infrastructure as Code (IaC) and Configuration Management (CM) automations
Develop and refine onboarding and automation scripts for streamlined operations
Facilitate Kafka setup for application teams including consumers, producers, and connectors
Respond to team requests efficiently, converting complex issues into CLOUD Tickets when required
Integrate new vendor features and capabilities in collaboration with relevant stakeholders

Requirements

Proven experience in the implementation and maintenance of Confluent Platform
Minimum 8 years of experience in Kafka administration
Knowledge of Helm and Kubernetes
Proficiency in deploying Kafka in Kubernetes
Skills in Python or Shell Scripting
Background in cloud technologies including AWS, GCP (compute, networking, storage, IAM)