System Architect - Kafka

Cloud Operations, Apache Kafka, Kubernetes, Python, UNIX shell scripting

Hyderabad, Bangalore, Pune, Chennai, Gurgaon

We are seeking a seasoned Kafka System Architect to join our team.

The selected candidate will be pivotal in installing, monitoring, troubleshooting, and maintaining our Kafka platform while ensuring optimal performance and security. This role demands a proactive approach to developing new features, automations, and integrations to continually enhance system capabilities.

Responsibilities

Manage installation and provisioning of new Kafka clusters and support components
Conduct regular monitoring to assess the health and performance of the Kafka platform and associated data pipelines
Troubleshoot and resolve platform issues related to data pipelines, network problems, and cloud or containerization resources failures
Implement performance tuning for Kafka platform components
Monitor and adjust the Kafka clusters to optimize cost and performance
Plan, test, and execute upgrades to newer versions of the Kafka platform
Enhance the security of the Kafka platform through access control, encryption, and regular security audits
Carry out regular backups and develop disaster recovery plans
Oversee capacity management for the Kafka platform, including scalability planning
Document system procedures, configurations, and troubleshooting guides
Coordinate with Confluent Support for unresolved issues
Maintain and enhance automated scripts for platform management and maintenance
Provide support to application teams with Kafka consumers, producers, connectors, and streams setup
Handle team support requests effectively, utilizing Slack and converting complex issues into CLOUD Tickets

Requirements

Proven experience in Confluent Platform implementation and maintenance
Minimum 10 years of experience in Kafka administration
Skilled in using Helm and Kubernetes for deploying Kafka
Background in programming with Python or Shell Scripting
Proficiency in Cloud technologies across AWS or GCP covering compute, networking, storage, and IAM aspects

Nice to have

Previous experience with infrastructure as code (IaC) and configuration management (CM) tools
Capability to streamline team processes through enhanced automation scripts and self-service tools