Staff Software Engineer, Data Ingestion
The Staff Software Engineer, Data Ingestion will be a critical individual contributor responsible fordesigning collection strategies, developing, and maintaining robust and scalable data pipelines.This role is at the heart of our data ecosystem, deliver new analytical software solution to access timely, accurate, and complete data for insights, products, and operational efficiency.
Key Responsibility
Required Skills
Extensive Python Expertise: Extensive experience in developing robust, production-grade applications with Python.
Data Collection & Integration: Proven experience collecting data from various sources (REST APIs, OAuth, GraphQL, Kafka, S3, SFTP, etc.).
Distributed Systems & Scalability: Strong understanding of distributed systems concepts, designing for scale, performance optimization, and fault tolerance.
Cloud Platforms: Experience with major cloud providers (AWS or GCP) and their data-related services (e.g., S3, EC2, Lambda, SQS, Kafka, Cloud Storage, GKE).
Database Fundamentals: Solid understanding of relational databases (SQL, schema design, indexing, query optimization). OLAP database experience is a plus (Hadoop)
Monitoring & Alerting: Experience with monitoring tools (e.g., Prometheus, Grafana) and setting up effective alerts.
Version Control: Proficiency with Git.
Containerization (Plus): Experience with Docker and Kubernetes.
Streaming Technologies (Plus): Experience with real-time data processing using Kafka, Flink, Spark Streaming
Apply to this Job