Senior Software Engineer - Data Engineering Team
About the Role
You will design and develop Java microservices running on Kubernetes, build customer-facing integrations into data pipelines, and scale systems to process billions of events in real time and batch. You will tackle data duplication, schema versioning, governance, and high availability; maintain end-to-end ETL workflows; optimize queries and data formats for downstream consumers; and instrument and monitor production systems to ensure data completeness and reliability.
Requirements
- BSc in Computer Science or equivalent
- Strong software engineering background with at least 3+ years experience with Java
- 5+ years in data engineering and data pipeline development on high-volume, low-latency production environments
- Experience with low-latency, real-time systems processing billions of events per day
- Experience with Apache Airflow, Kubernetes, ClickHouse, Snowflake, Redis, Spark, caching technologies and Kafka
- Proven experience with relational and non-relational databases
- Expert/proficient in SQL and query optimizations, preferably in ClickHouse
- Experience in fintech, crypto or trading industries and familiarity with FIX and market data is an advantage
- Experience with monitoring systems such as Prometheus, Grafana, Zabbix, Datadog
- Experience in object-oriented development and strong software engineering foundations
- Ability to work independently and proactively identify solutions
- Excellent verbal and written communication skills in a remote environment
Responsibilities
- Design and develop the data team's microservices in Java on Kubernetes
- Design and implement customer-facing integrations of various asset types into data pipelines
- Scale data pipelines to support billions of events in low-latency real-time and T+1 batch
- Address data duplication, velocity, schema adherence and schema versioning, high availability, and data governance
- Develop and maintain data pipelines and micro-service ETL workflows including ingestion and transformation logic
- Plan and communicate integrations with other teams that consume the data
- Improve the way data is stored and served and optimize queries and data formats for consumption
- Perform query optimizations in data warehouses to ensure data completeness and reliability at scale