Senior Data Engineer

About the role

Requirements

• - Bachelor’s degree in Computer Science, Software Engineering, Data Science, or a related field.
• - 4–6 years of professional experience in Data Engineering or a closely related role.
• - Strong hands-on experience designing and implementing production-grade ETL/ELT pipelines.
• - Strong proficiency in Python for data processing, automation, and data-engineering workflows.
• - Advanced SQL skills and a strong understanding of relational databases.
• - Practical production experience with Apache Airflow for workflow orchestration.
• - Hands-on experience with Apache Spark and distributed data processing.
• - Strong understanding of data modelling, transformation patterns, and data-pipeline architecture.
• - Experience with Redis, caching strategies, and high-performance data-access patterns.
• - Experience processing large datasets and optimizing pipeline and query performance.
• - Strong understanding of data quality, validation, monitoring, observability, and pipeline reliability.
• - Familiarity with Linux, Git, Docker/containers, and modern software-engineering practices.
• - Strong analytical, troubleshooting, communication, and cross-functional collaboration skills.
• Preferred Qualification
• - Experience with cloud data platforms and object storage services such as AWS, Azure, or GCP.
• - Experience working with PostgreSQL, data warehouses, or analytical databases.
• - Experience processing geospatial data or large-scale location-based datasets.
• - Familiarity with DuckDB, Apache Sedona, Trino, Presto, or similar analytical technologies.
• - Experience processing high-volume event, mobility, transactional, or geospatial data.
• - Familiarity with CI/CD pipelines and infrastructure-as-code practices.
• - Experience supporting data products, analytics platforms, or AI/ML pipelines.

Responsibilities

• - Design, develop, and maintain scalable ETL/ELT pipelines for data ingestion, transformation, validation, and delivery.
• - Build, schedule, and orchestrate production-grade data workflows using Apache Airflow.
• - Develop distributed data-processing jobs using Apache Spark.
• - Write efficient, reusable, and maintainable Python code for data processing, automation, and pipeline development.
• - Develop and optimize complex SQL queries for data transformation, analysis, and quality validation.
• - Design pipelines capable of processing large-scale structured, semi-structured, and unstructured datasets.
• - Implement Redis for caching, high-performance data access, and data-intensive application requirements.
• - Integrate data from APIs, relational databases, files, third-party providers, and other internal and external sources.
• - Implement data validation, monitoring, logging, error handling, alerting, and pipeline observability.
• - Optimize pipeline performance, data storage, processing time, and infrastructure costs.
• - Develop reusable data-ingestion and transformation frameworks instead of one-off scripts.
• - Troubleshoot pipeline failures, performance bottlenecks, and data-quality issues to ensure timely resolution.
• - Collaborate with BI, Product, Data Science, and Engineering teams to deliver reliable, production-ready datasets.
• - Establish and maintain data-engineering standards, technical documentation, and development best practices.

Skills

• Python
• SQL
• Apache Airflow
• Apache Spark
• Redis
• ETL/ELT pipelines
• data modeling
• data validation
• data quality
• distributed data processing
• Linux
• Git
• Docker
• cloud data platforms (AWS/Azure/GCP)
• PostgreSQL