Senior Technical Lead

Job Description Senior Technical Lead Others, Dubai Job Summary A PySpark Developer is responsible for designing, developing, and maintaining large-scale, distributed data processing systems and pipelines. They leverage Apache Spark’s Python API (PySpark) to ingest, transform, and analyze massive volumes of structured and unstructured data. Core Responsibilities Data Pipeline Development: Design and build scalable, efficient ETL/ELT pipelines to move and transform data from various sources into data lakes or warehouses. Big Data Processing: Write optimized PySpark scripts (using DataFrames, Datasets, and Spark SQL) to process terabytes/petabytes of data. Performance Tuning: Troubleshoot, debug, and optimize Spark jobs by managing partitions, caching, broadcast variables, and minimizing data shuffling. Data Modeling: Assist in designing data architectures, schemas, and storage formats (e.g., Parquet, Delta Lake, ORC ). Integration: Connect Spark applications with cloud storage (AWS S3, Azure ADLS, Google Cloud Storage) and databases (NoSQL, relational). CI/CD & Automation: Deploy data workflows using orchestration tools and maintain code quality through unit testing. Technical Skills Required Category Skills & Technologies Core Languages Python (Advanced), SQL (Advanced) Big Data Frameworks Apache Spark / PySpark , Hadoop, Hive Cloud Platforms AWS (EMR, Glue), Azure (Databricks, Synapse), or GCP (Dataproc) Storage & Formats Delta Lake , Parquet, Avro, HDFS, Snowflake, Amazon S3 Orchestration Apache Airflow , Prefect, or Luigi Key Responsibilities as above Skill Requirements as above Other Requirements Apply now Information at a Glance Why HCLTech? At HCLTech, you'll supercharge your potential. You'll find your career. And you'll find your spark. All at a place that knows that helping its customers stay on top starts by putting its people first. HCLTech is a global technology company, home to more than 223,000 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending June 2026 totaled $14.8 billion. Disclaimer Privacy Statement Terms of use Sitemap Raise a Grievance We use cookies on our site. Please read more about cookies policy here. Opens in a new tab. Opens in a new tab. Opens in a new tab. Opens in a new tab. Opens in a new tab. Opens in a new tab. Copyright © 2026 HCL Technologies Limited × Cookie Consent Manager When you visit any website, it may store or retrieve information on your browser, mostly in the form of cookies. Because we respect your right to privacy, you can choose not to allow some types of cookies. However, blocking some types of cookies may impact your experience of the site and the services we are able to offer. Required Cookies These cookies are required to use this website and can't be turned off. Required Cookies Show More Details Required Cookies Provider Description Enabled SAP as service provider We use the following session cookies, which are all required to enable the website to function: "route" is used for session stickiness "careerSiteCompanyId" is used to send the request to the correct data center "JSESSIONID" is placed on the visitor's device during the session so the server can identify the visitor "Load balancer cookie" (actual cookie name may vary) prevents a visitor from bouncing from one instance to another Cookies from provider SAPasserviceprovider are required and cannot be turned off Functional Cookies These cookies provide a better customer experience on this site, such as by remembering your login details, optimizing video performance, or providing us with information about how our site is used. You may freely choose to accept or decline these cookies at any time. Note that certain functionalities that these third-parties make available may be impacted if you do not accept these cookies. Consent to all Functional Cookies Show More Details Functional Cookies Provider Description Enabled Vimeo Vimeo is a video hosting, sharing, and services platform focused on the delivery of video. Opting out of Vimeo cookies will disable your ability to watch or interact with Vimeo videos. Cookie Policy Privacy Policy Terms and Conditions Consent to cookies from provider Vimeo YouTube YouTube is a video-sharing service where users can create their own profile, upload videos, watch, like, and comment on videos. Opting out of YouTube cookies will disable your ability to watch or interact with YouTube videos. Cookie Policy Privacy Policy Terms and Conditions Consent to cookies from provider YouTube Advertising Cookies These cookies serve ads that are relevant to your interests. You may

Technical Skills Required: Core Languages Python (Advanced), SQL (Advanced). Big Data Frameworks: Apache Spark / PySpark, Hadoop, Hive. Cloud Platforms: AWS (EMR, Glue), Azure (Databricks, Synapse), or GCP (Dataproc). Storage & Formats: Delta Lake, Parquet, Avro, HDFS, Snowflake, Amazon S3. Orchestration: Apache Airflow, Prefect, or Luigi. Additional responsibilities imply experience in designing ETL/ELT pipelines, data modeling, and CI/CD & automation.

Data Pipeline Development: Design and build scalable, efficient ETL/ELT pipelines to move and transform data from various sources into data lakes or warehouses. Big Data Processing: Write optimized PySpark scripts to process large datasets. Performance Tuning: Troubleshoot, debug, and optimize Spark jobs. Data Modeling: Assist in designing data architectures, schemas, and storage formats. Integration: Connect Spark applications with cloud storage and databases. CI/CD & Automation: Deploy data workflows and maintain code quality through unit testing.