Consultancies

Posted 15 days ago

No clicks

**Senior Data Engineer (Python/Spark/MS Fabric): Design, Optimize, and Operate Large-Scale Data Workloads** - Design, optimize, and manage cost-effective Spark and Delta Lake workloads on Microsoft Fabric or Databricks, demonstrating production troubleshooting and capacity-economics expertise. - Build and optimize Spark pipelines, diagnose performance using Spark UI and Delta statistics, and manage Delta Lake design, compute sizing, throttling exposure, and operational reliability. - Develop reusable Python libraries and Spark modules for ingestion, governance, security, monitoring, and optimization, exhibit solid production Python library development skills. - Collaborate with cross-functional teams, contribute to pipeline design, monitor SLAs, and recover failures, showcasing excellent communication and problem-solving abilities. - Must-haves: Microsoft Fabric/Databricks production engineering, Apache Spark & Spark UI troubleshooting, Delta Lake optimization, Fabric capacity economics, and operational pipeline design.

Compensation: Not specified
City: Not specified
Country: Not specified

Full Job Description

Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way youd like, where youll be supported and inspired by a collaborative community of colleagues around the world, and where youll be able to reimagine whats possible. Join us and help the worlds leading organizations unlock the value of technology and build a more sustainable, more inclusive world.

Job Description

Time Zone Requirement

Resources must be able to provide overlap with the customer's US Eastern Time working hours, including availability through approximately 4:30 PM ET.

About Client

Our client is expanding its data platform team as part of its ongoing data modernization initiatives. They are looking to add two hands-on Data Engineers to support the evolution of the Microsoft-based analytics and data platform.

Role Overview

Design, optimize, and operate reliable, large-scale, cost-aware Spark and Delta Lake workloads.

Build and optimize Spark pipelines on Microsoft Fabric or Databricks. Diagnose performance using Spark UI and Delta statistics. Manage Delta Lake design, Fabric capacity consumption, compute sizing, throttling exposure, and operational reliability. Create reusable Python libraries and Spark modules for ingestion, governance, security, monitoring, and optimization.

Demonstrated production troubleshooting and capacity-economics experience, not only notebook or ETL development.

Measures workload behavior before changing infrastructure; optimizes code and configuration before increasing compute; documents pool sizing, throttling risk, and operational requirements; explains capacity behavior to platform and engineering teams.

Must Have

Microsoft Fabric or Databricks production engineering

Apache Spark and Spark UI troubleshooting

Shuffle skew, spill, OOM, task-duration analysis, AQE, join strategy, and predicate pushdown

Delta Lake optimization: liquid clustering, partitioning, Z-order, file sizing, OPTIMIZE, statistics, VACUUM, and incremental MERGE

Fabric capacity economics: CU-seconds, bursting, smoothing, throttling stages, pool configuration, node sizing, concurrency, and workload isolation

Production Python library development, packaging, validation, logging, testing, versioning, and documentation

Operational pipeline design: idempotency, watermarks, quarantine, monitoring, failure recovery, and SLA awareness

Good-To-Have Skills

OneLake security and access roles

Governance metadata and TBLPROPERTIES

Reusable ingestion and maintenance frameworks

Secrets handling and thread safety

Direct Lake optimization awareness

What You Will Love About Working Here

We care about all our employees and want them to feel as comfortable as possible. That's why we offer them health insurance from the first days, regardless of the probationary period.

The gift from the company - Christmas holidays from 25 December to 31 December.

ooperation with Superhumans center and Veteran HUB. Capgemini Engineering has supported the launch of psychological rehabilitation department of Superhumans. Our team also donated over UAH 500 000 prosthetics for three Ukrainian defenders. Currently, we support psychological counseling provided by the Veteran Hub, and we have implemented an internal policy making the company friendly to military and veterans with the assistance of the Hub.

Capgemini is an AI-powered global business and technology transformation partner, delivering tangible business value. We imagine the future of organizations and make it real with AI, technology and people. With our strong heritage of nearly 60 years, we are a responsible and diverse group of 420,000 team members in more than 50 countries. We deliver end-to-end services and solutions with our deep industry expertise and strong partner ecosystem, leveraging our capabilities across strategy, technology, design, engineering and business operations. The Group reported 2024 global revenues of €22.1 billion.
Make it real | www.capgemini.com

Full Job Description