at Capgemini
ConsultanciesPosted 13 days ago
No clicks
Senior Data Engineer role in Mississauga, ON required. Key responsibilities include designing, building, and optimizing scalable data pipelines and ETL/ELT workflows using Databricks (Unity Catalog, Delta Live Tables, Workflows), Apache Spark, and Scala to handle large volumes of structured and unstructured data. This involves architecting and managing data lakehouse solutions with Apache Iceberg and Delta Lake for efficient data storage and retrieval. Additionally, develop robust backend data services and APIs for real-time and batch data consumption across various enterprise platforms. Collaborate with data architects, analysts, and engineering teams to define data models, establish best practices, and drive continuous improvement across the data platform. 3-5 years of relevant experience required.
- Compensation
- Not specified
- City
- Not specified
- Country
- Canada
Currency: Not specified
Full Job Description
Job Description
- Location: Mississauga, ON
- 3-5 years of experienced, Design, build, and optimise scalable data pipelines and ETL/ELT workflows using Databricks (Unity Catalog, Delta Live Tables, Workflows, Asset Bundles), Apache Spark, and Scala to process large volumes of structured and unstructured data.
- Architect and manage data lakehouse solutions leveraging Apache Iceberg and Delta Lake for table format management, versioning, time travel, and efficient data storage and retrieval at scale.
- Build and maintain robust backend data services and APIs, integrating Databricks with downstream systems, enabling real-time and batch data consumption across enterprise platforms.
- Collaborate with data architects, analysts, and engineering teams to define data models, establish best practices, and drive continuous improvement across the data platform
