LOG IN
SIGN UP
Canary Wharfian - Online Investment Banking & Finance Community.
Sign In
Forgot password?
Don't have an account?
or
Join Canary Wharfian
By signing up, you agree to our Terms & Conditions and Privacy Policy.
or

Lead solution architect - Data engineering

ExperiencedNo visa sponsorship
British Petroleum logo

at British Petroleum

Commodities

Posted 11 days ago

No clicks

**Lead Solution Architect - Data Engineering: Microsoft Azure & Databricks Expert** Drive bp's high-value data initiatives as our Lead Solution Architect. Design, build, and optimize modern data platforms using **Microsoft Azure** and **Databricks** to achieve significant operational efficiencies, safety, and environmental aims. Key responsibilities include: - **Cloud-Native Data Platform Architecture** using Azure services like Azure Data Lake, Azure Databricks, Azure Synapse, Azure SQL, and Azure Data Factory. - **Data Modeling & Delta Lake Architecture**, including dimensional, canonical, and domain-driven designs, and medallion (Bronze–Silver–Gold) frameworks. - **ETL/ELT Pipeline Development** using Databricks' PySpark/Spark SQL. - **Delta Lake Best Practices Implementation** and Spark workload optimization. - **Azure Data Ecosystem Integration**, API development, and CI/CD pipeline establishment. - **Data Governance & Security** policies, GDPR compliance, and observability frameworks. Bring 7+ years in Data Architecture or Data Engineering, proficiency in **Azure Databricks, Azure Data Lake Gen2, Azure Data Factory, and Synapse**, along with **PySpark, SQL, and Python**. Lead cross-functional teams, drive strategy, and mentor engineers. Relocation within the country and up to 10% travel required. Hybrid remote/office work environment.

Compensation
Not specified

Currency: Not specified

City
Not specified
Country
India

Full Job Description

Entity:

Technology


Job Family Group:

IT&S Group


Job Description:

Role Overview

We are a global energy business involved in every aspect of the energy system. We are working towards delivering light, heat and mobility to millions of people, every day. We are one of the very few companies equipped to solve some of the big complex challenges that matter for the future. We have a real contribution to make to the world's ambition of a low carbon future. Join us and be part of what we can accomplish together. You can participate in our new ambition to become a net zero company by 2050 or sooner, and to help the world get to net zero.

Critical to achieving bps digital ambitions is the delivery of our high value data and analytics initiatives, and the enablement of the technologies and platforms that will support those objectives. We are developing deep data capability to transform the access, supply, control and quality to our vast and ever-growing reserves of data. 

We are looking for an outstanding Data Architect with deep expertise in Microsoft Azure and Databricks to design, build, and optimize modern data platforms. The ideal candidate will have strong architectural leadership, hands-on technical proficiency, and the ability to influence internal customers to drive data strategy, governance, and scalable data solutions.. These use cases would drive significant operational efficiency, safety and environmental aims that are key to the transformation of bp. 

Key Responsibilities

Data Architecture & Platform Design

  • Architect end-to-end cloud-native data platforms using Azure services, including Azure Data Lake Storage Gen2, Azure Databricks, Azure Synapse, Azure SQL, Azure Data Factory, and Event/Service Bus.
  • Define and implement data modeling standards, including dimensional, canonical, and domain-driven designs.
  • Design Delta Lake architectures, including medallion (BronzeSilverGold) frameworks.
  • Develop scalable and secure ETL/ELT pipelines using Databricks (PySpark/Spark SQL).

2. Databricks & Delta Lake Engineering

  • Implement Delta Lake best practices, including OPTIMIZE, ZORDER, Change Data Feed (CDF), schema evolution, and table versioning.
  • Optimize Spark workloads for performance, cost, and cluster utilization.
  • Lead governance implementation using Unity Catalog, including permissions, data lineage, and metadata modeling.

3. Azure Data Ecosystem Integration

  • Integrate solutions across Azure Data Factory, Synapse Pipelines, Event Hubs, Functions, Key Vault, Azure DevOps, and Purview.
  • Drive CI/CD and Infrastructure-as-Code using Terraform, Bicep, or ARM templates.
  • Build APIs and microservices supporting data consumption.

4. Data Governance & Security

  • Define policies for data quality, master data management, metadata, and data lifecycle management.
  • Ensure GDPR-compliant data design, encryption (KMS/Key Vault), RBAC/ABAC, and secure network architecture.
  • Establish frameworks for observability, including logging, monitoring, and alerting.

5. Stakeholder Leadership

  • Collaborate with business, product owners, engineers, and analysts to gather data requirements.
  • Lead architectural reviews and provide long-term technology strategy.
  • Mentor engineering teams on Azure, Databricks, and scalable data design principles.

Required Skills & Qualifications

Technical Skills

  • 7+ years in Data Architecture or Data Engineering roles.
  • Expert-level knowledge of: 
  • Azure Databricks (Spark, Delta Lake, MLflow), Agent Bricks
  • Azure Data Lake Gen2
  • Azure Data Factory / Synapse Pipelines
  • Azure SQL / Synapse Dedicated SQL / Cosmos DB
  • Unity Catalog & Delta Live Tables
  • Strong programming experience in PySpark, SQL, and Python.
  • Hands-on with CI/CD (Azure DevOps / GitHub Actions).
  • Deep understanding of distributed systems & big data performance tuning.

Soft Skills

  • Ability to translate business requirements into technical architecture.
  • Strong documentation, presentation, and interpersonal skills.
  • Leadership in multi-disciplinary teams and sophisticated enterprise environments.

Formal education

  • A Bachelor's (or higher) degree, ideally in Computer Science, MIS/IT, Mathematics, chartered accountancy, engineering, or equivalent work experience.

Preferred Qualifications and Skills

  • Azure certifications (e.g., DP-203, AZ-305, Databricks Data Engineer Professional/Associate).
  • Courses on Generative AI in Data engineering
  • Experience with data mesh, domain-driven design, and event-driven architecture.
  • Familiarity with Power BI or other BI tools.
  • Experience integrating governance tools such as Azure Purview.
  • TOGAF 9 certification

Why join us?

At bp, we provide an excellent working environment and employee benefits! We offer an open and inclusive culture and a great work-life balance. Tremendous learning and development opportunities are available to craft a career path, life and health insurance, medical care package and many others!

We support our people to learn and grow in a diverse and exciting environment. We believe that our team is strengthened by diversity. We are committed to crafting an inclusive environment in which everyone is respected and treated fairly. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, perform crucial job functions, and receive other benefits and privileges of employment.


Travel Requirement

Up to 10% travel should be expected with this role


Relocation Assistance:

This role is eligible for relocation within country


Remote Type:

This position is a hybrid of office/remote working


Skills:

Agility core practices, Business modelling, Creativity and Innovation, Data Analysis, Data design, Digital Analytics, Emerging technology monitoring, Information Assurance, Integration, IT Management, IT strategy and planning, Network Design, Requirements definition and management, Risk Management, Service operations and resiliency, Solution Architecture, Systems Development Management, Technical specialism, User Experience Analysis

.


Legal Disclaimer:

We are an equal opportunity employer. We do not discriminate on the basis of protected characteristics like race, religion, color, sex, national origin, sexual orientation, veteran status or disability status. Individuals with an accessibility need may request an adjustment/accommodation related to bps recruiting process (e.g., accessing the job application, completing required assessments, participating in telephone screenings or interviews, etc.). If you would like to request an adjustment/accommodation related to the recruitment process, please contact us.

If you are selected for a position and depending upon your role, your employment may be contingent upon adherence to local policy. This may include pre-placement drug screening, medical review of physical fitness for the role, and background checks.

Lead solution architect - Data engineering

Compensation

Not specified

City: Not specified

Country: India

British Petroleum logo
Commodities

11 days ago

No clicks

at British Petroleum

ExperiencedNo visa sponsorship

**Lead Solution Architect - Data Engineering: Microsoft Azure & Databricks Expert** Drive bp's high-value data initiatives as our Lead Solution Architect. Design, build, and optimize modern data platforms using **Microsoft Azure** and **Databricks** to achieve significant operational efficiencies, safety, and environmental aims. Key responsibilities include: - **Cloud-Native Data Platform Architecture** using Azure services like Azure Data Lake, Azure Databricks, Azure Synapse, Azure SQL, and Azure Data Factory. - **Data Modeling & Delta Lake Architecture**, including dimensional, canonical, and domain-driven designs, and medallion (Bronze–Silver–Gold) frameworks. - **ETL/ELT Pipeline Development** using Databricks' PySpark/Spark SQL. - **Delta Lake Best Practices Implementation** and Spark workload optimization. - **Azure Data Ecosystem Integration**, API development, and CI/CD pipeline establishment. - **Data Governance & Security** policies, GDPR compliance, and observability frameworks. Bring 7+ years in Data Architecture or Data Engineering, proficiency in **Azure Databricks, Azure Data Lake Gen2, Azure Data Factory, and Synapse**, along with **PySpark, SQL, and Python**. Lead cross-functional teams, drive strategy, and mentor engineers. Relocation within the country and up to 10% travel required. Hybrid remote/office work environment.

Full Job Description

Entity:

Technology


Job Family Group:

IT&S Group


Job Description:

Role Overview

We are a global energy business involved in every aspect of the energy system. We are working towards delivering light, heat and mobility to millions of people, every day. We are one of the very few companies equipped to solve some of the big complex challenges that matter for the future. We have a real contribution to make to the world's ambition of a low carbon future. Join us and be part of what we can accomplish together. You can participate in our new ambition to become a net zero company by 2050 or sooner, and to help the world get to net zero.

Critical to achieving bps digital ambitions is the delivery of our high value data and analytics initiatives, and the enablement of the technologies and platforms that will support those objectives. We are developing deep data capability to transform the access, supply, control and quality to our vast and ever-growing reserves of data. 

We are looking for an outstanding Data Architect with deep expertise in Microsoft Azure and Databricks to design, build, and optimize modern data platforms. The ideal candidate will have strong architectural leadership, hands-on technical proficiency, and the ability to influence internal customers to drive data strategy, governance, and scalable data solutions.. These use cases would drive significant operational efficiency, safety and environmental aims that are key to the transformation of bp. 

Key Responsibilities

Data Architecture & Platform Design

  • Architect end-to-end cloud-native data platforms using Azure services, including Azure Data Lake Storage Gen2, Azure Databricks, Azure Synapse, Azure SQL, Azure Data Factory, and Event/Service Bus.
  • Define and implement data modeling standards, including dimensional, canonical, and domain-driven designs.
  • Design Delta Lake architectures, including medallion (BronzeSilverGold) frameworks.
  • Develop scalable and secure ETL/ELT pipelines using Databricks (PySpark/Spark SQL).

2. Databricks & Delta Lake Engineering

  • Implement Delta Lake best practices, including OPTIMIZE, ZORDER, Change Data Feed (CDF), schema evolution, and table versioning.
  • Optimize Spark workloads for performance, cost, and cluster utilization.
  • Lead governance implementation using Unity Catalog, including permissions, data lineage, and metadata modeling.

3. Azure Data Ecosystem Integration

  • Integrate solutions across Azure Data Factory, Synapse Pipelines, Event Hubs, Functions, Key Vault, Azure DevOps, and Purview.
  • Drive CI/CD and Infrastructure-as-Code using Terraform, Bicep, or ARM templates.
  • Build APIs and microservices supporting data consumption.

4. Data Governance & Security

  • Define policies for data quality, master data management, metadata, and data lifecycle management.
  • Ensure GDPR-compliant data design, encryption (KMS/Key Vault), RBAC/ABAC, and secure network architecture.
  • Establish frameworks for observability, including logging, monitoring, and alerting.

5. Stakeholder Leadership

  • Collaborate with business, product owners, engineers, and analysts to gather data requirements.
  • Lead architectural reviews and provide long-term technology strategy.
  • Mentor engineering teams on Azure, Databricks, and scalable data design principles.

Required Skills & Qualifications

Technical Skills

  • 7+ years in Data Architecture or Data Engineering roles.
  • Expert-level knowledge of: 
  • Azure Databricks (Spark, Delta Lake, MLflow), Agent Bricks
  • Azure Data Lake Gen2
  • Azure Data Factory / Synapse Pipelines
  • Azure SQL / Synapse Dedicated SQL / Cosmos DB
  • Unity Catalog & Delta Live Tables
  • Strong programming experience in PySpark, SQL, and Python.
  • Hands-on with CI/CD (Azure DevOps / GitHub Actions).
  • Deep understanding of distributed systems & big data performance tuning.

Soft Skills

  • Ability to translate business requirements into technical architecture.
  • Strong documentation, presentation, and interpersonal skills.
  • Leadership in multi-disciplinary teams and sophisticated enterprise environments.

Formal education

  • A Bachelor's (or higher) degree, ideally in Computer Science, MIS/IT, Mathematics, chartered accountancy, engineering, or equivalent work experience.

Preferred Qualifications and Skills

  • Azure certifications (e.g., DP-203, AZ-305, Databricks Data Engineer Professional/Associate).
  • Courses on Generative AI in Data engineering
  • Experience with data mesh, domain-driven design, and event-driven architecture.
  • Familiarity with Power BI or other BI tools.
  • Experience integrating governance tools such as Azure Purview.
  • TOGAF 9 certification

Why join us?

At bp, we provide an excellent working environment and employee benefits! We offer an open and inclusive culture and a great work-life balance. Tremendous learning and development opportunities are available to craft a career path, life and health insurance, medical care package and many others!

We support our people to learn and grow in a diverse and exciting environment. We believe that our team is strengthened by diversity. We are committed to crafting an inclusive environment in which everyone is respected and treated fairly. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, perform crucial job functions, and receive other benefits and privileges of employment.


Travel Requirement

Up to 10% travel should be expected with this role


Relocation Assistance:

This role is eligible for relocation within country


Remote Type:

This position is a hybrid of office/remote working


Skills:

Agility core practices, Business modelling, Creativity and Innovation, Data Analysis, Data design, Digital Analytics, Emerging technology monitoring, Information Assurance, Integration, IT Management, IT strategy and planning, Network Design, Requirements definition and management, Risk Management, Service operations and resiliency, Solution Architecture, Systems Development Management, Technical specialism, User Experience Analysis

.


Legal Disclaimer:

We are an equal opportunity employer. We do not discriminate on the basis of protected characteristics like race, religion, color, sex, national origin, sexual orientation, veteran status or disability status. Individuals with an accessibility need may request an adjustment/accommodation related to bps recruiting process (e.g., accessing the job application, completing required assessments, participating in telephone screenings or interviews, etc.). If you would like to request an adjustment/accommodation related to the recruitment process, please contact us.

If you are selected for a position and depending upon your role, your employment may be contingent upon adherence to local policy. This may include pre-placement drug screening, medical review of physical fitness for the role, and background checks.