LOG IN
SIGN UP
Canary Wharfian - Online Investment Banking & Finance Community.
Sign In
Forgot password?
Don't have an account?
or
Join Canary Wharfian
By signing up, you agree to our Terms & Conditions and Privacy Policy.
or

Lead Software Engineer

ExperiencedNo visa sponsorship
J.P. Morgan logo

at J.P. Morgan

Bulge Bracket Investment Banks

Posted 6 days ago

No clicks

**Lead Software Engineer** at JPMorganChase: Craft and lead high-throughput, low-latency data pipelines using Apache Spark, Delta Lake, and Databricks. Drive performance, reliability, and scalability; evolve Lakehouse patterns. Leverage AI-assisted engineering tools to improve code quality and delivery speed. Manage Databricks clusters and automate workflows. Build reusable libraries, enforce data quality, and optimize Spark performance. Implement CI/CD for data projects. Requires 5+ years of software engineering experience, expertise in Apache Spark on Databricks/EMR, and strong Python proficiency.

Compensation
Not specified

Currency: Not specified

City
Not specified
Country
United States

Full Job Description

Location: Plano, TX, United States

We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible.

 

As a Lead Software Engineer at JPMorganChase within the Corporate Sector Corporate Technology, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firms business objectives.

 

Job Responsibilities

  • Lead the architecture and delivery of high-throughput, low-latency data pipelines on Databricks using Apache Spark (Core, SQL, Structured Streaming), driving performance, reliability, and scalability.

  • Establish and evolve Lakehouse patterns with Delta Lake (ACID transactions, schema evolution, time travel, Z-ordering, compaction) to ensure performant, maintainable data platforms at scale.

  • Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.

  • Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.

  • Own Databricks cluster strategy and configuration, including runtime selection, autoscaling, driver/executor sizing, Spark configurations, init scripts, cluster policies, pools, and instance profiles.

  • Orchestrate and automate pipelines and jobs using Databricks Workflows, integrating with AWS eventing and orchestration services as needed

  • Design secure ingestion and transformation frameworks leveraging Databricks services, including Delta or unmanaged table design, ingestion task creation, and Airflow DAGs to produce trusted and refined datasets.

  • Enforce data quality, lineage, and governance using Unity Catalog and/or AWS Glue Catalog, embedding expectations and validation directly into pipelines.

  • Drive Spark and Databricks performance engineering and tuning (partitioning and file sizing, AQE, broadcast joins, shuffle tuning, caching, spill/memory control, job right-sizing, and liquid clustering/partitioning keys) to optimize cost and throughput.

  • Build and maintain reusable libraries, frameworks, and APIs in Python and/or Java, ensuring strong unit, integration, and data validation test coverage.

  • Implement CI/CD for data projects using Git-based workflows, Terraform-based infrastructure deployments and environment promotion, and automated releases; champion engineering standards, code reviews, and enterprise-authorized AI-assisted engineering practices (e.g., code review/refactoring, test acceleration, and incident/root-cause analysis) with consistent validation (secure coding, peer review, automated testing) and reuse of proven patterns.

 

Required qualifications, capabilities, and skills:

  • Formal training or certification on software engineering concepts and 5+ years applied experience.

  • Advanced experience in software engineering and Data Engineering, including significant production delivery with Apache Spark on Databricks and/or AWS EMR.

  • Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.

  • Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices.

  • Advanced hands-on Databricks expertise across Delta Lake, Unity Catalog, Workflows, Repos/notebooks, and SQL Warehouses, including cluster configuration and optimization.

  • Proven ability to architect, build, and operate reliable ETL/ELT data pipelines (batch and streaming), including schema design/evolution, SLAs, and reliability engineering practices.

  • Deep Spark performance tuning skills, with experience diagnosing bottlenecks and optimizing jobs for scalability, cost, and runtime efficiency.

  • Strong programming proficiency in Python for data processing, platform tooling, and automation.

  • Strong SQL and analytics data modeling expertise, including dimensional/star schema design and Lakehouse best practices.

  • Demonstrated experience leading effective use of approved AI-assisted software development tools (coding, code review, test acceleration, troubleshooting), including setting team expectations and validation standards for correctness, performance, and security of AI outputs.

  • Strong responsible-AI and security-first engineering mindset, including data sensitivity awareness, secure handling of inputs/outputs, roles/instance profiles, secrets management, encryption at rest/in transit, network controls, and adherence to resiliency and security expectations; experience coaching teams on safe, compliant adoption within delivery practices.

 

Preferred qualifications, capabilities, and skills

  • Experience with Delta Live Tables and advanced governance (catalogs, grants, auditing) in Databricks.

  • AWS networking knowledge (VPC, subnets, routing, security groups) and data egress controls.

  • Experience with Terraform for Infra deployments

  • Cost optimization experience: autoscaling strategies, spot vs on-demand, auto-termination, storage layouts and compaction; Familiarity with Airflow, Genie, Streamlit and React

  • Observability for data systems (freshness/completeness metrics, lineage, SLAs, alerting); Demonstrated leadership in code quality, reviews, testing strategy, CI/CD, and technical mentorship; excellent communication with stakeholders.

Carry out critical tech solutions across multiple technical areas as an integral part of an agile team

Lead Software Engineer

Compensation

Not specified

City: Not specified

Country: United States

J.P. Morgan logo
Bulge Bracket Investment Banks

6 days ago

No clicks

at J.P. Morgan

ExperiencedNo visa sponsorship

**Lead Software Engineer** at JPMorganChase: Craft and lead high-throughput, low-latency data pipelines using Apache Spark, Delta Lake, and Databricks. Drive performance, reliability, and scalability; evolve Lakehouse patterns. Leverage AI-assisted engineering tools to improve code quality and delivery speed. Manage Databricks clusters and automate workflows. Build reusable libraries, enforce data quality, and optimize Spark performance. Implement CI/CD for data projects. Requires 5+ years of software engineering experience, expertise in Apache Spark on Databricks/EMR, and strong Python proficiency.

Full Job Description

Location: Plano, TX, United States

We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible.

 

As a Lead Software Engineer at JPMorganChase within the Corporate Sector Corporate Technology, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firms business objectives.

 

Job Responsibilities

  • Lead the architecture and delivery of high-throughput, low-latency data pipelines on Databricks using Apache Spark (Core, SQL, Structured Streaming), driving performance, reliability, and scalability.

  • Establish and evolve Lakehouse patterns with Delta Lake (ACID transactions, schema evolution, time travel, Z-ordering, compaction) to ensure performant, maintainable data platforms at scale.

  • Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.

  • Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.

  • Own Databricks cluster strategy and configuration, including runtime selection, autoscaling, driver/executor sizing, Spark configurations, init scripts, cluster policies, pools, and instance profiles.

  • Orchestrate and automate pipelines and jobs using Databricks Workflows, integrating with AWS eventing and orchestration services as needed

  • Design secure ingestion and transformation frameworks leveraging Databricks services, including Delta or unmanaged table design, ingestion task creation, and Airflow DAGs to produce trusted and refined datasets.

  • Enforce data quality, lineage, and governance using Unity Catalog and/or AWS Glue Catalog, embedding expectations and validation directly into pipelines.

  • Drive Spark and Databricks performance engineering and tuning (partitioning and file sizing, AQE, broadcast joins, shuffle tuning, caching, spill/memory control, job right-sizing, and liquid clustering/partitioning keys) to optimize cost and throughput.

  • Build and maintain reusable libraries, frameworks, and APIs in Python and/or Java, ensuring strong unit, integration, and data validation test coverage.

  • Implement CI/CD for data projects using Git-based workflows, Terraform-based infrastructure deployments and environment promotion, and automated releases; champion engineering standards, code reviews, and enterprise-authorized AI-assisted engineering practices (e.g., code review/refactoring, test acceleration, and incident/root-cause analysis) with consistent validation (secure coding, peer review, automated testing) and reuse of proven patterns.

 

Required qualifications, capabilities, and skills:

  • Formal training or certification on software engineering concepts and 5+ years applied experience.

  • Advanced experience in software engineering and Data Engineering, including significant production delivery with Apache Spark on Databricks and/or AWS EMR.

  • Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.

  • Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices.

  • Advanced hands-on Databricks expertise across Delta Lake, Unity Catalog, Workflows, Repos/notebooks, and SQL Warehouses, including cluster configuration and optimization.

  • Proven ability to architect, build, and operate reliable ETL/ELT data pipelines (batch and streaming), including schema design/evolution, SLAs, and reliability engineering practices.

  • Deep Spark performance tuning skills, with experience diagnosing bottlenecks and optimizing jobs for scalability, cost, and runtime efficiency.

  • Strong programming proficiency in Python for data processing, platform tooling, and automation.

  • Strong SQL and analytics data modeling expertise, including dimensional/star schema design and Lakehouse best practices.

  • Demonstrated experience leading effective use of approved AI-assisted software development tools (coding, code review, test acceleration, troubleshooting), including setting team expectations and validation standards for correctness, performance, and security of AI outputs.

  • Strong responsible-AI and security-first engineering mindset, including data sensitivity awareness, secure handling of inputs/outputs, roles/instance profiles, secrets management, encryption at rest/in transit, network controls, and adherence to resiliency and security expectations; experience coaching teams on safe, compliant adoption within delivery practices.

 

Preferred qualifications, capabilities, and skills

  • Experience with Delta Live Tables and advanced governance (catalogs, grants, auditing) in Databricks.

  • AWS networking knowledge (VPC, subnets, routing, security groups) and data egress controls.

  • Experience with Terraform for Infra deployments

  • Cost optimization experience: autoscaling strategies, spot vs on-demand, auto-termination, storage layouts and compaction; Familiarity with Airflow, Genie, Streamlit and React

  • Observability for data systems (freshness/completeness metrics, lineage, SLAs, alerting); Demonstrated leadership in code quality, reviews, testing strategy, CI/CD, and technical mentorship; excellent communication with stakeholders.

Carry out critical tech solutions across multiple technical areas as an integral part of an agile team