Posted 7 days ago
No clicks
**Lead Cloud Platform Engineer (Data & Execution Platform)** Drive cloud-native infrastructure and services for large-scale data processing systems. This hands-on role requires deep expertise in Apache Spark and Airflow to translate defined architectures into production-grade platforms. Lead the implementation and operation of a modern execution platform built on Spark for distributed compute and Airflow for orchestration. Leverage AWS services (EKS, EC2, S3, etc.) and developer productivity tools like AI-assisted coding to ensure reliability, security, and performance. Mentor engineers and promote best practices. Eight plus years of software, platform, or cloud engineering experience required.
- Compensation
- Not specified
- City
- Not specified
- Country
- Not specified
Currency: Not specified
Full Job Description
Job Description:
Note: Fidelity is not providing immigration sponsorship for this position.
The Role
We are seeking a hands-on Lead Cloud Platform Engineer to implement, scale, and operate cloud-native infrastructure and services that power large-scale data processing systems. This role focuses on translating defined architectures into production-grade platforms that are reliable, observable, secure, and performant. You will lead the implementation and operation of a modern execution platform built on Apache Spark for distributed compute and an Airflow orchestration layer and DAG execution environment. The ideal candidate brings deep production experience in Spark and Airflow, and excels at troubleshooting, tuning, and operationalizing distributed systems in AWS environments, while leveraging modern developer productivity tools such as AI-assisted coding and LLM-based workflows.
The Expertise and Skills You Bring
Implement and operate cloud-native platform services for distributed data systems
Scale fault-tolerant, high-throughput systems aligned with architectural patterns
Own Spark data pipelines and Airflow orchestration layer and DAG execution
Tune Spark workloads (partitioning, memory, execution plans, shuffle optimization)
Troubleshoot Spark jobs and Airflow DAGs across performance and failures
Operate and optimize Kubernetes-based execution environments, including node group scaling, workload placement, and resource utilization
Troubleshoot Kubernetes infrastructure and workload issues, including scheduling, networking, and runtime performance
Leverage developer productivity tools (e.g., GitHub Copilot, LLMs) to accelerate development, debugging, and operational workflows.
Drive operational excellence including monitoring, incident response, and RCA
Implement observability (metrics, logging, tracing, dashboards, alerting)
Define and manage SLIs/SLOs for platform reliability
Deploy solutions using AWS services (EKS, EC2, S3, Lambda, RDS, etc.) (Implement secure networking (VPCs, IAM, subnets, load balancing)
Maintain CI/CD pipelines and deployment automation
Lead execution across planning, delivery, and cross-team coordination
Mentor engineers and promote reliability and scalability best practices
Strong understanding of distributed systems (fault tolerance, scalability, consistency
Expertise in Apache Spark (tuning, debugging, optimization)
Expertise in Apache Airflow (DAG execution, orchestration, troubleshooting)
Strong experience operating Kubernetes (EKS preferred) including cluster scaling and lifecycle management
Hands-on management of node groups, autoscaling, and capacity planning
Deep understanding of Kubernetes networking and security (security groups, network policies, ingress/egress)
Experience with Kubernetes resources (Deployments, StatefulSets, Jobs, CronJobs)
Familiarity with Custom Resources (CRDs) and advanced configuration via annotations and labels
Experience monitoring Kubernetes clusters (metrics, logs, events) and integrating with observability tools
Troubleshooting Kubernetes workloads (scheduling failures, resource contention, networking issues)
Experience with AWS services and cloud-native design patterns
Proficiency in Python, Java, or Go
Experience with Docker and Kubernetes
Hands-on observability (metrics, logging, tracing)
Experience with SLI/SLO-based reliability models
Practical experience using AI-assisted development tools (e.g., GitHub Copilot, LLMs) to improve code quality, debugging, and productivity
Networking fundamentals (DNS, TCP/IP, TLS, VPC design)
Strong troubleshooting and performance tuning skills
Strong communication and leadership skills
Bachelors or Masters degree in Computer Science or related field (or equivalent experience)
8 plus years in software, platform, or cloud engineering roles
Experience operating large-scale distributed systems in production
Strong experience with AWS cloud platforms
Mandatory hands-on experience with Apache Spark and Apache Airflow in production
Experience supporting ETL, data platforms, or workflow execution systems at scale
Fidelitys Onsite Working Model
Fidelity is transitioning to a full-time onsite working model through a phased rollout across regions and roles. Currently, some roles and locations require 100% onsite presence, while others require less. Onsite expectations are likely to evolve as the rollout continues. This transition does not apply to fully remote roles.
Certifications:
Category:
Information TechnologyPlease be advised that Fidelitys business is governed by the provisions of the Securities Exchange Act of 1934, the Investment Advisers Act of 1940, the Investment Company Act of 1940, ERISA, numerous state laws governing securities, investment and retirement-related financial activities and the rules and regulations of numerous self-regulatory organizations, including FINRA, among others. Those laws and regulations may restrict Fidelity from hiring and/or associating with individuals with certain Criminal Histories.
Apply
All fields are required. Candidates should limit the number of roles they apply to at any given time.
Benefits that balance life and work
From our fully paid parent leave to our on-site health and wellness centers, our benefits support the belief that more balance you have, the better you can achieve your goals.
Company overview
Company overview
At Fidelity, we are passionate about making our financial expertise broadly accessible and effective in helping people live the lives they want. We are a privately held company that places a high degree of value in creating and nurturing a work environment that attracts the best talent and reflects our commitment to our associates. We are proud of our diverse and inclusive workplace where we respect and value our associates for their unique perspectives and experience.
Reasonable accommodations
Fidelity will reasonably accommodate applicants with disabilities who need adjustments to participate in the application or interview process. To initiate a request for an accommodation contact the HR Accommodation Team by sending an email to accommodations@fmr.com, or by calling 800-835-5099, prompt 2, option 3.
Equal opportunity employer
Fidelity Investments is an equal opportunity employer. We believe that the most effective way to attract, develop, and retain a diverse workforce is to build an enduring culture of inclusion and belonging.
Applicant screening
At Fidelity, we value honesty, integrity, and the safety of our associates and customers within a heavily regulated industry. Certain roles may require candidates to go through a preliminary credit check during the screening process. Candidates who are presented with a Fidelity offer will need to go through a background investigation and may be asked to provide additional documentation as requested. This investigation includes but is not limited to a criminal, civil litigations and regulatory review, employment, education, and credit review (role dependent). These investigations will account for 7 years or more of history, depending on the role. Where permitted by federal or state law, Fidelity will also conduct a pre-employment drug screen, which will review for the following substances: Amphetamines, THC (marijuana), cocaine, opiates, phencyclidine.
AI Guidelines
Learn about our guidelines for use of AI when applying for a Fidelity job
