Bulge Bracket Investment Banks

Posted 6 days ago

No clicks

**Lead Software Engineer – LLM Ops Platform Reliability** at JPMorgan Chase in Glasgow. Seniors build and operate LLMs (like vLLM, lllmd) in production. They develop high-quality software, optimize GPU usage, and ensure reliability. Key skills include Python, AWS, Terraform, and site reliability engineering (SRE) practices. On-call experience a must.

Compensation: Not specified GBP
City: Not specified
Country: United Kingdom

Full Job Description

Location: GLASGOW, LANARKSHIRE, United Kingdom

Help shape how AI systems run reliably in production at scale. In this role, youll build and operate large language model serving infrastructure, bringing strong engineering fundamentals and site reliability practices to cutting-edge AI platforms. Youll work hands-on with cloud and Kubernetes-based deployments, deep observability, and cost-aware performance tuning. If you enjoy solving hard production problems and making platforms measurably better, youll find meaningful impact and growth here.

As a Lead Software Engineer at JPMorgan Chase in the AI and Machine Learning Platform team, you will build and scale AI infrastructure that modernizes traditional infrastructure management and site reliability engineering through applied AI. You will own the reliability, performance, and cost-efficiency of the LLM inference platform end to end. You will operate large language model serving stacks (such as vLLM and llm-d) in production at scale, with deep instrumentation and strong operational rigor. You will partner across engineering to deliver secure software, improve stability, and lead incident response and continuous improvement.

Job Responsibilities

Design, develop, troubleshoot, and deliver secure, high-quality production software and services for AI infrastructure

Build backend services and APIs that enable reliable operation of AI infrastructure in production

Operate and scale LLM serving infrastructure (such as vLLM and llm-d), including model hosting, request routing, continuous batching, and KV-cache optimization

Deploy, host, and lifecycle-manage open-source and proprietary LLMs on Amazon EKS and Amazon SageMaker, as well as on-prem and local GPU clusters, using reproducible infrastructure as code and continuous delivery pipelines

Implement observability (logs, metrics, traces) with dashboards and actionable alerting, including Prometheus metrics and Grafana/Alertmanager integration for LLM and GPU workloads

Tune GPU and accelerator capacity, autoscaling, and cost efficiency for LLM inference workloads using performance and optimization techniques (e.g., quantization, parallelism, speculative decoding)

Lead reliability engineering for LLM endpoints through capacity planning, load/soak testing, safe rollouts (blue/green, canary), failover, and incident response for outages and model-quality regressions

Participate in an on-call rotation, lead incident triage and mitigation, and produce clear post-incident root-cause analyses and follow-ups

Identify recurring operational issues and automate remediation to improve platform stability and developer experience

Build and maintain multi-agent systems with strong orchestration (planning, coordination, tool-calling, state/memory, and workflow control) where appropriate

Contribute to an inclusive team culture grounded in diversity, opportunity, inclusion, and respect, and help drive adoption of leading-edge technologies through communities of practice

Required Qualifications, Capabilities, and Skills

Formal training, certification, or equivalent practical experience in software engineering concepts

Hands-on experience with system design, application development, testing, and operational stability in production environments

Advanced proficiency in Python for building production-grade services and tooling

Proficiency with automation and continuous delivery methods

Hands-on experience with AWS and Terraform for infrastructure delivery and lifecycle management

Strong understanding of site reliability engineering practices, including incident management, root-cause analysis, runbooks, and reliability patterns

Practical knowledge of observability and instrumentation across metrics, logs, and traces

Comfort with on-call operations and production troubleshooting

Hands-on production experience operating LLM inference servers such as vLLM and llm-d (or directly equivalent serving stacks)

Hands-on experience hosting and serving LLMs on Amazon EKS and/or Amazon SageMaker, and on local GPU infrastructure

Knowledge of LLM reliability and risk considerations, including latency/throughput trade-offs, model and weight versioning, prompt/response logging, and safe rollout patterns

Preferred Qualifications, Capabilities, and Skills

Experience developing generative AI applications, AI agents, vector search, and retrieval-augmented generation patterns

Experience building AI agents using frameworks such as LangChain, CrewAI, LangGraph, or similar orchestration platforms

Experience operating or integrating serving platforms such as KServe, Ray Serve, NVIDIA Triton Inference Server, Text Generation Inference (TGI), alongside vLLM/llm-d

Familiarity with Amazon SageMaker JumpStart, SageMaker Endpoints, and Amazon Bedrock for managed model hosting

Experience with online LLM quality monitoring (e.g., hallucination, toxicity, drift detection) and tracing via OpenTelemetry conventions

Contributions to open-source LLM serving or inference projects (e.g., vLLM, llm-d, Ray, KServe, Triton)

Build and run production LLM infrastructure at scale, improving reliability, performance, and cost efficiency with applied AI.

Full Job Description