at Scotiabank Global Banking and Markets
Investment BankingPosted 4 days ago
No clicks
**DataOps Engineer Specialist - ScotiaTech, Bogotá** **Role Summary:** Integrate data across diverse sources into Azure Databricks (Delta Lake) while following DataOps practices. Collaborate with SMEs and engineers to understand data, improve quality, and ensure migration success. **Key Responsibilities:** - Integrate and validate data flow from legacy on-premises and GCP sources into Azure Databricks. - Deploy and reuse ingestion, transformation, and integration patterns following target design and standards. - Support DataOps (Git, CI/CD, automated testing, monitoring) for reliable, available, and performant pipelines. - Explore, map, and document legacy data to support migration, and articulate data meaning with SMEs. - Check data quality, cleanse, and prepare data across medallion layers (Bronze→Silver→Gold). - Build simple reports on data availability and migration progress. **Required Skills and Experience:** - 3+ years in DataOps, data integration, or data engineering support. - Proficient in Apache Spark (PySpark) and Python for large-scale data processing. - Strong SQL skills and experience with large-scale analytical databases. - Experience with data modeling, schema design, and data warehousing concepts. - Knowledge of data governance, security, monitoring, and observability practices within cloud environments. - Familiarity with Azure Databricks, Delta Lake, and Unity Catalog; experience with GCP is an asset. - Proficiency in working in Agile environments and collaborating with cross-functional teams. **Preferred Qualifications:** - Databricks certifications (Data Engineer Associate/Professional) or Azure DP-203 certification. - Experience with Lakehouse architecture and Delta
- Compensation
- Not specified
- City
- Not specified
- Country
- Not specified
Currency: Not specified
Full Job Description
ID de la solicitud: 271770
Gracias por tu inters en ScotiaTech, el nuevo e innovador centro tecnolgico de Scotiabank en Bogot.
nete a un equipo ganador que promueve la creatividad y la innovacin en un entorno cambiante, donde siempre estamos comprometidos con el logro de resultados, en una cultura inclusiva, diversa y de alto desempeo.
Purpose
The DataOps Engineer Specialist enables data integration and DataOps for International Banking, supporting the reliable flow of data across systems and into Harbour (Azure Databricks). Working hands-on with architecture and senior engineers, this role integrates data, deploys approved data patterns, and helps discover, map, and articulate legacy on-premises and GCP data so it can be migrated with confidence. Reporting to the Senior Data Engineer, the incumbent partners with SMEs and the engineering team to make data trusted, well-understood, and ready for use across International Banking markets.
Accountabilities
- Data Integration: Integrate data across sourcesincluding legacy on-premises into Azure Databricksand validate end-to-end data flow so data lands complete and usable.
- Pattern Deployment & Architecture: Deploy and reuse the teams approved ingestion, transformation, and integration patterns, partnering with architecture and senior engineers to follow the target design and standards.
- DataOps & Automation: Support DataOps practices (Git, CI/CD, automated testing, monitoring) to keep pipelines reliable, available, and performant.
- Discovery & Mapping: Explore legacy and GCP data, produce source-to-target mappings to the Delta Lake model, and document lineage and dependencies to support migration.
- Data Quality & Reconciliation: Check data quality and reconcile source vs. target (parity), cleansing and preparing data across the medallion layers (BronzeSilverGold).
- SME Collaboration & Articulation: Work with SMEs to understand and clearly articulate what data means, and support cataloguing in Unity Catalog so data is discoverable and trusted.
- Reporting & Visibility: Build simple reports/dashboards (e.g., Power BI) on data availability and migration progressby country, source, and owner.
Reporting Relationships
Primary Manager:
Direct Reports:
Shared Reports NA
Dimensions NA
Education / Experience / Other Information (include only those that are specific to the role)
Education
-
Bachelors degree in Computer Science, Data Engineering, Information Technology, Software Engineering, or a related field.
-
A Masters degree is considered a plus.
-
Relevant cloud or data engineering certifications (Databricks, GCP, Azure, AWS) are considered a plus.
Experience
-
3+ years of experience in DataOps, data integration, or data engineering support.
-
Experience integrating and mapping data, and supporting data migration or cloud modernization (on-prem or GCP to Azure Databricks is an asset).
-
Familiarity with DataOps practices (Git, CI/CD, automated testing, monitoring).
-
Experience working with SMEs and cross-functional teams in an Agile environment.
-
Strong proficiency in Apache Spark (PySpark) and Python for large-scale data processing.
-
Knowledge of data migration, cloud modernization, and platform transformation initiatives is highly desirable.
-
Strong SQL skills and experience working with large-scale analytical databases and data warehouses.
-
Experience with data modeling, schema design, and data warehousing concepts.
-
Understanding data governance, security, monitoring, and observability practices within cloud environments.
-
Experience working in Agile delivery environments and collaborating with cross-functional teams.
-
Understanding of data integration, ETL/ELT, and lakehouse/medallion patterns; Azure Databricks, Delta Lake, and Unity Catalog exposure is an asset.
Preferred Qualifications
-
Databricks certifications (Data Engineer Associate/Professional) are highly desirable; Azure DP-203 is an asset.
-
Experience with Lakehouse architecture and Delta Lake implementation.
-
Familiarity with infrastructure-as-code tools such as Terraform.
-
Exposure to data quality, data observability, and monitoring frameworks.
-
Experience supporting cloud migration initiatives from on-premises or legacy platforms.
Competencies
Working Conditions
Work in a standard office-based environment; non-standard hours are a common occurrence
Ubicacin(s): Colombia : Bogota : Bogota ScotiaTech es una unidad de negocio de ScotiaGBS, un grupo de empresas de Scotiabank, ubicado en Bogot, Colombia. ScotiaTech fue creado para apoyar diversos sistemas y procesos tecnolgicos del Banco. Ofrecemos un entorno de trabajo inclusivo y positivo, adems de ventajas competitivas. En ScotiaTech, valoramos las habilidades y experiencias nicas que cada persona aporta y nos hemos comprometido a crear y mantener un entorno inclusivo y accesible para todos. Los candidatos deben postularse directamente en lnea si desean ser tomados en cuenta para este puesto. Agradecemos a todos los candidatos por su inters en esta oportunidad profesional en ScotiaTech; sin embargo, solo contactaremos a quienes hayan sido seleccionados para una entrevista.
