Job Description
Senior Data Engineer
Veracity Group is the global outsourcing arm of St Anthony's Group of Industries, specializing in Business Process Management and Knowledge Process Management Services in key areas of Finance & Accounting, Process Automation, Data Science & AI, Mobility and Customer Experience Services. We cater to a diverse portfolio of clients from the US, UK, Australia, Middle East and Sri Lanka.
Veracity Digital creates software solutions for startups to mature organizations to bring their ideas into life! We are currently seeking a Senior Data Engineer to lead the design, development, and optimization of scalable data pipelines and analytical platforms using modern data engineering stacks. The ideal candidate will have experience with big data tools, cloud data platforms, and end-to-end data orchestration frameworks.
The Role
- Built and optimized ETL pipelines with Apache Spark for batch and streaming data.
- Developed and managed Apache Airflow DAGs for end-to-end data workflows.
- Led deployment of cloud data platforms on AWS and Azure (Redshift, Snowflake).
- Implemented DBT for modular, version-controlled in-warehouse transformations.
- Configured Trino (Presto) for federated querying across multiple data sources.
- Maintained CI/CD pipelines, containerized with Docker, deployed on Kubernetes.
- Supported ML workflows and tracking using MLflow, Scikit-learn, XGBoost, TensorFlow.
- Delivered BI dashboards in Power BI, Metabase, Superset, and QuickSight.
- Mentored engineers and established best practices for DevOps and data engineering.
The Person
- Bachelor’s degree in computer science, Engineering or a related discipline.
- Minimum of 4 years’ experience in a Data Engineer or equivalent role, with a proven record of delivering production-grade data platforms.
- Advanced Python and SQL proficiency for pipeline development, data transformation and performance tuning.
- Demonstrated expertise with Apache Spark, Apache Airflow, DBT and Trino (Presto).
- Hands-on experience architecting and managing Amazon Redshift and/or Snowflake environments.
- Familiarity with ML support libraries (Scikit-learn, XGBoost, TensorFlow) and ML lifecycle management in MLflow.
- Experience building and maintaining BI solutions in Power BI, Metabase, Apache Superset and AWS QuickSight.
- Exceptional problem-solving, communication and collaboration skills, with a track record of mentoring peers and driving complex, cross-team initiatives.
+94 112 074 146
www.veracitygp.com
Veracity Office - 275, Nawala Road, Nawala.
Hatch Works - 14 Sir Baron Jayatilaka Mawatha, Colombo.
