Main Job Purpose

Building robust and scalable data infrastructure and systems. Participating in the development of data pipelines, data models, and data integration solutions, ensuring the efficient processing, storage, and analysis of large volumes of data. Collaborating closely with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and implement solutions that enable effective data-driven decision-making.

Accountabilities

• Design, build, and maintain scalable and efficient data pipelines for extracting, transforming, and loading (ETL) data from various sources into target systems, data warehouses, or data lakes. Ensure the reliability, performance, and integrity of data pipelines.
• Develop data integration strategies to consolidate data from multiple sources, ensuring data consistency and accuracy.
• Implement data transformation processes to convert and cleanse data according to defined business rules and data quality standards. Handle data cleansing, data standardization, and data enrichment tasks to improve data quality and usability.
• Establish and maintain scalable data storage systems such as relational databases, NoSQL databases, or cloud-based data warehouses. Optimize data storage and retrieval for efficient data processing and analysis.
• Utilize big data technologies and frameworks such as Apache Hadoop, Apache Spark, or cloud-based services like Amazon Redshift, DMS…etc. to handle large volumes of structured and unstructured data. Implement distributed computing techniques for parallel processing and high-performance data processing.
• Develop and implement data quality assurance processes, including data validation rules, data profiling, and data quality monitoring. Identify and address data quality issues, anomalies, or inconsistencies in a proactive and timely manner to decide as per high quality data.
• Monitor and optimize data pipelines, data processing workflows, and data storage systems for performance and efficiency. Identify and resolve bottlenecks, optimize queries, and tune data processing operations for improved system performance to avoid any system failures.
• Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand their data requirements and provide technical expertise. Document data engineering processes, data flows, and system configurations for knowledge sharing and future reference.
• Stay updated with emerging technologies, trends, and best practices in the field of data engineering, continuously learn and evaluate new tools, techniques, and frameworks to improve data engineering capabilities and drive innovation.