- Building and optimizing data pipelines: Design and implement data ingestion, transformation, and analysis pipelines using tools like Spark, Kafka, and Hadoop.
- Building data infrastructure: Architect and deploy scalable and secure big data platforms on premises or in the cloud using technologies like Hadoop, AWS, Azure, or GCP.
- Prepare and transform raw data into a usable format for analysis, including data cleaning, validation, and transformation tasks.
- Collaborate with data scientists, and business stakeholders to understand their needs and translate them into actionable data solutions.
- Write scripts and automate routine tasks to streamline processes and boost efficiency.
Proficiency in Hadoop ecosystem (HDFS, Yarn, MapReduce, etc.), Spark, Kafka,Nifi,Hive and data warehousing tools.Experience with real-time data processing and streaming analytics.Strong programming skills in Python.Familiarity with scripting languages like Bash or Shell.Knowledge of data modeling, ETL/ELT processes, SQL, and statistical concepts.Understanding of distributed systems, cloud platforms, and infrastructure as code (IaC).Excellent analytical and troubleshooting skills to handle data challenges.Effective communication skills to collaborate with diverse teams and stakeholders.