Spark / Scala Developer

Role Overview We are seeking an experienced Spark/Scala developer to design, develop, and optimize large-scale data pipelines on big data platforms. The ideal candidate will have strong expertise in Hadoop ecosystem tools, data engineering practices, and scalable data processing frameworks. Key Responsibilities Design and develop scalable data pipelines using Spark and Scala Build and maintain data ingestion pipelines using Hadoop ecosystem tools (HDFS, Hive, Impala, Sqoop) Lead end-to-end implementation of data engineering solutions from development to production deployment Collaborate with data science and analytics teams to enable data-driven use cases Participate in requirement analysis, solution design, and architecture discussions Lead a team of 3–5 data engineers and ensure timely delivery of data products Implement data transformation and processing workflows on Hadoop platforms Ensure code quality through testing, CI/CD pipelines, and best practices Work in Agile environments using tools like Jira and Confluence

Required Skills & Experience 6–12 years of experience in Data Engineering / Big Data Strong hands-on experience with Spark and Scala (Mandatory) Expertise in Hadoop ecosystem: HDFS, Hive, Impala, Sqoop Advanced SQL and strong understanding of relational databases Experience with Unix/Linux environments Familiarity with CI/CD tools such as Git, Jenkins, Nexus Experience working in Agile methodologies Technical Competencies Strong knowledge of Big Data ingestion and processing frameworks Ability to build and optimize large-scale distributed systems Experience in designing and implementing data pipelines Strong coding, testing, and deployment skills Good to Have Experience with PySpark Exposure to Data Analytics use cases Knowledge of SAS and Tableau Strong command over Unix and HDFS operations