Data Engineer (Pyspark/SQL)

Job Title: Data Engineer Location: Indore, Bangalore, Raipur Job Type: Full-time Experience: 7-15 Years 

About the Role: 

We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. The ideal candidate will have a strong background in PySpark, SQL, and Unix technologies and will be responsible for designing, developing, and maintaining scalable data pipelines. You will collaborate with cross-functional teams to ensure the availability, reliability, and performance of our data infrastructure, enabling data-driven decision-making across the organization. Key Responsibilities: 

● Design, develop, and optimize large-scale data pipelines using PySpark. 

● Write efficient and complex SQL queries for data extraction, transformation, and analysis.

● Perform data extraction, transformation, and loading (ETL) processes from various data sources. 

● Manage and maintain Unix-based systems to ensure smooth operation of data workflows.

● Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and provide scalable solutions. 

● Ensure data quality, reliability, and security through effective monitoring and validation techniques. 

● Automate data workflows and processes to enhance efficiency and reduce manual intervention. 

● Troubleshoot and resolve issues related to data processing, storage, and performance.

● Stay up-to-date with the latest trends and technologies in data engineering and implement best practices. Required Skills & Qualifications: 

● Proficiency in PySpark for distributed data processing.

● Strong knowledge of SQL and experience with databases such as MySQL, PostgreSQL, or Oracle. 

● Hands-on experience with Unix/Linux environments, including shell scripting. 

● Experience in building and optimizing large-scale ETL pipelines. ● Solid understanding of data warehousing concepts and data modeling. 

● Familiarity with cloud platforms (e.g., AWS, Azure, or Google Cloud) is a plus. 

● Strong analytical and problem-solving skills.

● Excellent communication and teamwork abilities. Preferred Qualifications:

● Bachelor’s degree in Computer Science, Engineering, or a related field.

● Experience with big data technologies (e.g., Hadoop, Spark, Hive). 

● Knowledge of data security and governance practices.