Software Developer

Seeking Software Developer for our staff to:• Design, develop and modify data workflow software system using scientific analysis and mathematical models to predict and measure outcome and handle the consequences of

design.• Develop huge volume of data lake on amazon redshift by using analytics components of amazon web services (aws) including amazon s3, amazon elastic mapreduce (emr), amazon athena and bigdata ecosystem tools including hadoop, hive, hdfs, spark (pyspark).• Gather requirements from business analysts, develop and support it till the end of project lifecycle.• Create scheduling plan, job execution timings and sharing with scheduling team.• Build the code as per the design documents created.• Engage in the migration of etl processes from oracle to hive and develop code to importing and exporting data from oracle and db2 into hdfs and hive using sqoop.• Develop hive queries to load and process data in hadoop file system.• Develop hive udfs to sort structure fields and return complex data type.• Create hive tables with partitions such as static and dynamic with bucketing.• Develop spark jobs using python on top of yarn for interactive and batch analysis.• Convert hive/sql queries into spark transformations using spark sql (data frames) with python.• Import data from various formats like json, sequential, text, csv, avro and parquet to hdfs cluster and produced them with compression for optimization using pyspark.• Perform data analysis using pyspark (sparksql).• Use pyspark and developed scripts to filter & calculate aggregate data and write the resulted data to variety of file formats with compression for optimization as per business needs.• Test and debug.

Master's Degree in Computer Science, plus 12 months of work experience as a Software Developer or Software Engineer.