Cloud (AWS) / Databricks Technical Architect

Job Summary:

We are seeking an experienced and dynamic Cloud (AWS) / Databricks Technical Architect to join our team. The ideal candidate will have extensive expertise in designing, building, and deploying cloud-based solutions using AWS and Databricks, with a focus on data engineering, analytics, and machine learning. In this role, you will be responsible for driving the architecture and implementation of scalable, secure, and efficient cloud-based data solutions that support the organization's data-driven initiatives.

Key Responsibilities:

• Lead the design and implementation of cloud-based data architectures on AWS, utilizing a wide range of AWS services (e.g., S3, EC2, Lambda, RDS, Redshift, Athena, Glue).
• Architect and deploy scalable, secure, and cost-effective Databricks environments to process large volumes of data for analytics, data engineering, and machine learning.
• Provide leadership in designing modern data architectures, including real-time data pipelines, ETL/ELT workflows, and big data processing systems using Databricks and AWS technologies.
• Define and implement best practices for managing and optimizing data lakes, data warehouses, and data pipelines.
• Ensure architecture decisions align with business requirements, security policies, and compliance standards.

Requirements

• AWS Certification: Maintain and leverage AWS certification to design and implement cloud solutions.
• Cloud Architecture: Design and optimize cloud architecture for scalability and efficiency.
• Containers & Orchestration: Implement containers and orchestration tools for streamlined application deployment.
• Microservices Architecture: Design and manage microservices architecture for flexible and scalable systems.
• Cloud Environment Setup and Configurations: Set up and configure cloud environments to meet project requirements.
• Security & Access Management: Ensure secure access management and compliance within cloud environments.
• SQL, Python, Visualization & Analytical Tools: Use SQL, Python, and analytical tools for data processing and visualization.
• API Development & Management: Develop and manage APIs for seamless data integration and functionality.

Education & Experience:

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field.
• 8+ years of experience in cloud architecture, data engineering, or a similar technical role, with at least 5 years of hands-on experience working with AWS and Databricks.
• Proven track record in architecting and deploying large-scale data engineering solutions on AWS and Databricks.
• Experience working with various data processing frameworks (e.g., Apache Spark, Apache Kafka, Airflow) and cloud-based data storage solutions.

Technical Skills & Competencies:

• Deep expertise in AWS services, including but not limited to S3, EC2, Lambda, Glue, Redshift, Athena, and RDS.
• Strong experience with Databricks, including notebook creation, Spark-based processing, and managing Databricks clusters.
• Expertise in data engineering concepts, including ETL/ELT, data lakes, data pipelines, and real-time streaming architectures.
• Proficiency in programming languages such as Python, Scala, SQL, or Java for data processing and solution development.
• Experience with DevOps practices, CI/CD pipelines, containerization (e.g., Docker, Kubernetes), and infrastructure as code (e.g., Terraform, CloudFormation).
• Familiarity with machine learning workflows and tools, particularly those that integrate with Databricks (e.g., MLflow, Spark MLlib).
• Strong understanding of cloud security best practices, including IAM, encryption, and network security.

Preferred Qualifications:

• Experience with big data frameworks (e.g., Hadoop, Spark) and container orchestration platforms (e.g., Kubernetes).
• Familiarity with data governance, privacy, and compliance frameworks.