Cloud Data Engineer

We are seeking a highly skilled and motivated Cloud Data Engineer with a strong backend development focus and experience in Cloud platforms (AWS/GCP). As a member of our team, you will be responsible for the design, implementation, and maintenance of our data-centric, Cloud-hosted backend infrastructure and services. You will play a critical role in building and scaling our data processing pipelines and APIs, ensuring high availability, security, and performance.

Key Responsibilities:

• Build and optimize data pipelines using ELT frameworks such as Fivetran, dbt, Airflow, and PySpark, ensuring efficient data movement and transformation. Experience with schema design and data modeling is essential.

• Design, develop, and maintain high-throughput, low-latency backend services for data ingestion, processing, and transformation using languages such as Python and Java.

• Develop and maintain robust, scalable, and secure RESTful APIs using frameworks such as Spring Boot (Java) or Flask/FastAPI (Python). Experience with API gateway services (e.g., Apigee, Kong) a plus.

• Advise/design OData APIs.

• Implement and manage security protocols such as OAuth, SSO, and SSL.

• Design and implement event-driven architectures leveraging technologies like Kafka or Pub/Sub.

• Develop and implement CI/CD pipelines using tools like Jenkins, GitLab CI, or similar, to automate the deployment and scaling of backend services.

• Containerize applications using Docker and orchestrate them using Kubernetes. Experience with managing Kubernetes clusters (e.g., GKE, EKS) is preferred.

• Write and maintain infrastructure-as-code (IAC) using tools such as Pulumi, Terraform or CloudFormation.

• Collaborate with Architects and Security teams to ensure that the Cloud infrastructure and services are secure and compliant with industry standards (e.g., SOC 2, HIPAA, GDPR).

• Troubleshoot and resolve service-related issues, providing timely and efficient solutions.

• Stay up to date with the latest developments in Cloud technology, backend development best practices, and security vulnerabilities.

• Mentor and provide guidance to other team members as needed.

Requirements:

• Bachelor’s degree in Computer Science, Computer Engineering, or a related field.

• At least 5 years of experience in backend development in a Cloud environment (AWS or GCP preferred).

• At least 3 years of experience in Data Engineering in a Cloud environment.

• Strong proficiency in Python and Java; SQL proficiency is required.

• Extensive experience with RESTful API design, implementation, and testing using relevant frameworks. Experience with OpenAPI/Swagger specifications a plus.

• Experience with event-driven architectures and message queues (e.g., Kafka, RabbitMQ, Azure Event Hub, Amazon MSK).

• Experience with ELT data processing using frameworks such as Fivetran, dbt, Airflow, and/or PySpark.

• Proven experience with containerization (Docker) and orchestration (Kubernetes).

• Experience with CI/CD pipelines and infrastructure-as-code (Pulumi/Terraform).

• Working knowledge of Linux, shell scripting, and Git.

• Strong understanding of cloud security best practices, including IAM and data encryption.

• Excellent problem-solving and analytical skills.

• Excellent written and verbal communication skills.

• Ability to collaborate effectively in a team environment.