Cloud Data Engineer (The Data Pipeline Architect)
Tech Stack
Job Description
Are you passionate about building and optimizing data pipelines that drive data-driven decision-making in cloud environments?
Do you have the technical expertise to design robust data architectures that manage large-scale data processing and ensure seamless data flow?
If you’re ready to harness the power of cloud technology to create scalable data solutions, our client has the ideal role for you.
We’re seeking a Cloud Data Engineer (aka The Data Pipeline Architect) to design, develop, and manage cloud-based data infrastructures that support analytical and operational needs across the organization.As a Cloud Data Engineer at our client, you’ll collaborate with data scientists, analysts, and software engineers to build data pipelines and storage solutions that are secure, efficient, and scalable.
Your role will be vital in ensuring that data systems are optimized for performance and capable of supporting a range of data-driven initiatives.Key Responsibilities: Design and Build Data Pipelines: Create and manage scalable data pipelines that support ETL (Extract, Transform, Load) processes.
You’ll develop automated solutions to handle data ingestion, transformation, and integration in cloud platforms such as AWS, GCP, or Azure.
Develop and Maintain Data Architectures: Architect and maintain cloud-based data storage solutions, such as data lakes and warehouses, ensuring they align with business needs and best practices.
You’ll optimize data structures for performance and reliability.
Collaborate on Data Strategy and Solutions: Work closely with data scientists, business analysts, and application developers to understand data requirements and implement solutions that support analytics and machine learning models.
Ensure Data Security and Compliance: Implement and monitor security measures that protect data privacy and comply with industry standards (e.g., GDPR, HIPAA).
You’ll manage data access controls and audit logs to safeguard sensitive information.
Optimize Data Processing Workflows: Improve the performance of data processing workflows by optimizing resource allocation and parallel processing techniques.
You’ll work to reduce latency and enhance the scalability of data systems.
Automate and Streamline Data Operations: Develop automation scripts and tools using languages such as Python or Java to streamline data processes and reduce manual intervention.
You’ll contribute to the efficiency and reliability of data operations.
Monitor and Troubleshoot Data Pipelines: Use monitoring tools and techniques to ensure the health and performance of data pipelines.
You’ll identify and troubleshoot issues promptly to maintain smooth data flow and system uptime.