Data Engineer

Overview We are seeking a skilled and passionate Data Engineer to design, develop, and maintain data solutions that support Business Intelligence (BI) and Data Warehouse initiatives. In this role, you will work closely with Product Owners, Project Managers, and cross-functional teams to build scalable and secure data engineering products. You will play a key role in enabling data-driven decision-making across the organization through efficient data pipelines, integration of diverse data sources, and strong data governance practices. Responsibilities Collaborate with the Product Owner to develop, implement, and maintain data engineering products for BI and Data Warehouse projects. Design, develop, and deploy data solutions, including data tables, views, marts, pipelines, data warehouses, operational data stores, data lakes, and data virtualization. Perform data extraction, cleaning, transformation, and flow, including web scraping as required for data acquisition. Build and maintain large-scale, reliable batch and real-time data pipelines using modern data processing frameworks. Integrate and consolidate data silos into scalable and compliant systems. Ensure adherence to quality processes and governance standards across the data product lifecycle, with a focus on production environment standards. Implement and maintain data governance practices, ensuring compliance with security policies and regulatory requirements. Work closely with Project Managers, Data Architects, Business Analysts, Frontend Developers, Designers, and Data Analysts to deliver scalable, data-driven products. Develop backend APIs and work with databases to support applications. Contribute to Agile development practices, including Continuous Integration and Delivery, pair programming, and code reviews. Collaborate with business stakeholders to gather and analyse requirements, ensuring technical solutions align with business goals. Support business users in developing front-end reports, dashboards, and visualizations to enhance data analysis capabilities. Provide operational maintenance and technical support for deployed data engineering systems. Manage incident response and service requests, ensuring timely resolution within established SLAs. Participate in regular project audits and provide training support to team members and stakeholders as needed. Knowledge and Skills At least 5 years of experience as a Data Engineer. Proficient in data cleaning and transformation using SQL, Python (or R), and related tools. Strong experience in building ETL/ELT pipelines using SSIS, AWS DMS, Python, AWS Lambda, ECS, EventBridge, AWS Glue, or Spring. Proficient in database design and management across multiple databases such as SQL, PostgreSQL, AWS S3, Athena, MongoDB, MySQL, SQLite, VoltDB, Cassandra, etc. Hands-on experience with cloud platforms such as AWS, Azure, or Google Cloud. Experience in big data environments and cloud-based data engineering solutions. Proven experience in building production-grade data pipelines and integration workflows. Strong understanding of system design, data structures, and algorithms. Familiarity with data modelling, data storage solutions (Data Marts, Data Lakes, Data Virtualization, Data Warehouses) for optimal performance. Working knowledge of REST APIs and web protocols. Exposure to big data frameworks and tools (e.g., Hadoop, Spark, Kafka, RabbitMQ). Experience with web scraping tools and frameworks (e.g., BeautifulSoup, CasperJS, PhantomJS, Selenium, Node.js). Understanding of data governance policies, access control, and security best practices. Proficiency in at least one scripting language (e.g., SQL, Python). Comfortable working in both Windows and Linux development environments. Strong interest in bridging engineering and analytics functions. Excellent communication and collaboration skills, with the ability to work closely with stakeholders and technical teams. Seniority level Associate Employment type Full-time Job function Information Technology Industries IT Services and IT Consulting #J-18808-Ljbffr