Clement Demonchy

In a team responsible for exposition and anonymization of production tables

-Change Data Capture of relational database to iceberg table using Debezium

-Create mirror tables with Pyspark job on EMR serverless

-Create and maintain dataset on google cloud using Crossplane and Kopf

-Install resources on GKE using Argocd

In a team responsible for working with electric vehicle supply equipment (EVSE) and exposing it with REST API for multiple BtB companies

-Migration of data workflow from Azure synapse to AWS databricks

-Writing CI job using Github Actions and Databricks bundle

-Setting test for spark on dev machines to improve feedback

-Define alerts based on Lambda metrics using cloudwatch

-Expose metrics of data workflow with PostgreSQL Aurora and grafana

-Documentation on workflow using Mermaid on Github

-Analyzing and reducing cost on Azure and AWS, for example by dividing cost by 5 for AWS Lambda exploiting caching between service and improving batching of SQS elements

-Optimize geolocalization operation on PostgreSQL and detect invalid coordinates

I was responsible for a 4 person team of 3 Data engineer and 1 ML engineer at Skillup, an SaaS helping RH in the company to provide a better experience providing training for their employees and evaluating impact.

The team had a broad scope covering 4 principle subjects

-External data flux from customer with client import (user, booked training, …) & export (training evaluation)

-Scrapping & cleaning training website for our catalog

-Provide internal metrics & dashboard so products team can gain insights

-Use latest AI technologies (LLM) to implement new feature for customer

My first task when starting here was to put in place a CI/CD using github action to structure coding practice with automated testing and linting.

All the while implementing team's rituals (postmortem, retrospective, daily, ownership rotation) as well as mentoring & pairing to share best coding practice (ruff, pydantic, pytest, …).

This allowed us to refactor existing code more easily and made the team more resilient with people being able to work on other team member code.

The scrapping project required migration of Airflow orchestrated data pipelines that use Zyte for scrapping online training websites and extracting data using NLP models.

The frontend of Skillup being developed in Typescript we rely on GRPC API to interact with their system.

New services developed to allow complex importation of customer data were an Fastapi REST API to query Neo4j, MongoDB and PostgreSQL databases to reconcile existing data.

We then provided a website in intern developed in Dash for the product team in case a manual validation was required.

For the dashboarding we wanted to give a lot of autonomy for no tech savvy user and deployed both Metabase and Superset.

For the IA related project we needed to do some optimisation of OpenIA POC for generation of job's skills based on role title