Mouna Tayehi

Work with the team to understand legacy pipelines

Support legacy pipelines

Help the team design and implement new pipelines architecture

Explore new tech stack and suggest next steps

Data Curation tool:

The design and implementation of a data curation command line for data storage / manipulation and metadata maintenance.

Rewriting the data ingestion pipeline using DeltaLake to support incremental loads of data

Setup Airfow dags to run data pipelines

The development of a Python API that provides an alternate way to execute commands through Jupyter notebooks to give data scientists more flexibility to perform data analysis in a Python environment in a scalable way (pySpark env).

Creating Containers in Docker.

The configuration of the Jenkins pipeline to create, test and publish the docker images to custom registries.

Orchestrating services with both docker-compose and Kubernetes.

Data storage on Azure DataLake (Gen1 and Gen2).

Writing unit tests (Scala) and integration tests (Gherkin / Python)

Building and optimizing big data pipelines.

Client Daily batch pipeline:

The calculation of the attributes necessary for the application.

Investigating customer data anomalies on Azure DataBricks.

The implementation of an online forecasting service from the given attributes.

Retail Data Generation

Data profiling and statistics generation to be used in creating time series data as well as its different hierarchical attributes

Designing and implementing of a configuration per data type

Implementing a generic Python module that, based on a config, generates business data with meaningful patterns

Online Forecasting Service

Design and implementation of an API that queries a service to forecast promotions in real time.

Cache loaded data in-memory asynchronously.

Unit test the application.

Build of a module that queries the online forecaster and compares results against static results.

Dockerize and deploy the service with Helm on Kubernetes.

Budget forecasting system for IT projects:

Exploring the machine learning process.

Exploring the different types of machine learning and the different learning algorithms used.

Exploring data mining methods.

Exploring the Azure Machine Learning service and how it works.

Exploring recommendation systems to reach freelancers with employers.

Building a data science pipeline for estimating IT project prices that uses a predefined list of answered questions as attributes to predict the price tag using Azure Machine Learning.

Exploring natural language processing methods.

Exploring the different methods of extracting data from natural language.

Exploring question answering systems.

Performing Data Augmentation by synonymy in order to have a close distance between the description of the project and the questions predefined by the system.

Designing and implementing a question answering system for IT projects that extracts their functionalities from an additional description (or specifications) which will eventually be used in the price model of the IT project.