Ali Gouta

Cloud and Data software architect: data platform (phenix project-core team)

As a Member of the Core and Ops team of the phenix project, I endorsed the following roles:

Cloud (GCP) architect :

▪ Contribution to define the migration strategy of the data platform to GCP.

▪ Implementing the shared VPC strategy to connect the phenix GCP

platform to external Datacenters (IBM Datacenter).

▪ Configuring google LB4, Haproxy and Nginx instances at the host project

to forward traffic to service projects that hosts the phenix data platform mainly composed of offline architecture ( Cloudera) and a streaming architecture (Mesos)

▪ Configuring the firewall rules, DNSs zones to expose services, apis ,... between legacy platforms on external datacenters and platforms on GCP.

▪ Setting up a one way Kerberos trust between legacy datacenters and GCP to duplicate traffic through distcp.

▪ Defining the project strategy to create and secure the service accounts by leveraing google kms service to encrypt service accounts keys and pushing them into secured buckets. Also define naming convention of the buckets, bigquery datasets, and service accounts used by the projects.

▪ Full automation with ansible and google deployment manager.

Kubernetes Architect :

▪ Automate the deploying of GKE clusters + enabling rbacs to make kubernetes clusters inherit IAM roles and google Groups.

▪ Define the organization and deployment strategy on GKE clusters.

▪ Implementing Nginx ingress controllers to expose services and securing

the communications between internal and external components to GKE.

▪ Defining the project strategy on how to use Kustomize for deployments on k8s + help building the CICD workflow with Jenkins by leveraging the

kubernetes plugin.

▪ Enabling and leveraging workload Identity for authentication and

authorizing applications running on GKE.

Data and software Architect :

▪ Implement a new architecture to run spark streaming workloads on GKE

▪ Contributing and committing to google spark-on-k8s-operator github repository to run spark jobs on kubernetes and industrializing the operator

(with Golang) within carrefour:

o PR#952: Filter Custom resources on specific labels to allow

running multiple operator instances on GKE

o PR#935: exposing container Ports to Prometheus scraping

o PR#914: Support ingress configuration in crd to expose the

sparkUI in private networks.

▪ Monitoring the spark applications using the Prometheus operator

▪ Migrate the core phenix pipeline libraries developed in scala from spark

2.2.1/kafka 0.8 to spark2.4.5/kafka 2.4 with all breaking changes of using

kafka to manage consumers offsets instead of Zookeeper.

▪ Support building of spark applications with sbt to publish the assembly jars

into gcs instead of nexus.

▪ Leading the migration of more than 50 spark streaming pipeline

(normalizers and persisters) from Mesos/Marathon to GKE.

▪ Extending Kafka MirrorMaker 2 (MM2) which is based on the kafka connect framework to support specific topic naming policies, containerizing and running MM2 to duplicate flows between kafka

clusters.

Data software architect:

▪ Migration apis from scalatra to Spring boot

▪ Implementing the maven CICD pipeline to build and push Spring boot docker images

▪ Develp kafka sink bigtable connector

▪ Transforming and persisting Avro data in different backends

▪ Designing and implementing streaming pipelines based on kafka and schema registry

▪ Migrating from Azkhaban to Airflow and implementing the CICD pipeline to deliver airflow and python projects

▪ Implementing the maven CICD pipeline to build and push Spring boot docker images

Security referee :

▪ Reshaping authentication and authorization methods at phenix project (main SI project in Carrefour) by implementing an openLdap cluster with saslauthd enabled to proxy authenticated users to the Ldap Group. Groups are defined locally on the openldap.

▪ Installing and Securing Cloudera clusters by leveraging the ldap as a main entry point for authentication and authorization

▪ Proposing and implementing new methods to allow to clients outside the cluster to access to HDFS/Hive without the need to have a Kerberos ticket. This is by implementing and enabling Knox parcel on the cluster instead of HttpFs which requires Kerberos and configuring extra Hive servers with Ldap authentication. All of this while preserving the user impersonation.

▪ Extending a python client library to communicate with Cloudera Manager and to implement the required rest calls to install and configure Knox parcel.

▪ Providing support and expertise to the whole phenix team and its clients.

▪ Full automation with ansible of all kinds of deployments through rundeck.

Hortonworks Solution Architect at Société Générale: Hadoop (Hortonworks):

▪ Hadoop Security Expert: Designing and implementing of secured solutions for security requirements.

▪ Deploying a fully secured HDP cluster in Homologation (C3 cluster for sensitive data): Kerberos, Ranger, wire encryption, ...

▪ Installation and configuration of a new secured development/integration cluster for projects with ranger and Kerberos enabled.

▪ Synchronization ranger, with LDAPs, and Configuring sssd for ldap authentication

▪ Full automation of installation and configuration of components/products for the cluster with ansible

▪ Configuring backup cluster, and providing solutions for disaster recovery strategies.

▪ Configuring and running mirror-maker to backup streaming data in secured environments (Kafka Acls; SSL and Kerberos).

▪ Defining and implementing the migration strategy from using Kafka ACLs to Ranger policies and migration from self-signed certificates to CA signed certificates for Kafka SSL listener.

▪ Enabling wire encryption and managing SSL certificates on major Hadoop components.

▪ Installing and configuring Hue on a HA and kerberized cluster and synchronization with ldap.

▪ Installing and configuring Knox to connect reporting tools on Hive such as Tableau.

▪ Setup of Prometheus for monitoring and alerting of the most critical components: ldap, FS size, ...

Talend:

▪ Define and implementation of the Talend-Big Data architecture in all Societe Genreale environments.

▪ Connecting the different TAC instances to the Active Directory group and securing the communication with SSL.

▪ Implementing ansible playbooks to install TAC and jobservers.

▪ Define and implementation the logging strategy for Talend projects that

use Kafka (SASL)

▪ Defining best practices and security strategies to isolate jobservers with

cgroups for projects and to authenticate each jobserver with Kerberos.

▪ Configuration and installation of Talend Data Quality on a kerberized environment: Integration with Kafka for data dictionary service and HDFS

to import/export data.

Technical development leader of a regulatory project -Mesh contract- to address the IFRS 9.2 requirements in term of regulations using Big Data technologies at Société Générale:

▪ Hortonworks consultant

▪ Defining the software stack for the project.

▪ Contributing and leading the developments.

▪ Implementing the continuous delivery/integration process for the project ▪ Successful production deployment of the project