Data Engineering Consultancy
Data Engineering consultancy & POC development of an analytics system
Data Engineering Consultancy for a B2B IT e-learning platform. Our scope of work involved initial architecture assessment, solution research, architecture development, and POC conception.
AWS
Python
Elasticsearch
INE helps IT professionals improve their skills by offering world-class IT training and certifications. The e-learning platform provides hands-on practice and on-demand instructor-led video training. INE is committed to deliver the most advanced technical training on the planet. Hence, keeping their platform optimized is vitale. Their latest major initiative consisted in rebuilding some of the platform’s software solutions, including the analytics system. NaNLABS was brought in to provide expertise on Data Engineering and handle the development of a POC for the brand new analytics system.
Goals
INE’s analytics system had reached its limits in terms of scalability, and it wasn’t profitable to keep maintaining it. The e-learning platform needed data engineering experts to provide solutions and deliver a custom analytics system. Considering the high volume of data generated by thousands of simultaneous connections to the platform, the new architecture had to be scalable, and cost-effective. After modeling data of the events to be tracked, the goal was to develop a POC for the new analytics system.
Challenges and Solutions
From the very beginning, our client mentioned that they’d like the solution to be scalable and cost-efficient. To provide our client with viable solutions, we had to understand the initial analytics system first, by performing reverse engineering. It helped us define what needed to be improved and narrow down cost-efficient alternatives.
The solution implemented involved developing the back-end with Cloud technologies, such as AWS Kinesis & Firehose to process, analyze, and stream data. Exploration & visualization were powered by Apache Zeppelin and storage was enabled by ElasticSearch & TimescaleDB. Back-end was programmed using Python. We adopted an Infrastructure as Code (IaC) approach using AWS Cloud Formation & Terraform.
The initial infrastructure of INE’s analytics system wasn’t fulfilling the requirements of the platform anymore. This system had become expensive to maintain and wasn’t optimum to collect and manage the data & metrics that INE needed. Our experts in data modeling ensure to include logs, enabling the architecture to collect information and get insights from it.
Another particularity of this project was the high traffic supported by the initial system. Thousands of users log into the platform simultaneously, which generates millions of events every day. Hence, website traffic was a factor to consider when we designed the solutions since we would have to conceive an underlying infrastructure able to manage high traffic and high flow of data.
Results
We conceived, built, and delivered a state-of-the-art analytics system in 40 days.
Our client can rely on an analytics system able to manage millions of events every day.
We ensured to build a scalable and cost-efficient platform allowing future upgrades.
The new logs integrated into the analytics system enable it to collect, store & analyze data to provide insights and support INE performance enhancement initiatives.
Are you looking for a reliable partner to help you solve complex technical issues? Get in touch! We offer bespoke consulting services to support your team and propel your project forward. Yes, I want to propel my project forward!
