Data Engineering for an E-Learning Platform: From Struggling to Scalable in 40 Days

Read on to find out how NaNLABS' strategic approach empowered the INE team to achieve their ambitious goals and develop a winning solution.

Table of contents

• Meet INE, a data-led IT training platform

• The brief: Upgrading INE’s analytics system

• The challenge: Scaling for growth without scaling costs

• The solution: A cloud-powered tech transformation

• The result: Unlocking a new level of data analytics

• Ready to elevate your data game?

Buried in data but lacking actionable insights? Get every last drop of value out of your data with solutions that dig up the insights and boost your business decisions. 

Speak to a data engineering expert

Meet INE, a data-led IT training platform 

INE is in the business of boosting IT skills through training and certifications. Known for blending hands-on learning with accessible video training, the team aims to offer the best learning experience for their users, from IT professionals to Fortune 500 companies - which means keeping their platform cutting-edge. As the platform grew, the INE team realized their analytics system needed an overhaul for better functionality and cost efficiency so they set out to improve how they track and understand user interactions.

Offering data engineering services with decades of expertise, INE partnered with us to create a new analytics system that was both scalable and wouldn’t break the bank. They wanted to make sure they could keep delivering great training without compromising on the quality of the tech behind it and stay at the forefront of IT education.

The Brief: Upgrading INE’s analytics system

INE's plan was to design a system that's quick, reliable, and won't get bogged down no matter how many users log on. But, even with a large internal engineering team, they needed a hand tackling this project within their budget.

With five tasks, two weeks, and 40 hours of NaNLABS data engineering expertise, we took on the project together from the ground up, analyzing everything from how the current system was built to how much it would cost to make necessary changes. NaNLABS and INE kept the lines of communication open throughout, aiming for daily updates to keep the project on track and make sure everyone was focusing their energy where it was needed most.

The game plan

1. Analyze the current setup: Understand the current architecture and business limitations, what’s working and what’s not, and why it's not scaling or cost-effective.

2. Research alternatives: Perform technical feasibility analysis, looking into other tech and strategies that could do the job better for less.

3. Crunch the operational costs: Figure out what the new setup would cost to run day-to-day.

4. Plan the data tracking: Data modeling to decide on which user actions need tracking for the most valuable insights.

5. Build and test a Proof of Concept (POC): Create a scaled-down model of the new system to see if it's up to scratch. This includes setting up a cloud architecture, testing data events, and trying out report generation.

The Challenge: Scaling for growth without scaling costs

As INE set out to revamp their analytics system with NaNLABS, they encountered significant challenges that needed addressing to support their growth and maintain cost efficiency.

Project challenges

1. Scalability vs. cost efficiency: INE's existing analytics infrastructure was struggling to scale with its growth without incurring prohibitive costs. The balance between expanding capabilities and managing expenses became a critical issue, as the system could no longer support the platform's evolving needs without a financial toll.

2. High-volume data management: With thousands of users engaging with the platform simultaneously, the system generated millions of data events daily. The sheer volume of data overwhelmed the current setup, highlighting the need for an infrastructure capable of efficiently managing and processing this influx.

3. Traffic management: The high user traffic posed its unique challenges. The infrastructure had to not only accommodate the vast number of concurrent users but also ensure the seamless collection, storage, and analysis of the data they generated. This required a robust underlying infrastructure designed for high availability and performance under load for a seamless user experience.

Technical and operational hurdles

1. Deciphering the current system: Understanding the intricacies of the existing analytics system's architecture was a significant initial barrier. NaNLABS had to undertake a comprehensive analysis to pinpoint inefficiencies and scalability blockers.

2. Balancing stakeholder needs: Tailoring the analytics solution to meet the varied requirements of different stakeholders—ranging from business analysts to API end-users—added another layer of complexity. Finding a common ground that addressed all their needs while ensuring system efficiency was a delicate balancing act.

3. Estimating operational costs: Projecting the operational costs of a new, scalable solution included a lot of uncertainties. Without clear future usage scenarios and relying heavily on historical data, NaNLABS faced challenges in creating accurate cost estimates that aligned with INE's budgetary constraints.

The Solution: A cloud-powered tech transformation

Overcoming the challenges presented by INE’s existing analytics system required a combination of innovative thinking and technical expertise. Here's how NaNLABS tackled the project:

Reverse engineering for insights

The initial step involved a thorough reverse engineering process of the current analytics system. This critical analysis allowed the team to dissect and understand the limitations and inefficiencies of the existing setup. By identifying these key areas, NaNLABS could pinpoint what needed enhancement and begin formulating cost-efficient alternatives.

Leveraging cloud technologies

The key to the new solution was the development of a robust backend infrastructure utilizing cloud technologies. This approach included:

• AWS Kinesis & Firehose for real-time data processing and streaming capabilities, ensuring that data could be handled efficiently at scale.

• Apache Zeppelin was selected for data exploration and visualization, offering an intuitive interface for insights and analysis.

• ElasticSearch & TimescaleDB for data storage, chosen for their scalability and performance in handling time-series data, critical for analyzing user interactions and platform metrics.

• The backend was developed in Python, chosen for its versatility and strong support for data manipulation and analysis.

Adopting an IaC approach

Infrastructure as Code (IaC) using AWS Cloud Formation and Terraform was implemented to streamline the deployment and management of cloud resources. This method allowed for automated, repeatable, and error-free deployment of the necessary infrastructure, ensuring scalability and manageability without significant manual overhead.

Data modeling and log integration

A sophisticated data modeling strategy was employed to ensure that the architecture could efficiently collect, store, and analyze data. By integrating logs into the system, NaNLABS ensured that every interaction on the platform could be captured and analyzed, providing valuable insights into user behavior and platform performance.

Proactive solutions beyond the brief

NaNLABS took the initiative to propose solutions that went beyond the client’s initial request. Recognizing that merely adapting the existing system would not achieve the desired outcomes, the team advocated for a more scalable and sustainable solution. This included implementing real-time processing capabilities to enhance decision-making processes, such as offering video recommendations or tailoring course suggestions based on user interactions.

By embracing leading technologies and methodologies, NaNLABS delivered a solution that was scalable, cost-efficient, and capable of delivering real-time insights, significantly surpassing the initial expectations and requirements of the project.

The Result: Unlocking a new level of data analytics for INE

Within a remarkably short period of 40 days, NaNLABS conceived, built, and delivered a state-of-the-art analytics system POC for INE, addressing and surpassing the initial project goals.

Robust analytics capabilities

The newly implemented system now seamlessly manages millions of events each day, providing INE with the robust analytics capabilities it needs. This performance boost has revolutionized the way the platform collects, stores, and analyzes data at scale, supporting INE’s mission to deliver world-class IT training.

Scalable and cost-efficient platform

Key to the project's success was ensuring that the analytics system was scalable but also cost-efficient. NaNLABS achieved this through the strategic use of cloud technologies and a forward-thinking approach to data architecture. This foundation ensures that INE can confidently plan for future upgrades and expansions without the worry of prohibitive costs or technical limitations.

Client satisfaction

The client's response to our tech consultancy and delivered solutions was overwhelmingly positive, especially with the demonstrated potential for cost reductions of up to 80% in high-demand scenarios. While the consultancy phase concluded with the design, code for implementation, and infrastructure code ready for INE's use, the groundwork laid by NaNLABS has positioned INE for significant operational improvements and cost savings.

Ready to elevate your data game?

• Are you struggling with the scalability and costs of your current data system?

• Looking for a partner to help navigate the complexities of data engineering and deliver real business outcomes?

• Interested in how data-driven insights can transform your business strategy and operations?

Transforming raw data into actionable insights is no small feat, but it's exactly what we thrive on at NaNLABS.

Why choose NaNLABS for your data engineering needs?

The data transformation we undertook with INE, crafting a state-of-the-art analytics system that exceeds scalability and efficiency expectations, is all part of our commitment to excellence and innovation in data engineering platform development and consultancy. Trust us to think beyond simply managing your data, leveraging it to drive strategic decisions and enhance your business outcomes.

If you're facing challenges with your data systems or looking to explore the vast potential of your data, we're here to help.

Buried in data but lacking actionable insights? Get every last drop of value out of your data with solutions that dig up the insights and boost your business decisions. 

Speak to a data engineering expert