AppPress: Inflation Rate Insights
AppPress is a fintech company that provides innovative solutions for mobile payments, e-commerce, and digital banking. They aims to make finance more accessible, convenient, and secure for everyone.
Project description
Imagine a world where you can track the changes in the prices of everything you buy online, from groceries to gadgets, from clothes to cars. And AppPress, an e-commerce company, wants to know that. They have used our data engineering services to build a solution that can calculate the online inflation rate by following the internet pricing fluctuations for products and services. The project must also have ability to predict the price in the future by trained ML models.
Challenge: Handling Petabytes of Data
NeuSplend was challenged to develop the robust data pipeline that automatically crawl petabytes of data from Common Crawl bucket stored on AWS S3, calculate the inflation rates throughout multiple years and build the ML models on it. The solution must be fast, effective and high accuracy because client use these insights to make the important decisions in their business.
It required NeuSplend to handle multiple obstacles to deliver the excellent services to the customer:
Scalable processing of lots of crawled data. Estimated yearly data size is ~10 PB or about a billion events.
Processing data in the high-speed capacity. Since downloading is a long and expensive process, NeuSplend had to reduce this amount to a minimum.
Extracting key insights from the Petabytes of data. Because of being raw data, our team need to extract the proper goods price and calculate the inflation rate year by year.
Solution: Robust data pipeline based on the cloud platform
Leverage the power of the cloud platform that hosts the massive Common Crawl data, we chose AWS as our preferred cloud provider to deliver a complete end-to-end system solution to our customer.
NeuSplend tackled the project with a comprehensive plan that consisted of three main phases: Proof of Concept (PoC), Minimum Viable Product (MVP), and Continuous Robustness.
We started the project with the discovery phases, where we reviewed the customer’s existing system and designed a suitable data architecture for their business needs. Then we built and deployed the pipeline including ML models, testing it thoroughly and delivering the excellent services to the customer with the comprehensive documentation. The solution addressed all the challenges mentioned in the Challenge tab, while keeping the maintenance cost as low as possible for the customer. Below is the project architecture:
The result: Uncovering microeconomic insights through online inflation rate
AppPress could uncover the hidden patterns and trends of the online prices of goods and services through microeconomic analysis with this project. From these valuable insights, they have grown their business by several $1M in recent years.
Today, we continue to support AppPress for maintaining the pipeline to run effectively. With the significant changes of the market, the data architecture needs to adapt quickly, and NeuSplend will make it grow with the development of customer business.
Technologies we apply
Data Engineering & Cloud Tools
Databases & Distribution Systems
BigData tools
Data Science and Machine Learning tools
AWS Athena
AWS Lambda
AWS Cloudwatch
AWS API Gateway
AWS EC2
Apache Superset
AWS S3
Apache Hadoop
Apache Cassandra
Amazon EMR
Apache Hadoop
Apache Spark
AWS SageMaker
Jupyter notebook
Python
Our work
NeuSplend is a powerhouse of diverse and innovative solutions for any domain and customer. We don’t just work for you because of the project, we work with you to ensure your success and happiness.
