Techelse: Coin Streaming Pipeline
Techelse is a fintech company that leverages cutting-edge technology to offer financial solutions, such as payments, investing, and trading. Bitcoin is one of the innovative payment methods that they are developing based on the power of cryptography and decentralization.
Project description
Machine learning technology is transforming every domain of the market, and Bitcoin trading is no exception. Coin Streaming Pipeline is a project that aims to build a data system that seamlessly streams coin prices from customer-provided API, and displays them on the client website. The project also harnesses machine learning to forecast the future prices of the coins based on historical data and trends.
Challenge: Performance of Gigabytes streaming data
The coin market is diverse and dynamic, and each coin generates massive data per minute or even per second. When the customer approached NeuSplend, they wanted us to build a solution that could handle a large volume and variety of data from different sources and formats. The data architecture should be highly available even in case of a disaster or zone outage.
It required NeuSplend to build a solution that is smart, high availability and good performance:
The solution that was capable of processing a large amount of streaming data. The estimated data size was ~ 1 GB/minute.
The data architecture had to be resilient and recoverable in case of a disaster. NeuSplend planned to build every component in multiple zones to ensure the high availability of the system.
Choosing the best model for predicting coin prices was also a mathematical challenge that had to be solved. The optimal model had to fit the customer’s needs and expectations.
Solution: A robust streaming pipeline with the ability to predict coin prices.
After evaluating the customer’s existing infrastructure and the format of the customer-provided API, we decided to choose Azure cloud platform as the main cloud provider for our solution. Azure ML would also be used to train Random Forests, which are a powerful machine learning technique to predict the future coin prices.
NeuSplend’s QA tested the ML models to ensure that the customer can use it to forecast and trade cryptocurrencies under changing market conditions.
We believe that discovery is not a time-consuming step, but a valuable one that gives us the essential information that we need to build and deploy the excellent services to the customer. Thanks to the careful plans prepared beforehand, we deployed the solution in separate regions that give the solution high availability even in case of an outage or a downstream issue in one region. The ML models also fit with the data architecture, and provide the customer with accurate price predictions that they can use to make important decisions. Below is the project architecture we have deployed:
The result: Predicting the priceless information for customers
After using the new data streaming architecture integrated with Azure machine learning service, Techelse can now rely on the historical data and the streaming data they get from the pipeline to forecast the potential prices of their preferred coins in the future.
NeuSplend has designed the data architecture not only to be highly available in case of regional disasters, but also to optimize the cost for maintaining and developing. After completing this project, we still work hand in hand together to enhance and improve this data architecture to make it stronger and faster.
Technologies we apply
Data Engineering & Cloud Tools
Databases & Distribution Systems
BigData tools
Data Science and Machine Learning tools
Azure Managed Airflow
Azure Event Hub Cluster
Azure Key Vault
Azure Container Registry
Microsoft PowerBI
Grafana
Azure DB for PostgreSQL
Azure Blob Storage
Azure Storage Account (Logs and Data)
Azure Storage Account for Models
Azure Data Explorer
Azure Managed Airflow
Azure Machine Learning Studio Workspace
Azure Application Insights
Python
Our work
NeuSplend is a powerhouse of diverse and innovative solutions for any domain and customer. We don’t just work for you because of the project, we work with you to ensure your success and happiness.
