Integration of market data to a scalable cloud architecture
Objective
A UK-based alternative market firm wanted to integrate and analyze large volumes of data from various traditional and non-traditional sources.
Challenges
- Manual processing and updates for new data were conducted
- Inconsistent data formats and structures
Approach
- Extracted data from PDFs, Excel, using web scraping with DB design on AWS Cloud for faster processing and scalability
- Designed a data quality framework to perform data mapping and ensure data integrity
- Integrated various sources of alternate data to complement traditional sources
- Built interactive Tableau dashboards for data visualization
Impact
- Provided a central repository of cloud-based AWS scalable data models
- Completed automated solution from sourcing to visualization, reduced processing efforts by 90%
Questions