Skip to content
SPEC INDIA

SPEC INDIA

Request a Quote
Menu
  • Services
      • Custom Software Development
      • Enterprise Mobility
      • Technology Consulting
      • Business Intelligence & Analytics
      • On-Demand Team
      • Product Engineering
      • Legacy App Migration
      • Software Testing & QA
      • SLA Driven 24/7 Support
  • Technology
      • .NET core
      • ASP.NET
      • Java
      • Power BI
      • Tableau
      • Pentaho
      • DevOps
      • IoT
      • Beacons
      • Fullstack
      • MEAN Stack
      • React
      • AngularJS
      • NodeJS
      • VueJS
      • Kotlin
      • Swift
      • React Native
      • Flutter
      • Python
      • ML
      • Big Data
      • Salesforce
      • R
      • AI
      • Azure
      • AWS
    • Our Innovation

      Hire highly-skilled on-demand teams to build sustainable, secure, and strategic digital solutions.

      Let's Connect
  • Industries
      • Retail & FMCG
      • Healthcare
      • Supply Chain & Logistics
      • Education
      • Manufacturing
      • Advertising
      • Travel & Hospitality
      • Media & Entertainment
      • Energy & Utilities
      • BFSI
  • Business Solutions
      • Custom Mobile ERP
      • Mobile Sales Force Automation (SFA)
      • Enterprise CRM
      • Warehouse Management System
      • Enterprise Service & Maintenance Solution
      • Vehicle Tracking System
      • Help Desk Management System
      • Self Service BI Plugin
      • Spot Billing System
  • Our Work
    • Case Study
      Live BI Visualization
      Portfolio
  • Company
      • Overview
      • Quality Policy
      • GDPR
      • Testimonials
      • Life @ SPEC
      • CSR
      • Events & News
      • Engagement Models
      • Project Execution Models
      • Contact Us
      • Career
      • Blog
CLOSE

Stock Market Data Integration and Analytics Tool

SPEC INDIA/Case Study/Stock Market Data Integration and Analytics Tool

About the Customer

Industry: Finance

The client is a leading news agency of Australia. This is first of its kind – Global Stock Ranking tool that can search foreign exchanges, sectors and company value ranges and order the companies by quality and value for investment. It can use predefined formulas or be modified to allow users to input their own criteria and produce an attractive, easy to interpret graphical display.

Key Challenges

The key challenge that the client faced was of consolidating data from various sources to a single data source. There was a need to create a system which can rank the companies from stock exchanges all over the world. Each company will be ranked over the cost of accounts and their past performance. So, the user can search any foreign exchange, sector and company value ranges and order the companies by value and quality for investment. Following were the hurdles being faced:

  • Design a data warehouse for such a huge system where data is coming every second.
  • Data was coming from multiple sources like txt, csv, xml, xlsx, web service and legacy database system.
  • Processing historical data for 60K companies from all over the world of 40years (from 1970).
  • Need for ETL process to download multiple zip files each of 3gig from Reuters web api and extract these zip files and process approx 140K xml files each of approx 1Mb.
  • Total size of xml files in each run was approximately 30gig.
  • Extract live data of each stock from web api and populate them in data warehouse.
  • Design a data warehouse which can serve client on the fly with necessary analytic.
  • Validate xmls against xsd schema and clean data to populate warehouse.
  • Download large files from web api’s in a slow network with resume capabilities.
  • Calculate probabilities for each company stock using complex calculations.

Our Solution

SPEC INDIA developed a Data Warehouse and Analytics Solution for the customer which helped them in doing effective data analysis. The solution covered following features:

  • Consolidated Data Warehouse
    • Enterprise data warehouse design with help of Kimball bus architecture.
    • Consolidate global stock market data for web based analysis to user.
  • Forecasting Stock Prices
    • Apply algorithms to predict the stock prices for providing analytic to the end user.
    • Identify/Forecast the stocks to be purchased.
  • Develop Robust ETL process
    • Download zip file in chunks to speed up the download using third party down loader integration with Pentaho Data Integration.
    • Process 160K yearly and interim files in each cycle, with a highly distributed and robust process developed and executed over a Pentaho kettle cluster using carte server.
    • Generally, a single xml file with lot of data hierarchy and of approx 1Mb size take 5-6 second to process but our team managed to finish the whole process for each file in 500-800 milliseconds using custom architecture.
    • Each xml file contains data about company, issues, COAs, issue periods, statements, company contacts, company officers, COA values, issue prices and their calculated aggregates over the period.
    • An ETL process executes at every hour and populates stock prices for interim price changes from web api.
    • ETL process is highly flexible for exception handling, notification sending and logging the errors.
    • ETL job can be resumed from any failure point in community edition.
    • Designed various aggregates and process in night cycle.
  • Visualization Using Open Source
    • For meaningful analysis and to compare the companies over sector, market cap, issue, we have implemented Lobster JS.
    • Lobster JS is an open source JS which provides interactive graphics and enables user for meaningful analysis.

Tools and Technologies

Business Benefits

SPEC INDIA’s successfully implemented solution has helped its client to make meaningful analysis before purchasing stock from stock exchanges over the globe.

It benefited them in the following ways:

  • Strong analytical data representation in clear way from the large set of historical data.
  • Help in decision making of stock purchases for the end user.
  • Rich Data visualization helps in identifying performing and non-performing companies and compares them to take a wise decision before investment.
  • Predicting/Forecasting stock prices.
  • It gave answers to the following questions:
    • How does the company make money?
    • How promising is the overall economic environment for company?
    • What Does the Company Do?
    • How fast is the Company Growing?
    • How Profitable Is It?
    • Is It Worth the Price?


Download Now

Stock Market Data Integration and Analytics Tool


https://www.spec-india.com/wp-content/uploads/2018/12/stock-market.pdf

    Please prove you are human by selecting the Key.

    Contact Now

      Please prove you are human by selecting the House.

      Safe & Confidential

      INDIA

      “SPEC House”, Parth Complex, Near Swastik Cross Roads, Navarangpura, Ahmedabad 380009, INDIA.

      +91-79-26404031, 32, 33, 34

      [email protected]

      USA

      “SPEC Partner", 350 Grove Street, Bridgewater, NJ 08807, United States.

      +1 908-450-9862

      [email protected]

      Services

      • Custom Software Development
      • Enterprise Mobility
      • Business Intelligence And Analytics
      • Web Application Development
      • Mobile App Development
      • UI / UX Design
      • Technology Consulting
      • Cloud Deployment
      • DevOps

      Quick Links

      • Blog
      • Career
      • News
      • Life @ SPEC
      • Case Study
      • Newsletter
      • Contact Us
      • Sitemap
      • Privacy Policy
      • Terms of Use
      • Tech in 200 Words

      © 2021 SPEC INDIA. All Rights Reserved.

      DMCA.com Protection Status

      Request a Quote

        .doc, .docx, .pdf, .xls, .xlsx, .jpg and .png files with 5MB size allowed.

        Please prove you are human by selecting the Car.

        Safe & Confidential
        We use cookies to enhance your user experience. By continuing to browse, you hereby agree to the use of cookies.
        To know more; visit our Privacy Policy & Cookies PolicyAccept and Close
        Privacy & Cookies Policy

        Privacy Overview

        This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
        Necessary
        Always Enabled

        Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.

        Non-necessary

        Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.

        Scroll Up