Skip to content
SPEC INDIA

SPEC INDIA

Request a Quote
Menu
  • Services
      • Custom Software Development
      • Enterprise Mobility
      • Technology Consulting
      • Business Intelligence & Analytics
      • On-Demand Team
      • Product Engineering
      • Legacy Application Modernization
      • Software Testing & QA
      • SLA Driven 24/7 Support
  • Technology
      • .NET core
      • ASP.NET
      • Java
      • Power BI
      • Tableau
      • Pentaho
      • DevOps
      • IoT
      • Beacons
      • Fullstack
      • MEAN Stack
      • React
      • AngularJS
      • NodeJS
      • VueJS
      • Kotlin
      • Swift
      • React Native
      • Flutter
      • Python
      • ML
      • Big Data
      • Salesforce
      • R
      • AI
      • Azure
      • AWS
    • Our Innovation

      Hire highly-skilled on-demand teams to build sustainable, secure, and strategic digital solutions.

      Let's Connect
  • Industries
      • Retail & FMCG
      • Healthcare
      • Supply Chain & Logistics
      • Education
      • Manufacturing
      • Advertising
      • Travel & Hospitality
      • Media & Entertainment
      • Energy & Utilities
      • BFSI
  • Business Solutions
      • Custom Mobile ERP
      • Mobile Sales Force Automation (SFA)
      • Enterprise CRM
      • Warehouse Management System
      • Enterprise Service & Maintenance Solution
      • Vehicle Tracking System
      • Help Desk Management System
      • Laundry Management System
      • Spot Billing System
  • Our Work
    • Case Study
      Live BI Visualization
      Portfolio
  • Company
      • Overview
      • Quality Policy
      • GDPR
      • Testimonials
      • Life @ SPEC
      • CSR
      • Events & News
      • Engagement Models
      • Project Execution Models
      • Contact Us
      • Career
      • Blog
CLOSE

DevOps Principles in Product Data Distribution

SPEC INDIA/Case Study/DevOps Principles in Product Data Distribution

About the Customer

Industry:

In a typical enterprise application, product data is distributed across multiple systems – sales, purchase, POS, CRM, billing, warehouse, reporting etc. Since product data is central to any enterprise and mission critical for any type of workflows, most systems maintain a local copy of product data. Any change in the same requires an update in all the downstream systems.

DevOps-In-Data-Distribution

Key Challenges

Business Problem

With the increased volume of product data changes due to ever-changing business needs and requirement of multiple test environments creation driven by agile adoption, it becomes significantly difficult to support delivery of product changes on a daily basis. Majority of data management teams’ time occupied in supporting product data distribution and there is little focus on innovation. The end result is that product data distribution and deployment process becomes a bottleneck – impacting business agility and competitiveness.

Technical Problem

Existing data distribution process was an age-old process and had several drawbacks:

  • Product data distribution and deployment only happen once a day. Any subsequent change to product data requires 24-hrs window to reflect in all the systems.
  • With an increase in data volume over a period of time, systems often face stability issues due to high load.
  • It typically takes 5-6 hours to complete data distribution and deployment across all downstream systems – impacting e2e integration testing.
  • There are many manual touch points that require coordination among multiple dev and testing teams.
DevOps-In-Data-Distribution-Challenges

Our Solution

Solution Design Goals

As part of devops journey, teams decided to work closely on improving the process with an aim to achieve the following design goals:

  • Multiple, smaller, on-demand data deployments as against one large scheduled deployment
  • Delta refresh as against full refresh
  • Automate Data deployment with the one-click process as against manual touch-points
  • Near-zero downtime in test environments
DevOps-In-Data-Distribution-Solutions

Tools & Technologies

DevOps-in-Data-Distribution_Tools

Architecture Changes

There were several architectural challenges to meet the above design goals. Out-of-the-box thinking and use of open source tools allowed teams to meet stated objectives. Some of the key architecture changes are:

  • Download & Delta Refresh: In the existing process, all systems refresh local copy of data completely irrespective of the amount of change. With the new design, an intermediate layer introduced that enables all systems to have only delta refresh. This greatly helped to reduce time to download data from the source and refresh in the local system.
  • Synchronize Data Refresh: With multiple systems involved – each having its own data storage, format and ETL process, the major question was ‘how to synchronize data refresh across all impacted systems?‘ Considering each system taking its own time to refresh data, if one system completes refreshing while other is in progress causes the challenge of systems going out of sync.
  • To solve this problem, the notification process is implemented using an open source messaging broker; RabbitMQ. Every time there is a change in product data, a notification message is sent to each of the downstream system using RabbitMQ so each consumer can react to the event.
  • Zero downtime: To ensure that any data deployment does not impact ongoing testing, all systems initially deploy data in an offline database and upon confirmation of latest data available in all systems, a notification is sent to each system to flip from offline to online db.

Contact Now

    Please prove you are human by selecting the Heart.

    Safe & Confidential

    INDIA

    “SPEC House”, Parth Complex, Near Swastik Cross Roads, Navarangpura, Ahmedabad 380009, INDIA.

    +91-79-26404031, 32, 33, 34

    [email protected]

    USA

    “SPEC Partner", 350 Grove Street, Bridgewater, NJ 08807, United States.

    +1 908-450-9862

    [email protected]
    Goodfirms_Logo ISO_27001_Certified ISO_Icon top_mobile_app_developers_india_2020 Top_BI_Big_Data_Consultants_2020 Software_Developers_India_2020

    Services

    • Custom Software Development
    • Enterprise Mobility
    • Business Intelligence And Analytics
    • Web Application Development
    • Mobile App Development
    • UI / UX Design
    • Technology Consulting
    • Cloud Deployment
    • DevOps

    Quick Links

    • Blog
    • Career
    • News
    • Life @ SPEC
    • Case Study
    • Newsletter
    • Contact Us
    • Sitemap
    • Privacy Policy
    • Terms of Use
    • Tech in 200 Words

    © 2022 SPEC INDIA. All Rights Reserved.

    DMCA.com Protection Status

    Request a Quote

      .doc, .docx, .pdf, .xls, .xlsx, .jpg and .png files with 5MB size allowed.

      Please prove you are human by selecting the Flag.

      Safe & Confidential
      We use cookies to enhance your user experience. By continuing to browse, you hereby agree to the use of cookies.
      To know more; visit our Privacy Policy & Cookies PolicyAccept and Close
      Privacy & Cookies Policy

      Privacy Overview

      This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
      Necessary
      Always Enabled
      Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
      Non-necessary
      Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
      SAVE & ACCEPT