DataData SolutionsInnovationMar-Apr2019Spring 2019The ConnectionTheConnection Using Data Replication to Offload Online Host Processing and Enable New Business Applications by Paul Holenstein March 18, 2019 written by Paul Holenstein March 18, 2019 174 Data is valuable. And the more current the data is, the more valuable it is. For this reason, the resident data in online transaction processing systems is some of the most valuable. However, if this data is trapped in that system, and not available to enable other real-time business intelligence processes, then its full value is not being exploited, and competitive opportunities are missed. To avoid these missed opportunities, companies need to access their online data in real-time. This issue presents a problem. As the demand for real-time data increases, accessing it while the online system is processing transactions can significantly impact the throughput and response times of that system. This article discusses such a situation, and the clever way it was resolved using data replication. The Manufacturer’s Trapped Data A large European steel tube manufacturer runs its online shop floor operations on an HPE NonStop Server. To exploit the currency and value of this online data, the manufacturer periodically generates reports on Linux Servers using a customized application and connectivity tool that remotely queries the online NonStop Enscribe database and returns the results (Figure 1). With this architecture, every time a Linux query/report was run, processing on the NonStop Server was required, and as query activity increases, this workload started to significantly impact online shop floor processing. This impact was compounded due to the high volume of data transformation and cleansing required for converting the Enscribe data into a usable format for the reports. Periodically, the company needed to suspend the execution of reports due to these production impacts. In addition, the remote connectivity architecture was not very robust or scalable, and was also susceptible to network failures, timeouts, and slowdowns when operating at full capacity. Since the data adapter used was nonstandard, the manufacturer could not access the data using standard ODBC and OLAP¹ tools to take advantage of new analytical techniques (such as DSS²). Furthermore, the company had a new requirement to share the OLAP analysis with the online NonStop applications in order to optimize shop floor control, which was completely impossible with the original solution. Therefore, a new architecture was required to address these issues and meet the new requirements. ¹ Online Analytical Processing (OLAP) is the technology behind many Business Intelligence (BI) applications. OLAP is a powerful technology for data discovery, including capabilities for limitless report viewing, complex analytical calculations, and predictive “what if” scenario (budget, forecast) planning. ² A Decision Support System (DSS) is a computerized information system used to support decision-making in an organization or a business. A DSS lets users sift through and analyze massive reams of data and compile information that can be used to solve problems and make better decisions. Figure 1 – Original Remote Query Architecture Rearchitecting Operations The manufacturer completely rearchitected its Linux-based querying/reporting application. Rather than remotelyquerying the NonStop Enscribe data each time a report is run, the data is replicated by HPE Shadowbase software in real-time from the Enscribe database to a copy of that database hosted on a Linux system. This architectural change only requires the data to be sent across the network once (when it is changed), instead of every time a query/report is run. The raw Enscribe data is non-normalized, and full of arrays, redefines, and data types that do not have a matching SQL data type. As part of the replication process, the non-relational Enscribe data is transformed and cleansed by HPE Shadowbase software into a relational format, and written to an Oracle RAC database. This architectural change only requires the data to be cleansed or normalized once when the data is changed, rather than every time a query/report is run. Since the data is now local to the Linux Servers and presented in standard relational format, it is possible to use standardized SQL data query and analysis tools (Figure 2). In addition, HPE Shadowbase software allowed the manufacturer to reverse replicate the OLAP results and share those results back with the NonStop applications to better optimize the shop floor manufacturing process. Figure 2 – The Manufacturer’s New Architecture Featuring Local Database Access Key Project Benefits This new architecture completely resolves the issues withthe old mechanism by decoupling the data access and transformation/cleansing process from the querying/reporting process, and by presenting the data locally on the Linux system in standard relational format: Querying/reporting and data transformation/cleansing impact on the production NonStop system is eliminated. Online data replication has much lower overhead than executing queries and performing data transformation/cleansing directly on the NonStop Enscribe database every time a query is executed. This architecture dramatically reduces the workload on the NonStop Server, avoids throughput and response time issues with the shop floor applications, and eliminates the periodic need to suspend running reports. Querying/reporting performance is dramatically improved. It is much faster to run queries and reports against an already transformed/cleansed local relational database using standard ODBC access, than it is to remotely access the NonStop Enscribe database and spontaneously perform the necessary data transformation/cleansing every time a query or report is executed. Querying/reporting application availability is improved. A local Linux copy of the database is used so that an outage of a network or a NonStop Server will not impact the querying/reporting applications. The shop floor application’s scalability is improved. A local database access is inherently more scalable than executing remote queries across a network, especially when removing the need for data transformation/cleansing on every query. Competition for resources on the NonStop Server is also greatly reduced by offloading workload to the Linux systems. In addition, the original architecture typically only allowed a single report to run at any one time (to minimize the impact on the NonStop production system). Now, with the data hosted on a Linux Oracle RAC database and the NonStop system out of the query loop, multiple reports and advanced analytics can be executed in parallel. Analytics and data value are improved. Transforming and cleansing the Enscribe data into relational format and storing it in a local relational database enables the use of standard ODBC database access. It also enables additional off-the-shelf querying/reporting tools to extract even more value from the data (including OLAP and DSS). The ability to share the OLAP analysis with the NonStop production applications using Shadowbase bi-directional data integration enables new uses for the data. (In this case, to better optimize the shop floor manufacturing process.) Summary This case study only scratches the surface of data replication’s potential. It can: Integrate disparate applications, such as an online production shop floor application with a reporting application Eliminate data silos and enables the creation of new business processes Distribute data wherever it is needed with low overhead while it transforms and cleanses that data Distribute data where it can easily be consumed by other applications (typically without requiring any application changes) And much, much more³ Data replication is not only for providing backups for business continuity, but also for moving data in real-time as required in order to leverage its value wherever needed. Think about where you have valuable, yet isolated, data and consider how you could use data replication to unlock this value for competitive advantage and to build new solutions for your business. ³ For more information, please see the Gravic white papers, HPE Shadowbase Streams for Data Integration and HPE Shadowbase Streams for Application Integration. ConnectConvergeSpring2019 0 comment 0 FacebookTwitterPinterestEmail Paul Holenstein Paul J. Holenstein is Executive Vice President of Gravic, Inc. He is responsible for the HPE Shadowbase suite of products. The HPE Shadowbase replication engine is a high-speed, uni-directional and bi-directional, homogeneous and heterogeneous data replication engine that provides advanced business continuity solutions as well as moves data updates between enterprise systems in fractions of a second. It also provides capabilities to integrate disparate operational application information into real-time business intelligence systems. Shadowbase Total Replication Solutions® provides products to leverage this technology with proven implementations. HPE Shadowbase software is built by Gravic, and globally sold and supported by HPE. Please contact your local HPE account team for more information, or visit https://www.ShadowbaseSoftware.com. To contact the authors, please email: SBProductManagement@gravic.com. previous post Building Bulletproof Security for Your Hybrid Cloud next post For Auto Makers, Data is the New Fuel for Analytics and Innovation You may also like Nonstop Trends and Wins January 17, 2024 Continuous Data Protection: The new imperative March 27, 2022 New Data Integrity Architectures for Mission Critical Systems December 6, 2021 To bring strategy back into your security, turn... December 6, 2021 Getting Started with Data-centric Security December 6, 2021 HPE GreenLake edge-to-cloud platform brings the cloud to... December 6, 2021 Democratization of AI in the Enterprise December 6, 2021 Getting the most from your data-driven transformation: 10... December 6, 2021 How to Safeguard Your Data & Applications from... December 6, 2021 The Year in Review Around the Storage Block December 6, 2021