Big Data company StreamSets, that provides data ingest technology to enterprises, has announced the launch of its Dataflow Performance Manager (DPM) to manage operations of a company’s end-to-end dataflows. StreamSets’s DPM claims to unify visibility and control of dataflows, which could data management costs, improve data quality and enable IT agility. Enterprises can also map dataflows to topologies and support business or IT operations including Data Lakes, Customer 360, IoT and Cyber Security, according to the company.
San Francisco-based StreamSets, which aims to ease Big Data management believes that current approaches for managing data movement deprive operators of much needed visibility and hands-on control, something it calls ‘operational blindness’. Intangible management can also lead to Data Drift that StreamSets define as the unpredictable and unending mutation of data characteristics caused by the operation, maintenance and modernization of the systems that produce the data.
In an attempt to address these challenges, DPM hopes to help enterprises harness their data in motion together with data at rest to gain real-time business insights. This in fact, insists StreamSets, could pose as the foundation for organizations that plan to establish a data operations center (DOC).
Girish Pancha, C0-founder and CEO, StreamSets, said:
“StreamSets Dataflow Performance Manager uniquely addresses the pain enterprises experience when they try to manage a multitude of dataflows manually or with legacy technologies in the face of data drift and dynamic use cases.”
“Data is a key asset that requires the same performance management practices that have been adopted for networks and applications. DPM brings operational intelligence to dataflow management so enterprises can be sure that the data driving their key applications is always timely and trustworthy.”
In a company blog, Pancha commented that, “DPM will be generally be available on or before September 27, in time for Strata.”
DPM, a subsequent follow on to Data Collector — StreamSets’s first product — also lists features where customers can track changes in topologies over time and measure baselines for end-to-end and point-in-flow key performance indicators (KPIs) for data availability and accuracy. It also aids users with dataflow operations by creating Data SLAs (service level agreements) and detecting and remediating violations.
Its Data Collector platform, an open source software was designed to help companies in building and deploying complex dataflow pielines. It also vouches to be backed by Silicon Valley venture capital firms including Accel Partners, Battery Ventures, Ignition Partners and New Enterprise Associates (NEA).
“While StreamSets Data Collector is a best-in-class tool for data engineers designing complex pipelines in the face of data drift, that is only half the battle. Our customers don’t just struggle with building pipelines, but also with managing the day in and day out operations of their dataflows. This is where DPM comes in,” continued Pancha.