3 Tips to Go From Data Cleansing to Value Realization

The end of the year prompts us to analyze and reflect. What worked within the last year and what is best to leave behind? What should be optimized for the year ahead?

Yes, you're still reading an advanced analytics blog. 

When it comes to data management, let’s take a moment to analyze and reflect on data availability and quality.

Here's what we know: Data management doesn't have to be a daunting process.

But, in order to implement new cutting-edge technology, we need accurate data.

The Facts:

In order to implement a software solution that uses advanced analytics and techniques, like machine learning, organizations require accurate data to prove viability. However, due to the complexity of energy operations, performance engineers are often faced with industrial and technical difficulties that hinder data availability and accuracy.

Even if you’re using the best software in the industry, data availability, curation, and best practices are required. Otherwise, value realization is truly impossible....  now that’s daunting!  

Let’s use the end of the year as an opportunity to start fresh with your asset data and operational outputs. 

Quality Data Beats Fancy Algorithms

The basic goal of Data Curation is to automatically identify active and useful datasets while providing ongoing management through their lifecycle of usefulness. Data quality ensures and enhances the viability of advanced analytics into automated decisions and curated outputs that benefit operations and performance engineering teams.

Common Use Cases Include:

• Inaccurate and/or missing Data
• Root Cause: unreliable pyranometer.
• Resolution: automatically cleanse, normalize, interpolate, and backfill data using multiple 3^rd party weather sources.
• Inconsistent Data
• Root Cause: unreliable availability calculations due to inconsistent intervals of data.
• Resolution: create custom ETL (extract, transform, load) to aggregate data, creating consistent data intervals used for advanced analytics and insights.
  
  

Best Practices for Data Quality & Management:

A Data Curation strategy consists of three primary stages, the first is useful data identification and prioritization, the second is automating data curation through a knowledge management system, and the third is using visualization tools to regularly review data curation and management best practices.

1. Start Small to Validate Data Accuracy and Prioritization

There are different ways of reducing and prioritizing data within a useful dataset. Look for errors in the data - faulty sensors, duplicates, outliers, and missing data. Use historical data records to choose relevant or highest priority datasets via root cause and/or lessons learned. To streamline value, go backward from resolution to problem statement.

2. Use Tools to Automate Data Curation

The average person can no longer handle managing the vast amount of data that flows through SCADA systems daily.

An automated knowledge management system includes, but is not limited to:

• Collecting, consolidating, and integrating data from different sources
• Sifting through and assessing data accuracy
• Determining which data is useful based on insights to root cause via machine learning
• Categorizing and structuring data into manageable datasets for visualization tools
  
  

3. Perform Process QA Regularly

Data curation is not an implement-and-forget-it practice. Data Quality and Management best practices include regular Quality Assurance checks. Over time, as assets depreciate or more assets are acquired, it’s vital to ensure the processes still make sense.

Getting Started: 

Data curation can be a time-consuming process to implement, and most operators and engineers do not have the time to spend improving data quality. Luckily, software exists that allows you to partner with a team of experts on data quality, management, and operational success.  

At NarrativeWave, we partner with our clients on their data management strategy and execution. Our client facing teams train users on data quality and best practices specific to our product’s functionality because their asset data is technically and figuratively, the heartbeat of our product. 

We know our product; our clients know their assets. Our data management partnership enables our product to use known behavior, rather than assumptions, to unlock new insights and produce automated root-cause diagnostics. Now that’s value realization.