Spotless Data

How would your home look like if you let dirt and mess accumulate for years? It would be a health hazard and would also make it impossible to find what you need when you need it most. In the end, you would reach a point when the problem simply couldn’t be overlooked. This is the situation that many plant managers are facing after accumulating huge quantities of manufacturing data over the years. 

By implementing a data-driven company culture, manufacturers can exponentially improve virtually any aspect of production. Big data can be used, among other things, to maximise energy efficiency, improve the business’s predictive maintenance strategy, and prevent downtime caused by equipment failure. To do this, manufacturers need accurate and reliable data.    
 
But when data is collected and accumulated for several years, its quality can start to decline. Dirty or rogue data is data affected by issues such as duplicates, inaccuracies, inconsistencies, and out-of-date information. When plants reach this point, it’s time for a good clean-up. 

Not The Exception

Dirty data is the norm, not the exception. As companies evolve, the amount of data they collect grows in quantity and complexity. High employee turnover, the use of different enterprise resources planning (ERP) solutions across several departments, and lack of standard guidelines for data entry complicate the situation. For these reasons, achieving perfect data is almost impossible, especially in large organisations.  

Data cleansing, or cleaning, is the process of detecting and correcting or eliminating incomplete, inaccurate, out-of-date or irrelevant data. It differs from data validation in that the latter is automatically performed by the system at the time of data entry, while data cleaning is done later on batches of data that have become unreliable.

There are a lot of data cleansing tools available, such as Trifacta, Openprise, WinPure, OpenRefine and many more. It’s also possible to use libraries like Panda for Python, or Dplyr for R. The variety of solutions on the market means that manufacturers might want to consult a data analyst to choose the best one for their business case.

How Dirty, Exactly?

Regardless of the solution employed and the type of data being cleansed, the first step is assessing the quality of the existing data. In this phase, a data analyst will assess the company’s needs and establish specific KPIs for clean data. Legacy data is then audited using statistical and database methods to reveal anomalies and inconsistencies.  

This can be done using commercial software that allows the user to specify various constraints. The existing data will be uploaded and tested against these constraints, and data that doesn’t pass the test should be cleansed.  

During this phase, manufacturers should establish which input fields must be standardised across the company. Standardisation rules can help businesses prevent the build-up of dirty data in that they minimise inconsistencies and facilitate the uploading of clean data into a common ERP.

Keep It Clean

After the audit, the cleaning process can begin. Data will pass through a series of automated software programmes that discard what is not compliant with the specified KPIs. The result is then tested for correctness and incomplete data will be amended manually, if possible. A final quality control phase will ensure that the output data is clean enough to by seamlessly uploaded into the chosen ERP.

However, just like when cleaning our homes, a big clean-up every now and then is not enough. The best approach is to implement a culture of continuous data improvement, distributing tasks among each member of the team. Developing practices that support ongoing data hygiene is the key to success.

About the Author:  Neil Ballinger is head of EMEA at automation parts supplier EU Automation and for more information on how to use big data to optimise your business, visit www.euautomation.com

Image: Unsplash

You Might Also Read: 

Some Expert Predictions For Industrial Cyber Security:

 

« Myanmar’s Cyber Security Bill
A Successful Solar Winds Investigation »

CyberSecurity Jobsite
Perimeter 81

Directory of Suppliers

LockLizard

LockLizard

Locklizard provides PDF DRM software that protects PDF documents from unauthorized access and misuse. Share and sell documents securely - prevent document leakage, sharing and piracy.

Jooble

Jooble

Jooble is a job search aggregator operating in 71 countries worldwide. We simplify the job search process by displaying active job ads from major job boards and career sites across the internet.

XYPRO Technology

XYPRO Technology

XYPRO is the market leader in HPE Non-Stop Security, Risk Management and Compliance.

Alvacomm

Alvacomm

Alvacomm offers holistic VIP cybersecurity services, providing comprehensive protection against cyber threats. Our solutions include risk assessment, threat detection, incident response.

ON-DEMAND WEBINAR: What Is A Next-Generation Firewall (and why does it matter)?

ON-DEMAND WEBINAR: What Is A Next-Generation Firewall (and why does it matter)?

Watch this webinar to hear security experts from Amazon Web Services (AWS) and SANS break down the myths and realities of what an NGFW is, how to use one, and what it can do for your security posture.

Hedgehog Security

Hedgehog Security

The key objective of Hedgehog is to provide simple, effective and affordable information security improvements that support your drive to increase productivity and profitability.

Nexusguard

Nexusguard

Nexusguard is at the forefront of the fight against malicious Internet attacks, protecting organizations worldwide from threats to their websites, services, and reputations.

HvS Consulting

HvS Consulting

HvS Consulting is a specialist information security company offering a full range of services including IT security architecture, ISO 27001 audits, Pentesting, Security monitoring and Training.

CyberSec.sk

CyberSec.sk

CyberSec.sk is the Slovak portal bringing the latest cyber security news, politics, tips and instructions on how to protect the internet.

Center for Applied Cybersecurity Research (CACR) - University of Indiana

Center for Applied Cybersecurity Research (CACR) - University of Indiana

CACR serves Indiana and the nation by tackling cyber risk in research and other unusual environments through agile, holistic, principle-based cybersecurity.

Stage2Data

Stage2Data

Stage2Data is one of Canada’s most trusted cloud solution providers offering hosted Backup and Disaster Recovery Services.

ICT Reverse

ICT Reverse

ICT Reverse is one of the UK’s leading, fully accredited providers of ICT asset disposal and secure data erasure.

Utility Cyber Security Forum

Utility Cyber Security Forum

The Utility Cyber Security Forum offers a focused venue in which utility executives can network one-on-one with colleagues facing issues in protecting against cyber attacks.

Bolt Learning

Bolt Learning

Bolt's Cyber Security eLearning module provides users with an in-depth understanding of cybercrime, how it can occur and what everyone can contribute to preventing it.

Spohn Solutions

Spohn Solutions

Spohn combines highly-experienced staff with a vendor neutral approach to deliver optimal solutions for IT Security and Compliance.

IPKeys Cyber Partners

IPKeys Cyber Partners

IPKeys Cyber Partners, together with the IPKeys Power Partners unit, provide Cyber Security and CIP Compliance for utilities, grid operators and public safety organization across the USA.

Cyber Law Consulting

Cyber Law Consulting

Cyber Law Consulting is a Dynamic full service legal firm which offers complete services for Cyber Law, cyberlaw, Internet Law, Data Protection Act, Cyber Security, IPR, Drafting.

Bearer

Bearer

Bearer helps modern teams ship trustworthy products with the help of our code security solution built for security, privacy and engineering teams.

Backblaze

Backblaze

The Backblaze Storage Cloud provides a foundation for businesses, developers, IT professionals, and individuals to build applications, host content, manage media, back up and archive data, and more.

Hudson Rock

Hudson Rock

Hudson Rock’s products — Cavalier & Bayonet — are powered by our cybercrime database, composed of millions of machines compromised by Infostealers in global malware spreading campaigns.

DOT Europe

DOT Europe

DOT Europe is a consensus based organisation which brings a diverse membership together to agree on their collective stance on EU tech policy.