Big Data: The 4 Layers Everyone Must Know

Big Data still causes a lot of confusion in people's heads: What really is it? What is new and what is old wine in new bottles?

In order to bring a little more clarity to the concept it might help to describe the 4 key layers of a big data system - i.e. the different stages the data itself has to pass through on its journey from raw statistic or snippet of unstructured data (for example, social media post) to actionable insight.

The whole point of a big data strategy is to develop a system which moves data along this path. In this post, I will attempt to define the basic layers you will need to have in place in order to get any big data project off the ground.

Although people have come up with different names for these layers, as we’re charting a brave new world where little is set in stone, I think this is the simplest and most accurate breakdown:

Data sources layer

This is where the data is arrives at your organization. It includes everything from your sales records, customer database, feedback, social media channels, marketing list, email archives and any data gleaned from monitoring or measuring aspects of your operations. One of the first steps in setting up a data strategy is assessing what you have here, and measuring it against what you need to answer the critical questions you want help with. You might have everything you need already, or you might need to establish new sources.

Data storage layer

This is where your Big Data lives, once it is gathered from your sources. As the volume of data generated and stored by companies has started to explode, sophisticated but accessible systems and tools have been developed – such as Apache Hadoop DFS (distributed file system), which I cover in this article – or Google File System, to help with this task.

A computer with a big hard disk might be all that is needed for smaller data sets, but when you start to deal with storing (and analyzing) truly big data, a more sophisticated, distributed system is called for. As well as a system for storing data that your computer system will understand (the file system) you will need a system for organizing and categorizing it in a way that people will understand – the database. Hadoop has its own, known as HBase, but others including Amazon’s DynamoDB, MongoDB and Cassandra (used by Facebook), all based on the NoSQL architecture, are popular too. This is where you might find the Government taking an interest in your activities – depending on the sort of data you are storing, there may well be security and privacy regulations to follow.

Data processing/ analysis layer

When you want to use the data you have stored to find out something useful, you will need to process, and analyze it. A common method is by using a MapReduce tool (which I also explain in a bit more depth in my article on Hadoop). Essentially, this is used to select the elements of the data that you want to analyze, and putting it into a format from which insights can be gleaned. If you are a large organisation, which has invested in its own data analytics team, they will form a part of this layer, too. They will employ tools such as Apache PIG or HIVE to query the data, and might use automated pattern recognition tools to determine trends, as well as drawing their conclusions from manual analysis.

Data output layer

This is how the insights gleaned through the analysis is passed on to the people who can take action to benefit from them. Clear and concise communication (particularly if your decision-makers don’t have a background in statistics) is essential, and this output can take the form of reports, charts, figures and key recommendations.

Ultimately, your Big Data system’s main task is to show, at this stage of the process, how measurable improvement in at least one KPI that can be achieved by taking action based on the analysis you have carried out.

Data Science Central

« Cybersecurity Training, Military Style
Why Executives Need to Prioritise Cybersecurity »

CyberSecurity Jobsite
Perimeter 81

Directory of Suppliers

ON-DEMAND WEBINAR: What Is A Next-Generation Firewall (and why does it matter)?

ON-DEMAND WEBINAR: What Is A Next-Generation Firewall (and why does it matter)?

Watch this webinar to hear security experts from Amazon Web Services (AWS) and SANS break down the myths and realities of what an NGFW is, how to use one, and what it can do for your security posture.

LockLizard

LockLizard

Locklizard provides PDF DRM software that protects PDF documents from unauthorized access and misuse. Share and sell documents securely - prevent document leakage, sharing and piracy.

Cyber Security Supplier Directory

Cyber Security Supplier Directory

Our Supplier Directory lists 6,000+ specialist cyber security service providers in 128 countries worldwide. IS YOUR ORGANISATION LISTED?

Perimeter 81 / How to Select the Right ZTNA Solution

Perimeter 81 / How to Select the Right ZTNA Solution

Gartner insights into How to Select the Right ZTNA offering. Download this FREE report for a limited time only.

CSI Consulting Services

CSI Consulting Services

Get Advice From The Experts: * Training * Penetration Testing * Data Governance * GDPR Compliance. Connecting you to the best in the business.

Perforce Software

Perforce Software

Perforce helps companies build complex software products more collaboratively, securely, and efficiently.

Evidian

Evidian

Evidian, a Bull Group company, is the European leader and one of the major worldwide vendors of identity and access management software.

Technology Association of Georgia (TAG)

Technology Association of Georgia (TAG)

TAG's mission is to educate, promote, influence and unite Georgia's technology community to stimulate and enhance Georgia's tech-based economy.

Bayshore Networks

Bayshore Networks

Bayshore Networks was founded to safely and securely protect Industrial IoT (IIoT) networks, applications, machines and workers from cyber threats.

VTT Technical Research Centre of Finland

VTT Technical Research Centre of Finland

VTT is the leading research and technology company in the Nordic countries. Areas of activity include cyber security.

Ntrepid

Ntrepid

Ntrepid products provide protection from web threats and enable organizations to safely conduct their online activities.

Safetica

Safetica

Safetica Technologies is a Czech software company that delivers data protection solutions for businesses of all types and sizes.

TriagingX

TriagingX

TriagingX successfully created the first generation malware sandbox that is being used by many Fortune 500 companies for daily malware analysis.

Guardian Digital

Guardian Digital

Guardian Digital makes email safe for business. Threat-ready business email protection. Fully supported.

Content+Cloud

Content+Cloud

Content+Cloud is a leading technology services business and Managed Services Provider (MSP) with a genuine passion for helping your organisation to succeed, whatever your ambitions.

The ATOM Group

The ATOM Group

ATOM builds and secures technology for regulated industries. We design and build for a future we can all trust.

MillenniumIT ESP (MIT ESP)

MillenniumIT ESP (MIT ESP)

MillenniumIT ESP provides solutions and services around Core Infrastructure, Cloud, Cyber Security, Enterprise Applications, Intelligent Automation and Data, Smart Buildings, and Managed Services.

Eureka Security

Eureka Security

Eureka help organizations securely use any cloud data storage technology they need without having to compromise on security.

Cerby

Cerby

Your team uses unmanageable applications that put you, your company, and your data at risk. Protect, secure, and accelerate your business automatically with Cerby.

AKS iQ

AKS iQ

AKS iQ leads the RegTech sector with AI, automating regulatory compliance in the banking industry and ensuring paperless TBML and CFT adherence in finance.

Intracis

Intracis

Intracis is a 'Made in India' cyber incident management solution aimed at ‘Making Security Simple’ by simplifying cyber incident management for CERTS and CSIRTS.