Data Architecture and Data Lakes: The Crucial Enabler for AI & Analytics

Authored by Sameer Dhanrajani, Chief Strategy Officer, Fractal Analytics

We are witnessing a palpable excitement and energy around the fields of AI & analytics due to the transformative impact that they have on businesses. However, we often miss highlighting the foundational aspect that keeps these revolutionary applications running. Data engineering and data architecture are what fuel our most vaunted AI & analytics processes and without a robust engineering backbone, such applications would undoubtedly fall apart. Data engineering is the crucial enabler for the information infrastructure of global enterprises. Through this blog, I will be addressing the topics of data architecture – what it is and why it is important – with a specific primer on data lakes, which is an emerging paradigm in the way we store and utilize data to bring life to our AI applications. 

Let us look at the detailed scenario of data architecture and some primary elements that govern this science. 

Data Architecture 

Given the rapid strides made in our data-driven business world, data architecture is an extremely fast-evolving field. With the huge data deluge seen in the field of business today it has been crucial for data architecture paradigms to keep up. To ingest and govern the massive stream of data that is being analyzed, the field of data architecture has seen many innovations. But what is Data Architecture? 

To put it in terms that everyone can understand, Data Architecture is the conglomeration of models, policies, rules and standards that govern which data is collected, how it is used and how it is stored, arranged, integrated and put into use. In other words, Data Architecture is the science that sets the marker for how data is stored and put in motion for data-driven processes and applications. Regardless of the innovation, there are 3 typical and constant elements in the traditional data architecture process, namely: 

1. Conceptual
   The business entities and architectural standards that the business owner and data architect mandate be accounted for


2. Logical
   The framework detailing how the business entities specified by the business owner relate to each other and their semantic models 


3. Physical The enabling data processing mechanism for how the data flows from one system to the next and how it is processed between these systems 

The role of the Data Architect in defining and governing this architecture is critical. The Data Architect co-creates these three elements based on inputs and priorities of the business owner and ensures that prevailing data and information systems are aligned to deliver the output that the business owner expects, all while maintaining high standards of system performance and resilience. 

Data Lakes: The Emerging Paradigm in Data Architecture 

As I mentioned earlier, the explosion in the quantum of enterprise data we are seeing today has required data architects to stay on top of emerging technology. With business entities, their relationships and the associated data therein expanding exponentially, the physical elements of the stack managed by Data Architects continues to change very rapidly. The pace of business today requires a re-look at the systems that manage our data and the Data Lake is the latest paradigm to manage the data deluge effectively. 

What is a Data Lake? A Data Lake is a repository in which data is stored in its natural format. It is usually considered to be the single source for all enterprise data, whether it is structured, semi-structured or unstructured. Here are five reasons why data lakes are receiving great attention and traction of late: 

1. Suitable for Multiple Data Formats
   One of the strongest reasons behind the recent popularity of data lakes has to do with their ability to ingest all kinds of data. Data warehouses so far have been restrictive in their ability to handle unstructured data, but data lakes serve as an excellent repository to handle structured (relational), semi-structured (CSV, XML or JSON) and unstructured (image, video and documents) data formats with relative ease. 


2. Faster Ingestion and Analysis
   Unlike traditional information systems, data lakes require no pre-structuring or formatting before the data enters the lake. With the high and increasing velocity in data streams today, data lakes are highly scalable in handling real-time data and hence provide superior support for faster analytics and running machine learning processes on real-time data. 


3. Lower cost of storage
   Cost is another factor that weighs heavily in favor of data lakes. Data lakes today are largely available as open-source tools and hence attract very low licensing costs. Additionally, given the quantum of data they ingest, they also provide more bang for the buck. 


4. Highly Configurable and Agile
   As data lakes do not require data to be structured in specified formats before ingestion, they are highly configurable and agile. Data scientists and developers can easily re-configure their models and queries to adapt to a changing and dynamic business landscape. 


5. Availability of Data Scientists
   While managing data lakes can be a highly technical task, the increasing number of data scientists who are more that up for the task make it possible to manage these systems well. 

With the landscape of data and analytics shifting faster than ever, data lakes and other emerging paradigms in data handling and processing are acquiring more prominence than ever before. However, I will leave you with one caveat when it comes to data lakes. Due to data lakes’ ability to ingest higher volumes of multi-type data, they can often be at a risk of becoming data ‘swamps’, if not managed well. 

Organizations need to ensure that they do not fall into the trap of dumping all data (irrespective of value) in data lakes and turning it into a data swamp.