I remember not long ago, when I was an architect designing batch-based and real-time systems, that batch-based was typically the preferred route since real-time capabilities were generally not a necessity. Batch-based systems were also easier to build and maintain, whereas real-time systems were extremely complicated with many more moving parts.
Now that AI is here, the batch-based systems of yore are no longer cutting it, and real-time systems are now the standard. Whether you’re building an AI platform or augmenting one, lightning-fast response times are expected and necessary. Because of this, the data stack under the hood has become the linchpin for real-time capabilities and many organizations are rethinking the data stack to deliver the performance GenAI demands.
When we think of real-time, we think of the slew of data products that have flooded the market, usually prefixed with “Apache.” With names like Kafka, Pinot, and Airflow, you’ve likely come across them before. To work for AI, many of these different technologies need to be stitched together, which brings up concerns and challenges around the integrations and support between each platform. Of course, full-stack managed solutions have also hit the market to ease some of that complexity, but the problem remains: there are almost too many choices.
With all the product proliferation, how do you cut through the noise to optimize your data stack for building GenAI platforms and applications? Here are three ways I think we should be looking at it:
Prioritize Data Quality and Governance Over Speed
Although speed is one of the hallmarks of GenAI, no one wants something that sacrifices accuracy for speed, and that’s especially true in high-risk or highly-regulated industry use cases. It’s important to remember that GenAI models are only as good as the data they retrieve or are trained on.
To get your data into shape, you’ll need to ensure that data validation, lineage tracking, and quality monitoring happen throughout the data pipeline. Because of the speed at which real-time systems generate their insights and actions, poor data quality amplified through real-time systems can cause widespread issues much faster than batch systems ever could. Think of it this way: data quality and governance are a prerequisite and the foundation on which speed can be built.
Design for Retrieval-Augmented Generation (RAG) from Day One
Traditional data stacks optimized for analytics, for the most part, don’t naturally support the vector search and semantic retrieval patterns that GenAI applications require. Thus, real-time GenAI data architectures need native support for embedding generation and vector storage as first-class citizens. This could mean integrating data with vector databases like Pinecone, Weaviate, or Chroma as part of the core infrastructure. It may also mean searching for multi-modal databases that can support all of your required data types out of the box without needing a bunch of separate platforms. Regardless of the underlying infrastructure, plan for needing hybrid search capabilities that combine traditional keyword search with semantic similarity, and consider how you’ll handle embedding model updates and re-indexing. Taking these key architectural decisions into account early on will help to make sure the stack is ready for your GenAI use cases without the need for a major overhaul.
Build for Multi-Modal Data Integration
GenAI applications work with many different types of data. Text, images, audio, structured data, and code are often leveraged simultaneously. Traditional data stacks were built with relational or document data in mind, but GenAI demands much more flexibility. This means rethinking your data architecture to handle multi-modal ingestion, storage, and processing from the ground up. From the start, you need to plan for different storage patterns: object storage for media files, graph databases for complex relationships, and specialized systems for time-series data.
Maintaining data relationships and ensuring consistent access patterns across these different storage systems is the real challenge when working with these various data types. While some platforms are beginning to offer enhanced vector search capabilities that can work across different data types, most organizations still need to architect solutions that coordinate multiple storage systems. The key is to design these multi-modal capabilities into your data stack early, rather than trying to bolt them on later when your GenAI applications demand richer data integration. Just like any other large-scale project, you save a lot of time by considering these factors in the design stage versus later on when changes become more difficult and costly.
AI has made the data stack infinitely more complex. The demands of GenAI truly stretch the definition of “real time”: today, we measure in milliseconds or nanoseconds, not minutes. A shiny UI, great APIs, and developer experience mean very little if the underlying data stack can’t deliver the accuracy and speed required to power the newest generation of AI-enabled applications and platforms. By considering the three points above, you’ll be well prepared to succeed with GenAI.
