DataStax to deliver high-performance RAG solution with NVIDIA Microservices

Generative AI data company, DataStax is supporting enterprise retrieval-augmented generation (RAG) use cases by integrating the new NVIDIA NIM inference microservices and NeMo Retriever microservices with Astra DB to deliver high-performance RAG data solutions for superior customer experiences.

With this integration, users will be able to create instantaneous vector embeddings 20x faster than other popular cloud embedding services and benefit from an 80% reduction in cost for services.

Organizations building generative AI applications face the daunting technological complexities, security, and cost barriers associated with vectorizing both existing and newly acquired unstructured data for seamless integration into large language models (LLMs). The urgency of generating embeddings in near-real time and effectively indexing data within a vector database on standard hardware further compounds these challenges.

DataStax is collaborating with NVIDIA to help solve this problem. NVIDIA NeMo Retriever generates over 800 embeddings per second per GPU, pairing well with DataStax Astra DB, which is able to ingest new embeddings at more than 4000 transactions per second at single-digit millisecond latencies, on low-cost commodity storage solutions/disks. This deployment model greatly reduces total cost of ownership for users and performs lightning-fast embedding generation and indexing.

With embedded inferencing built on NVIDIA NeMo and NVIDIA Triton Inference Server software, DataStax AstraDB vector performance of RAG use cases running on NVIDIA H100 Tensor Core GPUs achieved 9.48ms latency embedding and indexing documents, which is a 20x improvement.

When combined with NVIDIA NeMo Retriever, Astra DB and DataStax Enterprise (DataStax’s on-premise offering) provide a fast vector database RAG solution that’s built on a scalable NoSQL database that can run on any storage medium. Out-of-the-box integration with RAGStack (powered by LangChain and LlamaIndex) makes it easy for developers to replace their existing embedding model with NIM. In addition, using the RAGStack compatibility matrix tester, enterprises can validate the availability and performance of various combinations of embedding and LLM models for common RAG pipelines.

DataStax is also launching, in developer preview, a new feature called Vectorize. Vectorize performs embedding generations at the database tier, enabling customers to leverage Astra DB to easily generate embeddings using its own NeMo microservices instance, instead of their own, passing the cost savings directly to the customer.

“In today's dynamic landscape of AI innovation, RAG has emerged as the pivotal differentiator for enterprises building genAI applications with popular large language frameworks,” said Chet Kapoor, chairman and CEO, DataStax. “With a wealth of unstructured data at their disposal, ranging from software logs to customer chat history, enterprises hold a cache of valuable domain knowledge and real-time insights essential for generative AI applications, but still face challenges. Integrating NVIDIA NIM into RAGStack cuts down the barriers enterprises are facing to bring them the high-performing RAG solutions they need to make significant strides in their genAI application development.”

“At Skypoint, we have a strict SLA of five seconds to generate responses for our frontline healthcare providers,” said Tisson Mathew, CEO and founder of Skypoint. “Hitting this SLA is especially difficult in the scenario that there are multiple LLM and vector search queries. Being able to shave off time from generating embeddings is of vast importance to improving the user experience.”

“Enterprises are looking to leverage their vast amounts of unstructured data to build more advanced generative AI applications,” said Kari Briski, vice president of AI software, NVIDIA. “Using the integration of NVIDIA NIM and NeMo Retriever microservices with the DataStax Astra DB, businesses can significantly reduce latency and harness the full power of AI-driven data solutions.”