The phenomenal growth of AI, especially Large Language Models (LLMs), in 2023 sparked keen interest in models as the primary building blocks for AI application development. Interestingly, this focus is rapidly transitioning towards complex systems composed of multiple components, instead of just singular, monolithic models.
Take the example of Google’s AlphaCode 2, which set a new standard in programming by using an expertly engineered system that employs LLMs to produce up to a million potential solutions for a task and then filter them down. AlphaGeometry, also a Google product, combines an LLM with a traditional symbolic solver to tackle olympiad problems. Furthermore, according to data from Databricks, 60% of LLM applications involve some form of retrieval-augmented generation (RAG), while 30% utilize multi-step chains.
This shift towards compound systems is not only intriguing from a design perspective but it’s also quite exciting because it suggests that exceptional AI results can be achieved through smart engineering, not just by scaling up training. In this article, we delve into the trend of compound AI systems, their importance for AI developers, and discuss the tools for developing and optimizing such systems—an area that has been somewhat under-researched compared to model training.
What exactly are Compound AI Systems?
Simplifying it, a Compound AI System is one that tackles AI tasks utilizing multiple interacting components. This can include multiple calls to models, retrievers, or external tools. This is in contrast to an AI Model, which is a statistical model predicting the next token in the text.
Even though AI models are getting better continuously, increasing numbers of state-of-the-art results are being obtained using compound systems. But why this shift? Well, there are several distinct reasons:
- Efficacy of system design: While LLMs appear to abide by remarkable scaling laws that predictably generate better results, in many applications, building a compound system can offer better returns-vs-cost than scaling.
- The dynamic nature of systems: Given that machine learning models are trained on static datasets, developers need to combine models with other components to include timely data.
- Stronger control and trust: Utilizing an AI system rather than a model can help developers control behavior more tightly. For instance, developers can filter model outputs to achieve specific behavior.
- Varying performance goals: Each AI model has fixed quality levels and costs. However, there's a wide discrepancy in desired parameters for different applications, which highlights the utility of compound AI systems.
The growing focus on compound systems aligns with the trends in other AI fields like self-driving cars: most of the top-notch implementations consist of systems with multiple specialized components.
Developing Compound AI Systems
Although compound AI systems can provide obvious benefits, the art of designing, optimizing, and managing them is still emerging. With an AI system being a combination of traditional software and AI models, multiple design questions arise. For example, should one invest resources more in the retriever, the LLM, or even in calling an LLM multiple times in a RAG pipeline?
Furthermore, how do we optimize an AI system end-to-end with discrete components to maximize a metric, the same way we can train a neural network? This topic alone opens up a world of complex challenges for designers of Compound AI Systems.
To conclude
All eyes have been on Generative AI recently for its potential in opening a wide range of capabilities with natural language prompting. However, the focus now seems to be shifting towards the development of compound AI systems, which are emerging as an effective way to both control and enhance the capabilities of LLMs. While the best practices for developing compound AI systems remain somewhat elusive, there are already exciting approaches for design, end-to-end optimization, and operation that hold immense potential in the ongoing evolution of AI.
Disclaimer: The above article was written with the assistance of AI. The original sources can be found on Berkeley Artificial Intelligence Research.
