Earlier today, speaking at the AI World Congress conference in London.
Based on our first-hand experience at Robotic Online Intelligence (ROI) in applying LLMs to research processes, this was a 'what works and what doesn't' session, including an example of a RAG (Retrieval Augmented Generation) deployment for an interrogation of annual reports of companies (Chinese property developers as a test case).
The main conclusion from our internal case is that it can work quite well but requires much human expertise to 'guide the LLMs'.
Most importantly, the optimization of so many of the parameters and steps in RAG (chunking, embedding, pre-processing and question decomposition, search, answer generation, and evaluation) for one case will not be portable to another case.
The human experience from doing so does help with any new cases, though.
There is also room for practical shortcuts. For example, no need to always jump into vector databases and complex search if a simpler text search with regular expressions can get the job done to retrieve the most relevant chunks of text.