2025 Kickoff | On LLMs

Views on LLMs / AI for the Year and Plans for Kubro(TM)

 

In this note, you will find our take on LLMs / AI in 2025 and the product plans for the Kubro(TM) platform and ROI services this year.

 

On the road in 2024: London, New York, San Francisco

 

For 2025, on the AI front, the big theme remains 'how to make LLMs work in practice' against the backdrop of rapid progress and new opportunities on one hand, and the over-hyped ideas and the 'enterprise reality' on the other, that is very different from consumer applications. Here are some of the major themes rolling into 2025, with our (subjective) views and plans in brief.

 

1. (Autonomous) AI Agents

 

Everyone talks about some form of Agentic AI, where a model can 'decide on its own' what action to take and/or what tools to use (from a web search API to placing an order) for a given objective or question. In our domain of information/data and investment research businesses, we think there are much bigger productivity gains to be earned first from a more comprehensive workflow automation that would be still deterministic in nature but with many LLM components 'talking' to each other. The 'decide on its own' part can be partly used in that context, but the human-determined architecture and the platform to orchestrate such workflows is what we plan to focus on.

 

In that context, many components are now ready in Kubro(TM) to serve as the orchestration layer, including e.g. new LLM classifiers in tagging and robo-reports (more on that soon). We are also looking to add features like 'chat to configure Kubro(TM)' to help users set up various modules in the system. The agentic structure can also lend itself to handling larger documents (for data extraction or analysis) - and here we are planning to further expand the Document Explorer launched in Q4.

 

2. The RAG Road To RAGginess

 

2024 was largely a RAG year. RAG (Retrieval Augmented Generation) methods carried a promise of combining Large Language Models' general abilities to 'understand'/process unstructured text, with specific knowledge bases - be it scientific papers, SEC filings, policy documents, or medical data. 

 

The reality in practical deployment is much more challenging and limiting, yet still promising and will remain a big theme in 2025.

 

In Q3/Q4, we spent a lot of our R&D time on deploying RAG in our use case for China property as a testing ground, adding to the 'Regexy RAG' shortcut we developed in early 2024. 

 

Building on that, in 2025 we are in a position to deploy advanced RAG methods. In many cases, this will be suitable for non-SaaS implementation, including fine-tuned open-source models on the client side.

 

3. What Works And What Doesn't With LLMs In Research

 

Through many use cases, 2024 taught us the following about the applications of LLMs to market research:

 

Focus on the domain- and use-case-specific deployments - To make LLMs work, you need to focus on a specific business problem (one at a time) and/or workflow and deploy LLMs on a highly domain-specific and use-case-specific basis. For example, making RAG work for China property developers use case required many optimizations of a few dozen parameters and domain-specific factors. While the knowledge we gained through that can be applied elsewhere, the particular 'solution' is not portable to other sectors. 

 

Domain specificity needs topic models - We've always been using the term 'topic model', not in relation to any AI but rather a human-expert-defined 'map' of a particular domain (expressed through the tagging logic in Kubro(TM), for example, China property, data centers, digital assets, or... AI). That's in terms of how to split the particular domain in terms of various categories as well as the types of significant content (e.g. significant developments in a market). Only then, overlaying the automation / LLMs can yield efficiency.

 

LLMs can help define such topic models (already in Kubro(TM)) but still need to be 'guided' by the human expert.

 

We see more and more talk about the significance of combining knowledge graphs with LLMs, including Graph RAG. There is much more talk now about the importance of ontologies and the application of graphs with LLMs, reinforcing the point on domain-specificity.

 

Summarization, classification, and data extraction with LLMs do work  - These are the areas where LLMs can work very well and, for all practical purposes, are the most likely to deliver strong ROI. It appears that true reasoning and AGI, while getting closer, are not a practical consideration near term.

 

Our own monthly use case in China property research at the sister venture Real Estate Foresight draws on all these to distil local market insights:

 

 

Multimodal LLMs can serve as an alternative to OCR and for chart documentation  - We've found Multimodal LLMs very useful in data extraction and documentation - for example, to get from a screenshot of a table to having that data in Excel, or to generate structured text descriptions of 1,000 analytical charts (hence making them 'searchable'). But the models are not yet at the stage to be able to consistently provide accurate analysis of a chart.

 

The cost of human quality control - This remains one of the biggest issues in high-accuracy use cases. Tracing back to source material in RAG or LLMs checking on LLMs, and reverse lookup are some tools we have been adding and these all help a lot in confirming that say the data extracted was accurate, but that's harder to do for more complex questions.

 

4. With A Model A Week, How Do You Tweak..?

 

Talking about the practical aspects. In 2024, there were weeks where the big tech like Google, OpenAI, Meta, Anthropic, NVIDIA, IBM, Amazon, or startups (e.g. the latest buzz with DeepSeek), would launch multiple new AI models, within a matter of days. Some of these would reset prior limitations and open new opportunities. That's fascinating progress but... how do you keep up with all the novelty if you are deploying applications within an enterprise? 

 

Our approach has been to focus on what doesn't really change - workflow and abstracted business problems, with the flexibility to plug in (and test) new models and methods. 

 

5. Let's Not Get Over-LLM-ed Either

 

For the productivity of data operations and research automation, in many cases there is no need for... any LLMs (or AI). The 'old-style' automation can work very well. While expanding Kubro(TM) with all the new LLM-empowered tools, we still see these as components of the broader workflow solution.

 

 

Let me also share a few highlights from 2024 - where we have been adding (and tweaking) Kubro(TM) features throughout the year on a weekly basis.

 

Our code base is now over 500,000 lines of in-house written proprietary code (excluding open source modules), with a multitude of modules for different use cases in Kubro(TM) and on a standalone basis. 

 

In 2024, the single biggest addition to Kubro(TM) was a new module to help clients automate the handling of larger documents - from SEC filings and UK Companies House, to generally PDF files, while keeping the human analyst still at the centre of the workflow. That's the Document Explorer, with its multiple components.

 

We have also gone into Retrieval Augmented Generation (RAG) as well, more on the R&D side so far and an internal use case.

 

But it's not just about the hot new thing. We have been continuously evolving the core Kubro(TM) capabilities in capturing, classifying, and extracting the information from the web and other sources, including the back-end health checks quality controls, and reporting.

 

We ran more projects on specialist data sets such as People Data, forms 990 and 5500 filings in the US, US REITs, and we started selling Lead Generation services, utilizing Kubro(TM) at the backend.

 

The 'robo-reports' / newsletters (like this one) backend toolkit has also been evolving with more sophisticated filtering methods, including a sequence of LLMs but at the same time adding more features for human curation of content.

 

There was a lot of travel in 2024, especially to London, also San Francisco, and New York, where in the first two we hosted the first CAST (Coffee And Showcase Tour) sessions - small group meetings in November and September. We plan for the next CAST sessions in Hong Kong and Singapore in Q1.

 

Thank you for your support in 2024, and wish you a productive 2025!

 

 

Robert Ciemniak
Founder-CEO

Robotic Online Intelligence Ltd