🤖 Multimodal LLMs can certainly help sort out and search through the family photos but they can also be useful for research businesses.

🔍 That moment, "Hey, where is that chart we produced some time ago about X and Y...", feels likely familiar to research firms. Producing a new chart sometimes takes less than finding that old gem...

💡 In this example, we use an LLM to 'read' the charts in a pre-defined manner (note that at this stage of LLM capabilities, this is not to analyze the data, only to document what is in the chart/image). In this case, the latest data on China housing markets.

Storing such semi-structured descriptions in a database with links to images allows a simple search (or a more complex RAG) to find relevant charts.

We used Gemini 1.5 Pro API, but llava-phi3 via Ollama on-device also works fine.