ChatGPT has been very interesting, but what really bothers me is that it is designed to provide an instant answer while the quality of that answer may vary.
For example, if I ask it to “find me a cheap lithium battery charging integrated circuit,” it will either give me an answer that is outdated because its training data cut-off is in 2021:
Or, if I enable the web search plugin, it will give me a very brief summary of the top five links it finds:
The answer looks useful, but it has not really done the research. It has given me a few places to begin doing the research myself.
However, I think this is a bit counter to how we should be using computers - computers are supposed to go crunch the numbers and figure stuff out while we are at the beach sipping margaritas. The reason ChatGPT needs to respond quickly is because of our culture of instant gratification, but this is counterproductive.
Instead, I wanted to develop a tool that would take its own time, do a fair bit of work for me, and then let me review its results as a starting point for my research.
How it works
After playing around for a weekend, this is the structure I came up with:
Enter an input query
Use the OpenAI API to generate search terms
Ask the LLM which websites should specifically be checked
Get the results for those search terms
Read through each result and extract any relevant information
Produce a summary of the relevant information on each page
Use all of those summaries to generate an overview
The user interface is good old email.
I send a question to the tool and it replies with the completed research. This means it can take as long as it needs without making me sit in front of a chat window and wait. People don’t expect emails to get instant replies so it works out from a user expectation perspective.
Trying it on real questions
Finding a battery charging IC
One of my test queries was:
Find me the cheapest battery charging integrated circuit for a single-cell lithium polymer battery.
The tool generated search terms, read through the results, produced an overall answer, and included a summary of every page it used. I also added a “lateral thinking” section that tries to reframe the question in case the original approach is wrong.
It does a lot of this asynchronously. If there are 50 search results, it gets the content for all of them in parallel and extracts the relevant information in parallel. Even then, because of API rate limits, it can be a pretty slow process - a deep query can easily take 15 to 20 minutes. But thinking about what is actually happening in that time, the LLM is doing the work I would have to do anyway.
Here is the full email it returned:
The result is not perfect. In fact, it failed to answer the question: it did not find a specific IC and compare actual prices. But it did collect much more context than the normal web search response, and the per-page summaries make it easier to see why the final answer is good or bad.
That is important because I do not want the output to be treated as a definitive answer. I want it to be a useful starting point that I can verify.
Slow does not automatically mean good
The biggest lesson so far is that giving the model more time and more information does not automatically produce a better answer.
I made “standard” and “deep” search modes, with options to either discard irrelevant pages or summarise everything. Surprisingly, deep search was sometimes worse than standard search. More sources also means more irrelevant information, more opportunities for a bad summary, and more confident nonsense in the final answer.
I found some particularly strange hallucinations about air pollution and climate change in completely unrelated research. After digging through the logs, I found that some summarisation prompts were being sent with no content to summarize. The model happily wrote a plausible summary anyway. I now ask it to respond with “Content missing” when there is nothing to summarise.
There is also a cost and processing-time problem. One experiment made 56 calls just to clean up individual source summaries and another 324 calls to improve partial answers:
A particularly deep query ran from around 11:07 p.m. until 2:03 a.m. and cost almost US$2. I cut that to about US$0.20 by skipping calls when the input text was too short, but the result was also garbage.
I also tried moving more of the work locally using Hugging Face summarisation and question-answering models. Facebook’s BART CNN model and Google’s Pegasus reduced the amount of text that needed to go through ChatGPT, but the quality was inconsistent. A simpler prompt made Pegasus much better, although there is still plenty of work to do.
This configuration - the processing pipeline - is just what seemed reasonable after a weekend of experimenting. I am sure there are better ways to do it.
A more directed approach
One alternative is a ReAct-style model, where the LLM researches sequentially:
Question: Find me a battery charging IC
Thought: What battery charging ICs exist?
Action: Search for battery charging ICs
Response: TP4056, BQ24075, etc.
Thought: Which one is cheapest?
Action: Search for the price of each candidate
Instead of obtaining all the information at once, processing all of it upfront, and then churning out an answer, this would produce more directed research. The primary advantage is that it should be cheaper and use fewer API calls.
What happens next?
I have not worked on this much beyond the experiments above. The processing time is long, the results are not always accurate, and there still are not enough data sources for some of the questions I care about.
More importantly, combining search with an LLM feels like an obvious native feature for search engines. It seems likely that they will integrate this directly, and I do not see much point in competing with that.
For now I am leaving AutoResearch here as an experiment.
I still think the basic idea is interesting: an AI does not always need to answer instantly. If it can spend 20 minutes doing work that would take me an hour, that seems like a very reasonable trade.