Even with the help of micro-phenomenology, however, wrapping up what’s going on inside your head into a neat verbal package is a daunting task. So instead of asking subjects to struggle to represent their experiences in words, some scientists are using technology to try to reproduce those experiences. That way, all subjects need to do is confirm or deny that the reproductions match what’s happening in their heads.
In a study that has not yet been peer reviewed, a team of scientists from the University of Sussex, UK, attempted to devise such a question by simulating visual hallucinations with deep neural networks. Convolutional neural networks, which were originally inspired by the human visual system, typically take an image and turn it into useful information—a description of what the image contains, for example. Run the network backward, however, and you can get it to produce images—phantasmagoric dreamscapes that provide clues about the network’s inner workings.
The idea was popularized in 2015 by Google, in the form of a program called DeepDream. Like people around the world, the Sussex team started playing with the system for fun, says Anil Seth, a professor of neuroscience and one of the study’s coauthors. But they soon realized that they might be able to leverage the approach to reproduce various unusual visual experiences.
Drawing on verbal reports from people with hallucination-causing conditions like vision loss and Parkinson’s, as well as from people who had recently taken psychedelics, the team designed an extensive menu of simulated hallucinations. That allowed them to obtain a rich description of what was going on in subjects’ minds by asking them a simple question: Which of these images best matches your visual experience? The simulations weren’t perfect, although many of the subjects were able to find an approximate match.
Unlike the decoding research, this study involved no brain scans—but, Seth says, it may still have something valuable to say about how hallucinations work in the brain. Some deep neural networks do a respectable job of modeling the inner mechanisms of the brain’s visual regions, and so the tweaks that Seth and his colleagues made to the network may resemble the underlying biological “tweaks” that made the subjects hallucinate. “To the extent that we can do that,” Seth says, “we’ve got a computational-level hypothesis of what’s happening in these people’s brains that underlie these different experiences.”
This line of research is still in its infancy, but it suggests that neuroscience might one day do more than simply telling us what someone else is experiencing. By using deep neural networks, the team was able to bring its subjects’ hallucinations out into the world, where anyone could share in them.
Externalizing other sorts of experiences would likely prove far more difficult—deep neural networks do a good job of mimicking senses like vision and hearing, but they can’t yet model emotions or mind-wandering. As brain modeling technologies advance, however, they could bring with them a radical possibility: that people might not only know, but actually share, what is going on in someone else’s mind.
A new training model, dubbed “KnowNo,” aims to address this problem by teaching robots to ask for our help when orders are unclear. At the same time, it ensures they seek clarification only when necessary, minimizing needless back-and-forth. The result is a smart assistant that tries to make sure it understands what you want without bothering you too much.
Andy Zeng, a research scientist at Google DeepMind who helped develop the new technique, says that while robots can be powerful in many specific scenarios, they are often bad at generalized tasks that require common sense.
For example, when asked to bring you a Coke, the robot needs to first understand that it needs to go into the kitchen, look for the refrigerator, and open the fridge door. Conventionally, these smaller substeps had to be manually programmed, because otherwise the robot would not know that people usually keep their drinks in the kitchen.
That’s something large language models (LLMs) could help to fix, because they have a lot of common-sense knowledge baked in, says Zeng.
Now when the robot is asked to bring a Coke, an LLM, which has a generalized understanding of the world, can generate a step-by-step guide for the robot to follow.
The problem with LLMs, though, is that there’s no way to guarantee that their instructions are possible for the robot to execute. Maybe the person doesn’t have a refrigerator in the kitchen, or the fridge door handle is broken. In these situations, robots need to ask humans for help.
KnowNo makes that possible by combining large language models with statistical tools that quantify confidence levels.
When given an ambiguous instruction like “Put the bowl in the microwave,” KnowNo first generates multiple possible next actions using the language model. Then it creates a confidence score predicting the likelihood that each potential choice is the best one.
The news: A new robot training model, dubbed “KnowNo,” aims to teach robots to ask for our help when orders are unclear. At the same time, it ensures they seek clarification only when necessary, minimizing needless back-and-forth. The result is a smart assistant that tries to make sure it understands what you want without bothering you too much.
Why it matters: While robots can be powerful in many specific scenarios, they are often bad at generalized tasks that require common sense. That’s something large language models could help to fix, because they have a lot of common-sense knowledge baked in. Read the full story.
—June Kim
Medical microrobots that travel inside the body are (still) on their way
The human body is a labyrinth of vessels and tubing, full of barriers that are difficult to break through. That poses a serious hurdle for doctors. Illness is often caused by problems that are hard to visualize and difficult to access. But imagine if we could deploy armies of tiny robots into the body to do the job for us. They could break up hard-to-reach clots, deliver drugs to even the most inaccessible tumors, and even help guide embryos toward implantation.
We’ve been hearing about the use of tiny robots in medicine for years, maybe even decades. And they’re still not here. But experts are adamant that medical microbots are finally coming, and that they could be a game changer for a number of serious diseases. Read the full story.
We haven’t always been right (RIP, Baxter), but we’ve often been early to spot important areas of progress (we put natural-language processing on our very first list in 2001; today this technology underpins large language models and generative AI tools like ChatGPT).
Every year, our reporters and editors nominate technologies that they think deserve a spot, and we spend weeks debating which ones should make the cut. Here are some of the technologies we didn’t pick this time—and why we’ve left them off, for now.
New drugs for Alzheimer’s disease
Alzmeiher’s patients have long lacked treatment options. Several new drugs have now been proved to slow cognitive decline, albeit modestly, by clearing out harmful plaques in the brain. In July, the FDA approved Leqembi by Eisai and Biogen, and Eli Lilly’s donanemab could soon be next. But the drugs come with serious side effects, including brain swelling and bleeding, which can be fatal in some cases. Plus, they’re hard to administer—patients receive doses via an IV and must receive regular MRIs to check for brain swelling. These drawbacks gave us pause.
Sustainable aviation fuel
Alternative jet fuels made from cooking oil, leftover animal fats, or agricultural waste could reduce emissions from flying. They have been in development for years, and scientists are making steady progress, with several recent demonstration flights. But production and use will need to ramp up significantly for these fuels to make a meaningful climate impact. While they do look promising, there wasn’t a key moment or “breakthrough” that merited a spot for sustainable aviation fuels on this year’s list.
Solar geoengineering
One way to counteract global warming could be to release particles into the stratosphere that reflect the sun’s energy and cool the planet. That idea is highly controversial within the scientific community, but a few researchers and companies have begun exploring whether it’s possible by launching a series of small-scale high-flying tests. One such launch prompted Mexico to ban solar geoengineering experiments earlier this year. It’s not really clear where geoengineering will go from here or whether these early efforts will stall out. Amid that uncertainty, we decided to hold off for now.
