Developing the capacity to annotate massive volumes of data while maintaining quality is a function of the model development lifecycle that enterprises often underestimate. It’s resource intensive and requires specialized expertise.
At the heart of any successful machine learning/artificial intelligence (ML/AI) initiative is a commitment to high-quality training data and a pathway to quality data that is proven and well-defined. Without this quality data pipeline, the initiative is doomed to fail.
Computer vision or data science teams often turn to external partners to develop their data training pipeline, and these partnerships drive model performance.
There is no one definition of quality: “quality data” is completely contingent on the specific computer vision or machine learning project. However, there is a general process all teams can follow when working with an external partner, and this path to quality data can be broken down into four prioritized phases.
Annotation criteria and quality requirements
Training data quality is an evaluation of a data set’s fitness to serve its purpose in a given ML/AI use case.
The computer vision team needs to establish an unambiguous set of rules that describe what quality means in the context of their project. Annotation criteria are the collection of rules that define which objects to annotate, how to annotate them correctly, and what the quality targets are.
Accuracy or quality targets define the lowest acceptable result for evaluation metrics like accuracy, recall, precision, F1 score, et cetera. Typically, a computer vision team will have quality targets for how accurately objects of interest were classified, how accurately objects were localized, and how accurately relationships between objects were identified.
Workforce training and platform configuration
Platform configuration. Task design and workflow setup require time and expertise, and accurate annotation requires task-specific tools. At this stage, data science teams need a partner with expertise to help them determine how best to configure labeling tools, classification taxonomies, and annotation interfaces for accuracy and throughput.
Worker testing and scoring. To accurately label data, annotators need a well-designed training curriculum so they fully understand the annotation criteria and domain context. The annotation platform or external partner should ensure accuracy by actively tracking annotator proficiency against gold data tasks or when a judgement is modified by a higher-skilled worker or admin.
Ground truth or gold data. Ground truth data is crucial at this stage of the process as the baseline to score workers and measure output quality. Many computer vision teams are already working with a ground truth data set.
Sources of authority and quality assurance
There is no one-size-fits-all quality assurance (QA) approach that will meet the quality standards of all ML use cases. Specific business objectives, as well as the risk associated with an under-performing model, will drive quality requirements. Some projects reach target quality using multiple annotators. Others require complex reviews against ground truth data or escalation workflows with verification from a subject matter expert.
There are two primary sources of authority that can be used to measure the quality of annotations and that are used to score workers: gold data and expert review.
- Gold data: The gold data or ground truth set of records can be used both as a qualification tool for testing and scoring workers at the outset of the process and also as the measure for output quality. When you use gold data to measure quality, you compare worker annotations to your expert annotations for the same data set, and the difference between these two independent, blind answers can be used to produce quantitative measurements like accuracy, recall, precision, and F1 scores.
- Expert review: This method of quality assurance relies on expert review from a highly skilled worker, an admin, or from an expert on the customer side, sometimes all three. It can be used in conjunction with gold data QA. The expert reviewer looks at the answer given by the qualified worker and either approves it or makes corrections as needed, producing a new correct answer. Initially, an expert review may take place for every single instance of labeled data, but over time, as worker quality improves, expert review can utilize random sampling for ongoing quality control.
Iterating on data success
Once a computer vision team has successfully launched a high quality training data pipeline, it can accelerate progress to a production ready model. Through ongoing support, optimization, and quality control, an external partner can help them:
- Track velocity: In order to scale effectively, it’s good to measure annotation throughput. How long is it taking data to move through the process? Is the process getting faster?
- Tune worker training: As the project scales, labeling and quality requirements may evolve. This necessitates ongoing workforce training and scoring.
- Train on edge cases: Over time, training data should include more and more edge cases in order to make your model as accurate and robust as possible.
Without high-quality training data, even the best funded, most ambitious ML/AI projects cannot succeed. Computer vision teams need partners and platforms they can trust to deliver the data quality they need and to power life-changing ML/AI models for the world.
Alegion is the proven partner to build the training data pipeline that will fuel your model throughout its lifecycle. Contact Alegion at email@example.com.
This content was produced by Alegion. It was not written by MIT Technology Review’s editorial staff.
Your microbiome ages as you do—and that’s a problem
These ecosystems appear to change as we age—and these changes can potentially put us at increased risk of age-related diseases. So how can we best look after them as we get old? And could an A-grade ecosystem help fend off diseases and help us lead longer, healthier lives?
It’s a question I’ve been pondering this week, partly because I know a few people who have been put on antibiotics for winter infections. These drugs—lifesaving though they can be—can cause mass destruction of gut microbes, wiping out the good along with the bad. How might people who take them best restore a healthy ecosystem afterwards?
I also came across a recent study in which scientists looked at thousands of samples of people’s gut microbe populations to see how they change with age. The standard approach to working out what microbes are living in a person’s gut is to look at feces. The idea is that when we have a bowel movement, we shed plenty of gut bacteria. Scientists can find out which species and strains of bacteria are present to get an estimate of what’s in your intestines.
In this study, a team based at University College Cork in Ireland analyzed data that had already been collected from 21,000 samples of human feces. These had come from people all over the world, including Europe, North and South America, Asia, and Africa. Nineteen nationalities were represented. The samples were all from adults between 18 and 100.
The authors of this study wanted to get a better handle on what makes for a “good” microbiome, especially as we get older. It has been difficult for microbiologists to work this out. We do know that some bacteria can produce compounds that are good for our guts. Some seem to aid digestion, for example, while others lower inflammation.
But when it comes to the ecosystem as a whole, things get more complicated. At the moment, the accepted wisdom is that variety seems to be a good thing—the more microbial diversity, the better. Some scientists believe that unique microbiomes also have benefits, and that a collection of microbes that differs from the norm can keep you healthy.
The team looked at how the microbiomes of younger people compared with those of older people, and how they appeared to change with age. The scientists also looked at how the microbial ecosystems varied with signs of unhealthy aging, such as cognitive decline, frailty, and inflammation.
They found that the microbiome does seem to change with age, and that, on the whole, the ecosystems in our guts do tend to become more unique—it looks as though we lose aspects of a general “core” microbiome and stray toward a more individual one.
But this isn’t necessarily a good thing. In fact, this uniqueness seems to be linked to unhealthy aging and the development of those age-related symptoms listed above, which we’d all rather stave off for as long as possible. And measuring diversity alone doesn’t tell us much about whether the bugs in our guts are helpful or not in this regard.
The findings back up what these researchers and others have seen before, challenging the notion that uniqueness is a good thing. Another team has come up with a good analogy, which is known as the Anna Karenina principle of the microbiome: “All happy microbiomes look alike; each unhappy microbiome is unhappy in its own way.”
Of course, the big question is: What can we do to maintain a happy microbiome? And will it actually help us stave off age-related diseases?
There’s plenty of evidence to suggest that, on the whole, a diet with plenty of fruit, vegetables, and fiber is good for the gut. A couple of years ago, researchers found that after 12 months on a Mediterranean diet—one rich in olive oil, nuts, legumes, and fish, as well as fruit and veg—older people saw changes in their microbiomes that might benefit their health. These changes have been linked to a lowered risk of developing frailty and cognitive decline.
But at the individual level, we can’t really be sure of the impact that changes to our diets will have. Probiotics are a good example; you can chug down millions of microbes, but that doesn’t mean that they’ll survive the journey to your gut. Even if they do get there, we don’t know if they’ll be able to form niches in the existing ecosystem, or if they might cause some kind of unwelcome disruption. Some microbial ecosystems might respond really well to fermented foods like sauerkraut and kimchi, while others might not.
I personally love kimchi and sauerkraut. If they do turn out to support my microbiome in a way that protects me against age-related diseases, then that’s just the icing on the less-microbiome-friendly cake.
To read more, check out these stories from the Tech Review archive:
At-home microbiome tests can tell you which bugs are in your poo, but not much more than that, as Emily Mullin found.
Industrial-scale fermentation is one of the technologies transforming the way we produce and prepare our food, according to these experts.
Can restricting your calorie intake help you live longer? It seems to work for monkeys, as Katherine Bourzac wrote in 2009.
Adam Piore bravely tried caloric restriction himself to find out if it might help people, too. Teaser: even if you live longer on the diet, you will be miserable doing so.
From around the web:
Would you pay $15,000 to save your cat’s life? More people are turning to expensive surgery to extend the lives of their pets. (The Atlantic)
The World Health Organization will now start using the term “mpox” in place of “monkeypox,” which will be phased out over the next year. (WHO)
After three years in prison, He Jiankui—the scientist behind the infamous “CRISPR babies”—is attempting a comeback. (STAT)
Tech that allows scientists to listen in on the natural world is revealing some truly amazing discoveries. Who knew that Amazonian sea turtles make more than 200 distinct sounds? And that they start making sounds before they even hatch? (The Guardian)
These recordings provide plenty of inspiration for musicians. Whale song is particularly popular. (The New Yorker)
Scientists are using tiny worms to diagnose pancreatic cancer. The test, launched in Japan, could be available in the US next year. (Reuters)
The Download: circumventing China’s firewall, and using AI to invent new drugs
As protests against rigid covid control measures in China engulfed social media in the past week, one Twitter account has emerged as the central source of information: @李老师不是你老师 (“Teacher Li Is Not Your Teacher”).
People everywhere in China have sent protest footage and real-time updates to the account through private messages, and it has posted them, with the sender’s identity hidden, on their behalf.
The man behind the account, Li, is a Chinese painter based in Italy, who requested to be identified only by his last name in light of the security risks. He’s been tirelessly posting footage around the clock to help people within China get information, and also to inform the wider world.
The work has been taking its toll—he’s received death threats, and police have visited his family back in China. But it also comes with a sense of liberation, Li told Zeyi Yang, our China reporter. Read the full story.
Biotech labs are using AI inspired by DALL-E to invent new drugs
The news: Text-to-image AI models like OpenAI’s DALL-E 2—programs trained to generate pictures of almost anything you ask for—have sent ripples through the creative industries. Now, two biotech labs are using this type of generative AI, known as a diffusion model, to conjure up designs for new types of protein never seen in nature.
Why it matters: Proteins are the fundamental building blocks of living systems. These protein generators can be directed to produce designs for proteins with specific properties, such as shape or size or function. In effect, this makes it possible to come up with new proteins to do particular jobs on demand. Researchers hope that this will eventually lead to the development of new and more effective drugs. Read the full story.
The Blue Technology Barometer 2022/23
The overall rankings tab shows the performance of the examined
economies relative to each other and aggregates scores generated
across the following four pillars: ocean environment, marine activity,
technology innovation, and policy and regulation.
This pillar ranks each country according to its levels of
marine water contamination, its plastic recycling efforts, the
CO2 emissions of its marine activities (relative to the size
of its economy), and the recent change of total emissions.
This pillar ranks each country on the sustainability of its
marine activities, including shipping, fishing, and protected
This pillar ranks each country on its contribution to ocean
sustainable technology research and development, including
expenditure, patents, and startups.
This pillar ranks each country on its stance on ocean
sustainability-related policy and regulation, including
national-level policies, taxes, fees, and subsidies, and the
implementation of international marine law.
Get access to technology journalism that matters.
MIT Technology Review offers in-depth reporting on today’s most MIT
Technology Review offers in-depth reporting on today’s most
important technologies to prepare you for what’s coming next.
MIT Technology Review Insights would like to thank the following
individuals for their time, perspective, and insights:
- Valérie Amant, Director of Communications, The SeaCleaners
- Charlotte de Fontaubert, Global Lead for the Blue Economy, World Bank Group
- Ian Falconer, Founder, Fishy Filaments
- Ben Fitzgerald, Managing Director, CoreMarine
- Melissa Garvey, Global Director of Ocean Protection, The Nature Conservancy
Michael Hadfield, Emeritus Professor, Principal Investigator, Kewalo Marine Laboratory, University of Hawaii
- Takeshi Kawano, Executive Director, Japan Agency for Marine-Earth Science and Technology
- Kathryn Matthews, Chief Scientist, Oceana
- Alex Rogers, Science Director, REV Ocean
- Ovais Sarmad, Deputy Executive Secretary, United Nations Framework Convention on Climate Change
- Thierry Senechal, Managing Director, Finance for Impact
- Jyotika Virmani, Executive Director, Schmidt Ocean Institute
- Lucy Woodall, Associate Professor of Marine Biology, University of Oxford, and Principal Scientist at Nekton
Methodology: The Blue Technology Barometer 2022/23
Now in its second year, the Blue Technology Barometer assesses and ranks how each of the world’s largest
maritime economies promotes and develops blue (marine-centered) technologies that help reverse the impact of
climate change on ocean ecosystems, and how they leverage ocean-based resources to reduce greenhouse gases and
other effects of climate change.
To build the index, MIT Technology Review Insights compiled 20 quantitative and qualitative data indicators
for 66 countries and territories with coastlines and maritime economies. This included analysis of select
datasets and primary research interviews with global blue technology innovators, policymakers, and
international ocean sustainability organizations. Through trend analysis, research, and a consultative
peer-review process with several subject matter experts, weighting assumptions were assigned to determine the
relative importance of each indicator’s influence on a country’s blue technology leadership.
These indicators measure how each country or territory’s economic and maritime industries have affected its
marine environment and how quickly they have developed and deployed technologies that help improve ocean
health outcomes. Policy and regulatory adherence factors were considered, particularly the observance of
international treaties on fishing and marine protection laws.
The indicators are organized into four pillars, which evaluate metrics around a sustainability theme. Each
indicator is scored from 1 to 10 (10 being the best performance) and is weighted for its contribution to its
respective pillar. Each pillar is weighted to determine its importance in the overall score. As these research
efforts center on countries developing blue technology to promote ocean health, the technology pillar is
ranked highest, at 50% of the overall score.
The four pillars of the Blue Technology Barometer are:
Carbon emissions resulting from maritime activities and their relative growth. Metrics in this pillar also
assess each country’s efforts to mitigate ocean pollution and enhance ocean ecosystem health.
Efforts to promote sustainable fishing activities and increase and maintain marine protected areas.
Progress in fostering the development of sustainable ocean technologies across several relevant fields:
- Clean innovation scores from MIT Technology Review Insights’ Green Future Index 2022.
- A tally of maritime-relevant patents and technology startups.
- An assessment of each economy’s use of technologies and tech-enabled processes that facilitate ocean
Commitment to signing and enforcing international treaties to promote ocean sustainability and enforce
MIT Technology Review was founded at the Massachusetts Institute of Technology in 1899. MIT Technology Review
Insights is the custom publishing division of MIT Technology Review. We conduct qualitative and quantitative
research and analysis worldwide and publish a wide variety of content, including articles, reports,
infographics, videos, and podcasts.
If you have any comments or queries, please
get in touch.