The AI Industry Series: Top Healthcare AI Trends To Watch


AI needs doctors. Big pharma is taking an AI-first approach. Apple is revolutionizing clinical studies. We look at the top artificial intelligence trends reshaping healthcare.

Healthcare is emerging as a prominent area for AI research and applications.

And nearly every area across the industry will be impacted by the technology’s rise.

Image recognition, for example, is revolutionizing diagnostics. Recently, Google DeepMind’s neural networks matched the accuracy of medical experts in diagnosing 50 sight-threatening eye diseases.

Even pharma companies are experimenting with deep learning to design new drugs. For example, Merck partnered with startup Atomwise and GlaxoSmithKline is partnering with Insilico Medicine.

In the private market, healthcare AI startups have raised $4.3B across 576 deals since 2013, topping all other industries in AI deal activity.

AI in healthcare is currently geared towards improving patient outcomes, aligning the interests of various stakeholders, and reducing healthcare costs.

One of the biggest hurdles for artificial intelligence in healthcare will be overcoming inertia to overhaul current processes that no longer work, and experimenting with emerging technologies.

AI faces both technical and feasibility challenges that are unique to the healthcare industry. For example, there’s no standard format or central repository of patient data in the United States.

When patient files are faxed, emailed as unreadable PDFs, or sent as images of handwritten notes, extracting information poses a unique challenge for AI.

But big tech companies like Apple have an edge here, especially in onboarding a large network of partners, including healthcare providers and EHR vendors.

Generating new sources of data and putting EHR data in the hands of patients — as Apple is doing with ResearchKit and CareKit — promises to be revolutionary for clinical studies.

In our first industry AI deep dive, we use the CB Insights database to unearth trends that are transforming the healthcare industry.

Rise of AI-as-a-medical-device

The FDA is fast-tracking approvals of artificial intelligence software for clinical imaging & diagnostics.

In April, the FDA approved AI software that screens patients for diabetic retinopathy without the need for a second opinion from an expert.

It was given a “breakthrough device designation” to expedite the process of bringing the product to market.

The software, IDx-DR, was able to correctly identify patients with “more than mild diabetic retinopathy” 87.4% of the time, and identify those who did not have it 89.5% of the time.

IDx is one of the many AI software products approved by the FDA for clinical commercial applications in recent months. was approved to analyze CT scans and notify healthcare providers of potential strokes in patients. Post FDA-approval, closed a $21M Series A round from Google Ventures and Kleiner Perkins Caufield & Byers.

GE Ventures-backed startup Arterys was FDA-approved last year for analyzing cardiac images with its cloud AI platform. This year, the FDA cleared its liver and lung AI lesion spotting software for cancer diagnostics.

Fast-track regulatory approval opens up new commercial pathways for over 70 AI imaging & diagnostics companies that have raised equity financing since 2013, accounting for a total of 119 deals.

The FDA is focused on clearly defining and regulating “software-as-a-medical-device,” especially in the light of recent rapid advances in AI.

It now wants to apply the “pre-cert” approach — a program it piloted in January — to AI.

This will allow companies to make “minor changes to its devices without having to make submissions each time.” The FDA added that aspects of its regulatory framework like software validation tools will be made “sufficiently flexible” to accommodate advances in AI.

Neural nets spot atypical risk factors

Using AI, researchers are starting to study and measure atypical risk factors that were previously difficult to quantify.

Analysis of retinal images and voice patterns using neural networks could potentially help identify risk of heart disease.

Researchers at Google used a neural network trained on retinal images to find cardiovascular risk factors, according to a paper published in Nature this year.

The research found that not only was it possible to identify risk factors such as age, gender, and smoking patterns through retinal images, it was also “quantifiable to a degree of precision not reported before.”

In another study, Mayo Clinic partnered with Beyond Verbal, an Israeli startup that analyzes acoustic features in voice, to find distinct voice features in patients with coronary artery disease (CAD). The study found 2 voice features that were strongly associated with CAD when subjects were describing an emotional experience.

A recent study from startup Cardiogram suggests “heart rate variability changes driven by diabetes can be detected via consumer, off-the-self wearable heart rate sensors” using deep learning. One algorithmic approach showed 85% accuracy in detecting diabetes from heart rate.

Another emerging application is using blood work to detect cancer. Startups like Freenome are using AI to find patterns in cell-free biomarkers circulating in the blood that could be associated with cancer.

AI’s ability to find patterns will continue to pave the way for new diagnostic methods and identification of previously unknown risk factors.

Apple disrupts clinical trials

Apple is building a clinical research ecosystem around the iPhone and Apple Watch. Data is at the core of AI applications, and Apple can provide medical researchers with two streams of patient health data that were not as easily accessible until now.

Interoperability — the ability to share health information easily across institutions and software systems — is an issue in healthcare, despite efforts to digitize health records.

This is particularly problematic in clinical trials, where matching the right trial with the right patient is a time-consuming and challenging process for both the clinical study team and the patient.

For context, there are over 18,000 clinical studies that are currently recruiting patients in the United States alone.

Patients may occasionally get trial recommendations from their doctors if a physician is aware of an ongoing trial.

Otherwise, the onus of scouring through ClinicalTrials.Gov — a comprehensive federal database of past and ongoing clinical trials — falls on the patient.

Apple is changing how information flows in healthcare and is opening up new possibilities for AI, specifically around how clinical study researchers recruit and monitor patients.

Since 2015, Apple has launched two open-source frameworks — ResearchKit and CareKit — to help clinical trials recruit patients and monitor their health remotely.

The frameworks allow researchers and developers to create medical apps to monitor people’s daily lives.

For example, researchers at Duke University developed an Autism & Beyond app that uses the iPhone’s front camera and facial recognition algorithms to screen children for autism.

Similarly, nearly 10,000 people use the mPower app, which provides exercises like finger tapping and gait analysis to study patients with Parkinson’s disease who have consented to share their data with the broader research community.

Apple is also working with popular EHR vendors like Cerner and Epic to solve interoperability problems.

In January 2018, Apple announced that iPhone users will now have access to all their electronic health records from participating institutions on their iPhone’s Health app.

Called “Health Records,” the feature is an extension of what AI healthcare startup Gliimpse was working on before it was acquired by Apple in 2016.

“More than 500 doctors and medical researchers have used Apple’s ResearchKit and CareKit software tools for clinical studies involving 3 million participants on conditions ranging from autism and Parkinson’s disease to post-surgical at-home rehabilitation and physical therapy.” — Apple

In an easy-to-use interface, users can find all the information they need on allergies, conditions, immunizations, lab results, medications, procedures, and vitals.

In June, Apple rolled out a Health Records API for developers.

Users can now choose to share their data with third-party applications and medical researchers, opening up new opportunities for disease management and lifestyle monitoring.

The possibilities are seemingly endless when it comes to using AI and machine learning for early diagnosis, driving decisions in drug design, enrolling the right pool of patients for studies, and remotely monitoring patients’ progress throughout studies.

Big pharma’s AI re-branding

With AI biotech startups emerging, traditional pharma companies are looking to AI SaaS startups for innovative solutions.

In May 2018, Pfizer entered into a strategic partnership with XtalPi — an AI startup backed by tech giants like Tencent and Google — to predict pharmaceutical properties of small molecules and develop “computation-based rational drug design.”

But Pfizer is not alone.

Top pharmaceutical companies like Novartis, Sanofi, GlaxoSmithKlein, Amgen, and Merck have all announced partnerships in recent months with AI startups aiming to discover new drug candidates for a range of diseases from oncology and cardiology.

“The biggest opportunity where we are still in the early stage is to use deep learning and artificial intelligence to identify completely new indications, completely new medicines. ” — Bruno Strigini, Former CEO of Novartis Oncology

Interest in the space is driving the number of equity deals to startups: 20 as of Q2’18, equal to all of 2017.

While biotech AI companies like Recursion Pharmaceuticals are investing in both AI and drug R&D, traditional pharma companies are partnering with AI SaaS startups.

Although many of these startups are still in the early stages of funding, they already boast a roster of pharma clients.

There are few measurable metrics of success in the drug formulation phase, but pharma companies are betting millions of dollars on AI algorithms to discover novel therapeutic candidates and transform the drawn-out drug discovery process.

AI has applications beyond the discovery phase of drug development.

In one of the largest M&A deals in artificial intelligence, Roche Holding acquired Flatiron Health for $1.9B in February 2018. Flatiron uses machine learning to mine patient data.

Today, over 2,500 clinicians use Flatiron’s OncoEMR — an electronic medical record software focused on oncology — and over 2 million active patient records are reportedly available for research.

Roche hopes to gather real world evidence (RWE) — analysis of data in electronic medical records and other sources to determine the benefits and risks of drugs — to support its oncology pipeline.

Apart from use by the FDA to monitor post-marketing drug safety, RWE can help design better clinical trials and new treatments in the future.

AI needs doctors

AI companies need medical experts to annotate images to teach algorithms how to identify anomalies. Tech giants and governments are investing heavily in annotation and making the datasets publicly available to other researchers.

Google DeepMind partnered with Moorfield’s Eye Hospital two years ago to explore the use of AI in detecting eye diseases. Recently, DeepMind’s neural networks were able to recommend the correct referral decisions for 50 sight-threatening eye diseases with 94% accuracy.

This was just the Phase 1 of the study. But in order to train the algorithms, DeepMind invested significant time into labeling and cleaning up the database of OCT (Optical Coherence Tomography) scans — used for detection of eye conditions —and making it “AI ready.”

Clinical labeling of the 14,884 scans in the dataset involved various trained ophthalmologists and optometrists who had to review the OCT scans.

Alibaba had a similar story when it decided to venture into AI for diagnostics around 2016.

“The samples needed to be annotated by specialists, because if a sample doesn’t have any annotation we don’t know if this is a healthy person or if it’s a sample from a sick person… This was a pretty important step.” Min Wanli, Alibaba Cloud, told Alizila News

According to Min Wanli, chief machine intelligence scientist for Alibaba Cloud, once the company partnered with health institutions to access the medical imaging data, it had to hire specialists to annotate the imaging samples.

AI unicorn Yitu Technology, which is branching into AI diagnostics, discussed the importance of having a medical team in an interview with the South China Morning Post.

Yitu claims it has a team of 400 doctors working part time to label medical data, adding that higher salary ranges for US doctors may make this an expensive option for US AI startups.

But in the US, government agencies like the National Institute of Health (NIH) are promoting AI research.

The NIH released a dataset of 32,000 lesions annotated and identified in CT images — anonymized from 4,400 patients — in July this year.

Called DeepLesion, the dataset was formed using images marked by radiologists with clinical findings. It is one the largest of its kind, according to the NIH.

Large enough to train a deep neural network, the NIH hopes that the dataset will “enable the scientific community to create a large-scale universal lesion detector with one unified framework.”

Private companies like GE and Siemens are also looking at ways to create large-scale datasets.

GE Healthcare was granted a patent in May discussing machine learning to analyze cell types in microscope images.

The patent proposes an “intuitive interface enabling medical staff (e.g., pathologists, biologists) to annotate and evaluate different cell phenotypes used in the algorithm and the presented through the interface.”

Although other algorithmic approaches have been proposed to make the process less manual, AI currently relies heavily on medical experts for training.

Making annotated datasets available to the public, similar to what DeepMind and NIH are doing, is lowering the barrier to entry for other AI researchers.

China climbs the ranks in healthcare AI

Chinese investors are increasingly investing in startups abroad, the local healthcare AI startup scene is growing, and Chinese tech giants are bringing products from other countries to mainland China through partnerships.

From negligible deal activity just a few years ago, China has quickly climbed the ranks in the global healthcare AI market.

In H1’18, China surpassed the United Kingdom to become the second most active country for healthcare AI deals.

With $72M in funding and investors like Sequoia Capital China, Infervision is the most well-funded Chinese startup focused exclusively on AI solutions for the healthcare industry.

In parallel, Chinese investment in foreign healthcare AI startups is on the rise.

More recently, Fosun Pharmaceutical took a minority stake in US-based Butterfly Network, Tencent Holdings invested in Atomwise, Legend Capital backed Lunit in South Korea, and IDG Capital invested in India-based SigTuple.

The Chinese government issued an artificial intelligence plan last year, with a vision of becoming a global leader in AI research by 2030. Healthcare is one of the 4 areas of focus for the nation’s first wave of AI applications.

The renewed focus on healthcare goes beyond becoming a world leader in AI technology.

The one child policy, though now lifted, has resulted in an aging population: there are over 158M people aged 65+, according to last year’s census. This, coupled with a labor shortage, has shifted the focus to increased automation in healthcare.

China’s efforts to consolidate medical data into one centralized repository started as early as 2016.

The country has issues with messy data and lack of interoperability, similar to the United States.

To address this, the Chinese government has opened several regional health data centers with the goal of consolidating data from national insurance claims, birth and death registries, and electronic health records.

Chinese big tech companies are now entering into healthcare AI with strong backing from the government.

In November 2017, the Chinese Ministry of Science and Technology announced that it will rely on Tencent to lead the way in developing an open AI platform for medical imaging and diagnostics, and Alibaba for smart city development (an umbrella term which would include smart healthcare).

E-commerce giant Alibaba started its healthcare AI focus in 2016 and launched an AI cloud platform called ET Medical Brain. It offers a suite of services, from AI-enabled diagnostics to intelligent scheduling based on a patients medical needs.

Tencent’s biggest strength is that it owns WeChat, the “app for everything.” It is the most popular social media application in China with 1B users, offering everything from messaging and photo sharing to money transfer and ride-hailing.

Around 38,000 medical institutions reportedly had WeChat accounts last year, of which 60% allowed users to register for appointments online. More than 2,000 hospitals accept WeChat payment.

WeChat potentially makes it easy for Tencent to collect huge amounts of patient and medical administrative data.

This year, Tencent partnered with Babylon Health, a UK-based startup developing a virtual healthcare assistant. WeChat users will have access to Babylon’s app, allowing them to message their symptoms and receive feedback and advice.

It also partnered with UK-based Medopad, which brings AI to remote patient monitoring. Medopad has signed over $120M in China trade deals.

Apart from these direct-to-consumer incentives, Tencent is focusing its internal R&D into developing the Miying healthcare AI platform.

Launched in 2017, Miying provides healthcare institutions with AI assistance in the diagnosis of various types of cancers and in health record management.

The initiative appears to be focused on research at this stage, with no immediate plans to charge hospitals for its AI-assisted imaging services.

DIY diagnostics is here

Artificial intelligence is turning the smartphone and consumer wearables into powerful at-home diagnostic tools.

Startup claims it’s making urine analysis as easy as taking a selfie.

Its first product,, uses the traditional urinalysis dipstick to monitor a range of urinary infections. Computer vision algorithms analyze the test strips under different lighting conditions and camera quality via a smartphone., which is already commercially available in Europe and Israel, was recently cleared by the FDA.

Smartphone penetration has increased in the United States in recent years. In parallel, the error rate of image recognition algorithms has dropped significantly, thanks to deep learning.

A combination of the two has opened up new possibilities of using the phone as a diagnostic tool.

For instance, SkinVision uses the smartphone’s camera to monitor skin lesions and assess skin cancer risk. SkinVision raised $7.6M from existing investors Leo Pharma and PHS Capital in July 2018.

The Amsterdam-based company will reportedly use the funding to push for a US launch with FDA clearance.

A number of ML-as-a-service platforms are integrating with FDA-approved home monitoring devices, alerting physicians when there is an abnormality.

One company, Biofourmis, is developing an AI analytics engine that pulls data from FDA-cleared medical wearables and predicts health outcomes for patients.

Israel-based ContinUse Biometrics is developing its own sensing technology. The startup monitors 20+ bio-parameters — including heart rate, glucose levels, and blood pressure — and uses AI to spot abnormal behavior. It raised a $20M Series B in Q1’18.

Apart from generating a rich source of daily data, AI-IoT has the potential to reduce time and costs associated with preventable hospital visits.

AI’s emerging role in value-based care

Artificial intelligence is beginning to play a role in quantifying the quality of service patients receive at hospitals.

A value-based service model is focused on the patient, where healthcare providers are incentivized to provide the highest quality care at the lowest possible cost.

This is in contrast to the fee-for-service model, where providers are paid in proportion to the number of services performed. The more procedures and tests that are prescribed, for example, the higher the financial incentive.

Conversations around quality of healthcare services date back to the 1960s. The challenge has been finding ways to assess healthcare quality with quantifiable, data-driven metrics.

Value-based service models got a fresh breath of life when the Patient Protection and Affordable Care Act was passed in 2010.

Some of the safeguards in place include providing a financial incentive to providers only if they meet quality performance measures, or imposing penalties for hospital-acquired infections and preventable readmission.

The goal of moving towards a value-based care system is to align providers’ incentives with those of the patient and payers. For instance, under the new system, hospitals will have a financial incentive in reducing unnecessary tests prescribed by physicians.

AI startup Qventus claims that Arkansas-based Mercy Hospital, which is shifting to a value-based care system, saw a 40% reduction in unnecessary lab tests in 4 months. The algorithm compared the behavior of physicians prescribing tests — even when they weren’t absolutely necessary — to those with their peers treating patients for the same condition.

Qventus has raised $43M in funding from investors like Bessemer Venture Partners, Mayfield Fund, New York–Presbyterian Hospital, and Norwest Venture Partners. The company has also developed an efficiency index for hospitals.

Georgia-based startup Jvion works with providers like Geisinger, Northwest Medical Specialties, and Onslow Memorial Hospital.

Some of Jvion’s case studies highlight successful use of machine learning in identifying admitted patients who are at risk of readmission within 30 days of hospitalization.

The care team can then use Jvion’s recommendation to educate the patient on daily, preventive measures. The algorithms combine patient health data with data on socioeconomic factors (like income and ease of transportation) and history of non-compliance, among other things, to calculate risk.

Another startup in this space is OM1, which has raised $36M from investors like General Catalyst and 7wire Ventures, focuses on real world evidence to determine efficacy of treatments.

Another approach is for insurance companies to identify at-risk patients and intervene by alerting the care provider.

In Q2’18, Blue Cross Blue Shield Venture Partners invested in startup Lumiata, which uses AI for individualized health spend forecasts. The $11M Series C round saw participation from diverse set of investors, including Intel Capital, Khosla Ventures, and Sandbox Industries.

AI for in-hospital management solutions is still in its nascent stages, but startups are focusing on helping providers to cut costs and improve quality of care.

What therapy bots can and can’t do

From life coaching to cognitive behavioral therapy to faith-based healing, AI therapy bots are cropping up on Facebook messenger.

High costs of mental health therapy and the appeal of round-the-clock availability is giving rise to a new era of AI-based mental health bots.

Early-stage startups are focused on using cognitive behavioral therapy — changing negative thoughts and behaviors — as a conversational extension of the many mood tracking and digital diary wellness apps in the market.

Woebot, which raised $8M from NEA, comes with a clear disclaimer that it’s not a replacement for traditional therapy or human interaction.

Another company, Wyse, raised $1.7M last year, and is available on iTunes as an “anxiety and depression” bot.

Startup X2 AI claims that its AI bot Tess has over 4 million paid users. It has also developed a “faith-based” chatbot, “Sister Hope,” which starts a conversation with a clear disclaimer and privacy terms (on messenger, chats are subject to FB privacy policy, and contents of conversations are visible to Facebook).

But accessibility to Facebook and a lack of regulations makes verification of some bots and their privacy terms difficult.

Users also have access to sponsored “AI” messenger bots and interactions that appear to be a string of pre-scripted messages with little to no contextual cues.

In narrow tasks like image recognition and language processing and generation, AI has come a long way.

But, as pioneering deep learning researcher Yoshua Bengio said in a recent podcast on The AI Element, “[AI] is like an idiot savant” with no notion of psychology, what a human being is, or how it all works.

Mental health is a spectrum, with high variability in symptoms and subjectivity in analysis.

In its current state, AI can do little beyond regular check-ins and fostering a sense of “companionship” with human-like language generation. For people who need more than a nudge to reconstruct negative sentences, the current generation of bots could fall short.

But our brains are wired to believe we are interacting with a human when chatting with bots, as one article in Psychology Today explains, without the complexity of having to decipher non-verbal cues.

This could be particularly problematic for more complex mental health issues, potentially creating a dependency on bots and quick-fix solutions that are incapable of in-depth analysis or the ability to address the underlying cause.

Jobs considered safest from automation are ones requiring a high level of emotional cognition and human-to-human interaction. This makes mental healthcare — despite the upside of cost and accessibility — a particularly hard task for AI.

The doctor’s office becomes wearable


Omron HeartGuide

The Omron HeartGuide is a blood pressure machine in wristwatch form. It works just like a full-size machine — a sphygmomanometer — and will be going for FDA clearance. It won’t require a prescription and comes at a time when more of us need it: New blood pressure guidelines issued in early 2018 suggest 46 percent of Americans have high blood pressure, up from 32 percent under the decades-old standard. The maladies hypertension can cause are almost too numerous to count.

If your last annual checkup showed that you have normal blood pressure, don’t sleep too well: Major studies have found that 18-33 percent of us have white coat hypertension, where high blood pressure only presents itself at the doctor’s office. And perhaps another 10 percent of us have masked hypertension, which doesn’t show up at all during a physical. Constant monitoring can surface these, so you’re neither taking pills you don’t need, nor not taking pills you should.

Dexcom G6

The Dexcom G6 is a small glucose monitor that wirelessly reports your blood sugar reading as often as every 5 minutes without finger sticks or calibration in most cases. Just carry a small Bluetooth reader or use the G6 app on your phone. The readings are 10-20 minutes delayed since the device doesn’t directly read blood, but its ability to record and reveal insights is, technically, the painless equivalent of 288 finger sticks per day.

See at Dexcom

The Medtronic Guardian Connect and Abbott Freestyle Libre round out this competitive category and all are aimed at eventually getting many of us to to wear them to avoid ever becoming Type II diabetic. That will require progress on retail pricing or Medicare coverage, which the Dexcom G6 does not yet enjoy.

See at Medtronic

Alivecor KardiaBand 

The $199 Alivecor KardiaBand is the first FDA-cleared watch band that functions as a simple electrocardiogram (ECG) machine, something you used to have to visit a clinic to benefit from. The band is 84 percent accurate at discriminating normal heartbeat from atrial fibrillation, a key contributor to future risk of stroke

Alivecor and Omron work together so their products can blend their respective health signals into a more meaningful snapshot, so you and your doctor get more insights, not just more data. For the majority of people who don’t have an Apple Watch, Alivecor offers the same functionality in a form that works with any smartphone and only costs $99.

See at Alivecor

Beyond Verbal

A groundbreaking study conducted at Mayo Clinic recently found the first evidence that voice may be an accurate indicator of whether a person has coronary artery disease. Eighty-one tonal features of voice were measured after patients spoke to a recording app using technology from vocal biomarker company Beyond Verbal. Pending further confirmation, this could open the door to you monitoring your circulatory health by just talking.

So the gear is here, but revolutionizing health is never as easy as that. Where does all this need to go next?

We need answers, not just information, and those answers need to be shaped to motivate us, not overwhelm or discourage us. The fitness band taught us that numbers alone aren’t very engaging after a while. And these devices will find a cold reception in the clinical world if they just upload lots of raw data to busy doctors.

A single dashboard of insights from all of our health signals will keep consumers engaged. Those signals will come from our wearables, phones, voice devices, connected cars, social graphs and smart home devices. They already speak to our wellness, we just don’t know their language yet.

Over the counter is the path to mass adoption. If personal health gear is unnecessarily encumbered with prescriptions it will stay virtually locked up in the doctor’s office that we visit once a year, at best. Note that widespread adoption of health monitoring tech could set up a tension between it and pharmaceuticals, whose business it is to treat what this tech may avert.

Somebody has to pay for it. There is significant incremental cost to consumers with this new gear and most will not want to pay for it. Employers, insurers and regulators need to act in unison to find the most useful tech and get it paid for. Continuous glucose monitors are typically covered, while other devices may only qualify for HSA or flex spend account dollars.

3 Ways AI Is Getting More Emotional


In January of 2018, Annette Zimmermann, vice president of research at Gartner, proclaimed: “By 2022, your personal device will know more about your emotional state than your own family.” Just two months later, a landmark study from the University of Ohio claimed that their algorithm was now better at detecting emotions than people are.

AI systems and devices will soon recognize, interpret, process, and simulate human emotions. A combination of facial analysis, voice pattern analysis, and deep learning can already decode human emotions for market research and political polling purposes. With companies like Affectiva,  BeyondVerbal and Sensay providing plug-and-play sentiment analysis software, the affective computing market is estimated to grow to $41 billion by 2022, as firms like Amazon, Google, Facebook, and Apple race to decode their users’ emotions.

Emotional inputs will create a shift from data-driven IQ-heavy interactions to deep EQ-guided experiences, giving brands the opportunity to connect to customers on a much deeper, more personal level. But reading people’s emotions is a delicate business. Emotions are highly personal, and users will have concerns about fear privacy invasion and manipulation. Before companies dive in, leaders should consider questions like:

  1. What are you offering? Does your value proposition naturally lend itself to the involvement of emotions? And can you credibly justify the inclusion of emotional clues for the betterment of the user experience?
  2. What are your customers’ emotional intentions when interacting with your brand? What is the nature of the interaction?
  3. Has the user given you explicit permission to analyze their emotions? Does the user stay in control of their data, and can they revoke their permission at any given time?
  4. Is your system smart enough to accurately read and react to a user’s emotions?
  5. What is the danger in any given situation if the system should fail — danger for the user, and/or danger for the brand?

Keeping those concerns in mind, business leaders should be aware of current applications for Emotional AI. These fall roughly into three categories:

Systems that use emotional analysis to adjust their response.

In this application, the AI service acknowledges emotions and factors them into its decision making process. However, the service’s output is completely emotion-free.

Conversational IVRs (interactive voice response) and chatbots promise to route customers to the right service flow faster and more accurately when factoring in emotions. For example, when the system detects a user to be angry, they are routed to a different escalation flow, or to a human.

AutoEmotive, Affectiva’s Automotive AI, and Ford are racing to get emotional car software market-ready to detect human emotions such as anger or lack of attention, and then take control over or stop the vehicle, preventing accidents or acts of road rage.

The security sector also dabbles in Emotion AI to detect stressed or angry people. The British government, for instance, monitors its citizens’ sentiments on certain topics over social media.

In this category, emotions play a part in the machine’s decision-making process. However, the machine still reacts like a machine — essentially, as a giant switchboard routing people in the right direction.

Systems that provide a targeted emotional analysis for learning purposes.

In 2009, Philips teamed up with a Dutch bank to develop the idea of a  “rationalizer” bracelet to stop traders from making irrational decisions by monitoring their stress levels, which it measures by monitoring the wearer’s pulse. Making traders aware of their heightened emotional states made them pause and think before making impulse decisions.

Brain Power’s smart glasses help people with autism better understand emotions and social cues. The wearer of this Google Glass type device sees and hears special feedback geared to the situation — for example coaching on facial expressions of emotions, when to look at people, and even feedback on the user’s own emotional state.

These targeted emotional analysis systems acknowledge and interpret emotions. The insights are communicated to the user for learning purposes. On a personal level, these targeted applications will act like a Fitbit for the heart and mind, aiding in mindfulness, self-awareness, and ultimately self-improvement, while maintaining a machine-person relationship that keeps the user in charge.

Targeted emotional learning systems are also being tested for group settings, such as by analyzing the emotions of students for teachers, or workers for managers. Scaling to group settings can have an Orwellian feeling: Concerns about privacy, creativity, and individuality have these experiments playing on the edge of ethical acceptance. More importantly, adequate psychological training for the people in power is required to interpret the emotional results, and to make adequate adjustments.

Systems that mimic and ultimately replace human-to- human interactions.

When smart speakers entered the American living room in 2014, we started to get used to hearing computers refer to themselves as “I.” Call it a human error or an evolutionary shortcut, but when machines talk, people assume relationships.

There are now products and services that use conversational UIs and the concept of “computers as social actors” to try to alleviate mental-health concerns. These applications aim to coach users through crises using techniques from behavioral therapy. Ellie helps treat soldiers with PTSD. Karim helps Syrian refugees overcome trauma. Digital assistants are even tasked with helping alleviate loneliness among the elderly.

Casual applications like Microsoft’s XiaoIce, Google Assistant, or Amazon’s Alexa use social and emotional cues for a less altruistic purpose — their aim is to secure users’ loyalty by acting like new AI BFFs. Futurist Richard van Hooijdonk quips: “If a marketer can get you to cry, he can get you to buy.”

The discussion around addictive technology is starting to examine the intentions behind voice assistants. What does it mean for users if personal assistants are hooked up to advertisers? In a leaked Facebook memo, for example, the social media company boasted to advertisers that it could detect, and subsequently target, teens’ feelings of “worthlessness” and “insecurity,” among other emotions.

Judith Masthoff of the University of Aberdeen says, “I would like people to have their own guardian angel that could support them emotionally throughout the day.”  But in order to get to that ideal, a series of (collectively agreed upon) experiments will need to guide designers and brands toward the appropriate level of intimacy, and a series of failures will determine the rules for maintaining trust, privacy, and emotional boundaries.

The biggest hurdle to finding the right balance might not be achieving more effective forms of emotional AI, but finding emotionally intelligent humans to build them.

How Companies Are Integrating Voice Recognition Into Medicine


Companies have been working to integrate voice recognition into healthcare since the technology’s inception. From physician dictations to patient engagement, voice recognition has an immense amount of potential for facilitating processes in medical practice. Here, we highlight some of the key uses of voice recognition in medicine and associated companies.

Senior Care

In senior care, voice recognition allows elderly patients who prefer to be stationary to improve their health within their homes. Lifepod is a caregiving service that provides day-to-day management assistance for seniors. It provides reminders for medications, schedules, activities, appointments, and even entertainment, facilitating the lives of not only the seniors themselves, but their caregivers as well. Another voice recognition technology, ElliQ offers an AI social robot that suggests activities for elders to partake in, promoting an active lifestyle. RemindMeCare is a similar companion software, offered to consumers through Amazon Alexa software.

Record Keeping

Voice recognition in physician notetaking is arguably the most commonly discussed use of voice recognition in healthcare, aiming to aid in electronic health record keeping. Kiroku is one such technology, capable of listening in on physician-patient conversations and automatically writing the notes. Another device, Notable, utilizes a wearable voice interface and artificial intelligence system to record clinical visits in a similar manner. Being that physician notes often consume a large portion of a doctors day, these devices are potential mechanisms of freeing up more time for practitioners to interact with patients rather than record notes.

Patient Evaluation

Software is being developed to determine patient condition based on vocal features as well. BeyondVerbal is one such company that has created a system that examines a patient’s voice in real time to anayze patient wellbeing, health condition, and provide emotional insight. Healthymize is utilizing similar speech monitoring through breath during voice calls, allowing audio recognition technology to work without the patient even being in the office. Corti uses a deep-learning artificial intelligence system that specializes in using voice recognition to aid a physician in making difficult medical decisions in real time.

With many patients calling on Alexa for weather, music, and news everyday within their homes, the use of voice recognition platforms is becoming a very common practice. Many companies like those discussed above are aiming to integrate this popular technology into the medical setting in a plethora of ways. With voice recognition being such a young concept and many artificial intelligence banks just starting to obtain information, the potential impact this technology will have on healthcare could be revolutionary.

37 Startups building voice applications for healthcare


As the next frontier in human-technology interfaces, voice-enabled and voice-first technologies are leading the way in many innovative applications across industries. Predictions that 50 percent of searches will be voice-based by 2020 and that 55 percent of US households will have a smart speaker by 2022 have entrepreneurs, developers, product managers, and marketers rushing to figure out how they can capture the upcoming surge of voice-based technology.

In healthcare, voice technology finds a market particularly rife with potential and impactful use cases. The high cost of labor for physicians and other skilled workers – who spent countless hours inputting data into their electronic health records – is one example of an opportunity for startups to disrupt the status quo. In fact, one landscape of B2B voice technology startups across all verticals found that 47.1 percent of companies that were focused on a single sector were focused on healthcare.

From “Voice Tech Landscape: 150+ Infrastructure, Horizontal and Vertical Startups Mapped and Analysed”, Savina van der Straten, Dec 13, 2017.

From “Voice Tech Landscape: 150+ Infrastructure, Horizontal and Vertical Startups Mapped and Analysed”, Savina van der Straten, Dec 13, 2017.

We sorted 37 startups building products at the intersection of voice and healthcare by how they are tackling this market, in hopes of giving those interested in learning more about this exciting frontier an opportunity to check out what voice-based innovations might hit their office, clinic, or home in the next several years.

Voice is uniquely positioned to be a valuable tool for seniors who wish to stay in their homes – especially for those who are unable to use other forms of technology that may require mobility, dexterity of the hands, and/or good vision (such as smartphones). These startups take advantage of voice-first technology for seniors.

  • Cuida Health helps seniors connect with family, get access to services, adopt healthy habits, and thrive independently.

  • ElliQ is a proactive AI-driven social robot designed to encourage an active and engaged lifestyle by suggesting activities and making it simple to connect with loved ones.

  • LifePod is a voice-first caregiving service designed to improve the quality of life for caregivers and their loved ones by monitoring and supporting their daily routines.

  • Memory Lane is a way for users to easily recollect their lives, improve their mood, and share stories with family and friends.
  • Reminder Rosie is a simple, hands-free, inexpensive solution to remember your medication, appointments, and every-day tasks.
  • RemindMeCare is a person-centered care, activities and companionship software, available as an app integrated with Alexa.

  • Senter combines the latest IoT and AI technologies with a heavy focus on thoughtful user experience to make the home healthier and safer for aging individuals.

Patient-Provider Communication

Many voice technologies are automating or simplifying communication between patients and providers. Intelligent bots can save clinical staff valuable time and complete tasks – like appointment scheduling and reminders in an outpatient setting, or care team coordination in an inpatient setting.

  • Aiva is a voice-powered care assistant that enables hands-free communication for happier patients and better workflow.

  • uses AI to provide around the clock assistance for scheduling, rescheduling, and cancellation for all appointment types for both new and existing patients.

  •’s virtual hospital assistant not only saves cost with intelligent interactions, but helps doctors and other staff to be more productive, while improving patient experience.

  • Syllable is a chatbot for healthcare that enables engaging, conversational experiences on your website or in your mobile app.

  • VoiceFriend is a simple yet powerful notification solution that enables you to easily keep seniors, staff and families informed of events and important information.

Physician Notes

Forty-two percent of physicians feel “burned out,” according to Medscape. One of the major causes is the amount of time clinicians inevitably spend behind a computer, entering information from their last patient interaction into their electronic health record (EHR). Several startups are using voice technology as a virtual scribe to enter physician notes into the EHR.

  • Kiroku’s sophisticated natural language system can pick up context in a conversation between you and the patient and automatically write your clinical notes for you.

  • MDOps dramatically reduces the documentation time with you dictating and filing clinical notes using your iPhone or iPad, allowing you to spend more time with more patients.

  • Notable uses wearable tech, voice interface, and artificial intelligence to enrich every patient-physician interaction.

  • Saykara is simplifying data capture with a new artificial intelligence-based virtual scribe solution that eliminates the hassle of working with EHRs.

  • Sopris Health is an intelligent clinical operations platform offering a pioneering A.I. medical scribe technology to tackle clinical inefficiencies.

  • Suki is a digital assistant for doctors that starts by helping lift the burden of medical documentation.

  • is an automated medical scribe that listens to your patient visits via a small microphone in the exam room and creates an accurate patient note in real time. 

Speech & Hearing Difficulty

Several startups use voice technology to help improve the lives of those with speech and/or hearing difficulties. Some developers use natural language processing to turn spoken words into text and vice versa. Additional innovations may track disease progression over time using this data, as well.

  • Ava empowers deaf & hard-of-hearing people to a 24/7 accessible life by showing them who says what.

  • VocaliD leverages voicebank and proprietary voice blending technology to create unique vocal personas for any device that turns text into speech.

  • Voiceitt is developing the world’s first speech recognition technology designed to understand non-standard speech.

Development Platforms

These companies make it easy for those who want to develop and publish voice applications, especially if they want to publish across multiple platforms (e.g. Amazon Alexa and Google Home) at once.

  • ConversationHealth creates powerful bots to support the clinical journey of all stakeholders.

  • Orbita is an enterprise-grade platform for creating and maintaining voice-powered healthcare applications, across both voice and chatbot interfaces.

Vocal Biomarkers

Vocal patterns such as pitch, tone, rhythm, volume, and rate can serve as powerful data points – “vocal biomarkers.” This information can aid care teams in their diagnosis of a variety of conditions — from cognitive disorders to heart attacks (and many more). 

  • BeyondVerbal has developed a technology that extracts various acoustic features from a speaker’s voice, in real time, giving insights on personal health condition, wellbeing, and emotional understanding.

  • Cogito improves care management with real-time emotional intelligence.

  • Corti is a digital assistant that leverages deep learning to help medical personnel make critical decisions in the heat of the moment.

  • Healthymize provides personalized speech monitoring based on analysis of patients’ voice and breathing during regular voice calls.

  • NeuroLex strives to be the world’s leading platform company to advance linguistics as a tool to characterize various health conditions.

  • Sonde is developing a voice-based technology with the potential to transform the way we monitor and diagnose mental and physical health.

  • WinterLight Labs has developed a novel AI technology that can quickly and accurately quantify speech and language patterns to help detect and monitor cognitive and mental diseases.

Patient Engagement

These startups take their voice applications to the patient’s home, and use a voice interface to keep patients engaged in their care in between visits with their providers. Many are designed for patients with chronic conditions, to help close gaps in care for the 99 percent of the time that a patient is not in their doctor’s office.

  • CardioCube voice-based AI software is an everyday companion to help manage your chronic heart disease. Your healthcare provider in the hospital or clinic gets your disease insights for better and faster decisions.

  • CareAngel is a patient-focused virtual nurse assistant that helps individuals maintain health and well-being to close gaps in care and improve outcomes.

  • HealthTap’s Doctor A.I. is a personal Artificial Intelligence-powered “physician” that helps route users to doctor-recommended insights and care immediately.

  • Sensely intelligently connects people with clinical advice and services, enhancing access without compromising empathy.

  • Kencor Health integrates with the latest AI technology to keep your patients engaged in their treatment plan while keeping your team connected to how they are doing.
  • Pillo is the digital health assistant for the home dedicated to the health of you and your loved ones.

We look forward to continuing to track the progress of these startups, and others who are sure to form in the coming months and years. Even better, we look forward to exploring their solutions live at the Voice.Health Summit on October 17 in Boston. Learn more about the summit, who will be there (in addition to many of the startups mentioned above), and how you can attend at

This piece was produced in collaboration with the Boston Children’s Hospital Innovation and Digital Health Accelerator, a multi-disciplinary team addressing the unmet needs of patients/families, clinicians, and health systems across the enterprise and around the globe though the power of digital health. Together with the Personal Connected Health Alliance and Modev, they are bringing a first-of-its-kind gathering of technologists, clinicians, innovators and industry together at the Voice.Health Summit in Boston on Oct. 17. The summit is an official co-located event of the Connected Health Conference and will showcase a number of the disruptive startups outlined below in an immersive patient journey experience.



This blog post is a roundup of voice emotion analytics companies. It is the first in a series that aim to provide a good overview of the voice technology landscape as it stands. Through a combination of online searches, industry reports and face-to-face conversations, I’ve assembled a long list of companies in the voice space, and divided these into categories based on their apparent primary function.

The first of these categories is voice emotion analytics. These are companies that can process an audio file containing human speech, extract the paralinguistic features and interpret these as human emotions, then provide an analysis report or other service based on this information.

Beyond Verbal

Beyond Verbal was founded in 2012 in Tel Aviv, Israel by Yuval Mor. Their patented voice emotion analytics technology extracts various acoustic features from a speaker’s voice, in real time, giving insights on personal health condition, wellbeing and emotional understanding. The technology does not analyze the linguistic context or content of conversations, nor does it record a speaker’s statements. It detects changes in vocal range that indicate things like anger, or anxiety, or happiness, or satisfaction, and cover nuances in mood, attitude, and decision-making characteristics.

Beyond Verbal’s voice emotion analysis is used in various use cases by clients in a range industries. These include HMOs, life insurance and pharma companies, as well as call centres, robotics and wearable manufacturers, and research institutions. An example use case would be to help customer services representatives improve their own performance, by monitoring the call audio in real-time. An alert can be sent to the agent if they start to lose his/her temper with the customer on the phone, making them aware of their change in mood, and affording them the opportunity to correct their tone.

The technology is offered as a API-style cloud-based licensed service that can be integrated into bigger projects. It measures:

  • Valence – a variable which ranges from negativity to positivity. When listening to a person talk, it is possible to understand how “positive” or “negative” the person feels about the subject, object or event under discussion.
  • Arousal – a variable that ranges from tranquility/boredom to alertness/excitement. It corresponds to similar concepts such as level of activation and stimulation.
  • Temper – an emotional measure that covers a speaker’s entire mood range. Low temper describes depressive and gloomy moods. Medium temper describes friendly, warm and embracive moods. High temper values describe confrontational, domineering and aggressive moods.
  • Mood groups – an indicator of speaker’s emotional state during the analyzed voice segment. The API produces a total of 11 mood groups which range from anger, loneliness and self-control to happiness and excitement.
  • Emotion combinations – A combination of various basic emotions, as expressed by the users voice during an analyzed voice section.

“We envision a world in which personal devices understand our emotions and wellbeing, enabling us to become more in tune with ourselves and the messages we communicate to our peers. Understanding emotions can assist us in finding new friends, unlocking new experiences and ultimately, helping us understand better what makes us truly happy.”
Yuval Mor, CEO

to read the full article press here

Daniel Kraft provides glimpse of health tech’s future


The digital age has thrown the healthcare world into a state of feverish change. Though certain elements of the brick-and-mortar hospital remain the same after years, other aspects of medicine are in rapid development. Through new technologies, multiple parts of healthcare have the chance to interact.

“We do have the opportunity now to connect a lot of this new information,” Dr. Daniel Kraft, Singularity University’s faculty chair for medicine and neuroscience and Exponential Medicine’s founder and chair, said in a keynote address at MedCity INVEST on May 1. “As we have these new opportunities … they’re all converging  — essentially super-converging. As entrepreneurs and investors, you want to be looking at this super-convergence because that’s where the opportunity is to innovate, reinvent, reimagine.”

Encouraging attendees to think exponentially instead of linearly, Kraft took a broad look at where healthcare is headed, particularly when it comes to technology. Though wide-ranging and fast-moving, his presentation narrowed in on a few areas.

Health and prevention
Individuals’ behaviors impact the majority of chronic costs in healthcare, Kraft noted. Wearables can play a role in assisting with this issue.

But it’s moved beyond only wearables — there are now technologies like “inside’ables” (chips underneath one’s skin that can track vital signs), “ring’ables” (which track aspects like sleep) and “breath’ables” (which monitor one’s oral health).

Mental health
Within the behavioral health space, companies like Woebot are leveraging technology to provide therapy chatbots to consumers, while entities like Beyond Verbal are using voice to provide insight on emotional health. Other companies are enabling consumers to “game-ify” their meditation experience.

“You can get your own genome done for about $1,000 today,” Kraft said. “It comes with an app.”

He also mentioned Helix, an Illumina spinout that set out to be a hub for consumers to obtain genetic tests, and the work it’s doing in the realm.

“Watch the whole ‘omics space,” Kraft suggested.

Despite the demise of Theranos, there are plenty of opportunities in the field to make a mark. The digital stethoscope is emerging as a new type of diagnostic tool. Even the Apple Watch is becoming a diagnostic, Kraft said. Platforms can make it easier to do a remote ear exam, and apps can listen to a cough and diagnose pneumonia.

Even a broad look at these few areas unveils the value in connecting the dots between technology and the healthcare environment. And Kraft appears to be taking his own advice. His Exponential Medicine program is moving into the prescription health app service space, he said.

Looking down the road, the goal is to collaborate and move from “sick care” to a more proactive approach.

“I think the future is going to be … data [and] convergence amongst many technologies,” Kraft said. “We can all become futurists. It’s our opportunity to go out there and not predict the future but hopefully create it together.”

Photo: Jack Soltysik

Your voice will guide your chores, healthcare and driving

In 5 years, voice tech will help doctors diagnose and operate, carmakers provide customized web content, HR professionals judge job applicants and more.


Back in 1995, Shlomo Peller founded Rubidium in the visionary belief that voice user interface (VUI) could be embedded in anything from a TV remote to a microwave oven, if only the technology were sufficiently small, powerful, inexpensive and reliable.

“This was way before IoT [the Internet of Things], when voice recognition was done by computers the size of a room,” Peller tells ISRAEL21c.

“Our first product was a board that cost $1,000. Four years later we deployed our technology in a single-chip solution at the cost of $1. That’s how fast technology moves.”

But consumers’ trust moved more slowly. Although Rubidium’s VUI technology was gradually deployed in tens of millions of products, people didn’t consider voice-recognition technology truly reliable until Apple’s virtual personal assistant, Siri, came on the scene in 2011.

“Siri made the market soar. It was the first technology with a strong market presence that people felt they could count on,” says Peller, whose Ra’anana-based company’s voice-trigger technology now is built into Jabra wireless sports earbuds and 66 Audio PRO Voice’s smart wireless headphones

“People see that VUI is now something you can put anywhere in your house,” says Peller. “You just talk to it and it talks back and it makes sense. All the giants are suddenly playing in this playground and voice recognition is everywhere. Voice is becoming the most desirable user interface.”

Still, the technology is not yet as fast, fluent and reliable as it could be. VUI depends on good Internet connectivity and can be battery-draining.

We asked the heads of Israeli companies Rubidium, VoiceSense and BeyondVerbal to predict what might be possible five years down the road, once these issues are fixed.

Here’s what they had to say.

Cars and factories

Rubidium’s Peller says that in five years’ time, voice user interface will be part of everything we do, from turning on lights, to doing laundry, to driving.

“I met with a big automaker to discuss voice interface in cars, and their working assumption is that within a couple of years all cars will be continuously connected to the Internet, and that connection will include voice interface,” says Peller.

“All the giants are suddenly playing in this playground and voice recognition is everywhere. Voice is becoming the most desirable user interface.”

“The use cases we find interesting are where the user interface isn’t standard, like if you try to talk to the Internet while doing a fitness activity, when you’re breathing heavily and maybe wind is blowing into the mic. Or if you try to use VUI on a factory production floor and it’s very noisy.”

As voice-user interface moves to the cloud, privacy concerns will have to be dealt with, says Peller.

“We see that there has to be a seamless integration of local (embedded) technology and technology in the cloud.

“The first part of what you say, your greeting or ‘wakeup phrase,’ is recognized locally and the second part (like ‘What’s the weather tomorrow?’) is sent to the cloud. It already works like that on Alexa but it’s not efficient. Eventually we’ll see it on smartwatches and sports devices.”

Diagnosing illness

Tel Aviv-based Beyond Verbal analyzes emotions from vocal intonations. Its Moodies app is used in 174 countries to help gauge what speakers’ voices (in any language) reveal about their emotional status. Moodies is used by employers for job interviewees, retailers for customers, and many other scenarios.

The company’s direction is shifting to health, as the voice-analysis platform has been found to hold clues to well-being and medical conditions, says Yoram Levanon, Beyond Verbal’s chief scientist.

“There are distortions in the voice if somebody is ill, and if we can correlate the source of the distortions to the illness we can get a lot of information about the illness,” he tells ISRAEL21c.

“We worked with the Mayo Clinic for two years confirming that our technology can detect the presence or absence of a cardio disorder in a 90-second voice clip.

“We are also working with other hospitals in the world on finding verbal links to ADHD, Parkinson’s, dyslexia and mental diseases. We’re developing products and licensing the platform, and also looking to do joint ventures with AI companies to combine their products with ours.”

Levanon says that in five years, healthcare expenses will rise dramatically and many countries will experience a severe shortage of physicians. He envisions Beyond Verbal’s technology as a low-cost decision-support system for doctors.

“The population is aging and living longer so the period of time we have to monitor, from age 60 to 110, takes a lot of money and health professionals. Recording a voice costs nearly nothing and we can find a vocal biomarker for a problem before it gets serious. For example, if my voice reveals that I am depressed there is a high chance I will get Alzheimer’s disease,” says Levanon.

Beyond Verbal could synch with the AI elements in phones, smart home devices or other IoT devices to understand the user’s health situation and deliver alerts.

Your car will catch on to your mood

Banks use voice-analysis technology from Herzliya-based VoiceSense to determine potential customers’ likelihood of defaulting on a loan. Pilot projects with banks and insurance companies in the United States, Australia and Europe are helping to improve sales, loyalty and risk assessment regardless of the language spoken.

“We were founded more than a decade ago with speech analytics for call centers to monitor customer dissatisfaction in real time,” says CEO Yoav Degani.

“We noticed some of the speech patterns reflected current state of mind but others tended to reflect ongoing personality aspects and our research linked speech patterns to particular behavior tendencies. Now we can offer a full personality profile in real time for many different use cases such as medical and financial.”

Degani says the future of voice-recognition tech is about integrating data from multiple sensors for enhanced predictive analytics of intonation and content.

“Also of interest is the level of analysis that could be achieved by integrating current state of mind with overall personal tendencies, since both contribute to a person’s behavior. You could be dissatisfied at the moment and won’t purchase something but perhaps you tend to buy online in general, and you tend to buy these types of products,” says Degani.

In connected cars, automakers will use voice analysis to adjust the web content sent to each passenger in the vehicle. “If the person is feeling agitated, they could send soothing music,” says Degani.

Personal robots, he predicts, will advance from understanding the content of the user’s speech to understanding the user’s state of mind. “Once they can do that, they can respond more intelligently and even pick up on depression and illness.”

He predicts that in five years’ time people will routinely provide voice samples to healthcare providers for analytics; and human resources professionals will be able to judge a job applicant’s suitability for a specific position on the basis of recorded voice analysis using a job-matching score.

Emotion AI: Why your refrigerator could soon understand your moods


Artificial intelligence is already making our devices more personal — from simplifying daily tasks to increasing productivity. Emotion AI (also called affective computing) will take this to new heights by helping our devices understand our moods. That means we can expect smart refrigerators that interpret how we feel (based on what we say, how we slam the door) and then suggest foods to match those feelings. Our cars could even know when we’re angry, based on our driving habits.

Humans use non-verbal cues, such as facial expressions, gestures, and tone of voice, to communicate a range of feelings. Emotion AI goes beyond natural language processing by using computer vision and voice analysis to detect those moods and emotions. Voice of the customer (VoC) programs will leverage emotion AI technology to perform granular and individual sentiment analysis at scale. The result: Our devices will be in tune with us.

Conversational services

Digital giants — including Google, Amazon, Apple, Facebook, Microsoft, Baidu, and Tencent — have been investing in AI techniques that enhance their platforms and ecosystems. We are still at “Level 1” when it comes to conversational services such as Apple’s Siri, Microsoft’s Cortana, and Google Assistant. However, the market is set to reach new levels in the next one to two years.

Nearly 40 percent of smartphone users employ conversational systems on a daily basis, according to a 2017 Gartner survey of online adults in the United States. These services will not only become more intelligent and sophisticated in terms of processing verbal commands and questions, they will also grow to understand emotional states and contexts.

Today, there are a handful of available smartphone apps and connected home devices that can capture a user’s emotions. Additional prototypes and commercial products exist — for example, Emoshape’s connected home hub, Beyond Verbal‘s voice recognition app, and the connected home VPA Hubble. Large technology vendors such as IBM, Google, and Microsoft are investing in this emerging area, as are ambitious startups.

At this stage, one of the most significant shortcomings of such systems is a lack of contextual information. Adding emotional context by analyzing data points from facial expressions, voice intonation, and behavioral patterns will significantly enhance the user experience.

Wearables and connected cars

In the second wave of development for emotion AI, we will see value brought to many more areas, including educational software, video games, diagnostic software, athletic and health performance, and autonomous cars. Developments are underway in all of these fields, but 2018 will see many products realized and an increased number of new projects.

Beyond smartphones and connected-home devices, wearables and the connected car will collect, analyze, and process users’ emotional data via computer vision, audio, or sensors. The captured behavioral data will allow these devices to adapt or respond to a user’s needs.

Technology vendors, including Affectiva, Eyeris, and Audeering, are working with the automotive OEMs to develop new experiences inside the car that monitor users’ behavior in order to offer assistance, monitor safe-driving behavior, and enhance their ride.

There is also an opportunity for more specialized devices, such as medical wristbands that can anticipate a seizure a few minutes before the actual event, facilitating early response. Special apps developed for diagnostics and therapy may be able to recognize conditions such as depression or help children with autism.

Another important area is the development of anthropomorphic qualities in AI systems — such as personal assistant robots (PARs) that can adapt to different emotional contexts or individuals. A PAR will develop a “personality” as it has more interactions with a specific person, allowing it to better meet the user’s needs. Vendors such as IBM, as well as startups like Emoshape, are developing techniques to lend such anthropomorphic qualities to robotic systems.

VoC will help brands understand their consumers

Beyond enhancing robotics and personal devices, emotion AI can be applied in customer experience initiatives, such as VoC programs. A fleet of vendors already offer sentiment analysis by mining billions of data points on social media platforms and user forums. Some of these programs are limited to distinguishing between positive and negative sentiments while others are more advanced, capable of attributing nuanced emotional states — but so far, only in the aggregate.

We are still at an early stages when it comes to enhancing VoC programs with emotion AI. Technology providers will have to take a consultative approach with their clients — most of whom will be new to the concept of emotion AI. While there are only a few isolated use cases for emotion AI at the moment, we can expect it to eventually offer tools that transform virtually every aspect of our daily lives.

Annette Zimmermann is the research vice president at Gartner, a research and advisory company.

$10 million XPRIZE Aims for Robot Avatars That Let You See, Hear, and Feel by 2021


Ever wished you could be in two places at the same time? The XPRIZE Foundation wants to make that a reality with a $10 million competition to build robot avatars that can be controlled from at least 100 kilometers away.

The competition was announced by XPRIZE founder Peter Diamandis at the SXSW conference in Austin last week, with an ambitious timeline of awarding the grand prize by October 2021. Teams have until October 31st to sign up, and they need to submit detailed plans to a panel of judges by the end of next January.

The prize, sponsored by Japanese airline ANA, has given contestants little guidance on how they expect them to solve the challenge other than saying their solutions need to let users see, hear, feel, and interact with the robot’s environment as well as the people in it.

XPRIZE has also not revealed details of what kind of tasks the robots will be expected to complete, though they’ve said tasks will range from “simple” to “complex,” and it should be possible for an untrained operator to use them.

That’s a hugely ambitious goal that’s likely to require teams to combine multiple emerging technologies, from humanoid robotics to virtual reality high-bandwidth communications and high-resolution haptics.

If any of the teams succeed, the technology could have myriad applications, from letting emergency responders enter areas too hazardous for humans to helping people care for relatives who live far away or even just allowing tourists to visit other parts of the world without the jet lag.

“Our ability to physically experience another geographic location, or to provide on-the-ground assistance where needed, is limited by cost and the simple availability of time,” Diamandis said in a statement.

“The ANA Avatar XPRIZE can enable creation of an audacious alternative that could bypass these limitations, allowing us to more rapidly and efficiently distribute skill and hands-on expertise to distant geographic locations where they are needed, bridging the gap between distance, time, and cultures,” he added.

Interestingly, the technology may help bypass an enduring hand break on the widespread use of robotics: autonomy. By having a human in the loop, you don’t need nearly as much artificial intelligence analyzing sensory input and making decisions.

Robotics software is doing a lot more than just high-level planning and strategizing, though. While a human moves their limbs instinctively without consciously thinking about which muscles to activate, controlling and coordinating a robot’s components requires sophisticated algorithms.

The DARPA Robotics Challenge demonstrated just how hard it was to get human-shaped robots to do tasks humans would find simple, such as opening doors, climbing steps, and even just walking. These robots were supposedly semi-autonomous, but on many tasks they were essentially tele-operated, and the results suggested autonomy isn’t the only problem.

There’s also the issue of powering these devices. You may have noticed that in a lot of the slick web videos of humanoid robots doing cool things, the machine is attached to the roof by a large cable. That’s because they suck up huge amounts of power.

Possibly the most advanced humanoid robot—Boston Dynamics’ Atlas—has a battery, but it can only run for about an hour. That might be fine for some applications, but you don’t want it running out of juice halfway through rescuing someone from a mine shaft.

When it comes to the link between the robot and its human user, some of the technology is probably not that much of a stretch. Virtual reality headsets can create immersive audio-visual environments, and a number of companies are working on advanced haptic suits that will let people “feel” virtual environments.

Motion tracking technology may be more complicated. While even consumer-grade devices can track peoples’ movements with high accuracy, you will probably need to don something more like an exoskeleton that can both pick up motion and provide mechanical resistance, so that when the robot bumps into an immovable object, the user stops dead too.

How hard all of this will be is also dependent on how the competition ultimately defines subjective terms like “feel” and “interact.” Will the user need to be able to feel a gentle breeze on the robot’s cheek or be able to paint a watercolor? Or will simply having the ability to distinguish a hard object from a soft one or shake someone’s hand be enough?

Whatever the fidelity they decide on, the approach will require huge amounts of sensory and control data to be transmitted over large distances, most likely wirelessly, in a way that’s fast and reliable enough that there’s no lag or interruptions. Fortunately 5G is launching this year, with a speed of 10 gigabits per second and very low latency, so this problem should be solved by 2021.

And it’s worth remembering there have already been some tentative attempts at building robotic avatars. Telepresence robots have solved the seeing, hearing, and some of the interacting problems, and MIT has already used virtual reality to control robots to carry out complex manipulation tasks.

South Korean company Hankook Mirae Technology has also unveiled a 13-foot-tall robotic suit straight out of a sci-fi movie that appears to have made some headway with the motion tracking problem, albeit with a human inside the robot. Toyota’s T-HR3 does the same, but with the human controlling the robot from a “Master Maneuvering System” that marries motion tracking with VR.

Combining all of these capabilities into a single machine will certainly prove challenging. But if one of the teams pulls it off, you may be able to tick off trips to the Seven Wonders of the World without ever leaving your house.

Image Credit: ANA Avatar XPRIZE