CFD 373 – Emotions drive everything we do; Go beyond verbal with Yuval Mor

CFD 373 – EMOTIONS DRIVE EVERYTHING WE DO; GO BEYOND VERBAL WITH YUVAL MOR

Emotions Analytics change the way we interact with our machines and ourselves – forever. By decoding human vocal intonations into their underlying emotions in real-time, Emotions Analytics enables voice-powered devices, apps and solutions to interact with us on an emotional level, just as humans do.

Beyond Verbal focuses on raw vocal modulations – probably the most expressive output our body produces. This makes our offering unique by being non-intrusive, continuous in nature and easy to implement. Enabling wearables and digital health applications and empowering international brands to better understand consumer brand interaction, Beyond Verbal is pioneering a brand new breed of emotionally powered devices set to change the way we do business, make decisions and manage our lives, forever.

http://cashflowdiary.com/cfd-373-emotions-drive-everything-we-do-go-beyond-verbal-with-yuval-mor/

致力于情绪分析技术,这家公司要让 Siri 和 Alexa 理解你的喜怒哀乐

  Beyond Verbal 在 2012 年成立于 Tel Aviv,团队拥有数十年的情感分析研究经验,并与芝加哥大学,Mayo Clinic,Scripps 和 Hadassah 医学中心等知名组织进行了合作。

  2014 年,该公司推出了 Beyond Wellness API,能够让用户的智能手机或是带有麦克风的其他可穿戴设备发挥情绪传感器作用,公司研发的分析技术并不是进行语音实际文本内容或上下文语境的分析,而是通过系统创建的算法识别音域及语调变化,分析出像愤怒、焦虑、愉快或满足等情绪,这样就可以通过用户语音样本来了解人体情绪及人体身心健康状况。

  通过一段时间的被动跟踪、量化和报告情绪状态,Beyond Verbal 能够提示用户一段时间内的全面情绪健康状态,进而帮助用户改进情绪健康。目前,Beyond Verbal收集了包括 40 种语言内的 250 多万条“情绪标签声音”数据。

  不过该技术尚处于起步阶段,潜在用例包括呼叫中心通过分析通话情绪来改善与用户的关系,或在健康领域,承担评估某人的心理健康的角色。同时,确定它是否可以有效地用于检测身体状况(如心脏问题)的研究也正在进行中。

  如何让 Siri 进行情感分析?

  这种情绪检测的技术可以被应用到各种场合中:游戏的客户服务、约会服务(帮助人们知道对方是否真的对自己感兴趣)。同时,Beyond Verbal 目前正致力于通过开发人员的新 API 向虚拟个人助理(VPA)领域介引进其情感分析技术。 虽然苹果 Siri 和亚马逊 Alexa 在理解话语上正在不断改进,它可以对“Alexa,播一首披头士的歌”做出反应,但它们却不擅长识别人们的情绪。而这就是 Beyond Verbal 新 API 的最终目标:为数字助理带来“情绪智能”。

  “今天的数字世界正在迅速改变我们与技术等其他方面交流的方式。”Beyond Verbal 首席执行官 Yuval Mor 说道,“虚拟私人助理已经开始采用个性化体验。我们非常希望能够将 AI 和 Beyond Verbal 情绪分析的突破性技术融合在一起,为个性化技术和远程监控提供独特的视角。”

  那这如何实现呢?为什么亚马逊 Echo 设备需要了解人们的心情? 鉴于 Alexa 和 Co. 现在可以为第三方支持的语音服务提供支持,Beyond Verbal 提到了一些可能的用例:如果你的声音比较消沉,Alexa 可能会播放积极的音乐,或者可以告诉你,你的朋友听起来不太高兴,或者给你推荐令人愉快的电影。

  需要注意的是,Beyond Verbal 大部分的想法都是放眼未来,还有更多潜在的重要用途尚未被发现,比如医疗健康。在未来,Siri 可以在几周内观察到你的情绪不佳。或者如果目前的研究能重视其承诺,它甚至可以帮助人们检测严重的身体疾病。“在不久的将来,我们的目标是为我们的功能集添加声乐生物标志物分析,使虚拟私人助理能够通过你的声音来分析特定的健康状况。”Mor 补充说。

  实现过程是复杂的

  该技术在基本层面上亟待改进。事实上,在 Beyond Verbal 激活虚拟个人助理(VPA)与用户之间的对话时,首先需要 13 秒语音来进行第一次分析,之后则需要每 4 秒进行一次情感分析 ,此过程适用于每一次谈话。但很难想象人们会与 Amazon Echo 或 Apple HomePod 聊足够长的时间来启用情绪检测。这就对为什么每天被动地收集一个人的声音对于该项技术的成功而言至关重要。

  “目前,命令风格的对话会因为声音太少而难以进行情感分析。”VentureBeat 公司的一位发言人表示,他们正在开发一个额外的功能集,这将减少反应时间,不过还是不可用于商业用途。改善情绪检测的另一种方法是通过更广泛的物联网方式,使用多种设备来(包括可穿戴式,手机,智能车等)分析语音,但这仍然需要一段时间。

  据悉,Beyond Verbal 正在寻求新一轮融资,目前已经从基金中投入了大约 1080 万美元(其中包括 2016 年 9 月的 300 万美元融资)。如果该公司要实现将真正的情绪智能带给 AI 的愿景,那么将需要一笔巨大的投入。

第九届中国鞋服行业供应链与物流技术研讨会

http://www.56products.com/News/2017-6-26/IFFB6ACFC3AH622314.html

Beyond Verbal Adds Emotions to Virtual Assistants

Beyond Verbal has launched an API for virtual private assistants to help identify user emotions in real time.

Beyond Verbal, a provider of voice-driven emotions analytics, is launching a cloud-based API engine that will enable virtual private assistants (VPAs) to reveal customized recommendations based on individual moods and emotional states.

Beyond Verbal enables these assistants to understand the emotional message, context, and intent carried by users’ vocal intonations, which represent 35 percent to 40 percent of the emotions people convey in their communication. By integrating with Beyond Verbal’s API technology, virtual private assistants will now be able to understand and react to their users’ emotions all by the tone in their voices. With Beyond Verbal, AI assistants can also change their behavior, personality, and even their tone of voice  to fit themselves to the context of the conversation and person with whom they are communicating.

Beyond Verbal’s Emotional Analytics technology takes raw voice input and analyzes it for mood and attitude. The technology only requires 10 seconds of continuous voice input to render an emotional analysis. The operating system then measures the speaker’s tone of voice and the results are distributed into groups and analyzed in real-time.

“Today’s digital world is rapidly transforming the way we interact with our technology and each other. Virtual private assistants have begun to take on a personalized experience,” said Yuval Mor, CEO of Beyond Verbal, in a statement. “We are very excited for this next step in fusing together the breakthrough technology of AI and Beyond Verbal’s Emotions Analytics, providing unique insight into personalized tech and remote monitoring. In the not so far future our aim is to add vocal biomarker analysis to our feature set, enabling virtual private assistants to analyze your voice for specific health conditions.”


http://www.speechtechmag.com/Articles/ReadArticle.aspx?ArticleID=118992

Beyond Verbal launches API to enable voice-based emotion detection by virtual private assistants

Your Voice Reveals You - Vocal Biomarkers

As virtual private assistants like Amazon’s Alexa become smarter, more popular and more useful, why not give it them tools to recognize emotions? That seems to be the thinking behind Israel-based Beyond Verbal, which makes voice recognition software to analyze human emotion and health indicators, releasing a cloud-based API engine to integrate with virtual private assistants (VPAs).

The intention for Beyond Verbal’s latest project is to make one’s connected home more sensitive, so to speak, rather than a sounding board that simply takes request after request. Operating on the knowledge that vocal intonations represent 35 to 40 percent of emotions conveyed in human communication, Beyond Verbal set out to eliminate the emotional blind spots of VPAs. The company’s Emotional Analytics technology takes 10 seconds of raw voice input and analyzes it for mood and attitude, then factors in that information with the user’s request. For example, if someone asks their VPA to play music, the Beyond Verbal-enabled bot will take note of the person’s emotional state  – say, stressed out – and respond with a list of calming music choices.

“Today’s digital world is rapidly transforming the way we interact with our technology and each other. Virtual private assistants have begun to take on a personalized experience,” Beyond Verbal CEO Yuval Mor said in a statement. “We are very excited for this next step in fusing together the breakthrough technology of AI and Beyond Verbal’s Emotions Analytics, providing unique insight into personalized tech and remote monitoring.”

Beyond Verbal, which was founded in 2012, has made a name for itself with its straightforward consumer technology built on years of hard science research. The company is actually the product of several research projects spanning 21 years, and the company has collected more than 2.5 million emotion-tagged voices in more than 40 languages to analyze human emotions. It launched its Beyond Wellness API in 2014, which turns any smartphone or mic-equipped wearable device into an emotional wellbeing sensor using technology that doesn’t consider the actual content or context of spoken word, but instead studies intonation in the voice. The company has two free, consumer-facing apps, Moodie and Empath, and one for clinicians called Beyond Clinic.

Just getting VPAs accustomed to their owner’s emotions is the first step, Mor said. Eventually, they want to layer in insights from their ongoing research on vocal biomarkers of health conditions, which has recently ramped up. The company has been working with the Mayo Clinic, Scripps, Haddassah Medical Center in Jerusalem and Tel Aviv University and others to expand research into health-indicating vocal intonations. In September, Beyond Verbal launched a research platform to identify physiological markers through the voice that may indicate various health-related issues called the mHealth Research Platform, the company will enable collaboration with other institutions, medical centers and commercial organizations.

Beyond Verbal is one of several health tech companies making moves to leverage VPAs as of late. In February, Lenovo Health and Orbita launched a voice-enabled home health assistant. As of last March, people who use Amazon’s VPA can launch the WebMD skill on any Alexa-enabled device (such as the Echo, Echo Dot and Amazon Fire TV) and ask a question about a range of health-related topics including conditions, medication, tests and treatments. Alexa will respond with WebMD-sourced answers in easy-to-understand language. 

In April, Merck and Amazon partnered up to launch a developer competition to incent startups and developers to create apps that harness Amazon’s Alexa technologies for people with type 2 diabetes. Boston Children’s Hospital officially launched its partnership with Amazon Echo that same month, giving the voice-powered home appliance a new “skill” that will allow it to give simple health advice to parents about their children’s’ fever and medication dosing. Called KidsMD, the Alexa app is just the first step in a plan to bring Boston Children’s medical knowledge to the consumer space, according to Chief Innovation Officer John Brownstein.

But all those are built on spoken requests, and Beyond Verbal wants to go deeper by actually recognizing health conditions rather than waiting for their owner to ask about it by name.

“In the not so far future, our aim is to add vocal biomarker analysis to our feature set, enabling Virtual Private Assistants to analyze your voice for specific health conditions,” Mor said in a statement.

http://www.mobihealthnews.com/content/beyond-verbal-launches-api-enable-voice-based-emotion-detection-virtual-private-assistants

Beyond Verbal Makes Sure Your Virtual Private Assistant Knows How You Feel

TEL AVIV, Israel–()–Beyond Verbal, (www.BeyondVerbal.com) the leader in voice-driven Emotions Analytics, is now launching a brand new cloud-based API engine dedicated to raising the Emotional Intelligence (EQ) and emotional understanding of Artificial Intelligence (AI) assistants. With this new launch, different virtual private assistants (VPAs) will be able to reveal customized recommendations based on individual moods, all from the privacy of your own home.

“Today’s digital world is rapidly transforming the way we interact with our technology and each other. Virtual private assistants, have begun to take on a personalized experience”

Virtual private assistant interfaces are the next stages in AI. However, until now, these platforms were ‘request based only’ and they did not take into account your emotional state. With this in mind, Beyond Verbal set out to eliminate virtual private assistant’s blind spots and enable Humanoids and Bots to understand the emotional message, context, and intent carried by our vocal intonations. Vocal Intonations represent 35-40% of the emotions we convey in our communication, making it a fundamental role in making Artificial Intelligent Assistants more Emotional

Beyond Verbal’s Emotional Analytics technology takes raw voice input and analyzes it for mood and attitude. The technology only requires 10 seconds of continuous voice input in order to render an emotional analysis. The operating system then measures the speaker’s tone of voice and the results are distributed into groups and analyzed in real-time.

“Today’s digital world is rapidly transforming the way we interact with our technology and each other. Virtual private assistants, have begun to take on a personalized experience,” says Yuval Mor, CEO of Beyond Verbal. “We are very excited for this next step in fusing together the breakthrough technology of AI and Beyond Verbal’s Emotions Analytics, providing unique insight into personalized tech and remote monitoring. In the not so far future our aim is to add vocal biomarker analysis to our feature set enabling Virtual Private Assistants to analyze your voice for specific health conditions”

About Beyond Verbal:

Since its launch in 2012, Beyond Verbal has been using voice-driven emotions AI to dramatically change the way we can detect emotions and reveal health conditions. The only input needed is the human voice, making this technology non-intrusive, passive and cost effective. Beyond Verbal’s technology has been developed based on ongoing research into the science of emotions that started in 1995. By combining the company’s patented technology with its proprietary machine learning-based algorithms and AI, Beyond Verbal is focusing on enabling devices to understand our emotions and health.

[ Original ]

Table of Mood Phrases

Getting a Table of Mood Phrases in your language

The Moods table provides mapping between mood Id (returned in Moods section of analysis result) and a text of the Mood phrases.

Example of mood section with Ids

“Mood”: {
“Group21”: { “Primary”: { “Id”: 1, // will be mapped to “Creative, Passionate”
},
“Secondary”: { “Id”: 7, // will be mapped to “Loneliness, Unfulfillment”
}
},
}

There is no need to fetch this table each time a particular phrase required. In order to reduce network traffic and CPU requirements, the application can pre-load this table into its memory and then use each time when the phrase text is required.

Please be informed that not all languages are supported yet. Please contact Beyond Verbal to get information on how to support the language of your interest.

Moods Request

GET URL: https://apiv3.beyondverbal.com/v3/moods/{groupeName}/{language}

Moods Request Parameters

Name Location (BodyUrlHeader) Optional Explanation
Group name Url No This field specifies the Mood type for which table is requested.
Supported values:
Group7
Group11
Group21
Composite
Language Url Yes Requested language according to ISO-639 and ISO-3166 standards. Default en-us
Alternatively you can set required language in standard HTTP Accept-Language header
Auth token Authorization Header No

Example

GET https://apiv3.beyondverbal.com/v3/moods/Group11/

Authorization: Bearer 21G2BA4iZJavSJQbsyuppWmfSMLgLn-**gDTCfguhzGa_k8

OK (200) Response:

[
{“Id”:1,”Phrase”:”Creative, Passionate”},
{“Id”:2,”Phrase”:”Criticism, Cynicism”},
{“Id”:3,”Phrase”:”Defensivness, Anxiety”},
{“Id”:4,”Phrase”:”Friendly, Warm”},
{“Id”:5,”Phrase”:”Hostility, Anger”},
{“Id”:6,”Phrase”:”Leadership, Charisma”},
{“Id”:7,”Phrase”:”Loneliness, Unfulfillment”},
{“Id”:8,”Phrase”:”Love, Happiness”}


]

API Metadata Guide

Using Metadata with Beyond Verbal REST API

Metadata field of the START request allows to attach arbitrary information to each analysis session.

Most common usage of the metadata field is to uniquely identify particular user, device or group of users for later aggregated analysis.
Examples of unique identifiers are: email address, mobile phone number, physical (manufacturer) device ID, facebookId, twitter ID.

Set the clientId field value to your unique identifier. Example

POST https://apiv3.beyondverbal.com/api/v3/recording/start

Authorization: Bearer 21G2BA4iZJavSJQbsyuppWmfSMLgLn-**gDTCfguhzGa_k8

{
“dataFormat”: {“type”: “WAV”},
“metadata”:{“clientId” : “+991199483679”}
}

Optionally you can also set this id as additional field of metadata object in order to specify it’s origin

Field
email Client email
phone Client phone (mobile)
deviceId Physical device ID
facebookId Facebook Id
trwitterId Twitter Id

 

Example where clientId is a phone number:

 

POST https://apiv3.beyondverbal.com/api/v3/recording/start

Authorization: Bearer 21G2BA4iZJavSJQbsyuppWmfSMLgLn-**gDTCfguhzGa_k8

{
“dataormat”: {“type”: “WAV”},
“metadata”:{“clientId”: “+991199483679”, “phone” : “+991199483679”}
}

 

Example where the clientId is email:

POST https://apiv3.beyondverbal.com/api/v3/recording/start

Authorization: Bearer 21G2BA4iZJavSJQbsyuppWmfSMLgLn-**gDTCfguhzGa_k8

{
“dataormat”: {“type”: “WAV”},
“metadata”:{
“clientId”: “someone@someplace.com”,
“email” : “someone@someplace.com”
}
}

API Analysis Result Interpretation

Analysis Result Interpretation Guide

The UPSTREAM and ANALYSIS requests returns a JSON object which contains analysis result.
Following table summarizes fields, values and their descriptions of the returned JSON object.

JSON Object Field Description Version Support Notes
{
“status”:”success”, The status of request. Can be “success” or “error”.
“result”:{ The object of analysis results.
  “duration”:”21513.25″, Duration of voice data processed in milliseconds
  “sessionStatus”:”Done”, Session status can be:
“Started” – no analysis data yet produced,
“Processing” – intermediate results , more analysis can be expected,
“Done” – analysis session has ended, the result has an analysis results for whole session.
  “analysisSegments”:[ The array containing analysis segments
  { First analysis segment object. Following fields are  properties of the segment
     “offset”:0, Offset of the segment in milliseconds from the beginning of the session.
     “duration”:10000, Segment duration in milliseconds
     “end”:10000, The end of the segment in milliseconds V4 and above
     “analysis”:{ Analysis object. Contains analysis values for the segment. The content of the object is provided as example. The real fields can vary depending on license type
         “Temper”:{ Temper Object
           “Value”:”21.00″, Value of Temper
           “Group”:”low”, Group of Temper, unknown – means that value cannot be calculated for the slice unknown value: V4 and above
           “Score”:”92.00″, Confidence score of Temper (92 % positive)
        },
        “Valence”:{ Valence Object. (similar to Temper object)
           “Value”:”23.00″, Value of Valence
           “Group”:”negative”, Group of Valence, unknown – means that value cannot be calculated for the slice unknown value: V4 and above
           “Score”:”94.00″, Confidence score of Valence (94 % positive) V4 and above
        },
        “Arousal”:{ Arousal Object. (similar to Temper object)
           “Value”:”24.00″, Value of Arousal
           “Group”:”low”, Group of Arousal, unknown – means that value cannot be calculated for the slice unknown value: V4 and above
           “Score”:”80.00″, Confidence score of Valence 80 % positive) V4 and above
        },
        “Mood”:{ Mood Object, Contains Mood Group objects
           “Group7”:{ Mood Group 7 Object
           “Primary”:{ Primary mood of Mood Group 7
              “Id”:7, Id of the phrase
              “Phrase”:”Worried” Phrase (Primary Mood Group 7 Phrase)
           },
           “Secondary”:{ Secondary
              “Id”:4, Id of the phrase
              “Phrase”:”Frustrated” Phrase (Secondary Mood Group 7 Phrase)
           }
        },
           “Group11”:{
           “Primary”:{
              “Id”:3,
              “Phrase”:”Defensivness, Anxiety”
           },
           “Secondary”:{
              “Id”:7,
              “Phrase”:”Loneliness, Unfulfillment”
           }
        },
        “Group21”:{
           “Primary”:{
              “Id”:21,
              “Phrase”:”unhappiness”
           },
          “Secondary”:{
             “Id”:16,
             “Phrase”:”loneliness”
           }
        },
        “Composite”:{ Composite Mood Object
           “Primary”:{
              “Id”:274,
              “Phrase”:”Painful communication. High sensitivity.”
           },
           “Secondary”:{
              “Id”:241,
              “Phrase”:”Longing for change. Seeking new fulfillment.Search for warmth.”
           }
          }
         }
        }
     },
     { Following analysis segment objects
     ……
     },
    ],
     “analysisSummary”:{ The object of analysis summary
     “AnalysisResult”: { The object of analysis summary results
     “Temper”: { Temper Summary Object
     “Mode”: “low”, Most frequent Temper Group
     “ModePct”: “100.00” The Percentage of the most frequent Temper group
     },
     “Valence”: { Valence Summary Object
     “Mode”: “negative”, Most frequent Valence Group
     “ModePct”: “100.00” The Percentage of the most frequent Valence group
     },
     “Arousal”: { Arousal Summary Object
     “Mode”: “low”, Most frequent Arousal Group
     “ModePct”: “100.00” The Percentage of the most frequent Arousal group
     }
     }
   }
 }
}

NEW! Confidence score

Confidence Score – Feature Description

Starting at API version V4, Beyond Verbal has added a new feature to its API output called confidence score.

This document explains the provision of a corresponding confidence score to each emotion output & how Beyond Verbal’s
user community will benefit.

Along with various emotions outputs (Temper, Valence, Arousal) & their group (e.g. low, med, high), Beyond Verbal is
now providing a corresponding confidence score (from 0 to 100) for each such output.
*See example of the API output in Exhibit A

The confidence score is expressed as a metric that reflects the distributions of likelihoods of the identified
emotion output group (e.g. low arousal).

Using this confidence metric, Beyond Verbal will now return the value “unknown” when the confidence score of an
emotion group does not exceed a predetermined threshold. Accordingly, Beyond Verbal excludes speech segments when
they have been deemed to be unanalyzable (e.g. due to audio quality issues like excessive background noise).

By excluding unanalyzable samples, Beyond Verbal will filter out ambiguous & weak speech segments, thereby
significantly improving overall emotions analytics accuracy & performance.

Additionally, the provision of the confidence score empowers our users to use this parameter during API integration &
when customizing their own application logic. For example, on the user’s end, the application can exclude any Beyond
Verbal output with a confidence score below a specific threshold e.g. 65. This may be helpful in use cases where
there is a large voice data set & emotions analytics accuracy is more important than granularity/frequency of
analysis.

Beyond Verbal’s API output overall accuracy ranges from 70% for more ambiguous speech segments to above 90% for more
unambiguous & high quality speech segments.

When Beyond Verbal API users setup & adjust their own confidence score filter rules, the following approximate rule
of thumb should be helpful to clarify the expected impact on accuracy at varying levels of confidence:

Accuracy increases approximately 4-5% for a 10-point increase of the selection threshold. For example, rejecting
samples with confidence score below 70 leads to an increase in Beyond Verbal engine accuracy of 5%.

Therefore, depending on the use case & user’s preference for accuracy over total number of analyzed segments, by
filtering out segments with lower confidence scores, user can achieve accuracy in the 90% range.

User community feedback is always welcomed via api@beyondverbal.com.

Exhibit A

Example of API output including Score
confidence_score1