Amazon, Google and Apple Recently Enabled Their Products To “Listen” for These 3 Reasons
iPhone, Apple Watch and Pixel Buds- What you can do with their new sound detection features
Humans are born to detect threats by staying alert to sounds.
We get nervous when we hear footsteps walking behind us in a dark street. We are all ears and alert when we are about to go to bed and hear something outside the bedroom.
With limited sight at night, or at home alone, we often depend solely on sounds to understand the surroundings.
But what about when we are distracted? Can you hear the door knock when you are listening to music wearing big headphones? Or how do I know the car behind me honked to warn me of approaching with my noise cancelling mode on? What about for those who have hearing loss? What if I want to know if my dog barks and upset my neighbors while I’m away?
Big tech companies have come up with solutions to these problems thanks to the recent breakthroughs with sound AI technology — and now they are able to build these Machine learning-based sound detection features into their devices as small as earbuds.
Starting from 2019, Amazon, Google and Apple have competitively released sound detection features, mainly for these three purposes.
#1. Home security: Smart speakers can listen for suspicious activities while you are away
In May 2019, Amazon rolled out Alexa Guard, enabling Amazon Echo speakers to transition into Guard Mode by saying a command like “Alexa, I’m leaving”.
If sounds like glass break, CO alarm or human activity including footfalls or door opening is detected, it will send a notification to your phone. You can also configure it to lock any doors with compatible smart locks, or deter unwanted visitors by turning on/off lights to make it appear as if you’re home, sound dog barking sound or a siren when activity is detected. Similarly, in November 2019, Google released glass break detection for its smart speaker, Nest Secure.
In fact, sound recognition technology applied to home security is not a brand-new concept. Prior to the deep-learning based sound detection method like the example of Alexa Guard, a rule-based sound detection method had been used conventionally, which basically recognizes sound by loudness and magnitude of frequency. Traditional glass break detectors or smoke alarm detectors are built with this rule-based technology.
“Rule-based” system, as its name implies, is confined by knowledge and rules given manually by human. As a result, it was not useful for complex environment where there were high frequency of exceptional patterns and thus, had poor performance.
Modern deep-learning based models on the other hand, doesn’t depend on given knowledge because it can continue learning on its own, constantly optimizing itself by adapting to changes and hidden patterns, resulting in incomparably better performance compared to the rule-based one.
#2. Safety: Your wearable will make sure you don’t miss a thing!
Smartphone manufacturers are constantly producing wearable and accessories to upsell with their smartphones.
Among them, wireless earbuds, since the launch of AirPods 1, have been a huge hit, and report says Apple alone has sold nearly 60 million AirPods. Airpods users love the Noise cancelling function, introduced with AirPods Pro, “cancels” the external sound before it could make you feel as if you are isolated from outside world.
While it’s a cool function, noise cancelling presents high risk of accidents by nature. If you are a cyclist or drive frequently, chances are you have encountered a passenger with earbuds who’s totally unaware of your bicycle bells or car honks. Some states in the US have introduced laws to forbid drivers to wear headphones or earbuds while driving.
Aware of this danger earbuds could bring, earbuds makers have been devising ways to avoid this danger. Google’s Pixel Buds have brought Attention Alerts in August 2020. If a baby cries nearby or an emergency vehicle passes by with its siren on, the Pixel Buds will automatically lower the volume temporarily so you can hear what’s happening around you.
Additionally, Apple Watch added another creative and well-timed safety measure amid the pandemic — the hand washing timer. With this feature enabled, Apple Watch listens for water running sound, and will automatically display 20 seconds countdown for proper hand washing.
#3. Accessibility: Designed to assist with your daily lives
Hearing loss is a common problem caused by noise, aging, disease, and heredity. Globally, there are 466-million people suffering from some level of hearing loss.
Hearing loss hugely impacts daily life in many ways. Take for example, sounds they are likely to miss out due to hearing loss — when you forget to turn off the water faucet, you can’t hear the constantly running water. No beep from the oven telling you your food is ready. Can’t tell if the smoke alarm went off. Missing out their babies crying, murmuring, and giggling… all these are huge pain they have to live with.
These people could hugely benefit from the sound detection technology if their mobile devices could capture the sound for them and give them notifications. Apple and Google were the first to add this feature into their smartphones, and users can configure sounds they want to detect under Accessibility setting — iOS 14 Sound Recognition (updated in September 2020) and Android Sound Notifications (updated in October 2020) respectively.
iOS users can set their devices to constantly listen for 11 sounds and receive alerts when these sounds are detected.The sounds are categorized as follows:
•Alarms: Fire, Siren, Smoke
•Animals: Cat, Dog
•Household: Appliances, Car Horn, Door Bell, Door Knock, Water Running
•People: Baby Crying
Android users can likewise detect 11 sounds including shouting and landline phone ringing.
Tech companies see accessibility as one of its core values and they put in a ton of work in assistive technology in order to ensure everyone can use their devices in the best ways to suit them. Some accessibility features such as VoiceOver (screen reader) or Voice Control, originally designed and developed with accessibility in mind, were adopted by mainstream after many people found value in these features. Sound detection too, will turn out to be useful for many people in general if smartphones could recognize diverse sounds and help them understand their surroundings better.
Progressing towards a smarter future
As it’s a growing market in the early stage, tech giants saw huge potential in the sound AI technology and started in investing in the field.
World’s largest annual workshop in the sound AI research field, IEEE DCASE (Workshop on Detection and Classification of Acoustic Scenes and Events), which used to be a flagship event mainly for academia, recently gained FAANG’s attention. In 2019, Facebook, Amazon, Google, and IBM sponsored DCASE, and Apple joined in the following year.
Predominant application for sound AI technology until now would be a simple sound recognition, like what Apple, Google and Amazon has built so far. What’s bound to come in the next few years however, is much more than that.
Researchers have spent a great deal of time studying how electronic devices can be augmented with sensors that allow them to perceive their environment. This body of work, often referred to as context-aware, stems from the idea that in order to build large-scale ubiquitous systems, individual devices must be aware of their environment (e.g. where they are, what is happening, where something is happening). Based on such information, applications and devices can make decisions proactively.
Sounds provide countless contextual information. Imagine what it would be like if devices could process perceive their surroundings by sound. Your cars could tell whether you are passing by a highway or crowded street, or forest surrounded by birds chirping, and change driving mode accordingly. Security cameras catch violent noise and spot a crime. Smart lamps knows you fell asleep and turn off the light. Autonomous car, smart city, home automation and healthcare — expansion of context-aware sound applications could be exponential.
With the speed of the research advancement in the field, context-aware sound applications will boom in the very near future. So let’s tune in and see what’s coming!
