How Machine Listening Transforms Smart Home Experience
Making your life easier, healthier, & safer with machine listening
Two claps, and you can turn on a lamp. Three claps, and you can switch on a TV. First released in 1985, the Clapper that allows its user to activate home appliances using sequences of claps, is often referred to as one of the earliest commercial uses of machine listening.
So what is machine listening?
“Machine listening” is the use of signal processing and machine learning for making sense of natural/ everyday sounds, and recorded music.”
In other words, machine listening enables computers to understand the meaning of sounds like humans.
Some might think of speech recognition when they hear the term “machine listening”. Machine listening however, is a broader term that encompasses research on a wide range of tasks, including speech recognition, sound event detection, content-based music analysis, and audio-based search.
In this article, we are going to focus on detection of non-verbal sounds, that is, sounds we hear in our everyday lives — other than human voice. These include clapping, keyboard-typing, dog-barking, laugh, cough, and more, and how it can affect your smart home experience.
Controlling Home Appliances
Home appliances starting from TV to curtains, speakers, bed, lights… more and more things we use at home are going smart. What if these could be controlled with sounds like clap, knock or whistle?
We are already familiar with smart home platforms that support voice control or virtual assistants like Amazon’s Alexa, that are usually activated by wake words like “Hey XX, turn on the music.” Many confess however it feels quite awkward and odd talking to a voice assistant, especially in public situations.
Apart from the embarrassment, voice controls have varying performance depending on the language. Alexa, developed by the US company, works best with standard American English, and has limitations understanding non-English languages. Many smart platforms thus naturally have limited language supports for voice control.
Unlike voice controls, we feel at ease when we use non-verbal sounds to control IoT platform. Clap to turn on the TV, whistle to play music — setting up shortcut commands for actions you frequently perform could come in handy, giving you better speed and convenience.
Indoor Context-Aware
We make noises in our daily routines that contain clues to be recognized by sounds. In the kitchen, we often make chopping sounds when cutting up ingredients, turn on the microwave to warm cold food, clattering sounds when we eat, and water-running sounds when doing the dishes.
So… what’s the deal? How does the fact that my IoT platform is aware of all these benefit me?
Generally, context-awareness is the ability of devices to sense their physical environment and adapt their behavior accordingly. Context-aware home can bring true automation by making subtle adjustments and suggestions that can help lift your moods and automatically control the environments according to the situation.
When you are back home after a long day at work, you deserve a quality rest. As you are having dinner, the lights are automatically adjusted and the speaker plays pleasant music. After dinner time, you watch your favorite TV show and fall asleep. As you start to snore, TV will be turned off and the lights will go off.
While recognizing most of these at-home behaviors could be achieved with cameras with computer vision enabled, we feel disturbed and feel uneasy that camera is watching our private space, even if the product says it does not save any private data. That is why hearing-enabled devices could be a better choice than vision-enabled ones. Users can feel much more secure and private, knowing that their device runs purely by on-device computing power, and does not send any data to the cloud.
Monitoring Your Loved Ones
Sounds could be a great source of information for those whose major method of communication is not a proper language.
Dogs and babies. Many people devote their time and resources and would do anything to keep them safe and happy. There are many dog cams and baby cams, but it is not possible to keep your eyes on the cams all the time. With machine listening ability, you can be notified when your baby cries or dog barks, and instantly check if anything went wrong, when you are at home or away.
Monitoring is also a crucial job needed for patients and the elderly. Even at hospitals, it is hard to manage all the patients given the limited number of medical staffs. In times of pandemic, detecting that there is regular cough coming from a particular room could help spotting a potential source of infection. Patients who collapse to the floor at night and can’t reach help, could use some help from machine listening as well.
Conclusion
Machine listening properly engaged can make your daily life easier, healthier, and safer. Major tech companies have recently started to release their own machine listening solutions. Amazon’s Alexa Guard launched in 2019 lets Amazon Echo smart speaker listens for glass break and smoke alarms. Apple’s new iOS 14 feature lets the iPhone alert you if it hears sounds like a doorbell or baby crying. Starting from smart speakers and smartphones, we are going to see more of these machine listening abilities integrated with home IoT platform — into your own bed, home cams, lights and more.
