Smart Headphones: Part 3 – Software

In this post I will explain all the software components involved in the solution and how they are connected. The goal is to get one signal (on/off) go from the sensor on top of the headphones to a Google Chrome Extension. That extension will be the one playing and pausing the media content open in the browser.

To continue, let’s explain all the modules with a picture.

Diagram1

 

As you can see, five different components interact, let’s explain them in detail. All the code explained next is included in my GitHub repository: https://github.com/fcortes9/bitsofinnovation/.

Force Sensitive Resistor

As mentioned in the previous post, the force sensitive resistor was the sensor that solved all the tricky situations I faced throughout all the research. The connections between the sensor and the Arduino are shown here.

Arduino

The code running in the arduino is responsible for getting the analog signals from the sensor and transmit them directly to the PC. I took the approach of having all the on/off recognition logic in only one software module, the desktop app. In that way, customization and logic updates are easy to perform. In summary, the code on the Arduino looks like this:

int pin = 0; // the force sensitive resistor is connected to a0
int reading; // the analog reading from the resistor
int status = 0;

void setup(void) {
// We'll send debugging information via the Serial monitor
Serial.begin(9600);
}

void loop(void) {
reading = analogRead(pin);

Serial.print("#"); // Start character
Serial.print(reading);
Serial.print("-"); // End character

delay(50);
}

I am delimiting the values with a start and end character. This is because the Arduino will be sending characters one by one to via serial port. Since the desktop software can start listening to the serial port at any time, it is possible that the first number being read is actually a number in the middle of the original reading value. Let’s say the value is 297, we send 2, then 9 and 7. The desktop app, at startup, can miss the number 2 and read number 9. This would trigger a false play/pause action. By sending start and end characters, #297- is sent. On the desktop app, the first number that start counting is the one immediately after the character “#”, and all number counts until finding the end character “-“. In this way the initial value is properly reconstructed.

Desktop application

It is developed in C++ and Qt and has several responsibilities:

  • Calibration
  • Configuration
  • Reading of sensor values
  • Action triggering
Calibration

The first action a new user of the headphones will need to do is the calibration. The calibration will measure the force applied to the sensor when the user has the headphones on. In that way, we will have a reference value to compare to when the sensor values change. The calibration process is asking the user to get their headphones on and click the calibration button. The sensor value is read and stored in a calibration file. Later on, that value will be used to calculate a threshold value needed for triggering play/pause actions.

Configuration

At the moment, the configuration feature of the app lets the user select the actions to be taken when the headphones are used. Controlling the media and muting the microphone are the two actions I have already implemented. They are not mutual exclusive and the option can be found in the Settings menu. By default, only the control of the media is enabled.

Recognition of sensor values

Previously I explained that the values coming from the sensor are in the format “#X-“, where X can be a number between 0 and 1000. The desktop app listens to the serial port and reconstructs the reading values from the sensor. Once the reading value is known, the logic to trigger the play or pause is executed.  The measured value is compared to two variables.

One of them is the actual status of the media. If the media is playing and the headphones indicate that they are just being taken on, no action is required. Same situation applies when the music is paused by the user and then he takes the headphones off. The other variable is the threshold, mentioned in the calibration section. The threshold limits the tolerance on the actual sensor readings. A sensor reading value surpassing the threshold (from greater to lesser or vice versa) will trigger a pause or play action, a long as the status of the media indicates that this action is actually required.

The calculation of the threshold value is related to the reference “on” value obtained during the calibration. Once we have the reference value for the user, the threshold is computed as a value 12% less than the reference. Why 12%? Well… there was a lot of try and error involved to get to that number.

If you remember, there where three main scenarios when the user takes their headphones off: hanging them on a hanger, letting them lay on the desk or letting them hang on their neck. The first two scenarios are the same in respect to this threshold feature. The aperture of the headphones will be the same, no matter their orientation. The scenario of them hanging on the neck is a little different. The aperture in this case might (not necessarily) be a little higher. That is why the threshold value needs to be quite similar to the reference “on” value. In my case the “off” reading sensor value is about 20, the value “on” is about 270 and the value when they are hanging on my neck is around 50. The threshold value was set to 237.

One question may arise, when the threshold value is so close to the reference value, if the “hanging on neck” value is only 50?. There are two reasons. The first one is that, while having your headphones on your neck, you can move your head while talking to people and hit them with your chin. That will make the value to rise, and we still don’t want to play music in this case.

The other reason is the user experience. I noticed that having the threshold close to the “on” reference performs good on stopping quickly the media when the headphones are taken off. In case of a podcast or video, no word is being lost in these milliseconds of reaction. Threshold values very close to the “on” reference value will make the media to randomly play and stop while you have the headphones on. This is because the force sensor is not perfect and have some error, meaning the “on” reference value is not always the same.

Taking the headphones off and on again will create an offset on the original reference value. This offset can be as much as ±3%. Since you don’t want the user to calibrate the headphones so frequently, you need to take into account the offset error plus the possible movements of the headphones when the user hit them with their chin, while being on their neck. So after a lot of tweaking and tuning, I found the -12% to be a pretty accurate value.

Action triggering

Once the logic has recognized that a change status is needed, it needs to be propagated to the Google Chrome extension. This is done by triggering a Shell command from within the cpp code. I created two shell scripts, on.sh and off.sh. I call these two scripts with the configuration parameters, since the script alone will not know what actions need to be triggered.

Shell Scripts

The two scripts make use of the library xdotool. This library provides a way to simulate mouse clicks or key presses programatically. All I needed to do is to is to trigger a specific combination of key press to inform the Chrome extension to play or pause the media. In the case of muting the microphone, the library also provides a way to do so.

Additionally, as mentioned before, the actions to take can vary upon configuration. This is why, along with the call to the proper shell script, I am passing the configuration as argument. Inside the shell scripts, the configuration is checked and proper keys are pressed (if needed).

on.sh

#!/usr/bin/env bash

if ((($1&1)==1)); then
xdotool key Ctrl+Shift+5
echo "PLAY!"
fi

if ((($1&2)==2)); then
xdotool key XF86AudioMicMute
echo "MUTE!"
fi

 

off.sh

#!/usr/bin/env bash

if ((($1&1)==1)); then
xdotool key Ctrl+Shift+6
echo "PAUSE!"
fi

if ((($1&2)==2)); then
xdotool key XF86AudioMicMute
echo "MUTE"
fi

I decided to set the automatic play shortcut to Ctrl-Shift-5 and pause Ctrl-Shift-6. I don’t think this combination would clash any other software the user might have installed, and I personally never needed to use them in the past. Nevertheless, improvements will come in future versions. An improvement will also be made in the way the microphone is muted. Due to the single key assign for mute and unmute microphone (XF86AudioMicMute), it might happend that the user manual mutes and unmutes it while using the headphones. That will cause an off-sync and the mute can happen when actually the user wants to unmute. The enhancement will consists on  using amixer to recognize the volume of the microphone before triggering an action. Once the keys are pressed, and if we want to control the media, the Google Chrome extension takes over.

Google Chrome Extension

My initial thought was to create an extension myself, but soon I realized that there were plenty of extensions to control media on the internet. One of them is called the StreamKeys. It supports the most popular media content providers (Spotify, Netflix, Youtube, etc). I took the code and improved it to support automatic play/pause actions.

The souce code is under MIT license, and the patch to support automatic play/pause is here.

120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
BaseController.prototype.autoPlay = function() {
    if(this.selectors.play !== null && this.selectors.pause !== null) {
        if(this.isPlaying()) {
            //Do nothing
        } else {
            this.click({action: "autoPlay", selectorButton: this.selectors.play, selectorFrame: this.selectors.iframe});
        }
    } else {
        if(this.isPlaying()) {
            //Do nothing
        } else {
            this.click({action: "autoPlay", selectorButton: this.selectors.playPause, selectorFrame: this.selectors.iframe});
        }
    }
};

BaseController.prototype.autoPause = function() {
    if(this.selectors.play !== null && this.selectors.pause !== null) {
        if(this.isPlaying()) {
            this.click({action: "autoPause", selectorButton: this.selectors.pause, selectorFrame: this.selectors.iframe});
        } else {
            //Do nothing
        }
    } else {
        if(this.isPlaying()) {
            this.click({action: "autoPause", selectorButton: this.selectors.playPause, selectorFrame: this.selectors.iframe});
        } else {
            //Do nothing
        }
    }
};

In the next update I will explain more about the extension and upload the full source code to the repository.

Comments

comments

Add a Comment

Your email address will not be published. Required fields are marked *