A well designed testing process is vital for developing successful voice applications. The benefit of a proper process will improve the overall user experience and limit bad user ratings. On the other hand you need to streamline the process to make it as easy and accessible for the developers to save costly development time and improve time to market.

In the end for a robust testing result there is no way around manual testing on a actual device. We have seen so many cases where the result on an hardware device did diverge from results achieved on simulators significantly and did break the application.

Since our team enjoys working in open office spaces to foster exchange and communication it is really important not to annoy the whole office with talking to your Alexa device and testing the same phrases over and over again. So we came up with an setup to allow us silent testing on actual devices.

Use Bluetooth headphones to connect to your device

We either use Bluetooth headphones or wired ones to listen to the output of our devices without letting the whole office listen to our conversation.

Use a headphone as an input source to your device

All smart speakers do have a very sensible omnidirection microphone to pick up human voices from all parts of the room. Therefore they don’t have a possibility to plug any line in source into the device. So we need to improvise to get sound into the device. We usually use our headphones (volume set to max) connected to our laptop or a very small Bluetooth speaker (volume tuned to minimum). So we can use the output of our laptop (see next step) to trigger commands to the smart speaker.

Use a TTS engine

For testing you can either use pre-recorded mp3s that are triggered by you or as a simpler solution use a Text-To-Speech (TTS) engine that reads text out loud. This is sufficient for most of our testing where we want to check the user flow and not the robustness of the Speech to Text or Natural Language Processing accurateness of our applications. We have develop our very own TTS Testing suite that is available at speechtest.169labs.com, for free. Have a look on the settings page there to configure your different projects and language. We will continue to improve the tool over time! Go ahead and bookmark the page – happy to receive your feedback!

Use a soundproof glass encapsulation

To be sure the smart speaker is not picking any other voice commands from other people in the room we have been searching a soundproof encapsulation to block out all other sources besides our headphone source. We first started off having the smart speaker in a drawer of out desk which is sufficient when not working on Applications that do not use the screen. In search of a solution that provided us with the screen of the device as well we came across this soundproof glass encapsulation which is intended for home decoration. It fits perfectly over an Amazon Echo Dot, an echo spot and a Google Home mini device.

So in the end we have built our very own silent testing suite to speed up our development process, ensure everything is working flawlessly on real devices and does not annoy any coworkers that are in the same room. We can even build some end to end tests with a whole staged conversation to do retests after we changed the source code.