Project PiLexa was an idea I had while researching NLPs (natural language processors) and voice assistants for phones and smart homes. Given there are most frequently used phrases as indicators of a certain action or task, or most frequently asked questions from an assistant, we can write NLP “code” to extract data and match to known handlers. The interface is voice-to-text and text-to-speech which are both well developed technologies with open source libraries and free local ML models.
One of my classes wanted us to make an Android app, so I made an app to interface with an instance of the server on a device on the local network. The phone and device (a raspberry pi with a microphone or laptop) both can forward audio or text to the server, and the server will by default output voice, but can be configured to send back an audio file or just text responses. The app has a background service to poll for events from the server and the events include voice commands to control the phone.
Tech Stack
Java / Kotlin, Android SDK
Architecture
- Android app client for executing commands and configuring server
- Android background server that can ping server for events or commands to run
- Kotlin server for text-to-speech, speech-to-text, and interpreting speech / commands
- The server has a small library of RegEx patterns to invocations so that particular patterns of speech can be parsed and executed
Reception
My professor was impressed how far beyond the scope I went for the project but that meant it was slightly more difficult to run and test.
Lessons Learned
I should have kept the school project in scope but otherwise I learned a lot about NLP, voice-to-text, voice assistants, and built something I am eager to talk about.