Documentation of the Google Assistant SDK says that you need to hit enter to talk to the assistant. Is there a native way to set up a wake word?
Also consider using Snowboy - https://snowboy.kitt.ai/ - it supports Python and a variety of other languages, runs on Mac/Linux (including Pi) and supports multiple hotword detection. And its free for non-commercial use.