Using Google Oauth2, the following scope is used to get API-access to a users email: https://mail.google.com/
Testing the scope via OAuth 2.0 Playground confirms that the app wants to "View and manage your email":
Is there a way to only ask for access to send emails on behalf of the user? A more specific scope perhaps?
The scope is the same if you use OAuth through SMTP or IMAP. But Google released new set of scopes (https://developers.google.com/gmail/api/auth/scopes) for Gmail API access. Here they are:
You can try using the 5th one and see if it works for you.