When I send an email using the GMail API, I get an Access Token for the user to whom the message is to be sent and then when sending the message, I put some value in the From Field.
But no matter what I put in the from field, the message in the email shows as From "Me" and is in the Sent mails as well as Inbox.
Is there a way, I could use the Service Account to send the email, so that it does not show up as From "Me" and is also not in the Sent Mails.
No. The Gmail API is for Gmail users and service accounts are just for doing auth to a real Gmail account, they don't have their own Gmail account, etc.
If you want to send the email from some service, you need some bulk-sending email service like at: https://cloud.google.com/compute/docs/tutorials/sending-mail/ or something you run yourself or from some other provider.