I built my project using the new Android App Bundle format. With APK files, I can download the APK to my device, open it, and immediately install the app. I downloaded my app as a bundle (.aab format) and my Nexus 5X running Android 8.1 can't open the file. Is there any way to install AABs on devices in the same convenient manner as APKs?
Not directly.
Android App Bundles is a publishing format. Android devices require .apk
files to install applications.
The PlayStore or any other source that you're installing from needs to extract apks from the bundle, sign each one and then install them specific to the target device.
The conversion from .aab to .apk is done via bundletool.
You can use Internal App Sharing to upload a debuggable build of your app to the Play Store and share it with testers.