I did it myself after playing with Spinner. Here is the solution which worked pretty well.
First create a Dropdown with indicator image of your choice.
<layer-list xmlns:android="http://schemas.android.com/apk/res/android">
<item>
<shape
android:shape="rectangle"/>
</item>
<item
android:width="24dp"
android:height="24dp"
android:gravity="right|center_vertical">
<bitmap
android:src="@drawable/ic_dropdown"
android:tint="@color/colorPrimaryDark"/>
</item></layer-list>
Then assign it to the background of Spinner.
<Spinner
android:id="@+id/basket_spinner_item_quantity"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:spinnerMode="dropdown"
android:padding="0dp"
android:background="@drawable/ic_spinner_dropdown"/>
Now, adjust the padding and alignment of spinner item as per your requirement.
<TextView xmlns:android="http://schemas.android.com/apk/res/android"
android:id="@+id/simple_spinner_text_quantity"
android:paddingTop="5dp"
android:paddingBottom="5dp"
android:paddingLeft="20dp"
android:paddingRight="25dp"
android:gravity="right"
android:textAlignment="gravity"/>