Get text content of the android screen

Question

Is there any way to get the text content that is being displayed on the android mobile screen. It may be by any android app.

for example in the image linked above, i wish to get all the text (starting from the "Larry Page" till the end "Fred Smith" and also the position of all the words on the screen) that is in the image and its respective position on the screen.

Answer 1

This can be achieved using the Accessibility feature of Android (as the Universal copy app does).

What you need to do is implement a class called AccessabilityService which can access all the screen UI through a listener.

In order to that you need to declare it in the manifest:

 <service android:name=".MyAccessibilityService">
 <intent-filter>
     <action android:name="android.accessibilityservice.AccessibilityService" />
 </intent-filter>
 <meta-data android:name="android.accessibilityservice" android:resource="@xml/accessibilityservice" />

the file @xml/accessibilityservice is where you configure the service like to which package to listen or what kind of events you want to get:

<accessibility-service
 android:accessibilityEventTypes="typeViewClicked|typeViewFocused"
 android:packageNames="com.example.android.myFirstApp, com.example.android.mySecondApp"
 android:accessibilityFeedbackType="feedbackSpoken"
 android:notificationTimeout="100"
 android:settingsActivity="com.example.android.apis.accessibility.TestBackActivity"
 android:canRetrieveWindowContent="true"/>

In the class override the method:

@Override
public void onAccessibilityEvent(AccessibilityEvent event) {
    Log.v(TAG, String.format(
            "onAccessibilityEvent: type = [ %s ], class = [ %s ], package = [ %s ], time = [ %s ], text = [ %s ]",
            getEventType(event), event.getClassName(), event.getPackageName(),
            event.getEventTime(), getEventText(event)));
}

In that method, you should find in the UI the desired text/TextView/EditText and extract from it the text you want.

NOTE: this way required the Accessibility permission which only the user can give you and it cannot be done in runtime.

You can read more in the Android documentation about how to implement an AccessabilityService

EDIT: after the question in the comment I'm adding the code to read from all the text components in the screen:

    private void printAllViews(AccessibilityNodeInfo mNodeInfo) {
        if (mNodeInfo == null) return;
        String log ="";
        for (int i = 0; i < mDebugDepth; i++) {
            log += ".";
        }
        log+="("+mNodeInfo.getText() +" <-- "+ 
        mNodeInfo.getViewIdResourceName()+")";
        Log.d(TAG, log);
        if (mNodeInfo.getChildCount() < 1) return;
        mDebugDepth++;

        for (int i = 0; i < mNodeInfo.getChildCount(); i++) {
            printAllViews(mNodeInfo.getChild(i));
        }
        mDebugDepth--;
    }

Just don't forget to initialize the AccessibilityNodeInfo and the depth counter like so:

public void onAccessibilityEvent(AccessibilityEvent event) {
    mDebugDepth = 0; 
    mNodeInfo = event.getSource();
    printAllViews(mNodeInfo);
 ...
}