How do you use "NextToken" in AWS API calls

Question

I've run into a little issue that I am really struggling to understand how it works. I have a tool I am writing that basically does a describe-organization to collect all the accounts in our AWS organization. Per the documentation here it says it responds with a json of the accounts which in my case will be hundreds and hundreds of accounts. So I wrote some very simple code to switch roles into our master account and make the call:

import boto3
import uuid
import pprint

iam_client = boto3.client('iam')
sts_client = boto3.client('sts')
org_client = boto3.client('organizations')


print("Starting in account: %s" % sts_client.get_caller_identity().get('Account'))

assumedRoleObject = sts_client.assume_role(
    RoleArn="arn:aws:iam::123456xxx:role/MsCrossAccountAccessRole",
    RoleSessionName="MasterPayer"
)

credentials = assumedRoleObject['Credentials']

org_client = boto3.client(
    'organizations',
    aws_access_key_id = credentials['AccessKeyId'],
    aws_secret_access_key = credentials['SecretAccessKey'],
    aws_session_token = credentials['SessionToken'],
)

getListAccounts = org_client.list_accounts(
    NextToken='string'
)

But when I execute the code, I get the following error:

"botocore.errorfactory.InvalidInputException: An error occurred (InvalidInputException) when calling the ListAccounts operation: You specified an invalid value for nextToken. You must get the value from the response to a previous call to the API."

I'm really stumped on what that means. I see the NextToken, and I can find many references to it in the AWS documentation but I can't figure out how to actually USE it. Like, what do I need to do with it?

Answer 1

Don't take the boto3 examples literally (they are not actual examples). Here is how this works:

1) The first time you make a call to list_accounts you'll do it without the NextToken, so simply

getListAccounts = org_client.list_accounts()

2) This will return a JSON response which looks roughly like this (this is what is saved in your getListAccounts variable):

{
    "Accounts": [<lots of accounts information>], 
    "NextToken": <some token>
}

Note that the NextToken is only returned in case you have more accounts than one list_accounts call can return, usually this is 100 (the boto3 documentation does not state how many by default). If all accounts were returned in one call there is no NextToken in the response!

3) So if and only if not all accounts were returned in the first call you now want to return more accounts and you will have to use the NextToken in order to do this:

getListAccountsMore = org_client.list_accounts(NextToken=getListAccounts['NextToken'])

4) Repeat until no NextToken is returned in the response anymore (then you retrieved all accounts).

This is how the AWS SDK handles pagination in many cases. You will see the usage of the NextToken in other service clients as well.