strtok() - Why you have to pass the NULL pointer in order to get the next token in the string?

Question

This is the explanation of strtok().

#include <string.h>
char* strtok( char* s1, 
              const char* s2 );*

The first call to strtok() returns a pointer to the first token in the string pointed to by s1. Subsequent calls to strtok() must pass a NULL pointer as the first argument, in order to get the next token in the string.

But I don't know, why you have to pass the NULL pointer in order to get the next token in the string. I searched about 15 minutes, but didn't find an explanation in the internet.

Answer 1

strtok() keeps some data inside of itself by using static variables. This way, strtok() can continue searching from the point it left off at during the previous call. To signal strtok() that you want to keep searching the same string, you pass a NULL pointer as its first argument. strtok() checks whether the first argument is NULL and if it is, it uses its currently stored data. If the first parameter is not null, it is treated as a new search and all internal data is reset.

Maybe the best thing you can do is to search for an actual implementation of the strtok() function. I've found one small enough to post it here, so you get an idea of how to handle this NULL parameter:

/* Copyright (c) Microsoft Corporation. All rights reserved. */

#include <string.h>

/* ISO/IEC 9899 7.11.5.8 strtok. DEPRECATED.
 * Split string into tokens, and return one at a time while retaining state
 * internally.
 *
 * WARNING: Only one set of state is held and this means that the
 * WARNING: function is not thread-safe nor safe for multiple uses within
 * WARNING: one thread.
 *
 * NOTE: No library may call this function.
 */

char * __cdecl strtok(char *s1, const char *delimit)
{
    static char *lastToken = NULL; /* UNSAFE SHARED STATE! */
    char *tmp;

    /* Skip leading delimiters if new string. */
    if ( s1 == NULL ) {
        s1 = lastToken;
        if (s1 == NULL)         /* End of story? */
            return NULL;
    } else {
        s1 += strspn(s1, delimit);
    }

    /* Find end of segment */
    tmp = strpbrk(s1, delimit);
    if (tmp) {
        /* Found another delimiter, split string and save state. */
        *tmp = '\0';
        lastToken = tmp + 1;
    } else {
        /* Last segment, remember that. */
        lastToken = NULL;
    }

    return s1;
}