Obtaining zero-length string from strtok()

c csv strtok
Mauren picture Mauren · Sep 16, 2013 · Viewed 8k times · Source

I have a CSV file containing data such as

value;name;test;etc

which I'm trying to split by using strtok(string, ";"). However, this file can contain zero-length data, like this:

value;;test;etc

which strtok() skips. Is there a way I can avoid strtok from skipping zero-length data like this?

Answer

Martin R picture Martin R · Sep 16, 2013

A possible alternative is to use the BSD function strsep() instead of strtok(), if available. From the man page:

The strsep() function is intended as a replacement for the strtok() function. While the strtok() function should be preferred for portability reasons (it conforms to ISO/IEC 9899:1990 ("ISO C90")) it is unable to handle empty fields, i.e., detect fields delimited by two adjacent delimiter characters, or to be used for more than a single string at a time. The strsep() function first appeared in 4.4BSD.

A simple example (also copied from that man page):

char *token, *string, *tofree;

tofree = string = strdup("value;;test;etc");
while ((token = strsep(&string, ";")) != NULL)
    printf("token=%s\n", token);

free(tofree);

Output:

token=value
token=
token=test
token=etc

so empty fields are handled correctly.

Of course, as others already said, none of these simple tokenizer functions handles delimiter inside quotation marks correctly, so if that is an issue, you should use a proper CSV parsing library.

Related questions

tokenizing a string twice in c with strtok()

I'm using strtok() in c to parse a csv string. First I tokenize it to just find out how many tokens there are so I can allocate a string of the correct size. Then I go through using the same …

Read More
read comma-separated input with `scanf()`

I have the following input: AG23,VU,Blablublablu,8 IE22,VU,FooBlaFooBlaFoo,3 and so on... I want it to "parse" with scanf() using some code like this: char sem[5]; char type[5]; char title[80]; int value; while(scanf("%s,%s,%s,%d", …

Read More
How do I parse out the fields in a comma separated string using sscanf while supporting empty fields?

I have a comma separated string which might contain empty fields. For example: 1,2,,4 Using a basic sscanf(string,"%[^,],%[^,],%[^,],%[^,],%[^,]", &val1, &val2, &val3, &val4); I get all the values prior to the empty field, and unexpected results from …

Read More