The documentation for OpenCV's floodfill function states:
The function uses and updates the mask, so you take responsibility of initializing the mask content. Flood-filling cannot go across non-zero pixels in the mask. For example, an edge detector output can be used as a mask to stop filling at edges. It is possible to use the same mask in multiple calls to the function to make sure the filled area does not overlap.
How does the function update the mask? Does it set all the pixels within the floodfill to some non-zero value?
Answer
All zero-valued pixels in the same connected component as the seed point of the mask are replaced by the value you specify. This value must be added to the flags parameter, left-shifted by 8 bits:
uchar fillValue = 128;
cv::floodFill(img, mask, seed, cv::Scalar(255) ,0, cv::Scalar(), cv::Scalar(), 4 | cv::FLOODFILL_MASK_ONLY | (fillValue << 8));
A simple, but perhaps enlightening example follows. Creating an image like so:
//Create simple input image
cv::Point seed(4,4);
cv::Mat img = cv::Mat::zeros(100,100,CV_8UC1);
cv::circle(img, seed, 20, cv::Scalar(128),3);
Results in this image:
Then, creating a mask and flood-filling it:
//Create a mask from edges in the original image
cv::Mat mask;
cv::Canny(img, mask, 100, 200);
cv::copyMakeBorder(mask, mask, 1, 1, 1, 1, cv::BORDER_REPLICATE);
//Fill mask with value 128
uchar fillValue = 128;
cv::floodFill(img, mask, seed, cv::Scalar(255) ,0, cv::Scalar(), cv::Scalar(), 4 | cv::FLOODFILL_MASK_ONLY | (fillValue << 8));
Gives this result:
The white pixels in the mask are the result of edge detection, while the grey pixels are the result of the flood-fill.
UPDATE:
In response to the comment, flag value 4 specifies the pixel neighborhood with which to compare the color value difference. From the documentation:
Lower bits contain a connectivity value, 4 (default) or 8, used within the function. Connectivity determines which neighbors of a pixel are considered.
When the cv::FLOODFILL_MASK_ONLY flag is not passed, both the image and the mask are updated, but the flood filling will stop at at any nonzero mask values.