This might be a mix of a R question and an algorithm question. The question is about both OPTICS in general and the R implementation of optics in the package "dbscan" ( https://cran.r-project.org/web/packages/dbscan/dbscan.pdf )
My main reason to move from using DBSCAN to OPTICS was that the dataset I have has variable density clusters, and they are of irregular shape. OPTICS produces a reachability plot, but for my use case the more interesting part is the extraction of clusters. There is some automatic cluster extraction described in the original paper that isn't just a single cut-point for eps. ( http://fogo.dbs.ifi.lmu.de/Publikationen/Papers/OPTICS.pdf).
So my two part qn: 1) Is there a way to use the R package in this way for the automatic extraction? 2) Is there an OPTICS implementation that supports this (python,elsewhere)?
1) Yes! The dbscan package has a function to extract optics clusters with variable density. ?dbscan::extractXi()
extractXi extract clusters hiearchically specified in Ankerst et al (1999) based on the steepness of the reachability plot. One interpretation of the xi parameter is that it classifies clusters by change in relative cluster density. The used algorithm was originally contributed by the ELKI framework, but contains a set of fixes.
See https://cran.r-project.org/web/packages/dbscan/vignettes/dbscan.pdf for extensive description of methods and examples