I'm looking to get Stack Overflowers' advice and suggestion on time-series libraries written in C++, some of the constraints and requirements for the library:
Have you looked at IT++ ? The documentation and feature list are quite compelling and applicable, given your question. Its development may have stalled, but is the primary C++ time series / signal processing library that is open source that I know of.
Otherwise, for straight-up linear algebra, both Armadillo and Eigen are good and actively maintained, and Armadillo can even be used along with IT++.
Last but not least, consider R and Octave both of which can also be extended with C++ code (see Rcpp for the R/C++ interface, and RcppArmadillo for an R/C++/Armadillo interface.