I'm looking for any modern resources for setting up a video streaming server. Preferably open source solutions.
My searching on this has lead to a lot of dead ends. I also do need to build my own instead of paying for a service.
To setup your own video streaming server you should get started with the following components:
FFMpeg (see the streaming guide)
VLC (see their wiki)
Maybe a content management solution to serve your streams such as OpenACS
However, please note: Your question is very broad, hence this broad answer.
It really depends on your requirements and your given infrastructure. You will need to script tasks such as preparing streams for HTTP Live Streaming or other formats. If possible maybe add some more details to your questions (formats, codecs, platforms, file restreaming or stream restreaming, real-time etc.)
Go here and here (long answer) for more input on HTTP Live Streaming.
Hopefully this will lead you to useful directions.