How can I build a small operating system on an old desktop computer?
This might be in vain, as I know writing an operating system is unbearably complicated (especially by oneself).
I don't expect to build the next linux, or windows.
I know it will be horrible, and buggy, and won't work, but that's fine.
I want to write everything myself, in Assembly, C, and (some) C++.
This is a future project, as I'm busy with some other things at the moment and don't have the time immediately, but I figured I would ask it now, so maybe I could get lots of answers to this, and it could build up and be a useful resource for this kind of approach (everything else I have seen involved building off of minix, using an existing bootloader, building it in a virtual booting program thing, etc).
I want to set up one of my older desktops with a monitor, keyboard and mouse, and start working on a blank hard drive.
I want to learn how to write my own bootloader (I've found lots of resources about this, but for completeness, please still add some good ones), my own USB driver (if that's necessary), a CD driver (if that's necessary), etc. Everything, from the ground up.
How do I put the code onto the computer? Is it best to do it with a floppy disk? Can most computers do it from a USB stick?
What drivers do I need, and can you suggest any references to building those?
After the booting sequence--then what? How do I get into protected mode etc.
How do I manage memory without the help of an operating system? Do I just use whatever addresses I want? No initialization necessary?
What will I undoubtedly run into that will confuse me?
How can I make it either a command line O/S, and a graphical one?
What is a graphical O/S built on? Like, how would I do something like, a command line, with a font, and a picture at the top?
Where can I read about setting up a multitasking environment? (ie., having two graphical-like command lines running side-by-side).
How would I set up a sort of windowing system? How do I display graphics on the screen once simple multitasking is set up?
Believe me, I understand that this is a very complicated project, and I probably will never get around to completing it or writing anything on it of any use.
There are lots of other pieces to this I haven't mentioned, if you think of any, feel free to add those too.
Please put one "topic" per answer--for example, USB drivers, and then maybe a list of resources, things to look out for, etc.
Also, please don't suggest building off of another O/S or pre-existing code. I know I will read a lot of pre-existing code (such as the linux kernel, or example resources, existing drivers, etc) but ultimately I want to do all the writing myself. I know I should build off of something else, and there are lots of other questions on SO about that that I can read if I change my mind and go that route. But this one is all about doing the whole thing from scratch.
Any suggestions on how to get it graphical? Different video modes and how to work with them, etc?
Answer
First things first. Read, read, read, read, read. You need to have a firm understanding of how the OS works before you can hope to implement your own.
Grab one of Andrew Tanenbaum's books on operating systems. This is the one we used in my OS class in college:
Modern Operating Systems on Amazon
Despite the ridiculous cover, it's a fantastic read, especially for a textbook. Tanenbaum is really an expert in this area and his explanations of how the OS works underneath the hood are clear and easy to understand. This book is mostly theory, but I believe he also has a book that discusses more of the implementation. I've never read it, though, so I can't comment on it.
That should help you bone up on process management, memory management, filesystems, and everything else your OS kernel needs to do to get it up to a bootable state. From that point on it's basically a matter of writing device drivers for the hardware you need to support, and offering implementations of the C library functions to make kernel calls for things like opening files and devices, reading and writing, passing messages between processes, etc.
Read up on x86 assembly (assuming you are designing this for an x86 machine). That should answer a lot of your questions with regards to moving between processor operating modes.
If you've got any electronics knowledge, it may be easier to start with writing an operating system for an embedded device that has ample documentation, because it will generally be simpler than an x86 PC. I've always wanted to write my own OS as well, and I'm starting with writing a microkernel embedded OS for This Development Board from Digilent. It can run the soft-core MicroBlaze processor from Xilinx, which has very thorough documentation. It's also got some RAM, flash data storage, LEDs, switches, buttons, VGA output, etc. Plenty of stuff to play around with writing simple drivers for.
One of the benefits of an embedded device is also that you may be able to avoid writing a VGA driver for a long time. In my case, the Digilent development board has an onboard UART, so I can effectively use the serial output as my console to get the whole thing up and booting to a command line with minimal fuss.
Just make sure that whatever you choose to target has a readily available and well-tested compiler for it. You do not want to be writing an OS and a compiler at the same time.