How to organize build, server, client and shared JavaScript code with NodeJS
One big benefit I've always perceived with using NodeJS on the server is the potential for sharing bits of code between the server and client side (ex. input validation). Now that I'm actually developing using NodeJS one difficulty that I've found is determining the responsibility and context in which each body of code is executed. Below I'll list a few of the difficulties I've had in hopes gain some enlightenment on conventions or guidance that I may be overlooking that could help elevate these issues.
Build-Time Code
Build time code for projects that use Gulp, Grunt, or vanilla NPM in a way that follow the basic documentation are generally pretty easy to follow. Most smaller projects tend to keep all of the code within a single file and the file tends to be named a conventional name like gulpfile.js, however with bigger projects I've seen these scripts begin to be split out. I've seen some cases where the gulp file is split into multiple files and placed under a separate directory. Even worse I've found cases where the gulpfile.js file isn't even named as such causing new developers to hunt around to find where the gulpfile is located and once it is located the gulp command always has to be run with the specific --gulpfile option.
Run-Time Server-Side Code
The entry point for basic node applications appear to simply require pointing out a specific JavaScript file when running the node command (ex. node script.js). For web server applications, such as those using Express, I've noticed that by convention the entry point file is often called server.js and can usually be found in the root directory of the application. In some other cases however such as when running the web server in a developer environment I've seen gulp tasks take on the responsibility of launching Node. In these cases there seems to be multiple ways to include the entry point but one example I've found is just starting up the webpack complier followed by a require statement for the entry point script. Figuring out how to incorporate normal guidance on how to accomplish a typical node debug command is non-trivial in this type of setup. Besides the entry point of the application, there doesn't seem to be any general guidance on directory structure for NodeJS/Express applications that keeps server-side specific code in it's place to help locate it and to keep it separate from build-time and client-side code.
The server-side story becomes even more complex in cases where the server side code is used both for the purpose for serving up static content, server-side generated views (such as with MVC), as well as for providing an API to the client side. My preference is to separate API from the application project but I get the feeling from others that there is a sense of overcomplexity involved in doing so where I see it as a reasonable separation of concerns.
Run-Time Client-Side Code
Since client-side code can often have various entry points based on the first page that is requested this can be tricky. However because of the general transparency of URLs with how they map to resources for typical cases it as well as how powerful the debugging tools have become in modern browsers, it isn't too much trouble following the follow of the scripts. The difficult instead for the client side code comes more for typical build processes which generally end up copying the files around and placing them into a production like structure under a different name. An example would be a project that has a folder called src or js that holds client-side and server-side code intermingled except for that only a portion of the files happen to be included in a build task which transforms and often concatenates the files and places them in a distribution folder. Common names of these distribution folders that I've seen are dist, public, www, and wwwroot. Often if not always these directories are at the root of the project which at least makes it a bit easier to locate without having to interrogate the build scripts.
My hope is that there is some general guidance on how to put all of this together in a sane way perhaps by an authoritative source mainly to give guidance to those like myself who may want start off on the right foot. As a side effect perhaps being able to reference some sort of standard even if it is a loose one may also reduce the amount of boilerplate a team has to invent and discuss as they get started. Within each of the contexts listed above there will obviously be some technology specific conventions such as those followed for AngularJS, Meteor, or ReactJS on the client-side. The conventions I'm looking for are more specific to separating the main highlevel contexts in end-to-end JavaScript applications where the language and platform no longer become obvious way to differentiate between each.
Answer
Build-Time Code
IMHO if you have so much build-time code that it's more than say 1000 lines and requires more than a handful of files, something has gone off the rails. Either you don't know how to make good use of existing packages from npm or you don't understand how to refactor generic code and publish as independent npm packages. If you feel like you need guidance about a project's build-time code because it is so large and complex, my suggestion is to modularize and split out into separate projects - each published independently to npm. Also just check your overall approach. What are you doing that is so custom and requires so much machinery?
Run-Time Server Side Code
Please see my other answer to ExpressJS How to structure an application?
Generally I'd rather see client side code and server side code either completely separate npm packages (separate git repos, separate package.json files, published independently) (if they are large enough) or otherwise co-mingled in the same module and grouped by coupling (all the code relating to a feature kept together including front and back end code), especially if your code base has a substantial amount of code that works in both environments.
I have an open-source full-stack node/JS application called mjournal that keeps browser code and node code alongside each other. You can have a look and see if it feels logical to you and easy to understand where code lives. It is by no means a popular approach, so many folks will dislike it, but it feels good to me personally since I've embraced "group by coupling" as a general principle.
Figuring out how to incorporate normal guidance on how to accomplish a typical node debug command is non-trivial in this type of setup
Yeah, that's nonsense. Your app should start with npm start or something like node server.js. Elaborate gulp/grunt setups that confuse new developers are unnecessary complexity you should just eliminate.
The server-side story becomes even more complex in cases where the server side code is used both for the purpose for serving up static content
In my experience, the code to serve up static content boils down to 5 lines or less, so no big deal. If you have a non-microscopic amount of code dealing with serving static content, again something has gone way off the rails.
Run-Time Client Side Code
My hope is that there is that there is some general guidance on how to put all of this together in a sane way perhaps by an authoritative source mainly to give guidance to those like myself who may want start off on the right foot.
There are some people in the node community that have adopted the "convention over configuration" approach in use by Ruby on Rails, EmberJS, and some other large projects. If you like that approach, check out tools that use it like SailsJS, EmberJS, Yeoman generators, etc.
But in general, looking for a "standard" is not how the node.js/javascript/web community rolls. Small npm packages. File layouts that are forced into obviousness due to smallness. I feel your frustration here as front-end toolchains are so complex, but ultimately it's because JavaScript in the browser took too many decades to create a reasonable module system. Things may start to standardize in the next few years now that ES6 modules are an official spec, but with so much code already written in CommonJS and it's terrible precursors like RequireJS/AMD, we'll be dealing with them probably for the foreseeable future.