Resizing point sprites based on distance from the camera

Question

I'm writing a clone of Wolfenstein 3D using only core OpenGL 3.3 for university and I've run into a bit of a problem with the sprites, namely getting them to scale correctly based on distance.

From what I can tell, previous versions of OGL would in fact do this for you, but that functionality has been removed, and all my attempts to reimplement it have resulted in complete failure.

My current implementation is passable at distances, not too shabby at mid range and bizzare at close range.

The main problem (I think) is that I have no understanding of the maths I'm using.
The target size of the sprite is slightly bigger than the viewport, so it should 'go out of the picture' as you get right up to it, but it doesn't. It gets smaller, and that's confusing me a lot.
I recorded a small video of this, in case words are not enough. (Mine is on the right)

Can anyone direct me to where I'm going wrong, and explain why?

Code:
C++

// setup
glPointParameteri(GL_POINT_SPRITE_COORD_ORIGIN, GL_LOWER_LEFT);
glEnable(GL_PROGRAM_POINT_SIZE);

// Drawing
glUseProgram(StaticsProg);
glBindVertexArray(statixVAO);
glUniformMatrix4fv(uStatixMVP, 1, GL_FALSE, glm::value_ptr(MVP));
glDrawArrays(GL_POINTS, 0, iNumSprites);

Vertex Shader

#version 330 core

layout(location = 0) in vec2 pos;
layout(location = 1) in int spriteNum_;

flat out int spriteNum;

uniform mat4 MVP;

const float constAtten  = 0.9;
const float linearAtten = 0.6;
const float quadAtten   = 0.001;

void main() {
    spriteNum = spriteNum_;
    gl_Position = MVP * vec4(pos.x + 1, pos.y, 0.5, 1); // Note: I have fiddled the MVP so that z is height rather than depth, since this is how I learned my vectors.
    float dist = distance(gl_Position, vec4(0,0,0,1));
    float attn = constAtten / ((1 + linearAtten * dist) * (1 + quadAtten * dist * dist));
    gl_PointSize = 768.0 * attn;
}

Fragment Shader

#version 330 core

flat in int spriteNum;

out vec4 color;

uniform sampler2DArray Sprites;

void main() {
    color = texture(Sprites, vec3(gl_PointCoord.s, gl_PointCoord.t, spriteNum));
    if (color.a < 0.2)
        discard;
}

Answer 1

First of all, I don't really understand why you use pos.x + 1.

Next, like Nathan said, you shouldn't use the clip-space point, but the eye-space point. This means you only use the modelview-transformed point (without projection) to compute the distance.

uniform mat4 MV;       //modelview matrix

vec3 eyePos = MV * vec4(pos.x, pos.y, 0.5, 1); 

Furthermore I don't completely understand your attenuation computation. At the moment a higher constAtten value means less attenuation. Why don't you just use the model that OpenGL's deprecated point parameters used:

float dist = length(eyePos);   //since the distance to (0,0,0) is just the length
float attn = inversesqrt(constAtten + linearAtten*dist + quadAtten*dist*dist);

EDIT: But in general I think this attenuation model is not a good way, because often you just want the sprite to keep its object space size, which you have quite to fiddle with the attenuation factors to achieve that I think.

A better way is to input its object space size and just compute the screen space size in pixels (which is what gl_PointSize actually is) based on that using the current view and projection setup:

uniform mat4 MV;                //modelview matrix
uniform mat4 P;                 //projection matrix
uniform float spriteWidth;      //object space width of sprite (maybe an per-vertex in)
uniform float screenWidth;      //screen width in pixels

vec4 eyePos = MV * vec4(pos.x, pos.y, 0.5, 1); 
vec4 projCorner = P * vec4(0.5*spriteWidth, 0.5*spriteWidth, eyePos.z, eyePos.w);
gl_PointSize = screenWidth * projCorner.x / projCorner.w;
gl_Position = P * eyePos;

This way the sprite always gets the size it would have when rendered as a textured quad with a width of spriteWidth.

EDIT: Of course you also should keep in mind the limitations of point sprites. A point sprite is clipped based of its center position. This means when its center moves out of the screen, the whole sprite disappears. With large sprites (like in your case, I think) this might really be a problem.

Therefore I would rather suggest you to use simple textured quads. This way you circumvent this whole attenuation problem, as the quads are just transformed like every other 3d object. You only need to implement the rotation toward the viewer, which can either be done on the CPU or in the vertex shader.