Need help with realtime local reflections

Posted By: Superku

Need help with realtime local reflections - 09/25/12 02:37

Inspired by this post
http://www.gamedev.net/blog/1323/entry-2254101-real-time-local-reflections/
I've written a GPU (depth) raytracer for screen space reflections on arbitrary surfaces that is surprisingly fast, even unoptimized (up to 1024fps on my 5870, depending on the amount of object shader pixels visible). However, I have a problem with the implementation that I cannot fix, the reflection image gets clinched when you decrease the distance to the surface:





The effect code:
Click to reveal..
Code:
float4x4 matWorldViewProj;
float4x4 matWorld;
float4x4 matViewProj;
float4x4 matView;
float4x4 matProj;
float4 vecViewPos;

texture bmp_camera_bmap;
texture bmp_depth_bmap;

sampler CameraSampler = sampler_state 
{ 
	Texture = <bmp_camera_bmap>; 
	AddressU  = Clamp; 
	AddressV  = Clamp; 
}; 

sampler DepthSampler = sampler_state 
{ 
	Texture = <bmp_depth_bmap>; 
	AddressU  = Clamp; 
	AddressV  = Clamp; 
}; 

// Vertex Shader: 
void ReflectVS( 
in float4 InPos: POSITION, 
in float3 InNormal: NORMAL, 
in float2 InTex: TEXCOORD0, 
out float4 OutPos: POSITION, 
out float2 OutTex: TEXCOORD0, 
out float3 OutNormal: TEXCOORD1,
out float4 ProjTex: TEXCOORD2,
out float4 OutPos2: TEXCOORD3) 
{ 
	OutPos = mul(InPos, matWorldViewProj);
	OutNormal = normalize(mul(InNormal, matWorld));
	OutTex = InTex; 
	ProjTex = OutPos;
	OutPos2 = mul(InPos, matWorld);
} 

// Pixel Shader: 
float4 ReflectPS( 
in float2 InTex: TEXCOORD0, 
in float3 InNormal: TEXCOORD1,
in float4 ProjTex: TEXCOORD2,
in float4 InPos: TEXCOORD3): COLOR 
{ 
	InNormal = normalize(InNormal);
	ProjTex.xy = ProjTex.xy/ProjTex.w;
	float3 projViewPos = ProjTex.xyz;
	ProjTex.xy = ProjTex.xy*0.5 + 0.5;
	ProjTex.y = 1-ProjTex.y;

	// calculate reflection vector in view space
	float3 viewpos = mul(InPos,matView).xyz;
	float3 dir = viewpos.xyz;
	dir = reflect(viewpos.xyz,mul(InNormal,matView));
	dir = normalize(dir);
	
	// transform from view to screen space
	float4 tPos = mul(float4(viewpos+dir,1), matProj);
	tPos.xy = tPos.xy/tPos.w;
	float3 screenDir = tPos.xyz-projViewPos;
	screenDir *= 0.05/length(screenDir.xy); // start with a rather big length and refine if necessary

	// transform from screen to texture coordinates
	float3 screenPos = projViewPos+screenDir;
	screenPos.xy = screenPos.xy*0.5 + 0.5;
	screenPos.y = 1-screenPos.y;
	screenDir.x *= 0.5;
	screenDir.y *= -0.5;

	float depth;
	int max_num = 30;
	int done = 0;
	float3 oldPos = ProjTex;
	for(int i = 0; i < max_num; i++)
	{
		depth = tex2D(DepthSampler, screenPos.xy).x;
		if(depth < screenPos.z) // refine search
		{
			screenPos = oldPos;
			screenDir *= 0.5;
			screenPos += screenDir;
			done++;
		}
		else
		{
			oldPos = screenPos.xyz;
			screenPos.xyz += screenDir;
		}
	}
	if(done < 1) return 0;
	return tex2D(CameraSampler, screenPos.xy);
} 

technique ReflectTechnique 
{ 
	pass P0 
	{ 
		VertexShader = compile vs_3_0 ReflectVS(); 
		PixelShader  = compile ps_3_0 ReflectPS(); 
	} 
}


Please help me and fix the code or tell me where the problem lies, I've tried everything (in return you get the source code for your own applications, of course).

Download the test scene from above: http://www.superku.de/reflect.zip
Thanks in advance for any help!
Posted By: Hummel

Re: Need help with realtime local reflections - 09/25/12 14:21

I'll take a look into this later.
Posted By: Superku

Re: Need help with realtime local reflections - 09/25/12 17:45

Great, thanks! :>
Posted By: Hummel

Re: Need help with realtime local reflections - 09/26/12 00:26

I wasn't yet able to solve your problem completely but at least I got a messy reference implementation running which does the raycasting in world space:

Code:
float4x4 matWorldViewProj;
float4x4 matWorld;
float4x4 matViewProj;
float4x4 matView;
float4x4 matProj;
float4 vecViewPos;
float4 vecViewDir;
float4 vecViewPort;

texture bmp_camera_bmap;
texture bmp_depth_bmap;

sampler CameraSampler = sampler_state 
{ 
	Texture = <bmp_camera_bmap>; 
	AddressU  = border; 
	AddressV  = border; 
}; 

sampler DepthSampler = sampler_state 
{ 
	Texture = <bmp_depth_bmap>; 
	AddressU  = Clamp; 
	AddressV  = Clamp; 
}; 

// Vertex Shader: 
void ReflectVS( 
in float4 InPos    : POSITION, 
in float3 InNormal : NORMAL, 
in float2 InTex    : TEXCOORD0, 

out float2 OutTex   : TEXCOORD0, 
out float3 OutNormal: TEXCOORD1,
out float3 wPos     : TEXCOORD2,
out float4 OutPos   : TEXCOORD3,
out float4 pPos     : POSITION

) 
{ 
	pPos = mul(InPos, matWorldViewProj);
	OutPos = pPos;
	
	OutNormal = normalize(mul(InNormal, matWorld));
	
	OutTex = InTex; 
	
	wPos = mul(InPos, matWorld);
} 

float2 calc_ScreenPos(float4 pPos)
{
	return (float2(pPos.x,-pPos.y)/pPos.w+vecViewPort.zw)*0.5+0.5;
}

// Pixel Shader: 
void ReflectPS( 
in float2 Tex    : TEXCOORD0, 
in float3 Normal : TEXCOORD1,
in float3 wPos   : TEXCOORD2,
in float4 pPos   : TEXCOORD3,

out float4 COL : COLOR0
)
{ 
	Normal = normalize(Normal);
	float3 View = normalize(vecViewPos.xyz - wPos);
	
	float3 R = -reflect(View, Normal);
	float3 sR = mul(R, matView);
	sR = R;
	
	float2 sTex = calc_ScreenPos(pPos);
	
	//--------------//
	const float srel = vecViewPort.y/vecViewPort.x;
	
	//	float2 PomOffDir=sR.xy*0.05;//*vecViewPort.zw;
	//	//	PomOffDir = sR.xy/-sR.z*0.1;
	//	PomOffDir.y *= -1.f;
	//	PomOffDir.y *= srel;
	
	//	float PomOff=IntersectionTest(DepthSampler, sTex, PomOffDir, 0.0f);
	//	sTex+=PomOffDir*PomOff;

	float3 screenPos = float3(sTex.xy, pPos.w);
	screenPos.xy = mul(float4(wPos, 1), matView).xy;
	screenPos = wPos;
	
	float3 screenDir = sR;

	//	screenDir.y *= -1.f;
	screenDir = normalize(screenDir);
	//	screenDir.xy /= abs(screenDir.z);

	const int step_count = 32;
	const float init_step_size = 20.f;
	
	int halve_again = 0;
	float step_size = init_step_size;
	float depth;
	float2 sUV;

	float3 oldPos = screenPos;
	float3 newPos = screenPos + screenDir * step_size;

	for(int i=0; i<step_count; ++i)
	{
		sUV = calc_ScreenPos(mul(float4(newPos, 1.f), matViewProj));
		depth = tex2Dlod(DepthSampler, float4(sUV,0,0)).x;
		
		if(dot(vecViewDir, newPos-vecViewPos.xyz) < depth)
		//		if(newPos.z < depth)
		{
			oldPos = newPos;
			
			if(halve_again)
			step_size *= 0.5f;
			halve_again = 0;
			
			newPos += screenDir * step_size;
		}
		else
		{
			step_size *= 0.5f;
			newPos = oldPos + screenDir * step_size;
			//			step_size *= 0.5f;
			halve_again = 1;
		}
	}

	sUV = calc_ScreenPos(mul(float4(newPos, 1.f), matViewProj));
	depth = tex2Dlod(DepthSampler, float4(sUV,0,0)).x;

	//	if(dot(vecViewDir, newPos-vecViewPos.xyz) < depth)
	//	if(newPos.z < depth)
	if(step_size == init_step_size)
	{
		COL = 0;
		return;	
	}
	//--------------//
	
	COL = tex2D(CameraSampler, sUV);
	
	//	COL.rgb = dot(vecViewDir, wPos-vecViewPos.xyz)*0.01f;
	//	COL = tex2D(CameraSampler, calc_ScreenPos(mul(float4(newPos, 1.f), matProj)));
	//	COL = tex2D(CameraSampler, calc_ScreenPos(mul(float4(screenPos, 1.f), matProj)));
	//		COL = tex2D(CameraSampler, sTex);
	//	COL = tex2D(DepthSampler, sTex).x*0.0005;
	//		COL = float4(sR.xy, 0, 1);
	//		COL.rgb = screenDir.z;
	//		COL.rgb = sR.z;
	//		COL.rgb = mul(float4(R,0), matViewProj).z;
	//		COL.rgb = abs(screenDir);
	//COL.rgb = mul(R, matView).z;
	//COL.rgb=mul(float4(wPos, 1), matView).y;
	//COL.rgb=pPos.w*0.01f;
	COL.a = 1;
} 

technique ReflectTechnique 
{ 
	pass P0 
	{ 
		VertexShader = compile vs_3_0 ReflectVS(); 
		PixelShader  = compile ps_3_0 ReflectPS(); 
	} 
}


Note that I changed the depth output in Depth.fx to linear depth.

Also, if you want to achieve multiple reflections don't forget to update your depthmap besides your colormap also.

Once the algo works right you should consider to add a quadtree like structure (min/max-depth mipmap pyramid) to your depthmap to speed up the raycasting process and make it more stable for extreme cases like long search distances and sudden changes in the depthmap. wink

Hope that helps.
Posted By: Superku

Re: Need help with realtime local reflections - 09/26/12 01:28

Thanks Hummel for your effort!
I forgot to mention that it already worked with a tracing in world coordinates but all the matrix multiplications inside the loop were way too slow on my notebook, that's why I switched to screen space tracing. My results looked pretty bad compared to your version, though (they can be seen in the WIP thread), so your effort hasn't been in vain.

Quote:
Note that I changed the depth output in Depth.fx to linear depth.

Hm what does that mean exactly? The issue with the clinched reflection has to result from something that is "not linear" (as one can easily see on the crates) and I've always doubt that my approach will work because of non-linearity somewhere, but the source I've linked in my first post proves me wrong.
I don't achieve linear depth when I remove the multiplication with the Projection matrix, do I?

Multiple reflections and the like have very low priority, but thanks for the hints.
Posted By: Hummel

Re: Need help with realtime local reflections - 09/26/12 02:44

Quote:
I forgot to mention that it already worked with a tracing in world coordinates.
I actually expected that. laugh

Quote:
I don't achieve linear depth when I remove the multiplication with the Projection matrix, do I?
You do. When being in WorldView-space the z component is already linear depth. Also you can achieve linear depth in world space as I did in the shader for depth comparison: dot(vecViewDir, wPos-vecViewPos.xyz). After transforming a vertex position to WorldViewProj-space you can find it's linear depth in the w element, z contains the non-linear depth which is used for z-testing. In the end it's a matter of precision distribution, the non-linear value offers more precision close to the near-plane, but since my reference implementation works in world space it was easier to work with linear depth. In other words all I changed in Depth.fx was "OutDepth = OutPos.z;" to "OutDepth = OutPos.w;".

In the end it should work in screen space also. I'll see what I can do.
Posted By: Superku

Re: Need help with realtime local reflections - 09/26/12 03:16

Thank you for the explanation.
It would be great if you could try to find the reason for the problem, I couldn't do it for 4 days.
Posted By: Hummel

Re: Need help with realtime local reflections - 09/26/12 21:38

Ok, I found a suitable solution which only requires a multiplication to get the screen space texcoords in the loop. I'll clean up and post the code later. Perhaps I'll even be able to take the last step to pure ss-texcoords in the meantime...
Posted By: Superku

Re: Need help with realtime local reflections - 09/26/12 22:03

Cool, I'm hoping to see a pure screenspace solution (or at least I want to know where the mistake in my implementation is)! Thanks again.
Posted By: Hummel

Re: Need help with realtime local reflections - 09/27/12 00:23

Still no luck, also it's actually a division not a multiplication in the loop. However:
Quote:
float4x4 matWorldViewProj;
float4x4 matWorld;
float4x4 matViewProj;
float4x4 matView;
float4x4 matProj;
float4 vecViewPos;
float4 vecViewDir;
float4 vecViewPort;

texture bmp_camera_bmap;
texture bmp_depth_bmap;

sampler CameraSampler = sampler_state
{
Texture = <bmp_camera_bmap>;
AddressU = border;
AddressV = border;
};

sampler DepthSampler = sampler_state
{
Texture = <bmp_depth_bmap>;
AddressU = Clamp;
AddressV = Clamp;
};

// Vertex Shader:
void ReflectVS(
in float4 InPos : POSITION,
in float3 InNormal : NORMAL,
in float2 InTex : TEXCOORD0,

out float2 OutTex : TEXCOORD0,
out float3 OutNormal: TEXCOORD1,
out float3 wPos : TEXCOORD2,
out float4 OutPos : TEXCOORD3,
out float4 pPos : POSITION

)
{
pPos = mul(InPos, matWorldViewProj);
OutPos = pPos;

OutNormal = normalize(mul(InNormal, matWorld));

OutTex = InTex;

wPos = mul(InPos, matWorld);
}

float2 calc_ScreenPos(float4 pPos)
{
return (float2(pPos.x,-pPos.y)/pPos.w+vecViewPort.zw)*0.5+0.5;
}

float2 calc_ScreenPos(float3 pPos)
{
return (float2(pPos.x,-pPos.y)/pPos.z+vecViewPort.zw)*0.5+0.5;
}

float2 calc_ScreenPos(float2 pPos_xy, float depth)
{
return (float2(pPos_xy.x,-pPos_xy.y)/depth+vecViewPort.zw)*0.5+0.5;
}

// Pixel Shader:
void ReflectPS(
in float2 Tex : TEXCOORD0,
in float3 Normal : TEXCOORD1,
in float3 wPos : TEXCOORD2,
in float4 pPos : TEXCOORD3,

out float4 COL : COLOR0
)
{
// Normal = float3(0,1,0);

float2 sTex = calc_ScreenPos(pPos);
float3 screenPos = float3(sTex.xy*pPos.w, pPos.w);
// screenPos = float3(sTex.xy, pPos.w);

Normal = normalize(Normal);
float3 View = normalize(vecViewPos.xyz - wPos);

float3 R = Normal * dot(Normal, View) * 2.f - View;

//--------------//

float3 sR;
sR = mul(R, matViewProj).xyz;
// sR = mul(float4(R,1), matViewProj).xyz;
sR.xy = calc_ScreenPos(sR.xyz) * sR.z;
// sR.xy = calc_ScreenPos(sR.xyz);
// sR.xy/=abs(sR.z);
// sR.y *= -1;

float3 screenDir = (sR.xyz);
screenDir = normalize(sR.xyz);
// screenDir = sR.xyz;
// screenDir.xy /= sR.z;
// screenDir.y *= -1.f;

float3 ori_screenDir = screenDir;

//-------------------------------------//

const int step_count = 32;
const float init_step_size = 40.f;
// const float init_step_size = 0.1f;

float halve_again = 1.f;

float step_size = init_step_size;
float depth;
float2 sUV;

float3 oldPos = screenPos;
float3 newPos = screenPos + screenDir * step_size;

for(int i=0; i<step_count; ++i)
{
sUV = newPos.xy / newPos.z;
// sUV = newPos.xy;

depth = tex2Dlod(DepthSampler, float4(sUV,0,0)).x;

if(newPos.z < depth)
{
oldPos = newPos;

step_size *= halve_again;
halve_again = 1.f;

newPos += screenDir * step_size;
}
else
{
step_size *= 0.5f;
newPos = oldPos + screenDir * step_size;

halve_again = 0.5f;
}
}

sUV = newPos.xy / newPos.z;
// sUV = newPos.xy;

//-------------------------------------//
COL.rgb = ori_screenDir.x;
COL.rgb = (abs(sR.x/sR.z-0)<0.0025);
COL.rgb = -sR.y;
// COL.rgb = sR.z;

if(step_size.x == init_step_size.x)
{
COL = 0;
return;
}

COL = tex2D(CameraSampler, sUV);

//--------------//

COL.a = 1;
}

technique ReflectTechnique
{
pass P0
{
VertexShader = compile vs_3_0 ReflectVS();
PixelShader = compile ps_3_0 ReflectPS();
}
}

It's bugging me since it really shouldn't be that hard to get it running.
Posted By: Superku

Re: Need help with realtime local reflections - 09/27/12 01:13

Well this is an almost perfect result, great work!
Still, I'm buffled too why I don't get the screenspace implementation to work. Have you had a look at the source code and the example from the Litheon guy from gamedev.net and compared it with my code from my first post?
Posted By: Hummel

Re: Need help with realtime local reflections - 09/27/12 13:48

I had a quick look but haven't compared it to yours. However, I've noticed that he is using screen space derivatives (ddx/ddy) somehow.
Posted By: Superku

Re: Need help with realtime local reflections - 09/27/12 14:03

Yes, but he only uses them for a gradient mip map selection.
I think an interesting part in the source code is the following:

Code:
// Transform the View Space Reflection to Screen Space
   // This because we want to ray march in to the depth buffer in Screen Space (thus you can use the default hardware depthbuffer)
   // Depth is linear in Screen Space per Screen Pixel
   float3 vspPosReflect = Input.ViewPos +  vspReflect;
   float3 sspPosReflect = mul(float4(vspPosReflect, 1.0), g_mProj).xyz / vspPosReflect.z;
   float3 sspReflect = sspPosReflect - Input.ScreenPos;


Two thoughts come to my mind: How does his projection matrix look like and esp. why does he divide by vspPosReflect.z (including the z-component of ssPosReflect)?
I had to adapt my code as follows to get somehow passable results:

Code:
// transform from view to screen space
	float4 tPos = mul(float4(viewpos+dir,1), matProj);
	tPos.xy = tPos.xy/tPos.w;
	float3 screenDir = tPos.xyz-projViewPos;

Posted By: Hummel

Re: Need help with realtime local reflections - 09/27/12 14:56

He is just dividing by linear depth since vspPosReflect is in view space. He could have also done the following:
Code:
float3 vspPosReflect = Input.ViewPos +  vspReflect;
   float4 sspPosReflect = mul(float4(vspPosReflect, 1.0), g_mProj).xyzw;
sspPosReflect.xyz /= sspPosReflect.w;
   float3 sspReflect = sspPosReflect.xyz - Input.ScreenPos;

And since Input.ScreenPos uses non-linear depth in the z-value too they cancel out each other.
I tried something alike (I didn't divide the z value, though...) but ended up with the same problem like you. So the magic seems to hide somewhere else.
Posted By: Kartoffel

Re: Need help with realtime local reflections - 09/30/12 16:17

I've got a simple suggestion to improve the quality (you see it on the right side):



I changed the line return tex2D(CameraSampler, screenPos.xy); in reflect.fx to
Code:
float Intensity = saturate(1.0f - distance(screenPos.xy, float2(0.5f, 0.5f)) * 2.0f);
Intensity = Intensity * (2.0f - Intensity);
return tex2D(CameraSampler, screenPos.xy) * Intensity;

Posted By: Superku

Re: Need help with realtime local reflections - 09/30/12 16:22

Thank you for the suggestion. The real shader of course blends out the reflection smoothly where no data is available, I've posted only the relevant part for the calculation of the reflection.
Posted By: Kartoffel

Re: Need help with realtime local reflections - 09/30/12 16:28

oh, ok smile
© 2024 lite-C Forums