I think vec_rotate is not needed when using vec_for_screen. If I use it the ray tracer will rotate away from where I'm actually pointing. I tried your code rayp but I doesn't make any difference, there is still no hit.vertex returned.

@3run, Thanks for pointing that out to me man. I was able to fix it by using ent_nextvertex. I need to keep the GS manual open all the time for the list of all the functions.
new code.
Code:
var vert; //to store vertex index next to hit position
void camera_scan()
{
	VECTOR tempv,temp;
	while(1)
	{
		if(mouse_left) 
		{
			tempv.x = mouse_pos.x;
			tempv.y = mouse_pos.y;
			tempv.z = 2000;
			vec_for_screen(tempv,camera);
			c_trace(camera.x, tempv, IGNORE_ME | SCAN_TEXTURE);
			
			if(HIT_TARGET) //needed to prevent crashing when not hitting something :P
			{
				
				vert = ent_nextvertex(hit.entity,hit.x);
				vec_for_vertex(temp,hit.entity,vert);
				
				//for debugging, create a point and line from the hit  position to the nearest index
				draw_point3d(temp.x,vector(0,0,255),100,300);
				draw_point3d(vector(hit.x,hit.y,hit.z),vector(0,0,255),100,300);
				draw_line3d(vector(hit.x,hit.y,hit.z),vector(0,0,255),100);
				draw_line3d(temp.x,vector(0,0,255),100);
			}
		}
		wait(1);
	}
}



thanks for helping me out rayp and malice.
Now I just have to figure how to get all the the vertex near the hit position at a given radius without looping through each indices. I use Markus's real time terrain deforming library in A7 but it no longer work for A8 so I have to make my own function for terraform.

Last edited by JerahYalyn; 04/24/15 04:19.