From GameStudio Wiki

This effect avoids the nasty pixelated borders which happen with overlay or d3d_autotransparency. (the WavingGrass shader does the same thing with a pixelshader.)

you have to use a 32bit tga texture. all pixels with alpha values above the treshold value (the first value of the vector in the starter function) will be rendered completely opaque and all other pixels will be rendered completely transparent. because there won't be any semi-transparent pixels then it's possible to enable the z-buffer, so you won't have any sorting errors. the advantage over using 3dgs' overlay flag is that there will be no ugly pixelated border because of the way the values in the alpha channel get interpolated.

entities using this effect should have my.flare=off; and my.transparent=off; so that the engine won't unnecessarily depth sort them because of the alpha channel!

with the cullmode variable you can specify if the polygons should be rendered two sided. (-> [cullmode|http://msdn.microsoft.com/archive/en-us/dx81_c/directx_cpp/GRAPHICS/Reference/CPP/D3D/Enums/D3DCULL.asp])

function mtl_vegetation_init
{
	vec_set(mtl.emissive_blue,mat_model.emissive_blue);
	vec_set(mtl.ambient_blue,mat_model.ambient_blue);
	vec_set(mtl.diffuse_blue,mat_model.diffuse_blue);
	vec_set(mtl.specular_blue,mat_model.specular_blue);
	mtl.power=mat_model.power;
	mtl.albedo=mat_model.albedo;
	mtl.skill1=pixel_for_vec(vector(128,0,0),0,8888); // the first value in the vector is the threshold
}

material mtl_vegetation
{
	event=mtl_vegetation_init;
	effect=
	"
	texture entSkin1;
	dword mtlSkill1;

	technique vegetation
	{
		pass p0
		{
			Texture[0]=<entSkin1>;
			ZWriteEnable=True;
			AlphaBlendEnable=False;
			AlphaTestEnable=True;
			AlphaRef=<mtlSkill1>;
			AlphaFunc=Greater;
			CullMode=CCW; // CCW or None

			ColorArg1[0]=Texture;
			ColorOp[0]=Modulate2X;
			ColorArg2[0]=Diffuse;
		}
	}
	technique fallback{pass p0{}}
	";
}

action vegetation
{
	my.transparent=off;
	my.flare=off;
	my.material=mtl_vegetation;
}

attach the action <vegetation> to your vegetation entities. experiment with the threshold value and the smoothness of the borders in your alpha channels. if you use double sided geometry (~CullMode=None;) the lighting often doesn't look correct. a material which doesn't react to any dynamic lights or a vertex shader which does double sided lighting would fix this.

Shorter Version (without adjustable threshold)

material mat_alphatest
{
  effect =
  "
    technique alpha_test
    {
      pass p0
      {
        zWriteEnable = true;
        alphaTestEnable = true;
        alphaBlendEnable = false;
      }
    }
  ";
}

action apply_alphatest_material
{
  my.material = mat_alphatest;
}