Infinite Procedural Terrain Engine

Return to Showcase

Infinite Procedural Terrain Engine

Postby AnimateDream » Tue Oct 26, 2010 1:19 pm

For details about the engine, videos, screenshots, or a playable browser demo, see its new web page.

There's a more complex terrain engine made by Craig here. It's modular with its tile sources and renderers. It also has some great performance techniques. Unfortunately it doesn't yet have anywhere near the visual quality of mine. We have actually discussed combining our efforts into one engine, but that process has just started. Perhaps I am just too much of a python and shader noob, but I find his engine very difficult to follow the structure and flow of. My project is I believe very straightforward. Anyways if you need a terrain engine I really recommend looking both over.
Last edited by AnimateDream on Tue Aug 30, 2011 11:28 pm, edited 9 times in total.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Re: perlin noise terrain algorithm + shader

Postby AnimateDream » Mon Nov 08, 2010 3:37 pm

deleted
Last edited by AnimateDream on Thu Mar 03, 2011 8:46 am, edited 1 time in total.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Postby AnimateDream » Sun Dec 05, 2010 7:45 pm

Progress is still continuing on the geomipmap terrain. I discovered that the normal data being recieved by the shader, while incorrect, is actually related to the correct values. By multiplying the x and y component of the normal by around -500 each I can get a very rough approximation of what the normal should be. Why this is necessary I have no idea.This has partially fixed terrain lighting and perhaps it will allow me to get my shader that textures based on slope partially working as well. Anyways I have a couple screenshots of the current state of the terrain for those interested.

http://stephenlujan.com/Panda3D/terrain1.jpg

http://stephenlujan.com/Panda3D/terrain2.jpg[/url]
Last edited by AnimateDream on Sat Dec 18, 2010 12:22 am, edited 3 times in total.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Postby rdb » Mon Dec 06, 2010 3:33 am

Just a hint: I advise using texture filtering like mipmapping for your terrain textures, that will smooth out the pixely artifacts in the distance.
rdb
 
Posts: 9482
Joined: Mon Dec 04, 2006 5:58 am
Location: Netherlands

Postby AnimateDream » Tue Dec 07, 2010 5:41 pm

Thanks rdb. I completely forgot about that. I have newer screenshots now with better texture filtering.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

something wrong?

Postby Eraslt » Wed Dec 08, 2010 1:40 am

i get very dark terrain, what might be wrong?
Image
Eraslt
 
Posts: 47
Joined: Tue Dec 08, 2009 2:29 pm
Location: Lithuania

Postby rdb » Wed Dec 08, 2010 3:51 am

You're using an old version of Panda3D, that doesn't support custom parameters without the k_ prefix.
rdb
 
Posts: 9482
Joined: Mon Dec 04, 2006 5:58 am
Location: Netherlands

Re: perlin noise terrain algorithm + shader

Postby darthrigg » Wed Dec 08, 2010 5:41 pm

AnimateDream wrote:So I've evolved my brief demo into an infinite procedural terrain engine here:
https://github.com/AnimateDream/Panda-3d-Procedural-Terrain-Engine
I'm very happy with the terrain quality, but the performance is a bit slow. It runs ok now with heavy use of threading, but panda3d only uses simple threads unless you rebuild the core to use true threads.

There's a more sophisticated terrain engine made by Craig here. Actually its two terrain engines. A geomipmap engine and a geoclipmap engine. The former utilizes the cpu more and the later calculates geometry right on the gpu. The geoclipmap wouldn't run on my radeon hd 3870, and I found the geomipmap terrain rather ugly personally, though its probably mostly related to the textures used. Also I found it rather difficult to understand, but that may just be me. My engine is still simpler, and hopefully sufficiently commented. Anyways if you need a terrain engine I really recommend looking both over.


The .7z post/link works. But when I cloned via git... I get errors for missing files:

:grutil(error): No valid heightfield image has been set!
:grutil(error): Failed to load heightfield image heightmaps/ID123X0.0Y128.0.png!
:grutil(error): No valid heightfield image has been set!
tile generated at 0.0 128.0
:grutil(error): Failed to load heightfield image heightmaps/ID123X128.0Y0.0.png!

Cheers,
Gary
User avatar
darthrigg
 
Posts: 160
Joined: Sat Oct 30, 2010 3:20 pm

Re: perlin noise terrain algorithm + shader

Postby Craig » Thu Dec 09, 2010 2:03 am

AnimateDream wrote:There's a more sophisticated terrain engine made by Craig here. Actually its two terrain engines. A geomipmap engine and a geoclipmap engine. The former utilizes the cpu more and the later calculates geometry right on the gpu. The geoclipmap wouldn't run on my radeon hd 3870, and I found the geomipmap terrain rather ugly personally, though its probably mostly related to the textures used. Also I found it rather difficult to understand, but that may just be me. My engine is still simpler, and hopefully sufficiently commented. Anyways if you need a terrain engine I really recommend looking both over.


Thanks for mentioning my setup. Just to clarify, my terrain system includes 2 renderers for the same underlying tile based system. The geoclipmap based one is very unfinished, and the geomip one is just trivial and poor use of Panda's existing terrain stuff as tiles.

The system has 2 parts, the renderer, and the tile source/bakery. The intention it to be able to use any renderer with any tile source. Currently there is only one tile source, and it is a very confusing, complicated and experimental one. As I'm a coder not an artist, I made the a tile source that procedurally generates terrain. As an experiment, I tried doing this fully on the GPU with a series of shaders (called maps) defined in a custom file format. It does work, but it has some huge performance issues (unnecessary copying of all intermediate steps to ram I believe). Regardless, its pretty quick, but horribly confusing and not very useful in practice.

With actual artist produced maps (or a decent map generator) and artist produced textures (ok, mine are from an artist, but they are for the wrong zoom level and taken from another project), and a decent renderer, it could be nice. Its missing a major feature: the ability to place meshes on the terrain, but its in the design, just not done.


As for passing params from vsahder to fshader, there is a comment here in my shader generator code that explains the issue (and there is code there that deals with it in the confusing context of my shader generator): https://github.com/Craig-Macomber/Panda3D-Shader-Generator/tree/master/shadereffects#L99

Edit: AnimateDream, is that project BSD licensed? AKA: can I use stuff from it, such as your shader and textures, and perhaps height map generator in my Terrain system which is BSD licensed? My terrain system would benefit from another tile source and another renderer (or an improvement to my geomip one). As my terrain system uses interchangeable tile sources and renderers, things get way more interesting when I have more than one of each that actually works.
Craig
 
Posts: 330
Joined: Thu Jul 02, 2009 8:55 pm

Re: perlin noise terrain algorithm + shader

Postby AnimateDream » Sat Dec 11, 2010 12:21 am

Craig wrote:Thanks for mentioning my setup. Just to clarify, my terrain system includes 2 renderers for the same underlying tile based system. The geoclipmap based one is very unfinished, and the geomip one is just trivial and poor use of Panda's existing terrain stuff as tiles.

The system has 2 parts, the renderer, and the tile source/bakery. The intention it to be able to use any renderer with any tile source. Currently there is only one tile source, and it is a very confusing, complicated and experimental one. As I'm a coder not an artist, I made the a tile source that procedurally generates terrain. As an experiment, I tried doing this fully on the GPU with a series of shaders (called maps) defined in a custom file format. It does work, but it has some huge performance issues (unnecessary copying of all intermediate steps to ram I believe). Regardless, its pretty quick, but horribly confusing and not very useful in practice.

With actual artist produced maps (or a decent map generator) and artist produced textures (ok, mine are from an artist, but they are for the wrong zoom level and taken from another project), and a decent renderer, it could be nice. Its missing a major feature: the ability to place meshes on the terrain, but its in the design, just not done.


Craig! Pleasure to see you've noticed my little tinker toy. It would be totally remiss of me not to mention any similar projects. Especially when my terrain engine is very immature. I just started the thread to provide some insight into how one might use panda3d's technology to start creating their own terrain system, and when I inevitably did I just decided to dump it here too.

Your post actually clarified a lot of things for me. I'll have to look over your code again with that in mind.

Craig wrote:As for passing params from vsahder to fshader, there is a comment here in my shader generator code that explains the issue (and there is code there that deals with it in the confusing context of my shader generator): https://github.com/Craig-Macomber/Panda3D-Shader-Generator/tree/master/shadereffects#L99

Ahhh. The whole project is pretty brilliant. I'm looking at your comment now and trying to digest it. This project is my first foray into shader languages, I wasn't anticipating this kind of resistance. I expected it to be no more finicky than C. If I understand you correctly shader output and input parameters can get mismatched unless you use a semantic? I need to figure out how to use semantics on the arbitrary float values I'm passing between them then.

Craig wrote:Edit: AnimateDream, is that project BSD licensed? AKA: can I use stuff from it, such as your shader and textures, and perhaps height map generator in my Terrain system which is BSD licensed? My terrain system would benefit from another tile source and another renderer (or an improvement to my geomip one). As my terrain system uses interchangeable tile sources and renderers, things get way more interesting when I have more than one of each that actually works.


I've been putting off picking a license, but I want a generally unrestrictive one. BSD sounds good. The textures I used are not mine and there are a couple unused shaders written by others I used for reference. I obviously have to exclude those from the license.

Feel free to cannabalize any and all parts of this project. I'm a fan of collaboration. I really just want a terrain implementation in panda that can provide infinite procedural terrain. I'd rather teach my engine how to make good maps than have to make them all by hand.

My project is pretty straightforward to understand I hope, but in its current state its so slow its almost broken. It almost requires you rebuild panda3d with true threads instead of simple threads to avoid serious stalls while its building or updating terrain. I know what to do to fix the performance, I'm just too busy being baffled by simple shaders refusing to work for me.
Last edited by AnimateDream on Sat Dec 11, 2010 4:08 am, edited 1 time in total.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Re: perlin noise terrain algorithm + shader

Postby Craig » Sat Dec 11, 2010 12:55 am

AnimateDream wrote:
My project is pretty straightforward to understand I hope, but in its current state its so slow its almost broken. It almost requires you rebuild panda3d with true threads instead of simple threads to avoid serious stalls while its building or updating terrain. I know what to do to fix the performance, I'm just too busy being baffled by simple shaders refusing to work for me.


Ah, performance issues. For panda's geoMipTerrain, I gave up on updates, and just make it brute force. I use rather small tiles to reduce generation stalls, and I generate the height map data on the GPU in multiple passes spread over several frames. This spreading adds a lot of code complexity, but it well worth it. No threading :)

I'm considering adding a third renderer to replace the geoMipTerrain based one with my own code so I can split the generation over a few frames, and optimize it for me usage. I'd do this in Cython for full compiled speed. I think there is a simpler height map class I should try first though.


The shader semantics tell the GPU which register to store the value in.
Ex:
float4 vTexCoords : TEXCOORD0;
float4 something : COLOR2;
just choose one, I usually use TEXCOORD# or COLOR# for random stuff.
Craig
 
Posts: 330
Joined: Thu Jul 02, 2009 8:55 pm

Re: perlin noise terrain algorithm + shader

Postby AnimateDream » Sat Dec 11, 2010 1:36 am

Craig wrote:Ah, performance issues. For panda's geoMipTerrain, I gave up on updates, and just make it brute force. I use rather small tiles to reduce generation stalls, and I generate the height map data on the GPU in multiple passes spread over several frames. This spreading adds a lot of code complexity, but it well worth it. No threading :)

Actually my system is setup to use either of two LOD techniques. Currently it uses brute force, but it still updates because tiles are loaded with a maxed out tile size, but nearby tiles have their tilesize set smaller which forces them to update. Using threading this doesn't really create noticeable lag on my computer.

The initial tile generation is still a killer though. Its set to use small tiles and a task will load them up in another thread one at a time, but it doesn't build them fast enough to let the player move very quickly without exposing unfinished terrain. Also it creates horrible stalls when there are a lot of tiles to build. I'm not sure yet why the threading isn't solving that.

Craig wrote:I'm considering adding a third renderer to replace the geoMipTerrain based one with my own code so I can split the generation over a few frames, and optimize it for me usage. I'd do this in Cython for full compiled speed. I think there is a simpler height map class I should try first though.


Panda's whole heightmap image thing isn't too practical for procedural terrain, but even when I cache the images on the drive and reuse them in subsequent runs it doesn't seem to yield much of a performance gain. I would be happy to hear if you found an alternative.

Craig wrote:The shader semantics tell the GPU which register to store the value in.
Ex:
float4 vTexCoords : TEXCOORD0;
float4 something : COLOR2;
just choose one, I usually use TEXCOORD# or COLOR# for random stuff.


Well i tried making them to uniforms and making my own semantics for them in panda and that apparently resulted in my fshader getting 0's on all the values. I suppose I'll pack them into some unused register... it just goes against all of my programmer instincts.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Re: perlin noise terrain algorithm + shader

Postby Craig » Sat Dec 11, 2010 1:46 am

AnimateDream wrote:Well i tried making them to uniforms and making my own semantics for them in panda and that apparently resulted in my fshader getting 0's on all the values. I suppose I'll pack them into some unused register... it just goes against all of my programmer instincts.


uniforms are uniform (meaning the same everywhere). Shader inputs for the model, from panda are uniform. Stuff thats per vertex can't be.
Craig
 
Posts: 330
Joined: Thu Jul 02, 2009 8:55 pm

Re: perlin noise terrain algorithm + shader

Postby AnimateDream » Sat Dec 11, 2010 2:55 am

Craig wrote:uniforms are uniform (meaning the same everywhere). Shader inputs for the model, from panda are uniform. Stuff thats per vertex can't be.


Ah. I had wondered about their name. Anyways I'm following your suggestions but my fshader is getting correct brightness values, but not the height of the terrain. Oddly it used to get the correct terrain height before I started messing with the semantics. I'll just post the whole shader here.
EDIT: updated code
Code: Select all
struct vfconn
{
    float2 l_texcoord0 : TEXCOORD0;
    float2 l_texcoord3 : TEXCOORD3;
    float4 l_position  : POSITION;
    float2 l_slope_brightness: TEXCOORD2;
    float3 l_mpos;//      : FOG;
    //float3 l_normal;
};

vfconn vshader( in float4 vtx_position : POSITION,
         in float3 vtx_normal : NORMAL,
              in float2 vtx_texcoord0 : TEXCOORD0,
              in float2 vtx_texcoord3 : TEXCOORD3,
              in uniform float4x4 mat_modelproj,
         in uniform float4x4 trans_model_to_world,
         in uniform float4 k_lightvec,
         in uniform float4 k_lightcolor,
         in uniform float4 k_ambientlight,
         in uniform float4 k_tscale
          //out vfconn OUT
            )
{
    vfconn OUT;

    OUT.l_position=mul(mat_modelproj,vtx_position);
    OUT.l_texcoord0=vtx_texcoord0*k_tscale;
    OUT.l_texcoord3=vtx_texcoord3;

    // worldspace position, for clipping in the fragment shader
    OUT.l_mpos = mul(trans_model_to_world, vtx_position);

    // lighting
    //WTF IS THIS NECESSARY
    vtx_normal.x *= -400;
    vtx_normal.y *= -400;
    //k_lightvec.z /= 400;
    float3 N = normalize( vtx_normal );
    float3 L = normalize( k_lightvec.xyz );

    float3 UP = float3(0,0,1);
    OUT.l_slope_brightness.x = 1.0 - dot( N, UP );

    OUT.l_slope_brightness.y = (max( dot( -N, L ), 0.0f )*k_lightcolor)+k_ambientlight;
    //OUT.l_normal = N;
    return OUT;
}

float calculateWeight( float value, float max, float min )
{
    if (value > max)
        return 0.0;
    if (value < min)
        return 0.0;

    //return 1.0;

    float weight = 0.0;

    weight = value - min < max - value ?
             value - min : max - value;

    //weight /= max - min;
    //weight *= weight;
    //weight = log2( weight );
    //weight = sqrt( weight );

    weight+= 0.001;
    //weight = clamp(weight, 0.001, 1.0);
    return weight;
}

float calculateFinalWeight( float height, float slope, float4 limits )
{
    return calculateWeight(height, limits.x, limits.y)
           //* calculateWeight(0.4, limits.z, limits.a);
           * calculateWeight(slope, limits.z, limits.a);
}

void fshader( in  vfconn IN,
              in uniform float4 region1Limits : REGION1LIMITS,
              in uniform float4 region2Limits : REGION2LIMITS,
              in uniform float4 region3Limits : REGION3LIMITS,
              in uniform float4 region4Limits : REGION4LIMITS,
              in uniform float4 k_waterlevel  : WATERLEVEL,
              in uniform sampler2D region1ColorMap : TEXUNIT0,
              in uniform sampler2D region2ColorMap : TEXUNIT1,
              in uniform sampler2D region3ColorMap : TEXUNIT2,
              in uniform sampler2D region4ColorMap : TEXUNIT3,
              in uniform sampler2D detailTexture   : TEXUNIT4,
              out float4 o_color : COLOR )
{
    // clipping
    //if ( IN.l_mpos.z < k_waterlevel.z) discard;

    //unpack some input
    // 0 = horizontal 1 = vertical
    float slope = IN.l_slope_brightness.x; //0.45;
    float brightness = IN.l_slope_brightness.y;
    //float slope = abs( l_slope );
    //float3 vertical = normalize(float3(0.0, 0.0, 1.0));
    //float slope =  1 - abs( dot( normalize(IN.normal), vertical) );
    float height = IN.l_mpos.z;

    vec4 weights = float4(0.0, 0.0, 0.0, 0.0);
    vec4 terrainColor = float4(0.0, 0.0, 0.0, 1.0);

    weights.x = calculateFinalWeight(height, slope, region1Limits);
    weights.y = calculateFinalWeight(height, slope, region2Limits);
    weights.z = calculateFinalWeight(height, slope, region3Limits);
    weights.a = calculateFinalWeight(height, slope, region4Limits);

    //--- Color terrain proportionately to weights
    float normalizer = (weights.x + weights.y + weights.z + weights.a + 0.000001);

    if (weights.x)
        terrainColor += weights.x / normalizer * tex2D(region1ColorMap, IN.l_texcoord0);
    if (weights.y)
        terrainColor += weights.y / normalizer * tex2D(region2ColorMap, IN.l_texcoord0);
    if (weights.z)
        terrainColor += weights.z / normalizer * tex2D(region3ColorMap, IN.l_texcoord0);
    if (weights.a)
        terrainColor += weights.a / normalizer * tex2D(region4ColorMap, IN.l_texcoord0);

    // detail texture
    float2 detailTexCoord= IN.l_texcoord0*8.0;
    terrainColor*= tex2D(detailTexture, detailTexCoord);
    // alpha splatting and lighting
    o_color=terrainColor*(brightness);
    //HDRL
    o_color = (o_color*o_color + o_color) / (o_color*o_color + o_color + float4(1.0, 1.0, 1.0, 1.0));
    o_color.a=1.0;
}


Once this is working I'm certain my terrain will look great. I guess I've gotten ahead of myself in the shader learning process though.
Last edited by AnimateDream on Sat Dec 11, 2010 3:54 am, edited 1 time in total.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Postby Craig » Sat Dec 11, 2010 3:21 am

Here is an example GC shader:
https://github.com/Craig-Macomber/Panda3D-Terrain-System/blob/master/geoClip.sha
A bit more complex on is here:
https://github.com/Craig-Macomber/Panda3D-Terrain-System/blob/master/geoClipGrass.sha

Those pass heights and normals from the vshader to the fshader.

You should only use standard semantics, and for uniforms that are given names via set shader input methods don't need them.

Also, you don't want to use if statements if you can avoid it. Conditionals are generally slow for shaders.

You might find the lerp function handy. I don't see you using it anywhere, its the common way to smoothly interpolate between different colors/positions etc. http://http.developer.nvidia.com/CgTutorial/cg_tutorial_appendix_e.html )

I've never worked with structs in shaders. I never needed them, so I don't really know how they work. Seemed like needless complexity to me.

I learned shaders mostly by guess and check. I know pretty much only one way to do them with a minimal feature set.
Craig
 
Posts: 330
Joined: Thu Jul 02, 2009 8:55 pm

Postby AnimateDream » Sat Dec 11, 2010 3:31 am

Thank you greatly for all of your suggestions Craig. I just chopped off the semantics for the world position and left it for the brightness and slope. Its ugly code, but it will do for now because the results are anything but ugly. :D

Image

Image

Also my shader can blend together an unlimited number of textures at the same vertex now. I fear I wouldn't figure out how to achieve the same effect using lerps.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Postby Craig » Sat Dec 11, 2010 3:53 am

AnimateDream wrote:
Also my shader can blend together an unlimited number of textures at the same vertex now. I fear I wouldn't figure out how to achieve the same effect using lerps.


Looks nice. You can blend unlimited textures using lerps. General approach is to start with a base texture, and lerp things into it one after another. This is equivalent to stacking alpha masked textures, and also lets you easily blend alpha masked textures (multiply the factor given to lerp by the alpha component). Its also possible to use the alpha to make different parts of the texture fad in first (so the grass blades are there or not rather than the texture fading in, or making nice transitions from scattered rocks to solidly rocky. Translucent rocks just look bad)

Anyway, looks quite nice. Well done.
Last edited by Craig on Sun Dec 12, 2010 10:10 pm, edited 2 times in total.
Craig
 
Posts: 330
Joined: Thu Jul 02, 2009 8:55 pm

Postby Craig » Sun Dec 12, 2010 9:55 pm

I tried to run it today, and got this output:

CraigsBook:src craigmacomber$ python main.py
Hello World
DirectStart: Starting the game.
Known pipe types:
osxGraphicsPipe
(all display modules loaded.)
Sun Dec 12 18:46:11 CraigsBook.local Python[6505] <Error>: kCGErrorIllegalArgument: CGSCopyRegion : Null pointer
Sun Dec 12 18:46:11 CraigsBook.local Python[6505] <Error>: kCGErrorFailure: Set a breakpoint @ CGErrorBreakpoint() to catch errors as they are logged.
running from:/Users/craigmacomber/Desktop/Panda-3d-Procedural-Terrain-Engine/src
instancing world...
:pstats(warning): Ignoring spurious connection_reset() message
:gobj(error): shaders/stephen4.sha: invalid parameter name (uniform in float4 region1Limits)
:gobj(error): shaders/stephen4.sha: invalid parameter name (uniform in float4 region2Limits)
:gobj(error): shaders/stephen4.sha: invalid parameter name (uniform in float4 region3Limits)
:gobj(error): shaders/stephen4.sha: invalid parameter name (uniform in float4 region4Limits)
:gobj(error): shaders/stephen4.sha: invalid parameter name (uniform in sampler2d region1ColorMap)
:gobj(error): shaders/stephen4.sha: invalid parameter name (uniform in sampler2d region2ColorMap)
:gobj(error): shaders/stephen4.sha: invalid parameter name (uniform in sampler2d region3ColorMap)
:gobj(error): shaders/stephen4.sha: invalid parameter name (uniform in sampler2d region4ColorMap)
:gobj(error): shaders/stephen4.sha: invalid parameter name (uniform in sampler2d detailTexture)
:gobj(error): Shader encountered an error.
setting up water plane at z=60.0
:grutil(error): Failed to load heightfield image heightmaps/ID123X0.0Y0.0.png!
:grutil(error): No valid heightfield image has been set!
tile generated at 0.0 0.0
:grutil(error): No valid heightfield image has been set!
:grutil(error): No valid heightfield image has been set!
:grutil(error): No valid heightfield image has been set!
:grutil(error): No valid heightfield image has been set!
:grutil(error): No valid heightfield image has been set!
:grutil(error): No valid heightfield image has been set!
:grutil(error): No valid heightfield image has been set!
:grutil(error): No valid heightfield image has been set!
:grutil(error): No valid heightfield image has been set!
:grutil(error): No valid heightfield image has been set!


and no visible terrain (I suspect because the shader failed to load). I suspect part of the issue is my using panda 1.7.0 which does not have all the new shader param options.

As for the height map issue, I had to make a heightmaps folder. You should make the code create it (os.mkdir I think) of include it in the repo (you might need to put some dummy file in it).

Once I get the shader working (update panda), I'll have some more feedback.

Edit:
I looked in terrain.py and say a loop. I changed it to this:
Code: Select all
        ys=self.image.getYSize()-1
        gh=self.terrain.getHeight
        sg=self.image.setGray
        yo= self.yOffset
        for x in range(self.image.getXSize()):
            xo=x + self.xOffset
            for y in range(ys+1):
                sg(x, ys-y, gh(xo, y + yo))


which cut the runtime of that code to about 66%. nothing special just localizing and avoiding some extra calls. I don't really get what that code does, but faster is better right?
Craig
 
Posts: 330
Joined: Thu Jul 02, 2009 8:55 pm

Postby AnimateDream » Tue Dec 14, 2010 11:41 pm

Craig wrote:and no visible terrain (I suspect because the shader failed to load). I suspect part of the issue is my using panda 1.7.0 which does not have all the new shader param options.


Sorry I built a custom panda out of the cvs to get true threads running.

Craig wrote:As for the height map issue, I had to make a heightmaps folder. You should make the code create it (os.mkdir I think) of include it in the repo (you might need to put some dummy file in it).


I fixed this unintentionally actually. The program no longer caches heightmap images to disk. It was almost always a net loss for performance, and I haven't gotten around to needing to add hand modifications to the procedural terrain.

Craig wrote:I looked in terrain.py and say a loop. I changed it to this:
Code: Select all
        ys=self.image.getYSize()-1
        gh=self.terrain.getHeight
        sg=self.image.setGray
        yo= self.yOffset
        for x in range(self.image.getXSize()):
            xo=x + self.xOffset
            for y in range(ys+1):
                sg(x, ys-y, gh(xo, y + yo))


which cut the runtime of that code to about 66%. nothing special just localizing and avoiding some extra calls. I don't really get what that code does, but faster is better right?


Yikes. Thank you for pointing that out. I really hadn't done much optimization work yet. Also I'm still used to c++ where the execution time wouldn't be too noticeably effected by some of those changes. Sorry.

Also the code just builds an image for a geomipmap out of my basic height/noise function.

Anyways, I understand better some of panda3d's geomipmaps functions and properties and corrected a few dumb mistakes. I also just committed my first comprehensive revision aimed at restructuring things for better performance. Its still too slow in my opinion, but its at least somewhat presentable now.

I still have some comments and refactoring to make it simple to grasp. Bear with me. Thank you for all of your help so far.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Postby Craig » Wed Dec 15, 2010 12:10 am

I already managed to copy out the water for my use, and I took your skybox too. When I'm done with finals with week, I'll probably dig into your code again. Then I'll look into your terrain shader and textures as they are much nicer than mine. Hopefully I can get my high performance system to look like your good looking one, then we all win :). Thanks for your work, and for freely sharing it!

I can merge several layers of perlin noise for a 515*512 texture in well under 0.01 seconds on the GPU, and thats were part of my performance comes from.

To make it run smoother, you can do one thing I did: I allow a few full speed frames to run between generating parts of the tiles. As full speed frames take drastically less time than ones bogged down generating terrain, you can have far higher frame rates with only a little slowdown in generation time. My app runs at about 30 fps while generating terrain (most of the load is making the GeoMipTerrain I think) before going to its regular idle 118 fps.


Just to give you a bit of insight into my plans, I intend to use my shader generator to inject rendering effects (like your nice terrain shader) into renderer specific templates (that handle placing the vertexes, computing slopes and such). Then it should be easy to swap both renderers, and terrain shading effects (perhaps in realtime). Its all about interchangeable parts.

I've done quite a bit of python optimization, so if you have questions, feel free to ask. Main issue in that case was python's large dynamic dispatch overhead. Each variable access and "." involves a dictionary lookup! Also, calls into panda are slow due to issues at the language barrier. (Not too slow, but a lot slower python-python calls). Cython can get around both those issues: http://www.panda3d.org/blog/?p=173

Edit: have some pictures: http://craig.p3dp.com/MiscPics/infinate/
This is starting to look ok, but it needs your better textures.
Image
Craig
 
Posts: 330
Joined: Thu Jul 02, 2009 8:55 pm

Postby AnimateDream » Wed Dec 15, 2010 3:32 am

Heh... well the water shader and skybox come from the YARR demo mentioned in my first post, and the textures mostly came from google image search. The shader is mine though, and the math I used to generate realistic heightmaps is worth at least looking at.

Anyways, I'm definitely liking the improvements I see in your screenshots. I can hardly wait to see what it looks like with prettier textures and shader.

I've updated the shader in my project to process two different sets of height and slope boundries for the same rock texture. The code got uglier, but the results got prettier. Obviously my shader and its python counterpart need more flexibility so that this kind of thing doesn't require hackish code. I'll do a rewrite soon.
Edit: I've realized that the shader's inability to store global variables means there is no way to make the shader more flexible without making it do possibly needless calculations on every pixel. The only good solution would be a shader generator like yours. Something much simpler would suffice for the purposes of customizing my terrain shader though.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Postby AnimateDream » Thu Dec 16, 2010 10:44 pm

There is now a simple shader generator to gracefully handle a variable number of textures and a variable number of slope/altitude boundaries per texture. This obviates the need to modify multiple lines in both the shader and python side to achieve such changes.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Postby AnimateDream » Wed Mar 02, 2011 1:02 am

Three significant updates.
1. Performance improved greatly.

2. My terrain shader got a major overhaul as I finally noticed proper documentation here. I had previously assumed input in the vertex shader came from the semantics, but it turns out they're totally irrelevant and only the variables' names seem to matter in practice. :roll: Anyways, I've combined hdr per-pixel lighting and bloom with my terrain shader, despite my lack of shader expertise. This still works with my shader generator

To do: Make these new effects configurable by the generator. Also I would like to switch my detail texture from an ugly blend to a normal map.

3. There is a simple GUI control in the demo that can be used to alter the terrain shader parameters on the fly. Of course, these shifts in textures can be achieved programmatically as well and smoothly transition a landscape to a totally different climatic appearance.

To do: Expand the gui and engine to allow new textures and regions to be added during run-time.

Just for fun... some screen shots of the latter feature at work.

http://stephenlujan.com/panda3d/terrain/menu1.jpg

http://stephenlujan.com/panda3d/terrain/menu2.jpg

http://stephenlujan.com/panda3d/terrain/menu3.jpg


Oh yeah, and here's a Ralph on a mountain I liked. Forgive the jpeg artifacts/blurriness. I didn't want this thread to drain everyone's bandwidth. :wink:

http://stephenlujan.com/panda3d/terrain/mountain1.jpg
Last edited by AnimateDream on Wed Aug 10, 2011 3:35 pm, edited 1 time in total.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Postby AnimateDream » Wed Aug 10, 2011 1:52 pm

I've had a couple dozen commits since my last post here. It was mostly minor improvements and organization, but a few obvious differences are:
* All shader issues are now resolved.
* Normal mapping is working beautifully.
* The time of day changes.
* There is a new sky made to be less static.
* A Ralph clone follows you around.
* Some loading screen feedback appears.

Check out the browser demo, or a couple of screen shots.

Sunset
Valley
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Postby drwr » Wed Aug 10, 2011 2:49 pm

The browser demo appears to be built against Panda3D 1.8, which isn't released yet, and doesn't run everywhere. It doesn't run on my Mac, for instance.

You should build p3d files meant for public consumption only against the standard Panda3D releases.

David
drwr
 
Posts: 11425
Joined: Fri Feb 13, 2004 12:42 pm
Location: Glendale, CA

Postby Nemesis#13 » Wed Aug 10, 2011 3:10 pm

Now those terrain pictures look awesome!
Keep up the good work.
User avatar
Nemesis#13
 
Posts: 1041
Joined: Mon Aug 04, 2008 8:09 pm
Location: Germany

Postby AnimateDream » Wed Aug 10, 2011 3:20 pm

Sorry, I forgot to mention I've been building with 1.8.

My shader fails to work using either of these.
http://runtime.panda3d.org/packp3d.p3d
http://runtime-dev.panda3d.org/packp3d.p3d

It only works properly with this one.
http://runtime-dev.panda3d.org/packp3d_dev.p3d



Oh, thanks Nemesis!
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

Postby AnimateDream » Tue Aug 30, 2011 11:35 pm

I've added exponential fogging to the shader and optimized it by removing all conditional statements. The terrain engine has almost all of the features I have intended for it at this point. I've finally made a proper web page for it here. You can go there for more details, screen shots, videos, and a browser demo.
AnimateDream
 
Posts: 89
Joined: Tue Feb 16, 2010 2:36 am

I've tried the most recent version and it crashes

Postby lord gengoro kitsune » Fri Sep 09, 2011 2:23 am

Tried the latest ver crashes right after the loading card Idle spews a lot of stuff about invalid shader params. The previous one which had a gui and would allow adjustment of the shader heights and slopes wouldn't run right either, stayed in monotexture shader, with NO terrain lighting, and with no error messages about not being able to load the slope texturing shaders.
I also cannot run Craig Macomber's terrain demo with the geo clip grass enabled, I get NO terrain whatsoever. I have a two year old HP DV7 1200 series with DX11 but an ATI Radeon HD 3200 GPU which seems to have a serious "mental handicap" and also seems to only want to run in single precision mode.
This is the second time I've posted in a long time ... after being roundly castigated by a bunch of "Ivy League" "professional developers" over my favorite choice of text colors ( 1.0, 0.0, 1.0, 1.0 ) over black and my stunning lack of professionalism... what I'm trying to do isn't about money to me its about supporting creativity which moneygrubbing VR sites like Second Life care little about.
Your slope based texturing system was of great interest to me as was Craig's infinite terrain system. Also having problems as my app needs functional collision detection which only seems to work with EGG based terrains and objects not geomip or procedural plants created on the fly.
At this point I'm about to chuck my whole project and this "brain dead" computer in the dumpster, don't tell me to just buy a newer one theres NO funds for that. Ready to throw in the towel and crawl back into my foxhole.
YIP YIP YIP

I work hard hopefully someday I'll have something to show for it
User avatar
lord gengoro kitsune
 
Posts: 67
Joined: Sun Aug 15, 2010 11:53 pm
Location: cleveland OH

update on collisions

Postby lord gengoro kitsune » Sun Sep 11, 2011 4:56 am

Had an instance of Alice beach house as a background piece to take snapshots in front of so it looked like I was farther along with meshes ( I kno cheating ). Went back and opened the EGG with TextPad, guess what I discover... the magic incantation <Collide> { Polyset keep descend } is not present. added that by cut and paste fronm the level file in Treeform's FPS demo and BINGO!!! We have contact. height setting code I ripped off from original Roaming Ralph still not working like it should but I'm showing collisions now at least with this one and only test object. Panda manual talks like the magick spell is not really needed but only to make collisions more efficient. BUT this appears to be an inaccuracy as IT seems to be REQUIRED. And NONE of the Alice EGGs I've looked at so far seem to have it but its easy enough to cut and paste. I really liked Craig's procedural trees but I will probably have to go back to my original concept of randomly placing EGGs for the trees and such. Still wish I could collide the terrains tho as editor would be so much cooler and more like what folks would expect it to do. Now I just wish I could figure how to do something like your slope based texturing shaders that could run on this machine. My code at this point is more Craig's than yours but I'd really like to make my hills rocky instead of grassy and be able to add a few more height levels. Craig's shader chain I still don't really understand well enough even to add in a shoreline texture. Actually at this point I wish I knew why your slope based shaders wont'run on this ATI HD3200 and why Craig's GEOCLIP won't either.
YIP YIP YIP

I work hard hopefully someday I'll have something to show for it
User avatar
lord gengoro kitsune
 
Posts: 67
Joined: Sun Aug 15, 2010 11:53 pm
Location: cleveland OH

Next

Return to Showcase

Who is online

Users browsing this forum: No registered users and 0 guests