It uses a quadtree, which is subdivided based on the projected size of the triangles of the chunks. Basically the algorithm tries to make every triangle a fixed size, e.g. 10 pixels.
If the triangles of a chunk match the requirements, its added to the list of instanced chunks, otherwise it is subdivided further.
You could do that, but it’d be very annoying to setup, and you would also have a lot of transform states, which would slow down the entire thing. Besides of that, you don’t have the advantage of hardware instancing then.
If you have a heightmap of 8192*8192 for example, then that would be 67.108.864 vertices, which will be much slower.
You should use a BulletHeightfieldShape for that, juzzuj added a patch so the triangles match up.
The heightmap can be any texture object, however since the culling happens on the CPU, one would have to regenerate the terrain areas which changed (the terrain caches each chunk average, min and max height for culling, those would have to get recomputed). Probably something for future versions, feel free to make a patch.