Usually grayscale .png files are used. Usually 8-bits or 16-bits, depending on the required precision.
If you need any suggestions for tools you can use, Terragen
is a popular tool that is widely used to generate pretty realistic results, although I've never used it myself. There is also L3DT
for generating pretty convincing large terrains based on erosion simulations, including climate-specific texture maps, or EarthSculptor
if you prefer to sculpt your terrain completely manually.
All of these are Windows-only programs, unfortunately, but L3DT seems to work great in wine.
Of course, if you just want something simple, you could also use your favourite image editor to generate a perlin noise image, which should create some decent-looking hills.