Applying gloss map

Not the way I implemented it. In my design, I intentionally made the gloss map a monochrome (alpha-format) map. If you want to set the specular highlight color, you set it in the material properties.