Tips for Creating Perfect Normal Maps Every Time
There are three key components that you must nurture in order to bake perfect normal maps: Smoothing Groups, UVs and the Projection Cage. Mistreating any one of these components can lead to ugly, normal maps and wasted time. Now if you're unsure of what a normal map is, you should take a look at this video to help clarify.
Support That Detail Do your high resolution mesh a favor and create a low resolution mesh that's worthy of its amazingly awesome detail! This is going to directly affect the silhouette of your game model. Perhaps you need a little more resolution around the curves of your model, if there isn't enough, you'll notice that the curved detail from your high resolution mesh, but the silhouette of the low resolution mesh throws off how your normal maps are viewed on the model. Don't be afraid of being generous with your polygons, within reason of course. Smooth Move Normal Map Whenever you create a low resolution mesh, you must set up your smoothing groups (3ds Max) or edge hardness (Maya). Some would tell you that you should use just a single smoothing group which can be done, but usually you need more geometry to help support it. Without that extra geometry, you'll get some unsightly gradients in your normal maps at lower texture resolutions. So multiple smoothing groups should be used, but only on hard edges. If You Like It, Put a Seam On It Hard surface game models can be riddled with hard edges and this can give you a hard time if you don't create your UVs properly. Let's just get the fundamental thought out of the way: "If it's a hard edge, put a seam on it." Otherwise, you'll see this terrible black line or artifact on the edge of your game mesh. The reason that this is happening is because it's attempting to take the very different pixel colors of the normal maps for each polygon and trying to blend them across the edge. So because they're so different, it creates the wrong normal orientation and causes the unsightly black line or "gap". Straighten the Cage When you're ready to bake your normal maps, you're going to project your high resolution detail onto your low resolution mesh. The cage is what you will use to help in that projection process. A common misstep isn't properly aligning your cage to your low resolution mesh. The goal is to have the cage engulf high resolution mesh while keeping it as close to the mesh as possible. If your cage is skewed, your detail in your normal maps will be skewed as well. Stay true to the original shape of the low resolution mesh as possible. It's important to note that these are all things to consider when getting ready to bake normal maps. While every model will be different, if you apply these rules as the foundation of how you bake normal maps, you will have more success than wasted hours of re-bakes. So remember these four rules: (1) make sure your low resolution mesh has enough supporting geometry for you high resolution detail, (2) don't be afraid to use multiple smoothing groups, (3) UV seams must be on hard edges if you use multiple smoothing groups, and (4) make sure your cage isn't skewed.