No matter how many times you wash your hands (and one should always do this before handling philatelic material), there is always a tiny amount of oily residue that adheres to one's skin---and which can migrate to any kind of paper item. When this oily residue gets onto a stamp or cover, it will stay there forever and, in the long run, promote the deterioration of the item.
COVERS can, of course, also be handled with tongs, too. However, this may prove to be a little awkward so the accepted way to handle them is to hold them only by the outer edges.
We might also mention that, in lieu of the above, some individuals use very thin rubber gloves which can be purchased in any drug or grocery story. Whichever manner you choose to handle your stamps and covers, be sure to keep in mind the oily residue that resides on your hands at all times.