For years I roamed the Dorset heaths thinking that the trees with silvery bark were silver birch until, one day, a friend pointed out they were frequently downy birch (Betula pubescens); it came as quite a surprise! Obviously I wanted to know how to tell the difference.
There are small differences between them but they still remain a challenge to me. Firstly downy birch has a much smoother bark that silver birch. On silver birch the silvery bark tends to split as the tree ages and areas of dark under bark start to show. Not only is the bark smooth on a downy birch it also can be, especially when young, a bronze colour rather than silver. If you look at the outgrowing branches the downy birch has branches and twigs that tend to point upwards whereas on silver birch they tend to droop downwards. Twigs on a downy birch are hairy (hence the 'downy') and plain coloured whereas on silver birch they are smooth but with silver diamond shaped patches. There, you see it is quite easy! Even knowing this I still struggle.
One thing with both species is that they attract various micro lichens that create patches on the bark, different species of lichen form different shaped patches. They are even harder to tell apart than the trees.