Does it matter that we celebrate the birth of Jesus on a pagan date?

To put it frankly, no, it does not matter which day we celebrate the birth of Christ on. It was a pagan date in the past, but if we do not celebrate it as one there is no harm done. Many things were once pagan including drums, birthday candles, Christmas trees and tattoos. If the meaning now has been well disconnected from the origin there should not be any negative moral implications.