In my opinion, letting go doesn't mean you stop caring about a person. Letting to doesn't even mean you stop loving the person. Letting go means you are giving them space and allowing them to carry on with their lives without your interference and also you moving forward to see where your life takes you.
I long ago let go of one of my exes that I loved very much. He is now married and has a son. However, I STILL think about him sometimes and I miss him too. He was a great soul. That doesn't mean I haven't let him go because I did a long while ago and was in two relationships since.
Letting go just means you don't sit around waiting for a person and carry on. Letting go means letting go of the pain that creates a block for you and prevents you from carrying on with your life.
Letting go means letting go of the fantasy or ideas you had of building a life with said person and accepting that it will not happen at this time, if ever. Letting go means allowing yourself to move forward and experience other things.
You can still love and care about someone even though you've let go of other things. That's just been my experience.