They may not "remember" it as a specific memory, but every experience that they have will build their personality & character, so it's worth it to give them happy experiences.