If someone comes to you in distress to share their troubles with you they often complain about some person who has wronged them. If you decline to enter into unrighteous judgment against this person are you accused of being uncaring?
If you have peace and refuse to panic or be distraught yourself are you accused of the same thing? I do not respond in these ways yet I am more loving and caring than anyone else. You will not be helpful to your friend if you respond the way they expect. A true expression of caring is to remain in my presence and peace that you may provide godly comfort and counsel.