No it's not true, you are fine, you can understand and answer (generally communicate)
But you need to contribute to communicate "faster", that's all.
So don't be - ( negative! ) , it will be Ok, by time.
But one quick tip, if you be afraid of English, English will be afraid of you. So just "use" it, even if you think it's a bit wrong, believe you will understand your mistakes little by little .
And someday you will notice your progress!