Doesn't sound correct to me, but "brains" is sometimes used in a sense that implies singular, rather than an actual plural noun. As an implied singular form, "brains" is quite common in English. For example: "He has the brains to solve that problem."
Acknowledging the usual Google Ngrams cautions, see https://waywordradio.org/discussion/topics/my-brains/has%20the%20brains,has%20the%20brai n" target="_blank">this comparison.
But in the context of the example you cite, it just doesn't sound right.
The plural there sounds natural to me. I don't have a problem with it. In my speech there is a lot of overlap between singular and plural uses. Both can be literal or figurative with nuances.
When used literally, the plural suggests brain matter rather than an intact organ. When used figuratively, the plural suggests more generic capacity than specific skill.
The ME removed and weighed the brain.
The suicide shot his brains out.
She has the brain for theoretical physics.
He doesn't have the brains for college.
In this case ("... more to do with my body than my brains."), the plural suggests to me that general intellect is what is being referenced.
Left brain - right brain - riddle solved - two brains make a brain 🙂