To be honest with you, ...

I don't think you are wrong. It makes my spidey sense tingle when I hear it. I don't necessarily conclude that what follows is less true, but it does cast a shadow of doubt over everything the person has said, especially the statements that came BEFORE the speaker was being (perfectly) honest with me.

Still, the phrase does serve an honest function. A lot of what we say is a simplification of reality. Even with the best of intentions we leave off details that we deem to be irrelevant or less relevant to the point at hand. "To be (perfectly) honest" can be a useful phrase to employ to introduce a more nuanced discussion of something that was previously simplified. It acknowledges in advance that what is to follow might in some ways contradict previous implications and, thus, avoids some confusion over the coming contradictions. It informs the audience to give priority to the new statements over the more simplified ones that came before.

It may not specifically reflect intentional dishonesty.

But I'd still be a bit wary.

There are others. "To tell the truth", "Frankly" to name two. These are idioms which really have nothing to do with truth or honesty. It is a way of saying, "What I have been saying has had somewhat of an agenda and has been scripted, if only in my mind.   But I am going to leave that agenda and tell you things that I was planning to keep from you."

If you keep this in mind when listening to people speak you will see that most people use it properly, but some insert it inappropriately.   I think these people are just looking for something to say and don't realize that this makes no sense.

In either group, honesty has nothing to do with what they are saying. Personally, I try to avoid using this phrase because it is overused and misused.

Glenn posted while I was writing this and I completely concur. He gave a more complete explanation.

Mort Sahl used to pause before one particular anecdote in his act to say "Everything I'm telling you is true, but this is actual".

If I recall correctly, he credited the turn of phrase to Lenny Bruce.

Glenn said

Still, the phrase does serve an honest function. A lot of what we say is a simplification of reality. Even with the best of intentions we leave off details that we deem to be irrelevant or less relevant to the point at hand. "To be (perfectly) honest" can be a useful phrase to employ to introduce a more nuanced discussion of something that was previously simplified. It acknowledges in advance that what is to follow might in some ways contradict previous implications and, thus, avoids some confusion over the coming contradictions.

Glenn,

The process you describe here is related to a description of my profession I once heard (and accepted), "Education is the art of telling progressively smaller lies." Maybe the reason I have accepted it is because the truth of physics is so much more complicated than the classical approximations which work so well.

Emmett