E. Cobham Brewer 18101897. Dictionary of Phrase and Fable. 1898.
It is all fiddle-faddle. Rubbishy nonsense; talk not worth attention. A ricochet word, of which we have a vast number, as flim-flam, helter-skelter, wishy-washy, etc. To fiddle is to waste time in playing on the fiddle, and hence fiddle means a trifle, and fiddle-faddle is silly trifle or silly nonsense.
Pitiful fool that I was to stand fiddle-faddling in that way.