E. Cobham Brewer 18101897. Dictionary of Phrase and Fable. 1898.
Go to Bungay with you!i.e. get away and dont bother me, or dont talk such stuff. Bungay, in Suffolk, used to be famous for the manufacture of leather breeches, once very fashionable. Persons who required new ones, or to have their old ones new-seated, went or sent to Bungay for that purpose. Hence rose the cant saying, Go to Bungay, and get your breeches mended, shortened into Go to Bungay with you!