its like when your in a car and the wheel balance is off, the wheel starts sawing side to side by itself and the oscillations grow.
It's not uncommon, and it's called shimmy. Your bike frame is a spring, not a completely rigid structure, and certain combinations of mass (you, plus or minus any luggage) and distribution result in specific natural frequencies of vibration. Usually the remedy is to damp the oscillation by standing on the pedals, or laying your leg against the frame, or another change from the excitation position. Both my old Tange Prestige steel-tubed mtb and my True Temper steel SWB recumbent have springy frames that vibrate maddeningly if I ride no-handed with my panniers loaded out back.
Check out Part 4, Section 8h.5 of the rec.bicycles.tech FAQ for Jobst Brandt's take on it.