20.5 / 13.75 is pretty short for trails.
I am about the same height as you and riding 21/14.1/74 with a big front tyre and a skinny rear to slacken things up even more.
Most folk at ours ride at least a 21.25, in fact I believe I am the only 21.
The sketchyness you describe is almost certainly too short a bike.
Look at a modern trails frame like a credence or deluxe and you'll see lots of super low bb heights, slacker head angles, long back ends and longer top tubes.
A cool thing I've noticed about a longer bike is you can get away with massive bars, which allows more rooooooast.....!!! As the whole thing needs more leverage to pull up and get near the "looping out" point, making everything bigger gives you finer control over the bike.
Anyway I have finished doing my poo now so I will sum up and say get a bigger trails frame with 9" bars and pull up more.