I've been through this same cycle and thought process, the smaller frame is hurting your off road ability. The bmxer in you wants a tight feeling bike but for true MTB rocks and steeps you need a big enough bike to move around on.
As for wheel size, I love 29" on a hardtail. 27.5" is just marketing and is effectively the new 26". Unless you're on your game already, moving from 26" to 27.5" with all else constant won't change much of anything for you.
Find a complete for sure. $1500-$2000 will get you a nice new bike like a Kona Honzo or Explosif. You can probably still find a leftover 2014 of most brands for a discount. A slacker HT angle and lower BB will also help you feel better in the rough stuff.
Good luck!
I got into DJ mtb's in 2006, then singlespeed hardtail, then various borrowed and rented XC and AM bikes (FS and hardtail). Also have a Transition Double 4x/slope with 1X9.
I'm not sure I understand what you're saying by "For true MTB rocks and steeps you need a big enough bike to move around on."
If anything, I find that the many people in the mtb world are STILL stuck in the dirt roadie mentality. When you are stretched out over a long frame with a long stem, 29 wheels, a high seat and ultra low bb, that is the most dirt roadie position you can be in. And is NOT a help for technical riding in the sense of getting over rocks and definitely not on steep descents. The only thing it is great for is climbing, which many dirt roadies are obsessed with.
I think 29"s are sluggish. And I think 27.5 is a great compromise.
I rode two 27.5" demo bikes at Winter Park this past summer. The 27.5" DH bike felt big/awkward/non-nimble. But the 27.5" Specialized Enduro Carbon felt great.
i crank around regular xc trails on a 4x/slope bike pretty regularly (Transition Double with 1X9 gearing), and i keep up with geared-up 29'er dudes pretty well. i realize that full leg extension setup does make for more efficient riding... but i also don't like giving up some of the 4x/bmx style handling for hitting the berms rollers jumps that i have built out here, mixed in to the regular mtb trails. or even just the feeling of being able to bunnyhop a log in the way that i'm used to etc. etc. i may actually buy a true xc/am mtb soon, but i will definitely ride a size a little smaller and set it up a little more bmx/DJ-ish. probably with a 60-70mm stem, bmx platform pedals, etc.
(i know i failed to do an impressive trick here but it's one of my few riding shots on this bike)