I think the magic ingredient is the sheaves, balancing grip, wedging, rope punishment, and sensitivity to rope variation.
I agree and thanks for the insights. It feels like I'm following you down some of the same paths. Unfortunately I just don't have the machinery to make sheaves from scratch, but the stock pulleys are OK for now. Below is my rig for shaping their innards. (Stock Disclaimer: Don't try this at home.)
This is stock pulley #2, the one with the rounded bottom made by Congress. The tapered end mill is 3 flute, 15 degrees on a side. This pulley worked a little better than the one carved with the Dremel, but only because it's narrower on the bottom for gripping thinner ropes. It still doesn't grab the 11mm ropes as well as it should. For a couple samples (Kernmaster, Mercury) I had to remove the fairlead stop to prevent slipping, which essentially turns the rope into the stop. This seems to put an extra strain on the gearbox and drill.
All of the other ropes worked well enough with the thicker kernmantles working the best. My favorites among climbing lines are Xstatic, Drenaline, and Scandere. For a dedicated ascending rope it's hard to beat the 1/2" HTP Static with 1/2" Atlas not too far behind. The thicker, stiffer ropes not only increase the pitch diameter, making ascending faster, but their cross section gets less mangled by the pulley. Rope stretch isn't an issue since it's all removed once you leave the ground, and the steady pull of the ascender doesn't bounce.
I'm officially declaring the gearbox as fully broken in. Last weekend it ran through another 600' of ascending/descending, and one of the starting torques, with 200 lbs. on the line, measured only 10 inch-pounds. That's the lowest measurement on the torque wrench. Consider there could be more than a hundred cordless drills on the market with torques over 400 inch-pounds.
The gearbox manufacturer included parameters for when the gearbox should be replaced based on initial backlash, aka slop in the gearing. At the edge of the rope pulley it first measured around 1/16". After all the testing it still measures around 1/16". According to the instructions the gearbox is worn out at 4x initial backlash. My guess is around 50,000', but who knows. I still don't know how this ascender works with loads over 200 lbs. Maybe the next test is adding a few more cinder blocks to the stack after breaking out the steel toe boots. Ouch!