I was going to guess Almond, based on your clues (too small and not hardy in Jersey), but come to think of it, if it's a Prunus, it could be a cultivar of virginiana. I have found two in Providence that I'm unable to correctly ID, even with other expert help. If you can get a close up of the twigs and leaves, and possibly the flowers (next year), I'd like to see if it's the same as the one I'm seeing.
-Tom
(PS - Yours also looks like Eucommia ulmoides, Hardy Rubbertree. Wrong family, though.)