Mine does this sometimes. He seems to favour reverse to going forward - so he spends a lot of time exploring backwards.
I found that if you hold the base of his tail while he stands up and steady him until he's balancing by himself, it helps him get into a more stable mode of operation, and I get fewer reversing issues.
I realise the OP posted 2 years ago, but hope that helps someone who gets here via Google :-)