Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies