Never had any issues.
The whole cup holder unit complete with the surround comes away very easily, no tools required.
Get around the lip of the surround, pull it away squarely and you will be able to see the mechanism that performs the locking function.