Here is a photo of the shaft the handle slides onto. I think you’re on the right track for removing it there shouldn’t be anything other than the bolt you already removed that was holding it on.

B6322E55-09C2-4395-8D1F-722C079E6D00.jpeg