go to the building control table, and you should be able to drag a follower up to an open spot on the top of that building's page.
This. Also, once assigned to a building, that follower will be busy and unable to do Garrison missions. If you need that follower for a mission, you can simply remove him/her from the assigned building, and then assign to the mission.