If I assign a user the a share point specific site with Site admin rights, will the user be able to change user permission for that site? or need to be a share point admin to do that (which I really don't want).
I believe you will have to make a user a sharepoint admin...but you can also see below that under manage roles, you can limit access and assign only the permissions they need.