Groups within groups works. Are you using the Admin Tool or The User Sync Desktop App to set his up.
Keep in mind that if the users are in a child domain, it will not work. Only users listed in the connected domain level will synchronize since the pointer to their user attributes are located there. In this scenario you would have to set up a sync at each domain level.