Is there anyway I can set the permissions where avatars show so only the ones not set to private show and the ones set to private either show another image....
No
or can I have that the set photo as your avatar option only appears in the main personal gallery...
No, not the main
personal gallery, but you can have it where the set avatar option appears only in the main
public categories
OPEN
album_mod/album_hierarchy_sql.php
FIND (or similar line)
'AVATAR_PIC' => ( ($album_config['personal_allow_avatar_gallery'] == 1) && ($userdata['user_id'] == $picrow[$j]['pic_user_id']) ) ? '<br /><a href="'. append_sid('album_avatar.' . $phpEx . '?pic_id=' . $picrow[$j]['pic_id']) . '">' . $lang['Avatar_Set'] . '</a>' : '',
REPLACE WITH
'AVATAR_PIC' => ( ($album_config['personal_allow_avatar_gallery'] == 1) && ($userdata['user_id'] == $picrow[$j]['pic_user_id']) && ($picrow[$j]['cat_user_id'] == 0) ) ? '<br /><a href="'. append_sid('album_avatar.' . $phpEx . '?pic_id=' . $picrow[$j]['pic_id']) . '">' . $lang['Avatar_Set'] . '</a>' : '',
Doing the changes above could still give you problems in a main category if it is set to private.
It should be possible to disable the option on any private gallery (public or personal), but would require additional changes from those listed above.