To do the baby sign for nephew, use your dominant hand and form the ASL sign for 'N' with your index and middle fingers while pointing to your head and twisting your hand twice. Do it at or above the level of your eyes. If you do it lower, it might be interpreted as niece.
We mainly use nephew to help our toddlers understand and embrace family structure.
Start using nephew when your toddler is ready to use qualifiers for family members.