In most sci-fi, I think this might be easier...aliens have physiology that sets them apart from humanity in very obvious ways, be it gills, claws, etc.. This in contrast to differentiating between humans and the other "core" fantasy races presented, such as the elf, dwarf, gnome, and halfling...who are all usually put forth as "short humans" or "humans with funny ears."
In the Warhammer 40k universe's supporting literature, there are several mentions of people not being able to look directly at glyphs or even the physical forms of, or created by, Chaos or the various warp (read: alien) species. I find this to be a unique side-effect of the racial indoctrination practices of the Imperium of Man, and something that can be adapted for use in various, but not all, campaigns. (My Wilderlands game, for example, would be a bad place to use these alternate rules, as Judges Guild has always supported fairly free mixing of more races than any other campaign I've ever known of.)
Now, how to apply this in-game? There are a couple of ways.
The first, and easiest, I would say, is to ignore any mechanics that might be associated with the reactions to races, and incorporate the feelings and involuntary reactions into the descriptions of what the PCs see.
For example, let's say that someone running a D&D game wishes to have the PCs meet an Elf in a grove outside of town. Upon approach, the demihuman is spotted wearing a hooded cloak, and brooks no reaction, but when they turn around and doff their hood, the alien perfection of the creature could stir any number of reactions in your PCs, and it should be described as such. Don't strongarm your players into having their characters feel a certain way. State to them that strong emotions well inside them at the sight, some might be filled with lust, others rage or melancholy at not being as perfect.
This can, and should, be different for every race. Using the same narrative system, let's replace the elf with an orc. Orcs in this campaign may be so hideous and terrifying that no man may truly look into the face of one, their eyes naturally skirting the beast's features, leaving only the impression of hellish terror or "death itself."
This could cause a bit of fun trouble within the campaign when trying to describe a nonhuman to a human. If the guard is in pursuit of a halfling who has absconded with some treasure or another, the guard is going to have a hard time picking out a person who seemed "very mouselike" or "like any alleycat" or "gave the impression of shadows."
This is a fairly extreme example of using the narrative, but can be lightened or even made more heavy-handed, to suit any campaign style. As I mentioned before, my Wilderlands game, to me, seems a poor fit for an assertive approach with this, so I use a dialed-back version of the narrative style, describing the various races with unique flairs, trying to give them something that's more than just "it appears much like a human, except with pointy ears." Elves, for example, have at least one feature that ties them to nature and their fey origins...a recent character, for instance, had hair made of vines.
A gnome, on the other hand, might have eyes that appear to be faceted gems, a dwarf's voice may crack with the sound of rocks clattering together. Simply using your GM-granted power of narrative can go a long way toward creating interesting contrasts with races in your games, and it might encourage your players to pick up a nonhuman race as their next character.
Next, we'll delve into a mechanical representation of the same thing. Obviously, there are a couple of ways to do this, as well, and by far, the easiest is to just impose a penalty to social rolls versus those of various races, but I like fiddling with crunchy mechanics, so let's see what kind of damage we can do.
For the purpose of our rules, I thought I would use a stripped-down version of the Sanity rules from the 3.x Unearthed Arcana with a splash of Predator's Taint from Vampire the Requiem and Fear & Damnation from Dark Heresy for flavor. I'll call this the Composure Roll for these purposes.
First, it's obvious that none of these systems is d20-based, so that's the first change we'll make (thus each 5pt incremement will be represented by a single number on the die.) This will make the mechanic fit more seamlessly into the game, by making it intuitive and easier to recall. Mechanics like this can already be a burden, one of the goals is to make using it as easy as possible for maximum use.
Since Wisdom is usually tied to mental composure, the goal, when meeting the selfsame elf mentioned before would be to roll equal to or less than one's Wisdom score on a d20, subtracting 1 from the Wisdom score for each instance of roll made in the current game session. Optionally, a GM may lower the score an additional point to accentuate the otherness of non-core races, and/or could also subtract one point per 2 or 5 class levels or hit die the trigger has to represent a further departure from the norm with age and experience.
As noted, to succeed, you must roll equal to or lower than the target number, whose base value is equal to one's Wisdom score. Upon failure, however, certain other effects take place. A failure of 4 or less indicates that the character visibly balks, and they take a -2 to all social rolls associated with their trigger.. Failing by 5 or more means that the character is obviously aghast with horror, raising the severity of the social penalty to -5.
Upon rolling a 20, however, the character's mind fails to hold together any composure. Have the player roll another roll, and if another failure comes up, a mental stress occurs.
Here's a nice d10 table for mental stresses, based on "The Shock Table" from Dark Heresy.
- The character is badly startled, and are treated as nauseated for 1 round.
- Fear grips the character, and they are shaken for d6 rounds unless another composure roll is successfully made.
- The afflicted scrabbles away and cannot willingly approach, as the frightened condition, save that they cannot fight, and may act normally otherwise once they are out of sight of the trigger. Lower your Wisdom by 1 for the purposes of Composure from this point on when dealing with a similar trigger. (i.e., another elf)
- The character is cowering for d6 rounds, and is shaken for a like amount thereafter. Lower your Wisdom by 1 for the purposes of Composure from this point on when dealing with a similar trigger.
- The character is panicked for d6 rounds. If unable to flee, may only take half actions, and are at -4 to all attack rolls, weapon damage rolls, saving throws, skill checks, and ability checks. Lower your Wisdom by 1 for the purposes of all Composure rolls from this point on.
- The character faints, remaining unconscious for d6 rounds. Upon waking, they are also shaken for a like amount of time. Lower your Wisdom by 1 for the purposes of all Composure rolls from this point on.
- The character is completely overcome with despair and sickness and is helpless for d6 rounds. Afterwards, the character is shaken until they can rest. Lower your Wisdom by 2 for the purposes of all Composure rolls from this point on.
- The character flies into frenzy, laughing hysterically and attacking allies and enemies alike with all weapons at their disposal, but not with spells or special abilities. This lasts until the character is made to snap out of it (as with a spell) or until they are unconscious. Lower your Wisdom by 2 for the purposes of all Composure rolls from this point on.
- The character crumples to the ground for d6 rounds, tearing at their own flesh (inflicting melee weapon damage per round). Afterwards, they are exhausted and shaken until recovered with rest. Lower your Wisdom by d4 for the purpose of all Composure rolls from this point on.
- The character's mind snaps and they become catatonic for d6 hours and may not be roused. Lower your Wisdom by d4 for the purpose of all Composure rolls from this point on. In addition, roll another Composure roll, if this one fails, step this result up to 11.
- In addition to the effects of 10, your character also suffers a minor mental disorder (worked out between you and your GM, or at GM's discretion) that lasts for d10 rounds, and d6 rounds after each other Composure roll made from now on. Lower your Wisdom by d6 for the purpose of Composure rolls from this point on, unless the additional Composure triggered by 10 comes up a 20, in which case you lower your permanent Wisdom score by 1. (This replaces, and is not in addition to, the lowering of Wisdom from 10.)
This system was tweaked from a d% system with a d10 system bolted on to make a working d20 system mechanic, and it wouldn't take much work to dig up the original sources and go from there, or take what I bashed together to retrofit into something more suiting your own game's system.