ToME WikiKhymChanur: Right now, the monster spell casting code just randomly picks a spell and casts it, so a monster might cast a healing a fleeing spell when at full HP, or cast "teleport to" when it should be fleeing. The spell info structure should have a field who's bits describe the spell, like "this spell is a fleeing spell", "this spell is an attack spell", and so on, so that the engine can narrow down the choices before making a random pick.
Orvin: A monster's choice of spell should also depend on its perception of who it is attacking (X), such as:
X's physical health (hp, cuts) X's mental state (confusion, sanity) X's class (which classes are generally stronger at melee vs. missile attacks) X's race (racial resists, susceptibilities)
What you could do is create a function f(monster,X,attack) that returns a number, evaluate this over all available attacks, and then assign a probability of choosing an attack based on the difference between max(f) and f(monster,X,attack), so attacks with similar f-evaluations are of equal probability and attacks with lower evaluations are of lesser probability. It would probably take a lot of experimentation with f to create the proper balance, but this could add quite a strategic challenge to the game.
Also, f could depend on the success of previous attacks. So, if a fire-based attack did much/little damage, increase/decrease the value of f for fire-based attacks.