max physical damage,
max fire damage,
max lightning damage,
max magic damage,
max cold damage,
max poison damage divided by 256.
Then he adds them all together. He doesn't check for min damage at all, so if you'd have 1-500 damage then I guess you'll get picked over your teammate if he has 498-499 damage.
There's more he checks for, but I'm not sure how everything is used in his decision-making.
He checks for current hitpoints, if the target is chilled, distance of course, cold resistance, fire resistance, lightning resistance, physical resistance, and magic resistance.
Since he checks for these resistance types, I assume he potentially factors those in when he chooses which attack to use, since his attacks have all of those types.
He also checks specifically if the target is (his own) bone prison.
I can imagine there must've been an alpha version at some point where he started to attack his own prison..
I think highest from gear, not skills. Not sure though.
Right.
His AI does check for which skills are selected and their skill levels. There's a small formula that uses the selected skills and their levels as input, and the result seems to get added to those max damage values. The AI doesn't seem to care what the skill actually does though, so even non-offensive spells will contribute.
(The formula takes the "AttackRank" of the skill and multiplies it by the skill level. It does this for both left and right, adds the results together, and divides that by 4.)
The damage he checks for seems to be before any (off-weapon) +%, which would explain why Harmony-wearers tend to become targets. Grief's extra damage might not apply at all since I think that might be "normal damage" and not "min - max"?
Anyway, looking at "sheet damage" would not be appropriate because (if I read it right) Diablo does not factor in +% damage from things like skills and strength/dexterity. He checks the weapon's max damage after any +% and +max that's on it though.
Oh, and poison damage is stored as damage/frame, so the division by 256 pretty much makes sure it always ends up as 0, unless you deal 25+ poison damage per second. But I'm looking at 1.00, so maybe this was changed later.