The first person to hit the mob will lock it to their group - there will be a visible indicator above the enemy showing that the enemy is locked to them.
When the enemy dies, the loot goes to the group that has the lock regardless of who did the most damage to the enemy.
If the enemy resets because the attacking group wipes out or the enemy dies, the lock is removed.
If the group attacking the enemy wipes and other groups are also fighting the enemy, the group that wipes will lose the lock and it will transfer to the group that has done the most damage to the enemy.
I am glad this is being considered. Personally, I have never experienced a KS. Not once. But I understand the concern, particularly in some of the very long boss fights.
That said, here is how I would exploit the target locking, if I wanted to.
A tank and a healer can survive many of the bosses for a long time, as long as you want in many cases, but may not be able to kill it in any reasonable amount of time, if at all. Even a strong tank by themselves, or a kiting ranger or Druid could lock a target and then wait for the rest of their group to get there, even if there was another full group on site forming to take the boss. One of the things this means is that small higher level groups could potentially dominate bosses even more than at present, amplifying the 'monopoly' problem, while larger, pug groups of lower levels who need the quests are still forming. It would even be possible for a clan to lock down multiple bosses and then rotate through, even though there were full teams organizing to take one boss while the lock down clan was killing another... I think this could even lead to griefing, where a kiting ranger just keeps a boss locked down without killing it for as long as they want, taunting the players who actually want to kill the boss, even when other groups are there and ready to go. C'mon, you can immediately think of at least three players on your server who are going to try to do this.
One thing then you might want to do is put a reasonable time limit on the lock. Or tie such a time limit to damge. If the locking player or group is not actually killing the boss within the first X minutes they lose the lock. If they are unable to kill the boss or wipe, they lose the lock. Once you lose lock on a boss you cannot attempt to regain it on that boss until it Respawns. Even this is exploitable by multiple rangers, etc.
Just saying an unintended consequence of a very positive idea. As you know the real solution for this is to put some bosses in instances of some sort. This has it's own unintended consequences and dynamics though, as well as being harder to code and such.
Another solution is to have triggered bosses at a distance from the trigger and only attackable by the group that triggers it. Just moves the KS question though and has consequences about changing group formations and such. One way to do this is to make the trigger a dropped object, like a horn or scroll to summon the boss whenever you want. Only one player need the summoning item and these items become very valuable. I would make them NO TRADE i think. That will also have questions, but could be fun.
To get really crazy, you could make a set of bosses that can only be summoned in the arena. The summoning item drops in the world, but can only be used in the arena. When you summon them there is then a global message that says the boss has been summoned in the arena. Players not in the summoning group could not KS, but could attack the group fighting the boss... That is, players outside the group cannot KS the boss, but they serve as ads for your boss fight and have to be dealt with accordingly. The boss on the other hand can aoe effect and maybe even target everyone. This could get pretty crazy if people summon multiple bosses. Lol... Actually, you would need to think through how to avoid the arena becoming so full of rampaging bosses that players could not survive. Probably means a time limit on bosses staying in existence if they are not taking damage. Might also mean only one summons at a time in the arena, though that eliminates the strategy of summoning your boss while another group is occupied with their own. I know... I am a bad, bad man... But this could be a lot of fun, though upsetting for people not used to PvP.
To do this right you would also have to make idols non useable in the arena and create a penalty time of at least 5-10 minutes or so for dead players before they can re-enter arena combat. If people can rez or respawn Zerg this idea is not feasible. I think these last restrictions would be a huge improvement to the arena with or without the bosses. Idols and immediate Rez Zerg ruin open pvp completely. Allow druid rezzes with the current penalties.