I don't think much has been done. I think there are different issues,
depending on your model of agent interaction. One model (the more
common) has agents sending each other declarative messages. The other
model has agents moving from platform to platform or at least sending
each other more procedural messages which are to be "executed" by
other agents.
In the declarative message model, we've discussed some basic security
measures for the KQML agent communication language
("http://www.cs.umbc.edu/kqml/") but nothing has been written up on it
yet alone implemented. There is a clear need for both agent
authentication and privacy of communication. It may turn out that most
or even all of the issues are best dealt with by the low-level
message transport layer. For example, HTTP has built-in provisions
for security and could be used to transport messages between agents.
I' not familiar with the security features of other distributed
processing/messaging frameworks (e.g., CORBA, OLE, ToolTalk), but
most will have to address security later if not sooner.
The TeleScript language of General Magic is an example of an agent
model in which agents remotely execute on a number of platforms. It
has been designed with security in mind, but the concern seems to be
to allay fears that an agent might do mischief or introduce viruses.
Both of these models are important since, in practice, even a language
like KQML+KIF can allow one to encode requests for a receiving agent
to place inference rules in its knowledge base.
Tim