I've been spending a fair amount of time in Second Life recently, and so I've had a chance to play with LSL, the Linden Scripting Language, which lets you program the behavior of objects in the world of Second Life. That means I've had the chance to look at LSL's solution to a problem I've given some thought to: how do you invite people to contribute code to a common environment, without worrying too much that malicious (or incompetent) contributors will hijack or crash the system?
This is an old question, and there are some well-established answers to it. From the beginning, the Java programming language has included a sandbox security model suitable, for example, for running applets in a browser. The default security level for an applet prevents most possible types of misbehavior—for example, it prohibits access to the local file system—but it doesn't have any provision for different applets to interact directly with each other; the security restrictions more or less make different applets running in the same process invisible to one another (although they could, for example, communicate with a common server). The Java security model also does not have provisions for restricting either the CPU time or memory consumption of untrusted code, and so does not deal with denial-of-service attacks that overuse those resources.
LSL, by contrast, includes explicit CPU time and memory restrictions on running scripts. Like Java, it doesn't let you access any of the underlying system (like local files)—but it does this not by having a security manager that explicitly prevents such operations, but just by omitting the operations from the language. Unlike Java, LSL doesn't also have to run general-purpose applications in a low-security environment.
And unlike Java applets, LSL does provide a fairly straightforward way for different programs to communicate with one another. Communication among running LSL scripts takes the form of text strings sent through channels, each channel denoted by a 32-bit integer. Publish the channel number and message format, and any object in the vicinity can talk with you. Objects in Second Life can interact with each other in ways independent of LSL, too. For example, objects with the physical attribute can collide with each other and push each other around.
Exciting though it is to be able to program things as visual and interactive as the content of Second Life, LSL has plenty of drawbacks. It is not a particularly full-featured language: it lacks, for example, proper arrays, which makes it impossible to build efficient higher-level data structures like hash tables. Worse, there is no way to share code across scripts; “libraries”, to the extent that such exist, are just pieces of source code that you can paste into a script.
LSL has security holes, too: for example, how do a set of cooperating LSL scripts open a private channel among themselves? Obviously, if you want to break some code that's communicating on a particular channel, and you know the channel, you can spoof a valid message, so there's a way to prevent that, right? Well, not really, no. The best you can do is set the permissions on a script that chooses a channel number so that other people can't edit the script and see the channel-choosing algorithm you used, but there's never any truly reliable way to guarantee that your channel won't collide with one used by some other object in the vicinity.
Another problem with interobject communication in LSL is speed: channels appear to be implemented with operating system sockets or some other relatively slow mechanism—certainly they don't seem to have the throughput you would expect if they were implemented as a message-delivering data structure within a single process. This means you can't treat message-sending as a sort of interobject method invocation: it's orders of magnitude too slow.
Neither Java nor LSL makes it really easy to create systems where pieces can cooperate closely with other pieces that are not fully trustworthy. What kind of implementation would be required to do that, and what kinds of things would you be able to build with it?