Edit me
Champion
Proposal
To create a portable threads module. My (Will's) goal is to create a portable threads module that relies on the underlying threads implementation of the various Lisp implementations. In other words, we're mostly creating a standard package, with standard functions/classes/methods, translating into each implementation's native threading code. There may be a few extra utility functions that are provided as well.Issues
- Dan Corkill has packaged the GBBopen's multiprocessing interface (general doc) into a stand alone system for: Allegro, CMUCL, Digitool MCL, the experimental ECL/threads (mostly there), Lispworks, OpenMCL, SBCL, and SCL (as well as non-threaded CLISP, CMUCL, ECL, and SBCL versions). Gary King has made this ASDF-Installable and added a page to the CLiki about it. Contact Gary if something seems amiss.
July 30, 2007 -- Version 2.0 of the Portable Threads Interface from GBBopen has been released. In Version 2.0, process-wait-style operations are depreciated in favor of condition variables and thread hibernate/awaken operators. [Scheduled functions (V2.1) and periodic functions (V2.2) have now been added.]
Support for other implementations will need volunteers. Dan wants to maintain overall support for this file (as part of GBBopen), so please send bugs, comments, criticism, suggestions, enhancements, and extensions to him at corkill@GBBopen.org
- Will asks: Do we really need the atomic-* forms? They seem too 'strong' in that they underlying implementation uses, iirc, one lock for the system as a whole. They seem too 'weak' in that they don't affect the general setf situation. They're also easy to write using with-process-lock.
Dan says: The atomic-* forms are certainly not required (since they can be built using other thread primitives), but they are handy. They suffer from the same weakness as anything else that is protected using with-process-lock in the sense that if someone interacts without the lock, all bets are off. (The underlying as-atomic-operation macro is now exported and documented to facilitate extension to other atomic operators.) The *atomic-operation-lock* is used on CLs that don't provide thread/process scheduling control. Having one lock could be a contention issue, but in practice the atomic forms are short enough relative to other computations that there isn't significant blockage. If people are uncomfortable about having them in the interface, I'd be happy to move the atomic-* forms out of the portable-threads interface (and into another part of GBBopen).
Will says: I wonder if we think up a 'base' vs. 'utility' package for portable threads, moving the atomic forms into the utility package. There might be other useful things to put there as well (such as process queues? other ideas?)
Dan says: Unless the 'utility' package becomes very large, my recommendation would be to avoid the added complexity of multiple portable-threads levels. Since I see the goal of the portable threads package to be a single set of interfaces that are 'blessed' and can be relied upon across multiprocessing/threaded CLs (and a subset of those entities that 'do the right thing' on all CLs), it makes sense to keep them all together. If the 'utility' entities can be implemented using the 'base' entities alone, having them all together won't add more work to supporting a new or changed CL over just the 'base' level.
Volunteers
Add your name here ...- Joseph Manby / Lispworks 4.1.0
- Creighton Hogg / CMUCL,CLISP
- Dan Corkill -- Split out GBBopen's Portable Threads Interface
Tasks
Will- To do:
Flesh out this page...Provide examples from the portable threads module I already have.See above.- Think about what a 'portable threads' test suite would look like. (A set of tests for GBBopen's Portable Threads Interface is provided in this file.)
- Volunteers
- To do:
- Write somewhere on this page which implementations of Lisp you have available.
- Contribute to the discussion.
- Test, test, test. Be aware of inconsistencies in the documentation, implementations, etc.
- Fill out the list of threads implementations under the 'Resources' section.
Lisp thread implementations
- Allegro Common Lisp. See Multiprocessing.
- CMU Common Lisp
- Steel Bank Common Lisp
- GNU clisp
- OpenMCL
- Xanalys Lispworks
- MCL
- Scieneer Common Lisp
- Corman Common Lisp
- Thread Interfaces for Common Lisp
Other compatibility layers
- ACL-Compat ACL Compatibiity Layer.
- BORDEAUX-MP
- CLOCC multiprocessing
- Sven Van Caekenberghe's s-sysdeps
Categories
Gardeners Projects'Archived' discussions
- Should we just adopt the ACL multiprocessing conventions?
Will says: Apparently not. - Can you really sanely put an identical interface over native and non-native
threads? The latter will block in foreign calls, which in turn leads to different
semantics.
Will says: The idea (that is, my idea) is just to layer over the implementation's best effort at threads. My general experience with threading is that you shouldn't rely the 'semantics' to be all that well defined across operating systems, releases of the OS, implementations, etc. All I really mean by this is that it's hard to believe that SBCL running under Windows is likely to do the same kind of thing as ACL under Linux, just 'mostly' the same. In any case, we push the problem down from the namespace down to the implementation. - Where should a clueless volunteer start? Any recommended reading? --Creighton
Will says: You might try grokking Allegro's multiprocessing documents. If multiprocessing itself it brand new to you, you'll just want to do the usual Google search. - I'm a fan of gpl stuff but many are not (as evidenced by some particularly nasty exchanges on the c.l.l mailing list). I only bring this up because multiprocessing will likely have to deal with each implementation's internals somewhat intimately. What are the triggers for transmitting the gpl license for code in clisp or gcl, and how can this code be properly isolated to prevent the portability layer from aquiring a license not everyone agrees with and therefore rendering it ineffective? -- Fred LeMaster
Dan says: A portability layer is unlikely to insulate the user from the license of the underlying CL implementation, but I assume that is not the issue. If the license of a portability layer (such as GBBopen's Apache 2.0 license) is used with a CL with a different license (such as GPL), then the portability layer may be tainted by the CL license in the context of that use. There isn't a lot of code in the portability layer, and its authors' would need to accept that possible license interpretation. (In GBBopen's case, the Apache 2.0 license doesn't prevent someone from using the code in a GPL'ed context, however that use doesn't prevent someone else from using the code in another context that is not GPL'ed.)
Fred: I see, thanks for the clarification. - So on implementations that have no native threads, how much do we want to fake multiprocessing? Aside from wrestling with the garbage collector, an explicitly cooporative system in which threads run until they yield could probably be implemented pretty uniformly, but how attractive would such a system be? --Fred LeMaster
Will says: I'm completely uninterested in writing CL-based multiprocessing, but if someone wanted to take up the challenge, great. It shouldn't be on the main line, though.
Fred: Well, that makes the job simpler at least. - Should we remove support for hibernate/awaken (suspend/resume)?
[Note: hibernate-thread and awaken-thread have been retained in Version 2.0 of GBBopen's Portable Threads Interface, as the issues that were discussed below have now been addressed by using sleeping rather than "deep hibernation" where needed.]
Dan says: These are heavy handed mechanisms that can easily get one into trouble. They are not supported for SBCL/sb-threads now, and I have learned from Gary Byers that (documentation notwithstanding), a process cannot hibernate itself in OpenMCL (without deadlocking the resumer). See the file simple-test.lisp for a simple native (non-portable-threads) example of this problem. I'm thinking very seriously about removing hibernate-process/awaken-process from the GBBopen portable-threads interface. Opinions?
Will says: I can see giving up awaken-process, but it seems like hiberate-process is still useful, the OpenMCL issue notwithstanding ('regular' MCL used to support this under the old Mac OSes). Basically, I think of this as a hint to the scheduler that says the current process may be (but is not required to be) rescheduled. If all the underlying systems essentially had ACL's process-allow-schedule, I would support the stronger semantics. We also need to support process-specific sleep, even if it's:
(defun process-sleep (secs) (process-wait "wait" (lambda () nil) secs))
On the other hand, awaken-process does seem heavy-handed.
Dan comments: I see hibernate-process as the heavy-handed one (especially if called from another process). One has be to be very careful of the state of the process being hibernated (such as locks, I/O, ...) and of what can be done with that process until it is awakened. BTW, the GBBopen Portable Threads Interface does provide the scheduler hint via thread-yield. Also, the Common Lisp sleep function does the right thing in all multiprocessing CLs that I'm aware of (sleeping only the calling process/thread). As for hibernate-process, I'm leaning to implementing a portable semaphore layer to use in place of hibernate/awaken. On some CLs, it might use hibernate/suspend internally, but only in a relatively safe and tested context.
Will says: Ok, I understand now. Yes, let's remove hibernate/awaken.
Dan says: I do want to be sure that the interface provides an efficient means of having a process wait for a long period without consuming processing resources. Having a process/thread 'sleep' until 'awakened' by some other process/thread is a fairly common pattern that needs to be supported effectively across implementations. [Version 2.0 of the Portable Threads Interface uses sleeping rather than hibernating where needed to provide a consistent 'hibernate' state that can run interrupt and symbol-value operations.]
