Better support for C extensions in TruffleRuby.

The first way to avoid problems with the translation of Ruby objects to native pointers is to avoid doing as much as possible. By default we'd replace every array of VALUE objects (which would be stored on the native heap) with a managed object that could store the object without conversion. We could also allocate managed structs instead of native one, but this became complicated with complex structures which had nested members of fused arrays.

The second way to solve these problems was to use the functions Sulong provided to convert between managed objects and native handles. These functions were hard to use without introducing memory leaks however, and did require that we patched every point at which a Ruby object needed to be converted to or from native memory.

The idea is fairly simple. C extensions will never see raw Ruby objects, they will only ever see wrappers that know how to convert themselves to native pointers, and at every point where a Ruby object needs to be extracted from a wrapper we know there should only ever be a wrapper or a native pointer. This makes it easy to convert back from a native pointer to a wrapper. Best of all C extensions don't have to know this is happening, so although it required a lot of changes to our C code to wrap and unwrap values that is as far as the changes go.

It was also important that these wrappers didn't cause the objects they wrapped to live longer than expected. This was a real problem with the handle conversion we used to do, and we didn't want to make it worse. Wrappers obviously need to keep a strong reference to the object they wrap, and objects should also keep a strong reference to their wrapper, but converting an object to a native pointer should not stop it from getting garbage collected at some point, but equally it mustn't be collected too soon.

MRI keeps objects alive in two ways when they are being used in a C extension. Any object still on the stack will be seen by the GC and kept alive, but that isn't enough to preserve values which may have been assigned to a field in a structure. MRI allows these to be kept alive by associating the structure with a Ruby object, and allowing that object to mark other objects it has references to. So, when the garbage collector traverses all the objects in your Ruby heap it calls these custom mark functions and the objects will be marked as live as long as the owners are. There's just one problem, we don't have a GC which can call custom mark functions, we have to work with any GC on the JVM. We also can't change the GC to look for native pointers on the stack which should also keep their respective objects alive.

We can solve this by keeping two lists of objects that need to be kept alive. Then each time we convert a wrapper to a native pointer we will add the wrapper to the lists, and it will in turn keep its object alive. One list is for those objects with pointers on the stack. We can create this list whenever we enter a C extension, and destroy it again when we finish the call. The other list is a fixed size buffer of every wrapper converted to a pointer. Whenever this list becomes full we'll run any marking functions associated with live objects and attach lists of marked objects to their owners. Let's see what that looks like with the example we've used before (I've left off the stack list to keep the diagrams from becoming too cluttered):

As you can see, since the handle table now only holds weak references to objects, once the preservation table has become full and the markers have been run there are no strong references from the preservation table or the handle table to the actual objects. We no longer need to write special finalisers for the object we create, as long as we ensure the marking functions do not maintain any strong references to the objects.

130 additions, and 1,402 deletions is the best kind of commit to be able to merge. This new approach has allowed us to remove almost all our patches for C extensions, even for complex ones such as zlib, OpenSSL, or pg.

Notice we've only removed almost all patches. There are however still some fundamental differences between us and MRI, but they are much smaller than they were.

Translating between Ruby objects and native pointers requires updating a global hash table, which is relatively expensive. We reduce that cost by tagging the pointers for common types, so true, false, nil, and so forth always convert to the same native value and never need to touch the hash table. Likewise fixnums can be tagged to cover most of their range, and we can probably use a similar technique for floating point numbers.

We are also doing work to reduce the costs of our GC marking technique. We can make an assumption that GC marking functions are not used and avoid those costs as long as we can safely recover as soon as markers are introduced, and we can reduce the cost of preserving objects on the stack in many situations.

Having applied some of these techniques we have benchmarked repeated calls to C extensions being up to 3 times faster than MRI 2.4 in some cases, though not yet for all our tests. We'll continue to work on performance in this area.

As I mentioned above we're still working on some changes to improve compatibility even more, and we'll continue to benchmark and improve performance. We expect any remaining problems to be related to specific functions in the Ruby C APIs rather than being more fundamental compatibility issues, and we'll be expanding our testing of gems considerably in the near future to help find and resolve these.