Product News from the Core Team

LINQ for the Compact Framework 3.5

2008-09-24T11:40:00Z

Db4o for .NET 3.5 ships with an optimizing LINQ provider. This provider is able to optimize the execution of a query by analyzing the LINQ expression tree emitted by the C# (or VB.NET) compiler to produce an equivalent SODA query which can be executed by db4o at full speed.

The Compact Framework 3.5 however includes no support for expression trees. No expression trees, no optimizing LINQ provider. A saddening state of affairs for Compact Framework db4o users.

As it turns out, expression trees are provided by the System.Linq.Expressions namespace which is not available in the standard Compact Framework 3.5 distribution.

Mono to the rescue

The Mono project provides a free software implementation of the .net framework including LINQ and the System.Linq.Expressions namespace.

Hypothesis: will the Compact Framework compiler emit expression trees if it sees a reference to a compatible assembly that provides a System.Linq.Expressions namespace?

Yes it will. LINQ for Compact Framework seemed to be possible.

There was a remaining issue. When the db4o LINQ provider cannot optimize a query down to SODA it can simply execute it as a LINQ for Objects query. This is accomplished by invoking the Compile method on the expression tree. The returned delegate can be used to evaluate the expression whenever needed. But the Compile method relies on System.Reflection.Emit which is again not supported by the Compact Framework. Hmpf.

The final piece of the puzzle appeared when Mainsoft contributed an expression tree interpreter to Mono. Expression tree evaluation that didn't rely on System.Reflection.Emit. Yummy.

So thanks to the hard and loving work of the Mono and Mainsoft guys we are able to ship to you the first optimizing LINQ provider for the Compact Framework.

A nice hack indeed.

Collected Generics, Volume 1

2008-09-12T11:49:00Z

Finally we are starting to harvest the fruits of our work on pluggable type handlers. As a first step, please welcome .NET generic collections to the exclusive club of types natively supported by db4o in the upcoming 7.6 build.

In principle, .NET generic collections could already be persisted in previous versions, but there was no dedicated handling in place for them. db4o would not recognize them as collections and would simply handle them as any other object - inspect their internal layout and persist the field values found. This approach came with some pitfalls. Most notably, db4o would not recognize the collection's internal state objects as parts of a single, atomic unit - in C/S mode, concurrent updates from multiple clients against the same collections could have resulted in inconsistent internal collection state in the worst case. Furthermore, issues could arise with specific collection implementations: Dictionaries, for example, would persist hash codes alongside keys/values and could stumble when attempting to reload this data in a different runtime version (as GetHashCode() implementations may change yielding different hash values). Plus, last but not least, dedicated collection handling opens the path for performance optimizations.

With the current type handler infrastructure in place, providing custom handling for .NET generic collections as its first "real world" application went pretty smooth - actually the bulk of the effort was spent on the implementation of combinatorial regression testing of permutations of collection, field and element types - take a look at GenericCollectionTypeHandlerTestSuite, if you're interested. As of now, dedicated type handler support is available for the following collection flavors:

List<T>
LinkedList<T>
Stack<T>
Queue<T>
Dictionary<K,V>
SortedList<K,V>
SortedDictionary<K,V>

These collections now should blend in seamlessly with the behavior of non-generic collections. More to come - but if you're missing a dedicated type handler for a specific flavor, of course you're also invited to contribute. ;)

Latest production build: 7.4.58.11547

2008-09-02T10:51:00Z

here is a short info to resolve a race condition between our newsletter sendout and the very latest production build:

We got informed about a serious (blocker) bug with a deadlock in committed callbacks after our newsletter had already been released to marketing: COR-1377

This bug received the highest priority and it has has been fixed immediately.

A new production build with the build number 7.4.58.11547 is online in the meanwhile. If you downloaded 7.4 right after the newsletter and if you use committed callbacks or UUIDs, we recommend to check the build number you are using (call Db4o#version() / Db4oFactory#Version() ) and to download the very latest build once again if it is lower than 7.4.58.

To our knowledge this bug has been present in all 7.2 builds.

Performance Contest Results

2008-08-27T10:16:00Z

Two months ago we invited the db4o community to take part in the first db4o performance contest. Now the contest has been completed. Here are the results.

Three entries are eligible for prizes:

June 15
Andrew Zhang submitted a patch that improves the speed to sort queries.

June 19
Andrew Zhang submitted a UTF-8 string encoder.

July 17
Erik Putrycz submitted a patch to reuse query results in subsequent similar queries.

Andrew and Erik wrote own Poleposition circuits for their patches to be able to measure performance improvements. They have been incorporated into the db4opolepos project in our SVN as circuits "Hockenheim" and "Hungaroring".

The three patched jars that Andrew and Eric supplied have been renamed to db4o-7.2-sortedquery.jar, db4o-7.2-utf8.jar, and db4o-7.4-cachedqueries.jar to make it easy to distinguish the results. These jars have also been checked into the db4opolepos project. Anyone can easily reproduce the performance race by running org.polpos.AllRacesRunner from this project.

The results from running this benchmark on our hardware are attached to this posting as PerformanceContest.pdf.

Our impression from reviewing the results and the patches:

- The effect of the tuned quicksort algorithm on query sorting is absolutely amazing. Improvements are indeed in the range of 20x to 300x as you can see in Hockenheim#query_ascending and #query_descending on pages 52 and 53 of the attached Poleposition results. The patch is very straightforward and very nice, so it can immediately go into production code as is. A perfect submission.

- The UTF-8 string encoder reduces the required slot length for stored strings. Shorter slots result in faster writes as you can see in Nurburgring#write on page 24 of the attached Poleposition run. A performance improvement of about 20% is measurable. Incorporation of the patch code into the db4o core would require some more work: Probably we would add UTF-8 as a third configurable string encoding. Possibly we could make it the default. In any case we would have to continue to support the existing Unicode string encoding. The submission shows very nicely how easy a UTF-8 string encoder is. We should make one available soon.

- The idea to cache query results is great. A working implementation is certainly an order of magnitude more difficult to write than the other two submitted patches. Somehow the Hungaroring benchmark shows a slowdown instead of an improvement. After Erik's comments that our Trunk code got slower for concurrent queries, we have also compared running the patched Jar against Trunk and Trunk still turns out to be faster. Erik's submission also does not exactly follow the rules of the contest:
The patch is not applied to the 7.2 workspace, so a direct comparison run to other submissions is not completely fair. In any case we really liked the idea and we have to give credit for trying.

So how do we judge the three submissions and who gets first, second and third prize?

Andrew's submissions both are measurable and they are both in compliance with the rules that we have set up. Erik's submission is a great idea, but it is not 100% complete and we have not yet seen the improvement clearly documented in a Poleposition run.

The core team reviewers suggest to award prizes as follows:
1st USD 3000 Andrew Zhang
2nd USD 2000 Andrew Zhang
3rd USD 1000 Erik Putrycz

If anyone in our community has a different opinion, please post it to our performance forum.

Andrew and Erik: Thank you very much for your submissions. You have deserved the prizes!

For all other developers that tried to improve performance but did not complete a submission on time: Thank you for trying!
If you have suggestions how to improve the Performance Contest, please tell us about them. Maybe it would make sense to let the contest run for a longer period of time than 2 months? What do you think?

We are very happy to have received usable patches in this first db4o performance contest. Hopefully we will get even more contributions the next time.

It is great to work with such a nice community!

Wanted: TypeHandlers

2008-08-22T16:29:00Z

You were playing with the idea to contribute to db4o? So far you did not find a nice, easy and short task to take on? Here is your chance: Write a TypeHandler for db4o and get your code included in the db4o core distribution tomorrow.

db4o comes with a big promise: Just call ObjectContainer#store() and any object and all attached objects will be automagically stored. The challenge to do this is high: Every possible class out there needs to be "understood" by db4o and an efficient byte array representation needs to be created on storage media. Indexing of "parts" of objects and querying is part of the task. With evolving languages and an increasing number of class constructs that need to be supported we decided to make it easier to write the code for marshalling and unmarshalling: Enter TypeHandlers.

TypeHandlers define how objects of a specific class are stored, loaded and queried by db4o.

By default db4o stores and restores all non-transient fields of an object. By writing TypeHandlers it is possible to alter the behaviour. A TypeHandler could decide to store only some members of a class or it could decide to store a completely different and more efficient representation.

The possible benefits of using a Typehandler:
- Storing and restoring objects can be faster.
- The database file can be smaller.
- Objects that consist of sub-objects (Example: collections) can be guarded against inconsistencies in concurrent multi-user environments by preventing "the object" from being split, but using one single contiguous slot as the marshalled representation. ( We will soon also demonstrate a stateful collection implementation over multiple pages, also powered by a Typehandler.)
- Fields that are intended to be transient do not need to be stored.
- Typehandlers can alter behaviour even if the sources of the persistent classes are not available or can not be altered.
- Querying support.

TypeHandlers are expected to do the complete marshalling of an object by writing/reading directly to/from the marshalling buffer.

About a year ago the TypeHandler4 interface consisted of more than 40 methods. Today only 5 methods remain. To make sure that the TypeHandler interface is ready for heavy duty use, we have already used it ourselves to implement some collection typehandlers. See:
com.db4o.typehandlers.CollectionTypeHandler (Java)
com.db4o.typehandlers.MapTypeHandler (Java)
Db4objects.Db4o.Typehandlers.ListTypeHandler (.NET)
Db4objects.Db4o.Typehandlers.MapTypeHandler (.NET)

In the latest development build (7.5.x.y) these TypeHandlers are already turned on to do real work storing, loading and querying some selected collections. To see how internal TypeHandlers are configured, take a look at:
com.db4o.internal.TypeHandlerConfigurationJDK_1_2 (Java)
Db4objects.Db4o.Internal.TypeHandlerConfigurationDotNet (.NET)

You will notice that many TypeHandlers are still missing if the full featureset of JDK and .NET collections is to be supported. Here is where we would like to ask you for your help:
Could you write a TypeHandler for us for a collection class that you think we need to support?

Adapting and registering one of the already existing typehandlers for your favourite collection will probably take you less than an hour of work. You can simply follow the pattern that is already there.

However writing the TypeHandler code is not the complete story. Since we want to be absolutely sure that a new TypeHandler works correctly, the bigger part of the implementation work is required for testing. We want to make sure that objects stored with an older version of db4o can be read and stored even after your new TypeHandler is installed.

For this purpose we developed the "Db4oMigrationTestSuite" test.
On Java you can run this class directly. You will find it in our SVN in the db4ojdk1.2 project.
( If you have not worked with the db4o sources before, here is a guide to get started. )

If you develop for .NET you would check out the db4o.net project from our SVN. You would typically work with the Db4o-2008.sln solution in the top level folder. To run the tests, the Db4objects.Db4o.Tests-2008 project would be your startup project. To save time by not running all the tests every time you can edit Db4objects.Db4o.Tests.AllTests in the Db4objects.Db4o.Tests-2008 project: If you want to run migration tests only, you can comment out all the entries in the #TestCases() method and add the following instead:
typeof(Db4objects.Db4o.Tests.Common.Migration.Db4oNETMigrationTestSuite)

Both for Java and .NET:
If you want the Db4oMigrationTestSuite to truely run against all relevant old db4o engines, you should check out the db4o.archives project from our SVN also.

Once you have testing set up and running, you are ready to go to write your own test for updating your favourite collection.

For an example how an update test could look like, please see:
com.db4o.db4ounit.common.handlers.ArrayListUpdateTestCase (Java)
Db4objects.Db4o.Tests.CLI2.Handlers.GenericListVersionUpdateTestCase (.NET)

Only when you have the test in place it makes start to implement your own TypeHandler as a more efficient method to store the collection. Test first, remember?

Should you have any problems to get started, please ask for help in the db4o product developer forum. It is a good idea to announce any work that you are about to do in the forum in any case. That will ensure that the work for a specific collection is only done once.

Now that you are all set to start:
Good luck for writing your TypeHandler!

All the new Typehandlers we have built so far are now available in the TRUNK of our SVN and with our continous build.

They help to make storage of collections more efficient:
- Less state is pushed around than with our previous translator approach.
- Maps no longer store an internal Entry object for each entry.
- All .NET collections, including all .NET generic collections will soon be handled efficiently with Typehandlers. Current issues with .NET generic collections in concurrent multi-user setups will soon be history.
- Old database files can be read with new Typehandlers installed. Future marshalling format changes are easily possible by writing and installing yet further new Typehandlers.
- We have a very nice hook into collection treatment that we will be able to use for much faster collections.

Enjoy!

Synchronization issues in networking C/S fixed

2008-08-02T13:32:00Z

We have fixed some thread synchronization issues in the C/S protocol and put new versions (rev. x.y.54.11278) for all branches online in the download center. Users with concurrent networking C/S in connection with lazy queries are recommended to upgrade. For more information, please refer to the corresponding tracker entry.

Debugging TA Enhanced Code and the Observer Effect

2008-07-29T00:03:00Z

Most of those who pair programmed with me know that I'm not very fond of debugging code by stepping through breakpoints inside an IDE. I'd much rather construct hypotheses on why the code is failing and systematically test them with carefully placed assertions and print statements.

One reason for that is the observer effect of debugging: the execution environment can be a little different, timings are usually changed, object inspectors might end up calling methods with unwanted side-effects and so on.

The last point is a particularly important one to keep in mind when dealing with code that's been enhanced for Transparent Persistence.

Let's say we have built the latest and greatest library management application and now want to put a web front end on top of it. We've been following web development trends and so we know AJAX is a must these days. We just need some simple way to transfer the objects from the server to the browser javascript runtime: JSON to the rescue. Good thing we can reuse some reflection based JSON serialization code we had laying around.

We write a quick spike to see how well it works:

    Dim book = New Book("The God Delusion")
    book.AddAuthor(New Author("Richard Dawkins"))
    
    Console.WriteLine(New JSONSerializer().Serialize(book))

Which give us the expected output:

	{ _title: "The God Delusion", _authors: [{ _name: "Richard Dawkins" }] }

We try one more spike to convince us we can get JSON out of a db4o query result:

    Using db = OpenDatabase()
        For Each book In db.Query(of Book)()
            Console.WriteLine(New JSONSerializer().Serialize(book))
        Next
    End Using

Unfortunately this gives us something completely unexpected and it's not the Spanish Inquisition:

	{ _title: null, _authors: null }

We place a breakpoint inside the loop so we can inspect the book instance:

It sure looks right and oddly enough we get the right output this time:

	{ _title: "The God Delusion", _authors: [{ _name: "Richard Dawkins" }] }

What's going on?

Transparent Activation is going on or rather NOT going on:

Our Book and Author classes were enhanced for Transparent Activation, Db4oTool injected the code required to implement IActivatable as well as the code that transparently activates the object before any field access;
JSONSerializer uses reflection to access objects fields;
Reflection completely bypasses any code introduced by Db4oTool which explains the "{ _title: null, _authors: null }" output, the object was never activated;
When inspecting an object in the debugger watch window its properties are accessed;
Accessing a property of our enhanced Book object causes its *transparent* activation which also explains why we got the right output aftewards;

Ok, now that we have a logical explanation for what's happening how do we get our application to work as expected?

Since JSONSerializer bypasses activation we have to ensure the object graph is completely activated before JSONSerializer gets to see it:

    Using db = OpenDatabase()
        For Each book In db.Query(of Book)()
            db.Activate(book, Integer.MaxValue)
            Console.WriteLine(New JSONSerializer().Serialize(book))
        Next
    End Using

And that's it.

Hopefully that will save you some frustration next time you need to debug Transparent Persistence enhanced code or to have it interacting with reflection based libraries.

The complete code can be found here.

Have fun!

Performance Contest - for one more month

2008-07-17T10:38:00Z

Do you like profiling? Do you like tuning code? If you do, how about taking part in the db4o performance contest? We offer USD 6000 in prizes for the best contributions to make db4o faster.

We have made it very easy for you to get started by providing an Eclipse workspace set up with the db4o sources and the Poleposition benchmark.

Links to get you started:
Performance Contest page
"Getting Started" video
Performance Contest Forum

The contest has been running for a month already and we are seeing first contributions coming in.

Andrew has improved sorted queries and provides a UTF-8 string encoder.
Erik is working on a query cache to make queries reusable.

Can you beat their patches? I am sure you can.
The contest is still open for contributions for one more month until August 15.

Implications of instrumenting assemblies for Transparent Activation / Persistence in db4o - Part I

2008-07-14T14:28:00Z

As the popular saying tell us, there's no such thing as free lunch. It is no different for work related subjects.

When we introduced Transparent Activation (TA) concept we knew that it could make developer's life easier (by presenting a simpler model for object life cycle): developers would not be required to think about activation depths again; just let db4o activate your objects when it's need.

IMHO, it was a great improvement since it not only simplifies developer's life as well brings some performance gains also (once objects get activated only when needed there's no wasted time activating objects not needed). But this was not without coasts: to benefit from this simpler model, developers were required to implement IActivatable interface for each object that they wanted to be TA. Also, once enabled, any non TA aware object will be fully activated at retrieval time.

Time passed and we introduced a way relieve developers from the need to implement Activatable interface, i.e, to make classes TA aware with no effort from developers. This is accomplished through a technique called instrumentation in which we dive into byte code level doing something like:

Find each type declared in a specific assembly (.net) / jar or class (java)
Analise each type found in step 1 to check whether it is a potential candidate for TA or not
If we found a potential candidate for TA

Make the type to implement IActivatable interface
Implement IActivatable methods
Ensure that the object will be activated prior to any field access by inspecting each field access in this type and inserting the required Activate() / activate() call prior to it.

Save the binary component

A pretty straight approach, I'd say.

But even when using instrumentation there are some implications that developers should be aware of. To illustrate, think about assembly signing; this is a process in which a developer "seals" an assembly contents by digitally signing it (the whole concept is beyond this post).

Basically, by signing an assembly a developer is providing ways to administrators to selectively grant access to assemblies based on the private key (related to the public key) used to sign it; also, signed assemblies are tamper resistant which means that changes (in byte code level) don't go unnoticed.

By the above description it's clear that instrumenting a signed assembly turns it invalid afterward (since its contents gets changed) and any tentative of loading it will fail with a invalid signature error.

So, after instrumenting signed assemblies you will need to sign them again but, IMHO, you should never, ever, instrument assemblies that are not under your control, so I'd suggest you to just change your build scripts so instrumentation takes place prior to signing.

To help developers detect this, starting from version 7.4.52 Db4oTool will emit an warning when asked to instrument (whether for TA/TP or Native Queries optimizations) a signed/delay signed assembly.

Just for the sake of completeness I've included the following walk through on how to create assemblies that are instrumented and signed (for a in depth discussion on signing assemblies please, read this).

In Visual Studio, open project properties window and select signing tab.
Select Sign the assembly;
Expand Choose a strong name key file drop down listbox and either, select a previously created key or create a new one by selecting <New...> option;
Select Delay sign only checkbox;
Save your project and build
Instrument your assembly (either running Db4oTool or through Db4oToolEnhancerMSBuildTask)
After instrumentation, run sn tool as follows to finish the sign process:

sn -Ra myassembly.dll mykey.snk

That's it! Your assembly is instrumented and signed.

In the next post I'll address some other implications / pitfalls regarding instrumentation for TA/TP.

Thoughts?

db4o-7.4 Development Release

2008-06-24T16:03:00Z

The following Jira tasks and bugs were resolved for db4o-7.4 development release:

DRS-98 - NullPointerException in replicate method
DRS-97 - Ensure dRS.NET can gracefully handle delegates in untyped fields and arrays
DRS-95 - dRS.NET fails with arrays/untyped fields
DRS-94 - Replication fails if the java platform doesn't support serializable constructor and the objects to be replicated don't have default constructors
DRS-93 - Replication fails for arrays in untyped fields
COR-1297 - Spike: Class derived from ArrayList using custom Typehandler
COR-1295 - Convert ListTypeHandlerTestSuite to .NET
COR-1291 - Backport "[Linq] Enum comparisons are not optimized" to 7.2
COR-1289 - Clean up db4obuild for opening to the public
COR-1284 - Defragment against unknown classes runs into NPE on btree index lookup
COR-1283 - Update old multidimensional arrays to null bitmap handling
COR-1282 - [decaf] functional test for resolving db4oj/db4ojdk1.2 conversion issues
COR-1281 - Running out of disk space causes unrecoverable data corruption
COR-1279 - Delegate in untyped arrays cause ObjectNotStorableException
COR-1276 - prepare Jprobe paircast and blog
COR-1275 - Db4o should not wrap exceptions thrown during event handling
COR-1265 - ArrayHandler: Defragment old databases
COR-1264 - ArrayHandler: Updating from old databases to new format
COR-1263 - ArrayHandler: Bitmap Marshalling Format
COR-1262 - Pass configuration on to reflector on startup
COR-1261 - Move constructor handling into reflector
COR-1255 - Provide Feedback on latest OMG SBQL vs, LINQ comparison
COR-1254 - _blob.ReadFrom() throws Object reference not set to an instance of an object. in C/S mode
COR-1247 - Create releae notes for production release 7.2
COR-1246 - Create release notes for 6.4 stable release
COR-1227 - Prepare 7.2 "Production" release (freeze trunk)
COR-1216 - TracerBullet: Deletion for Java ArrayList using the new Typehandler
COR-1214 - Db server closed when upgraded from 5.5 version to 6.4.14.8331 with btree based freespace management system
COR-1153 - [Linq] Enum comparisons are not optimized
COR-1129 - TracerBullet: Special use cases for Java ArrayList using the new Typehandler
COR-897 - Concurrency tests for close/timout scenarios

flushFileBuffers(false) and the C in ACID

2008-06-18T15:24:00Z

The short story:
Configuration.flushFileBuffers(false) is no more. If you still want to configure the behaviour that used to be available in db4o 6.1, you can do so by setting up IoAdapters as follows:

RandomAccessFileAdapter randomAccessFileAdapter = new RandomAccessFileAdapter();
NonFlushingIoAdapter nonFlushingIoAdapter = 
                                 new NonFlushingIoAdapter(randomAccessFileAdapter);
CachedIoAdapter cachedIoAdapter = new CachedIoAdapter(nonFlushingIoAdapter);
Configuration configuration = Db4o.newConfiguration();
configuration.io(cachedIoAdapter);
Db4o.openFile(configuration, PATH);

The long story:
We do a lot of things for consistent transactions (for the C in ACID). A commit of a db4o transaction consists of 4 write phases:

Writing a commit log to the database file
Switching the database file to commit mode
Committing all slots
Switching the database file to normal mode

Because a cache at any level may turn around the write order of individual slots and because a failure may end up in a partial write, it is essential that all data is flushed to the device after each one of these 4 phases.

This flush call takes a lot of time. It is the slowest part of the commit. On a system where you are sure enough (100% ? ) that the commit process always is successful, a tuning switch that skips flushing sounds like a smart way to improve performance.

That's what we thought and why we added the #flushFileBuffers(false) configuration switch in the first place.

Starting with db4o 6.4 we have added a new CachedIoAdapter to the IO layer as the default. This IoAdapter uses the #sync() calls as a signal to flush it's pages to the database file. What we did not think about well enough when we added this functionality: If users use the #flushFileBuffers(false) tuning switch, they also skip the flush of the CachedIoAdapter.

This issue does not surface as long as the database file is always shut down correctly, so it took us a long time to realize how serious it can be. Since we have switched db4o 6.4 to "stable", there have been a couple of support discussions like this one:

User: "We have switched to db4o 6.4. We are sometimes getting new exceptions like IncompatibleFileException."
db4objects: "Uhoh. Are you maybe using #flushFileBuffers(false) in productive use? Please don't do that. Don't tell us that you are."
User: "Yes we are."

After investing some time and thought we have figured that the CachedIoAdapter makes it much much more likely that corruption can happen if a database file is not shut down correctly (with the #flushFileBuffers(false) configuration call). Because this is so serious (it leads to corrupted database files), we have decided to turn off this configuration switch completely.

A similar functionality as in 6.1 is still available, if you really know what you are doing and if you know that you are taking the risc of corrupted database files if your system is halted because someone hits the power-off switch by accident in the middle of a commit. With the layer of IoAdapters, as outlined at the beginning of this posting, the risk of corruption is fairly small, just as it used to be in 6.1. Corruption will only happen if the OS cache or disc cache changes around the order of writes.

The changes are effective and the new IoAdapter is available from iteration 48 and revision 10977 onwards, e.g. from db4o versions:
6.4.48.10977
7.2.48.10977
7.4.48.10977

For previous stable 6.4 versions we strongly recommend not to use the #flushFileBuffers(false) setting in production use.

Many thanks to our users for reporting their problems!

Committed callbacks, pushed updates and read committed isolation revisited

2008-06-17T15:34:00Z

Some time ago we presented Pushed Updates as a sample application for Committed Callbacks in this blog. Recently we put a similar mechanism in a slightly different context. Since we encountered some perceived ambiguity concerning db4o transaction semantics, and since we were in for a little surprise ourselves, I'd like to briefly rehash this topic.

Pushed Updates basically are a mechanism to provide READ COMMITTED isolation from the point of view of your application. Wait - doesn't db4o provide that out of the box?!? Well, it does. Any query is guaranteed to return objects in their most recently committed state - unless they are still in memory on the requesting client, that is. db4o cannot change the state of of a live object that may be involved in application level processes at the same time. Only the application can judge whether it is safe to modify a given object, and Committed Callbacks are the way to transfer this responsibility.

Note that "in memory" above does not mean "in use by the application", and not even necessarily "referenced by the application". An object may still survive in the reference system some time after it has been discarded by the application, until the GC eventually hits. There is no remotely performant way to check whether there exist hard references to a given object.

So much for the theory, let's look at the code. The core of the Pushed Updates example looked like this:

private EventListener4 createCommittedEventListener(final ObjectContainer objectContainer){
return new EventListener4() {
    public void onEvent(Event4 e, EventArgs args) {
      ObjectInfoCollection updated = ((CommitEventArgs)args).updated();
      Iterator4 infos = updated.iterator();
      while(infos.moveNext()){
        ObjectInfo info = (ObjectInfo) infos.current();
        Object obj = info.getObject();
        objectContainer.ext().refresh(obj, 2);
      }
    }
};
}

EventListener4 committedEventListener = createCommittedEventListener(client);
EventRegistry eventRegistry = EventRegistryFactory.forObjectContainer(client);
eventRegistry.committed().addListener(committedEventListener);

This code registers a callback for committed events. When invoked, it will traverse all the objects that have been updated during this commit and refresh them. (The magic depth of two is a reverence to collections, forcing them to refresh their elements, too.)

Now this works nicely - in networking C/S mode, where the communication layer between client and server takes care of the correct "translation" of object identities. In embedded C/S mode, however, there is no translation - clients run in the same "live object space" as the server, the only thing they don't share is the weak reference system that defines their notion of object identities. And so the ObjectInfo will contain the server's instance of this persisted object, while the client will have its own version that it won't be able to map to the "stranger".

The fix is easy (and works for networking C/S, too, of course): We have to take care of the translation ourselves via ID lookup. In code:

public void onEvent(Event4 e, EventArgs args) {
Transaction trans = ((InternalObjectContainer)client).transaction();
ObjectInfoCollection updated = ((CommitEventArgs)args).updated();
Iterator4 infos = updated.iterator();
while(infos.moveNext()){
    ObjectInfo info = (ObjectInfo) infos.current();
    Object obj = trans.objectForIdFromCache((int)info.getInternalID());
    if(obj == null) {
      continue;
    }
    client.refresh(obj, 2);
}
}

Here we have to fall back on the InternalObjectContainer interface. Don't be too scared of this - it's not really under the hood stuff, but it's indeed intended to be used for 3rd party implementations of pluggable db4o features intended to be run inside the core, like type handlers, reflectors - and callbacks. Note that this variant also keeps the handler from further processing of objects that haven't been known to this client, anyway.

Bottom line: Transaction semantics and live in-memory objects are not entirely trivial to align, object identities can be tricky, and this is the recommended generic way for doing Pushed Updates. For networking C/S, the original Pushed Updates example is fine as it was.

Sharpen your Java app now: Java to C# converter released as free software

2008-05-20T16:27:00Z

As of today, db4objects releases its Eclipse based Java to C# source code conversion tool "sharpen" as free software.

sharpen is the "secret sauce" that enables us to provide the [db4o object database engine] in native versions for the Java and .NET platforms from a single code base.

Features include:

mapping between Java and .NET native type systems
compliance with C# coding conventions
user defined namespace/class/method mappings
method to property mappings
Generics support
partial conversions
mixing native and converted C# sources
VS solution file support
Ant integration

sharpen is used extensively in the db4o build system, generating most of the db4o engine core code and the unit test suites from the Java sources.

Before you start dreaming: sharpen is not a "magic wand". Don't expect to feed it an arbitrary Java app and receive a running .NET version at the push of a button. The translation process will require some design compromises on the Java side, and parts of the .NET code still have to be hand crafted and integrated into the converted sources. Still, we believe sharpen provides an amazingly smooth way to bridge the gap between Java and .NET and create real cross platform applications.

We are looking forward to see which sharpen propelled applications will cross the language chasm, and we're hoping on your feedback and your contributions that will help to cover automatic conversion of even more Java constructs and idioms, moving Java/.NET cross platform integration to a new level.

sharpen is released under the GPL. The svn repository, documentation and issue tracker are online. All that's missing now is you bringing your Java app to the .NET platform. To get you started, here is the full anatomy of a simplistic application developed with sharpen. Have fun!