db4o Developer Community

db4o open source object database, native to Java and .NET
Welcome to db4o Developer Community Sign in | Join
in Search
More Search Options

Product News from the Core Team

This blog features product news right from the core developer team, once new features and functions get checked into Subversion, available as Continuous Build every 2 hours.

  • Latest production build: 7.4.58.11547

    here is a short info to resolve a race condition between our newsletter sendout and the very latest production build:

    We got informed about a serious (blocker) bug with a deadlock in committed callbacks after our newsletter had already been released to marketing: COR-1377

    This bug received the highest priority and it has has been fixed immediately.

    A new production build with the build number 7.4.58.11547 is online in the meanwhile. If you downloaded 7.4 right after the newsletter and if you use committed callbacks or UUIDs, we recommend to check the build number you are using (call Db4o#version() / Db4oFactory#Version() ) and to download the very latest build once again if it is lower than 7.4.58.

    To our knowledge this bug has been present in all 7.2 builds.

  • Performance Contest Results

    Two months ago we invited the db4o community to take part in the first db4o performance contest. Now the contest has been completed. Here are the results.

    Three entries are eligible for prizes:

    June 15
    Andrew Zhang submitted a patch that improves the speed to sort queries.

    June 19
    Andrew Zhang submitted a UTF-8 string encoder.

    July 17
    Erik Putrycz submitted a patch to reuse query results in subsequent similar queries.

    Andrew and Erik wrote own Poleposition circuits for their patches to be able to measure performance improvements. They have been incorporated into the db4opolepos project in our SVN as circuits "Hockenheim" and "Hungaroring".

    The three patched jars that Andrew and Eric supplied have been renamed to db4o-7.2-sortedquery.jar, db4o-7.2-utf8.jar, and db4o-7.4-cachedqueries.jar to make it easy to distinguish the results. These jars have also been checked into the db4opolepos project. Anyone can easily reproduce the performance race by running org.polpos.AllRacesRunner from this project.

    The results from running this benchmark on our hardware are attached to this posting as PerformanceContest.pdf.

    Our impression from reviewing the results and the patches:

    - The effect of the tuned quicksort algorithm on query sorting is absolutely amazing. Improvements are indeed in the range of 20x to 300x as you can see in Hockenheim#query_ascending and #query_descending on pages 52 and 53 of the attached Poleposition results. The patch is very straightforward and very nice, so it can immediately go into production code as is. A perfect submission.

    - The UTF-8 string encoder reduces the required slot length for stored strings. Shorter slots result in faster writes as you can see in Nurburgring#write on page 24 of the attached Poleposition run. A performance improvement of about 20% is measurable. Incorporation of the patch code into the db4o core would require some more work: Probably we would add UTF-8 as a third configurable string encoding. Possibly we could make it the default. In any case we would have to continue to support the existing Unicode string encoding. The submission shows very nicely how easy a UTF-8 string encoder is. We should make one available soon.

    - The idea to cache query results is great. A working implementation is certainly an order of magnitude more difficult to write than the other two submitted patches. Somehow the Hungaroring benchmark shows a slowdown instead of an improvement. After Erik's comments that our Trunk code got slower for concurrent queries, we have also compared running the patched Jar against Trunk and Trunk still turns out to be faster. Erik's submission also does not exactly follow the rules of the contest:
    The patch is not applied to the 7.2 workspace, so a direct comparison run to other submissions is not completely fair. In any case we really liked the idea and we have to give credit for trying.

    So how do we judge the three submissions and who gets first, second and third prize?

    Andrew's submissions both are measurable and they are both in compliance with the rules that we have set up. Erik's submission is a great idea, but it is not 100% complete and we have not yet seen the improvement clearly documented in a Poleposition run.

    The core team reviewers suggest to award prizes as follows:
    1st USD 3000 Andrew Zhang
    2nd USD 2000 Andrew Zhang
    3rd USD 1000 Erik Putrycz

    If anyone in our community has a different opinion, please post it to our performance forum.

    Andrew and Erik: Thank you very much for your submissions. You have deserved the prizes!

    For all other developers that tried to improve performance but did not complete a submission on time: Thank you for trying!
    If you have suggestions how to improve the Performance Contest, please tell us about them. Maybe it would make sense to let the contest run for a longer period of time than 2 months? What do you think?

    We are very happy to have received usable patches in this first db4o performance contest. Hopefully we will get even more contributions the next time.

    It is great to work with such a nice community!

  • Wanted: TypeHandlers

    You were playing with the idea to contribute to db4o? So far you did not find a nice, easy and short task to take on? Here is your chance: Write a TypeHandler for db4o and get your code included in the db4o core distribution tomorrow.

    db4o comes with a big promise: Just call ObjectContainer#store() and any object and all attached objects will be automagically stored. The challenge to do this is high: Every possible class out there needs to be "understood" by db4o and an efficient byte array representation needs to be created on storage media. Indexing of "parts" of objects and querying is part of the task. With evolving languages and an increasing number of class constructs that  need to be supported we decided to make it easier to write the code for marshalling and unmarshalling: Enter TypeHandlers.

    TypeHandlers define how objects of a specific class are stored, loaded and queried by db4o.

    By default db4o stores and restores all non-transient fields of an object. By writing TypeHandlers it is possible to alter the behaviour. A TypeHandler could decide to store only some members of a class or it could decide to store a completely different and more efficient representation.

    The possible benefits of using a Typehandler:
    - Storing and restoring objects can be faster.
    - The database file can be smaller.
    - Objects that consist of sub-objects (Example: collections) can be guarded against inconsistencies in concurrent multi-user environments by preventing "the object" from being split, but using one single contiguous slot as the marshalled representation. ( We will soon also demonstrate a stateful collection implementation over multiple pages, also powered by a Typehandler.)
    - Fields that are intended to be transient do not need to be stored.
    - Typehandlers can alter behaviour even if the sources of the persistent classes are not available or can not be altered.
    - Querying support.

    TypeHandlers are expected to do the complete marshalling of an object by writing/reading directly to/from the marshalling buffer.

    About a year ago the TypeHandler4 interface consisted of more than 40 methods. Today only 5 methods remain. To make sure that the TypeHandler interface is ready for heavy duty use, we have already used it ourselves to implement some collection typehandlers. See:
    com.db4o.typehandlers.CollectionTypeHandler  (Java)
    com.db4o.typehandlers.MapTypeHandler (Java)
    Db4objects.Db4o.Typehandlers.ListTypeHandler (.NET)
    Db4objects.Db4o.Typehandlers.MapTypeHandler (.NET)

    In the latest development build (7.5.x.y) these TypeHandlers are already turned on to do real work storing, loading and querying some selected collections. To see how internal TypeHandlers are configured, take a look at:
    com.db4o.internal.TypeHandlerConfigurationJDK_1_2 (Java)
    Db4objects.Db4o.Internal.TypeHandlerConfigurationDotNet (.NET)

    You will notice that many TypeHandlers are still missing if the full featureset of JDK and .NET collections is to be supported. Here is where we would like to ask you for your help:
    Could you write a TypeHandler for us for a collection class that you think we need to support?

    Adapting and registering one of the already existing typehandlers for your favourite collection will probably take you less than an hour of work. You can simply follow the pattern that is already there.

    However writing the TypeHandler code is not the complete story. Since we want to be absolutely sure that a new TypeHandler works correctly, the bigger part of the implementation work is required for testing. We want to make sure that objects stored with an older version of db4o can be read and stored even after your new TypeHandler is installed.

    For this purpose we developed the "Db4oMigrationTestSuite" test.
    On Java you can run this class directly. You will find it in our SVN in the db4ojdk1.2 project. 
    ( If you have not worked with the db4o sources before, here is a guide to get started. )

    If you develop for .NET you would check out the db4o.net project from our SVN. You would typically work with the Db4o-2008.sln solution in the top level folder. To run the tests, the Db4objects.Db4o.Tests-2008 project would be your startup project. To save time by not running all the tests every time you can edit Db4objects.Db4o.Tests.AllTests in the Db4objects.Db4o.Tests-2008 project: If you want to run migration tests only, you can comment out all the entries in the #TestCases() method and add the following instead:
    typeof(Db4objects.Db4o.Tests.Common.Migration.Db4oNETMigrationTestSuite)

    Both for Java and .NET:
    If you want the Db4oMigrationTestSuite to truely run against all relevant old db4o engines, you should check out the db4o.archives project from our SVN also.

    Once you have testing set up and running, you are ready to go to write your own test for updating your favourite collection.

    For an example how an update test could look like, please see:
    com.db4o.db4ounit.common.handlers.ArrayListUpdateTestCase (Java)
    Db4objects.Db4o.Tests.CLI2.Handlers.GenericListVersionUpdateTestCase (.NET)

    Only when you have the test in place it makes start to implement your own TypeHandler as a more efficient method to store the collection. Test first, remember?

    Should you have any problems to get started, please ask for help in the db4o product developer forum. It is a good idea to announce any work that you are about to do in the forum in any case. That will ensure that the work for a specific collection is only done once.

    Now that you are all set to start:
    Good luck for writing your TypeHandler!

    All the new Typehandlers we have built so far are now available in the TRUNK of our SVN and with our continous build.

    They help to make storage of collections more efficient:
    - Less state is pushed around than with our previous translator approach.
    - Maps no longer store an internal Entry object for each entry.
    - All .NET collections, including all .NET generic collections will soon be handled efficiently with Typehandlers. Current issues with .NET generic collections in concurrent multi-user setups will soon be history.
    - Old database files can be read with new Typehandlers installed. Future marshalling format changes are easily possible by writing and installing yet further new Typehandlers.
    - We have a very nice hook into collection treatment that we will be able to use for much faster collections.

    Enjoy!

  • Synchronization issues in networking C/S fixed

    We have fixed some thread synchronization issues in the C/S protocol and put new versions (rev. x.y.54.11278) for all branches online in the download center. Users with concurrent networking C/S in connection with lazy queries are recommended to upgrade. For more information, please refer to the corresponding tracker entry.
  • Debugging TA Enhanced Code and the Observer Effect

    Most of those who pair programmed with me know that I'm not very fond of debugging code by stepping through breakpoints inside an IDE. I'd much rather construct hypotheses on why the code is failing and systematically test them with carefully placed assertions and print statements.

    One reason for that is the observer effect of debugging: the execution environment can be a little different, timings are usually changed, object inspectors might end up calling methods with unwanted side-effects and so on.

    The last point is a particularly important one to keep in mind when dealing with code that's been enhanced for Transparent Persistence.

    Let's say we have built the latest and greatest library management application and now want to put a web front end on top of it. We've been following web development trends and so we know AJAX is a must these days. We just need some simple way to transfer the objects from the server to the browser javascript runtime: JSON to the rescue. Good thing we can reuse some reflection based JSON serialization code we had laying around.

    We write a quick spike to see how well it works:

        Dim book = New Book("The God Delusion")
    book.AddAuthor(New Author("Richard Dawkins"))

    Console.WriteLine(New JSONSerializer().Serialize(book))

    Which give us the expected output:

    	{ _title: "The God Delusion", _authors: [{ _name: "Richard Dawkins" }] }

    We try one more spike to convince us we can get JSON out of a db4o query result:

        Using db = OpenDatabase()
    For Each book In db.Query(of Book)()
    Console.WriteLine(New JSONSerializer().Serialize(book))
    Next
    End Using

    Unfortunately this gives us something completely unexpected and it's not the Spanish Inquisition:
    	{ _title: null, _authors: null }

    We place a breakpoint inside the loop so we can inspect the book instance:

    Debugging TA With VB

    It sure looks right and oddly enough we get the right output this time:

    	{ _title: "The God Delusion", _authors: [{ _name: "Richard Dawkins" }] }

    What's going on?

    Transparent Activation is going on or rather NOT going on:

    1. Our Book and Author classes were enhanced for Transparent Activation, Db4oTool injected the code required to implement IActivatable as well as the code that transparently activates the object before any field access;
    2. JSONSerializer uses reflection to access objects fields;
    3. Reflection completely bypasses any code introduced by Db4oTool which explains the "{ _title: null, _authors: null }" output, the object was never activated;
    4. When inspecting an object in the debugger watch window its properties are accessed;
    5. Accessing a property of our enhanced Book object causes its *transparent* activation which also explains why we got the right output aftewards;

    Ok, now that we have a logical explanation for what's happening how do we get our application to work as expected?

    Since JSONSerializer bypasses activation we have to ensure the object graph is completely activated before JSONSerializer gets to see it:

        Using db = OpenDatabase()
    For Each book In db.Query(of Book)()
    db.Activate(book, Integer.MaxValue)
    Console.WriteLine(New JSONSerializer().Serialize(book))
    Next
    End Using
    And that's it.

    Hopefully that will save you some frustration next time you need to debug Transparent Persistence enhanced code or to have it interacting with reflection based libraries.

    The complete code can be found here.

    Have fun!

  • Performance Contest - for one more month

    Do you like profiling? Do you like tuning code? If you do, how about taking part in the db4o performance contest? We offer USD 6000 in prizes for the best contributions to make db4o faster.

    We have made it very easy for you to get started by providing an Eclipse workspace set up with the db4o sources and the Poleposition benchmark.

    Links to get you started:
    Performance Contest page
    "Getting Started" video
    Performance Contest Forum

    The contest has been running for a month already and we are seeing first contributions coming in.

    Andrew has improved sorted queries and provides a UTF-8 string encoder.
    Erik is working on a query cache to make queries reusable.

    Can you beat their patches? I am sure you can.
    The contest is still open for contributions for one more month until August 15.

  • Implications of instrumenting assemblies for Transparent Activation / Persistence in db4o - Part I

    As the popular saying  tell us, there's no such thing as free lunch. It is no different for work related subjects.

    When we introduced Transparent Activation (TA) concept we knew that it could make developer's life easier (by presenting a simpler model for object life cycle): developers would not be required to think about activation depths again; just let db4o activate your objects when it's need.

    IMHO, it was a great improvement since it not only simplifies developer's life as well brings some performance gains also (once objects get activated only when needed there's no wasted time activating objects not needed). But this was not without coasts: to benefit from this simpler model, developers were required to implement IActivatable interface for each object that they wanted to be TA. Also, once enabled, any non TA aware object will be fully activated at retrieval time.

    Time passed and we introduced  a way relieve developers from the need to implement Activatable interface, i.e, to make classes TA aware with no effort from developers. This is accomplished through a technique called instrumentation in which we dive into byte code level doing something like:

    1. Find each type declared in a specific assembly (.net) / jar or class (java)
    2. Analise each type found in step 1 to check whether it is a potential candidate for TA or not
    3. If we found a potential candidate for TA
      • Make the type to implement IActivatable interface
      • Implement IActivatable methods
      • Ensure that the object will be activated prior to any field access by inspecting each field access in this type and inserting the required Activate() / activate() call prior to it.
    4. Save the binary component

    A pretty straight approach, I'd say.

    But even when using instrumentation there are some implications that developers should be aware of. To illustrate, think about assembly signing; this is a process in which a developer "seals" an assembly contents by digitally signing it (the whole concept is beyond this post).

    Basically, by signing an assembly a developer is providing ways to administrators to selectively grant access to assemblies based on the private key (related to the public key) used to sign it; also, signed assemblies are tamper resistant which means that changes (in byte code level) don't go unnoticed.

    By the above description it's clear that instrumenting  a signed assembly turns it invalid afterward (since its contents gets changed) and any tentative of loading it will fail with a invalid signature error.

    So, after instrumenting signed assemblies you will need to sign them again but, IMHO, you should never, ever, instrument assemblies that are not under your control, so I'd suggest you to just change your build scripts so instrumentation takes place prior to signing.

    To help developers detect this, starting from version 7.4.52 Db4oTool will emit an warning when asked to instrument (whether for TA/TP or Native Queries optimizations) a signed/delay signed assembly.

    Just for the sake of completeness I've included the following walk through on how to create assemblies that are instrumented and signed (for a in depth discussion on signing assemblies please, read this).

    VS Project Properties VS Project Properties 2
    1. In Visual Studio, open project properties window and select signing tab.

    2. Select Sign the assembly;

    3. Expand Choose a strong name key file drop down listbox and either, select a previously created key or create a new one by selecting <New...> option;

    4. Select Delay sign only checkbox;

    5. Save your project and build

    6. Instrument your assembly (either running Db4oTool or through Db4oToolEnhancerMSBuildTask)

    7. After instrumentation, run sn tool as follows to finish the sign process:

      sn -Ra myassembly.dll mykey.snk
    That's it! Your assembly is instrumented and signed.

    In the next post I'll address some other implications / pitfalls regarding instrumentation for TA/TP.

    Thoughts?

  • db4o-7.4 Development Release

    The following Jira tasks and bugs were resolved for db4o-7.4 development release:

    • DRS-98 - NullPointerException in replicate method
    • DRS-97 - Ensure dRS.NET can gracefully handle delegates in untyped fields and arrays
    • DRS-95 - dRS.NET fails with arrays/untyped fields
    • DRS-94 - Replication fails if the java platform doesn't support serializable constructor and the objects to be replicated don't have default constructors
    • DRS-93 - Replication fails for arrays in untyped fields
    • COR-1297 - Spike: Class derived from ArrayList using custom Typehandler
    • COR-1295 - Convert ListTypeHandlerTestSuite to .NET
    • COR-1291 - Backport "[Linq] Enum comparisons are not optimized" to 7.2
    • COR-1289 - Clean up db4obuild for opening to the public
    • COR-1284 - Defragment against unknown classes runs into NPE on btree index lookup
    • COR-1283 - Update old multidimensional arrays to null bitmap handling
    • COR-1282 - [decaf] functional test for resolving db4oj/db4ojdk1.2 conversion issues
    • COR-1281 - Running out of disk space causes unrecoverable data corruption
    • COR-1279 - Delegate in untyped arrays cause ObjectNotStorableException
    • COR-1276 - prepare Jprobe paircast and blog
    • COR-1275 - Db4o should not wrap exceptions thrown during event handling
    • COR-1265 - ArrayHandler: Defragment old databases
    • COR-1264 - ArrayHandler: Updating from old databases to new format
    • COR-1263 - ArrayHandler: Bitmap Marshalling Format
    • COR-1262 - Pass configuration on to reflector on startup
    • COR-1261 - Move constructor handling into reflector
    • COR-1255 - Provide Feedback on latest OMG SBQL vs, LINQ comparison
    • COR-1254 - _blob.ReadFrom() throws Object reference not set to an instance of an object. in C/S mode
    • COR-1247 - Create releae notes for production release 7.2
    • COR-1246 - Create release notes for 6.4 stable release
    • COR-1227 - Prepare 7.2 "Production" release (freeze trunk)
    • COR-1216 - TracerBullet: Deletion for Java ArrayList using the new Typehandler
    • COR-1214 - Db server closed when upgraded from 5.5 version to 6.4.14.8331 with btree based freespace management system
    • COR-1153 - [Linq] Enum comparisons are not optimized
    • COR-1129 - TracerBullet: Special use cases for Java ArrayList using the new Typehandler
    • COR-897 - Concurrency tests for close/timout scenarios
  • flushFileBuffers(false) and the C in ACID

    The short story:
    Configuration.flushFileBuffers(false) is no more. If you still want to configure the behaviour that used to be available in db4o 6.1, you can do so by setting up IoAdapters as follows:

    RandomAccessFileAdapter randomAccessFileAdapter = new RandomAccessFileAdapter();
    NonFlushingIoAdapter nonFlushingIoAdapter = 
                                     new NonFlushingIoAdapter(randomAccessFileAdapter);
    CachedIoAdapter cachedIoAdapter = new CachedIoAdapter(nonFlushingIoAdapter);
    Configuration configuration = Db4o.newConfiguration();
    configuration.io(cachedIoAdapter);
    Db4o.openFile(configuration, PATH);
    

    The long story:
    We do a lot of things for consistent transactions (for the C in ACID). A commit of a db4o transaction consists of 4 write phases:

    • Writing a commit log to the database file
    • Switching the database file to commit mode
    • Committing all slots
    • Switching the database file to normal mode 

    Because a cache at any level may turn around the write order of individual slots and because a failure may end up in a partial write, it is essential that all data is flushed to the device after each one of these 4 phases.

    This flush call takes a lot of time. It is the slowest part of the commit. On a system where you are sure enough (100% ? ) that the commit process always is successful, a tuning switch that skips flushing sounds like a smart way to improve performance.

    That's what we thought and why we added the #flushFileBuffers(false) configuration switch in the first place.

    Starting with db4o 6.4 we have added a new CachedIoAdapter to the IO layer as the default. This IoAdapter uses the #sync() calls as a signal to flush it's pages to the database file. What we did not think about well enough when we added this functionality: If users use the #flushFileBuffers(false) tuning switch, they also skip the flush of the CachedIoAdapter.

    This issue does not surface as long as the database file is always shut down correctly, so it took us a long time to realize how serious it can be. Since we have switched db4o 6.4 to "stable", there have been a couple of support discussions like this one:

    User: "We have switched to db4o 6.4. We are sometimes getting new exceptions like IncompatibleFileException."
    db4objects: "Uhoh. Are you maybe using #flushFileBuffers(false) in productive use? Please don't do that. Don't tell us that you are."
    User: "Yes we are."

    After investing some time and thought we have figured that the CachedIoAdapter makes it much much more likely that corruption can happen if a database file is not shut down correctly (with the #flushFileBuffers(false) configuration call). Because this is so serious (it leads to corrupted database files), we have decided to turn off this configuration switch completely.

    A similar functionality as in 6.1 is still available, if you really know what you are doing and if you know that you are taking the risc of corrupted database files if your system is halted because someone hits the power-off switch by accident in the middle of a commit. With the layer of IoAdapters, as outlined at the beginning of this posting, the risk of corruption is fairly small, just as it used to be in 6.1. Corruption will only happen if the OS cache or disc cache changes around the order of writes.

    The changes are effective and the new IoAdapter is available from iteration 48 and revision 10977 onwards, e.g. from db4o versions:
    6.4.48.10977
    7.2.48.10977
    7.4.48.10977

    For previous stable 6.4 versions we strongly recommend not to use the #flushFileBuffers(false) setting in production use.

    Many thanks to our users for reporting their problems!

  • Committed callbacks, pushed updates and read committed isolation revisited

    Some time ago we presented Pushed Updates as a sample application for Committed Callbacks in this blog. Recently we put a similar mechanism in a slightly different context. Since we encountered some perceived ambiguity concerning db4o transaction semantics, and since we were in for a little surprise ourselves, I'd like to briefly rehash this topic.

    Pushed Updates basically are a mechanism to provide READ COMMITTED isolation from the point of view of your application. Wait - doesn't db4o provide that out of the box?!? Well, it does. Any query is guaranteed to return objects in their most recently committed state - unless they are still in memory on the requesting client, that is. db4o cannot change the state of of a live object that may be involved in application level processes at the same time. Only the application can judge whether it is safe to modify a given object, and Committed Callbacks are the way to transfer this responsibility.

    Note that "in memory" above does not mean "in use by the application", and not even necessarily "referenced by the application". An object may still survive in the reference system some time after it has been discarded by the application, until the GC eventually hits. There is no remotely performant way to check whether there exist hard references to a given object.

    So much for the theory, let's look at the code. The core of the Pushed Updates example looked like this:
    private EventListener4 createCommittedEventListener(final ObjectContainer objectContainer){
      return new EventListener4() {
        public void onEvent(Event4 e, EventArgs args) {
          ObjectInfoCollection updated = ((CommitEventArgs)args).updated();
          Iterator4 infos = updated.iterator();
          while(infos.moveNext()){
            ObjectInfo info = (ObjectInfo) infos.current();
            Object obj = info.getObject();
            objectContainer.ext().refresh(obj, 2);
          }
        }
      };
    }
    EventListener4 committedEventListener = createCommittedEventListener(client);
    EventRegistry eventRegistry = EventRegistryFactory.forObjectContainer(client);
    eventRegistry.committed().addListener(committedEventListener);
    This code registers a callback for committed events. When invoked, it will traverse all the objects that have been updated during this commit and refresh them. (The magic depth of two is a reverence to collections, forcing them to refresh their elements, too.)

    Now this works nicely - in networking C/S mode, where the communication layer between client and server takes care of the correct "translation" of object identities. In embedded C/S mode, however, there is no translation - clients run in the same "live object space" as the server, the only thing they don't share is the weak reference system that defines their notion of object identities. And so the ObjectInfo will contain the server's instance of this persisted object, while the client will have its own version that it won't be able to map to the "stranger".

    The fix is easy (and works for networking C/S, too, of course): We have to take care of the translation ourselves via ID lookup. In code:
    public void onEvent(Event4 e, EventArgs args) {
      Transaction trans = ((InternalObjectContainer)client).transaction();
      ObjectInfoCollection updated = ((CommitEventArgs)args).updated();
      Iterator4 infos = updated.iterator();
      while(infos.moveNext()){
        ObjectInfo info = (ObjectInfo) infos.current();
        Object obj = trans.objectForIdFromCache((int)info.getInternalID());
        if(obj == null) {
          continue;
        }

        client.refresh(obj, 2);
      }
    }
    Here we have to fall back on the InternalObjectContainer interface. Don't be too scared of this - it's not really under the hood stuff, but it's indeed intended to be used for 3rd party implementations of pluggable db4o features intended to be run inside the core, like type handlers, reflectors - and callbacks. Note that this variant also keeps the handler from further processing of objects that haven't been known to this client, anyway.

    Bottom line: Transaction semantics and live in-memory objects are not entirely trivial to align, object identities can be tricky, and this is the recommended generic way for doing Pushed Updates. For networking C/S, the original Pushed Updates example is fine as it was.
  • Sharpen your Java app now: Java to C# converter released as free software

    As of today, db4objects releases its Eclipse based Java to C# source code conversion tool "sharpen" as free software.

    sharpen is the "secret sauce" that enables us to provide the [db4o object database engine] in native versions for the Java and .NET platforms from a single code base.

    Features include:
    • mapping between Java and .NET native type systems
    • compliance with C# coding conventions
    • user defined namespace/class/method mappings
    • method to property mappings
    • Generics support
    • partial conversions
    • mixing native and converted C# sources
    • VS solution file support
    • Ant integration
    sharpen is used extensively in the db4o build system, generating most of the db4o engine core code and the unit test suites from the Java sources.

    Before you start dreaming: sharpen is not a "magic wand". Don't expect to feed it an arbitrary Java app and receive a running .NET version at the push of a button. The translation process will require some design compromises on the Java side, and parts of the .NET code still have to be hand crafted and integrated into the converted sources. Still, we believe sharpen provides an amazingly smooth way to bridge the gap between Java and .NET and create real cross platform applications.

    We are looking forward to see which sharpen propelled applications will cross the language chasm, and we're hoping on your feedback and your contributions that will help to cover automatic conversion of even more Java constructs and idioms, moving Java/.NET cross platform integration to a new level.

    sharpen is released under the GPL. The svn repository, documentation and issue tracker are online. All that's missing now is you bringing your Java app to the .NET platform. To get you started, here is the full anatomy of a simplistic application developed with sharpen. Have fun!
  • Smart java to c# conversion for the masses with sharpen

    As of Today our so far internal java to c# conversion utility, sharpen, is being released as free software.

    As an example of its usage we're going to write a simple contact management application in java and convert it to c#.

    While it is possible to use sharpen to convert a complete application it is advisable to handle this as an iterative process: java a little, sharpen a little. The reason for this is that not all java constructs and idioms can be handled out of the box by sharpen, thus the translation might influence the design of the original java code.

    As a first step we'll come up with a minimum skeleton for our console application, ignoring any "business logic" for now:

    package contacts;

    public class Program {

    public static void main(String[] args) {
    boolean running = true;
    while (running) {
    String option = Console.prompt("(a)dd new entry, (l)ist entries, (q)uit");
    if (option.isEmpty()) continue;
    switch (option.charAt(0)) {
    case 'q':
    running = false;
    break;
    default:
    System.out.println("'" + option + "' DOES NOT COMPUTE!");
    }
    }
    }
    }
    package contacts;

    import java.io.*;

    public class Console {

    public static String prompt(String prompt) {
    System.out.println(prompt);
    return readLine();
    }

    private static String readLine() {
    try {
    return new BufferedReader(new InputStreamReader(System.in)).readLine();
    } catch (IOException e) {
    throw new RuntimeException(e);
    }
    }
    }

    Here we can already see the first design decision triggered by sharpen. The console APIs for java and .net are quite different and although sharpen can handle IO primitives gracefully, it doesn't support translating our specific use of "System.in". The obvious way of handling situations like this is to factor out the required functionality into a specialized class and provide dedicated implementations for java and .net.

    With the first java iteration complete let's get to sharpening it.

    Before we can get any conversion done, the sharpen.core plugin must be installed. From inside eclipse:

    • Checkout sharpen.core from db4o's subversion repository at https://source.db4o.com/db4o/trunk/sharpen/
    • Right click the freshly checked out project in the "Package Explorer" and choose "Export" from the context menu;
    • Expand the "Plug-in Development" folder and select "Deployable plug-ins and fragments";
    • Set "Destination" to the root folder of your eclipse installation and click "Finish";

    We're ready to automate the conversion from java to c# with an ant script. In order to make this task easier, we will reuse a few sharpen related macros defined in sharpen-common.xml.

    <project name="ContactList" default="build-dotnet">

    <import file="build-properties.xml" />
    <import file="sharpen-common.xml" />

    <target name="build-dotnet" depends="sharpen">
    <exec executable="${nant.exe}">
    </exec>
    </target>

    <target name="sharpen" depends="init">

    <prepare-sharpen-workspace project="ContactList" dir="${sharpen.workspace.dir}" />

    <sharpen workspace="${sharpen.workspace.dir}" resource="ContactList/src">
    </sharpen>

    </target>

    <target name="init">

    <reset-dir dir="${sharpen.workspace.dir}" />

    </target>

    </project>

    Here we are using the common strategy of externalizing environment specific properties to a dedicated file (build-properties.xml):

    <project name="build properties">
    <property name="sharpen.workspace.dir" value="build" />
    <property name="eclipse.home" value="c:/java/eclipse" />
    <property name="nant.exe" value="c:/dotnet/nant-0.85/bin/NAnt.exe" />
    </project>

    We're not using a plain java properties file because the nant script which is used for compiling the converted sources also needs some of these definitions:

    <?xml version="1.0"?>
    <project name="ContactList" default="build">

    <include buildfile="build-properties.xml" />

    <property name="src.dir" value="${sharpen.workspace.dir}/ContactList.net" />
    <property name="bin.dir" value="${src.dir}/bin" />

    <target name="build">
    <csc target="exe" output="${bin.dir}/ContactList.exe">
    <sources basedir="${src.dir}">
    <include name="**/*.cs" />
    </sources>
    </csc>
    </target>

    </project>

    Running ant we get good news and bad news. The good news are sharpening went fine:

    sharpen:
    [mkdir] Created dir: build\ContactList
    [copy] Copying 2 files to build\ContactList
    [echo] org.eclipse.core.launcher.Main -data build -application sharpen.core.application ContactList/src
    [java] project: ContactList
    [java] source folder: src
    [java] Pascal case mode: None
    [java] Console.java
    [java] Program.java
    [java] Conversion finished in 3110ms.

    C# compilation didn't go so well:

    [exec] [csc] Compiling 2 files to 'ContactList.net\bin\ContactList.exe'.
    [exec] [csc] ContactList.net\src\contacts\Console.cs(15,16): error CS0246: The type or namespace name 'java' could not be found (are you missing a using directive or an assembly reference?)

    ...

    [exec] [csc] ContactList.net\src\contacts\Program.cs(12,16): error CS0117: 'string' does not contain a definition for 'isEmpty'

    Before we fix those errors let's take a look at how Program.java got translated to c#:

    namespace contacts
    {
    public class Program
    {
    public static void Main(string[] args)
    {
    bool running = true;
    while (running)
    {
    string option = contacts.Console.prompt("(a)dd new entry, (l)ist entries, (q)uit"
    );
    if (option.isEmpty())
    {
    continue;
    }
    switch (option[0])
    {
    case 'q':
    {
    running = false;
    break;
    }

    default:
    {
    System.Console.Out.WriteLine("'" + option + "' DOES NOT COMPUTE!");
    break;
    }
    }
    }
    }
    }
    }

    It looks good except for javaisms like the lower case namespace and method names which can be fixed with the introduction of the "pascalCase+" sharpen command line option.

    To get rid of the c# compilation errors we must provide a platform specific version of the Console class and tell sharpen to ignore its java counterpart using the @sharpen.ignore javadoc annotation.

    /**
    * @sharpen.ignore
    */
    public class Console {

    ...

    As a very useful convention we like to keep native and converted c# sources separate from each other by introducing a subfolder named "native".

    Here's our native c# Console implementation:

    namespace Contacts
    {
    public static class Console
    {
    public static string Prompt(string message)
    {
    System.Console.WriteLine(message);
    return System.Console.ReadLine();
    }
    }
    }

    The remaining compiler error is caused by calling the String.isEmpty which has no equivalent on the .net side. We'll sneak around this issue for the time being by rewriting the java side condition in terms of String.length:

                if (option.length() == 0) continue;

    Our build runs green now and the .net application is functional. We're ready to move on to the business logic.

    We'll start with a collection based implementation to be enhanced with persistence later in the process.

    package contacts;

    public class Contact {

    private String _name;
    private String _email;

    public Contact(String name, String email) {
    _name = name;
    _email = email;
    }

    public String name() {
    return _name;
    }

    public String email() {
    return _email;
    }
    }
    package contacts;

    import java.util.*;

    public class ContactList {

    private List<Contact> _entries = new ArrayList<Contact>();

    public void add(Contact contact) {
    _entries.add(contact);
    }

    public Iterable<Contact> entries() {
    return _entries;
    }
    }
    package contacts;

    public class Program {

    private ContactList _contacts = new ContactList();

    public void readEvalLoop() {
    boolean running = true;
    while (running) {
    String option = prompt("(a)dd new entry, (l)ist entries, (q)uit: ");
    if (option.length() == 0) continue;
    switch (option.charAt(0)) {
    case 'a':
    addEntry();
    break;

    case 'l':
    listEntries();
    break;

    case 'q':
    running = false;
    break;
    default:
    System.out.println("'" + option + "' DOES NOT COMPUTE!");
    }
    }
    }

    private String prompt(String message) {
    return Console.prompt(message);
    }

    private void listEntries() {
    for (Contact c : _contacts.entries()) {
    System.out.println(c.name() + " <" + c.email() + ">");
    }
    }

    private void addEntry() {
    String name = prompt("Name: ");
    String email = prompt("Email: ");
    _contacts.add(new Contact(name, email));
    }

    public static void main(String[] args) {
    new Program().readEvalLoop();
    }
    }

    A quick look at the converted object model shows that although correct the code still looks a bit alien to .net:

    namespace Contacts
    {
    public class Contact
    {
    private string _name;

    private string _email;

    public Contact(string name, string email)
    {
    _name = name;
    _email = email;
    }

    public virtual string Name()
    {
    return _name;
    }

    public virtual string Email()
    {
    return _email;
    }
    }
    }
    namespace Contacts
    {
    public class ContactList
    {
    private System.Collections.Generic.IList<Contacts.Contact> _entries = new System.Collections.Generic.List
    <Contacts.Contact>();

    public virtual void Add(Contacts.Contact contact)
    {
    _entries.Add(contact);
    }

    public virtual System.Collections.Generic.IEnumerable<Contacts.Contact> Entries()
    {
    return _entries;
    }
    }
    }

    Contact.Name, Contact.Email and ContactList.Entries would look better as properties and the qualified names used everywhere look rather untidy.

    We can solve the first issue using the @sharpen.property javadoc annotation:

        /**
    * @sharpen.property
    */
    public Iterable<Contact> entries() {
    return _entries;
    }
    Proper "using" directives can be enforced with the "-organizeUsings" command line option:
            <sharpen workspace="${sharpen.workspace.dir}" resource="ContactList/src">
    <args>
    <arg value="-pascalCase+" />
    <arg value="-organizeUsings" />
    </args>
    </sharpen>
    The c# version of ContactList is now indistinguishable from hand crafted code:
    using System.Collections.Generic;
    using Contacts;

    namespace Contacts
    {
    public class ContactList
    {
    private IList<Contact> _entries = new List<Contact>();

    public virtual void Add(Contact contact)
    {
    _entries.Add(contact);
    }

    public virtual IEnumerable<Contact> Entries
    {
    get
    {
    return _entries;
    }
    }
    }
    }

    We now have a complete cross platform contact management application. The only feature missing for the huge marketing success it deserves is persistence which can be easily implemented with db4o.

    We downloaded the db4o production packages for both java and .net 2.0 from here and added the corresponding paths to our build-properties.xml file:

        <property name="db4o.jar" value="c:/java/db4o-7.2/lib/db4o-7.2.39.10644-java5.jar"  />
    <property name="db4o.dll" value="c:/dotnet/db4o-7.2/bin/net-2.0/Db4objects.Db4o.dll" />
    <property name="eclipse.home" value="c:/java/eclipse" />

    All we have to do now is to change ContactList to use a db4o ObjectContainer instead of an ArrayList as its storage backend:

    package contacts;

    import com.db4o.*;

    public class ContactList {

    private ObjectContainer _container = Db4o.openFile("contacts.db4o");

    public void add(Contact contact) {
    _container.store(contact);
    }

    /**
    * @sharpen.property
    */
    public Iterable<Contact> entries() {
    return _container.query(Contact.class);
    }

    public void close() {
    _container.close();
    }
    }

    We also have to make sure that the ObjectContainer is shutdown cleanly on application exit, for that purpose we introduced the ContactList.close method above which must be called from the main application class:

        public static void main(String[] args) {
    Program program = new Program();
    try {
    program.readEvalLoop();
    } finally {
    program.close();
    }
    }
    Getting the c# version to work requires additional configuration switches for the sharpen task. First the java code now depends on the db4o jar which must be known at translation time. Second, the .net db4o interface operates against the native .net type system (this is a great candidate for a default setting) and uses namespace and interface names more inline with the .net design guidelines:
            <sharpen workspace="${sharpen.workspace.dir}" resource="ContactList/src">
    <args>
    <arg value="-pascalCase+" />
    <arg value="-organizeUsings" />

    <arg value="-cp" />
    <arg file="${db4o.jar}" />

    <arg