db4o Developer Community

Hi, 

I want to let you know that during the last few days we have made some enhancements and fixed some bugs / issues related to our LINQ implementation.

Some of them were just annoyances that forced developers to use "not so idiomatic" styles as workarounds, some of them either resulted in crashes or wrong results being returned or even bad performance. Just in case you are curious, here are the tickets to the related issues.

• COR-1651(bug): LINQ 'Where Not' clause against boolean field throws exception when native query optimization is enabled

• COR-1652 (bug): DateTimeOffset field in ORDER By throws exception

• COR-1666 (enhancement) : Support method/property chaining on LINQ queries

• COR-1667 (enhancement) : Support simple method calls on LINQ queries 
  

I'll use the class definition bellow to discuss these issues (Assume we have a bunch of objects of this class stored in the database):

The next two issues were related to our Linq -> Soda translator. In both cases we (wrongly) concluded that the expressions were not convertible to Soda falling back to LINQ for Objects. You would still get the right results but the performance could be seriously affected depending on the actual expression. The following sample shows the failing constructs:

from Perosn p in db
where p.Parent.Name == "John Doe"
select p;

In the first one, GetName() just returns the contents of a property (or a field). The second example just happened to have two indirections (p.Parent.Name).  In both cases we used to fail to recognize these patterns as "SODA compliant" and felt back to LINQ for Objects. So, for instance, in the second sample, instead of getting something like:

IQuery q = db.Query();
q.Constrain(typeof(Person));
q.Descend("_parentBackingField").Descend("_nameBackingField").Constrain("John Doe");

foreach(var p in q.Execute())
{
}

we would get something like:

IQuery q = db.Query();
q.Constrain(typeof(Person));

foreach(var p in db.Execute().Where(candidate => candidate.Parent.Name == "John Doe"))
{
}

which is less efficient since all objects of type Person will be retrieved.

The last issue was not directly related to LINQ; instead it was a Soda bug regarding ordering based on value types that implements IComparable / IComparable<T>.

So queries like (note that Id is a Guid, which implements these interfaces):

from Person p in db
orderby p.Id
select p;

would result in an exception like:

Db4objects.Db4o.Internal.IllegalComparisonException: Exception of type 'Db4objects.Db4o.Internal.IllegalComparisonException' was thrown.
   at Db4objects.Db4o.Internal.Handlers.StandardReferenceTypeHandler.PrepareComparison(IContext context, Object source)

It was fixed by making StandardReferenceTypeHandler aware of these interfaces.

Well, the good notice (as you already know :) is that all of them were fixed in revision 13278, so if you faced any of these issues you may wish to check it out again :)

As usual, you are invited to download a console application (attached to this post) that accepts a base folder as its argument and runs some tests against all db4o versions found as subdirectories under this base folder.

We can identify all the issues discussed before in the following screen shot (run against version 7.10.96) 

And confirm that they were gone on version 7.10.99!

I could not finish this post without saying that three of these issues were reported by community members. To you, our sincere thanks you!

Best.

Adriano

The LINQ framework design guidelines [1] describes the "... three means by which a type can be designed to participate in LINQ queries: implementing IEnumerable<> ..., implementing IQueryable<>, or by defining the Query Pattern on the type".

For db4o we chose the Query Pattern implementation because only a subset of the available LINQ operators could be reasonably supported. By implementing the pattern for the operators we support any code that references an unsupported operator automatically falls back to LINQ to Objects.

Yes, implementing IQueryable<> [2] used to be that slightly more complicated approach recommended when most LINQ operators were supported by the provider. Things have changed and IQueryable<> is now being used by APIs everywhere as the lowest common denominator for optimizing LINQ providers.

Which is why we decided to make LINQ to db4o support the IQueryable interface as well.

Just as with LINQ to db4o, it's very easy to use and completely integrated:

        using Db4objects.Db4o.Linq;
        
        // ...
        
        IObjectContainer container = Db4oEmbedded.OpenFile("HR.db4o");
        IQueryable<Person> query = container.AsQueryable<Person>();

Note: if you want to check out a demos that use these new features see this page

        using Db4objects.Db4o.Linq;
        
        // ...
        
        IObjectContainer container = Db4oEmbedded.OpenFile("HR.db4o");
        IQueryable<Person> query = container.AsQueryable<Person>();

db4o version 7.10 is out and here is the list of modifications, improvements and bug fixes that you will find in it:

In our latest development build we provide a first implementation of Transparent Activation and Transparent Persistence for standard Java collections.

In case you don't know what these terms mean:
Transparent Activation (TA) loads members of persistent objects lazily when they are accessed.
Transparent Persistence (TP) flags objects as dirty when their members are modified, so these objects are stored upon the next commit.

If you decide to work with TA and TP, you will not need any ObjectContainer#store() calls to update objects and no
ObjectContainer#activate() calls to activate object graphs. db4o does all the work nicely for you: transparently.

In order for TA and TP to work, persistent classes need to implement the Activatable interface.

Our Ant enhancer task com.db4o.enhance.Db4oEnhancerAntTask can add the necessary code to your persistent classes, as demonstrated in the Ant script further down in this posting.

Because we could not change the implementation of the standard Java collections we decided to derive classes
from them and to add the Activatable code in these derived classes.

The com.db4o.collections package now contains the following implementations:
ActivatableArrayList
ActivatableHashMap
ActivatableHashSet
ActivatableHashtable
ActivatableLinkedList
ActivatableStack
ActivatableTreeSet

The above is a complete list of all the collections that we support so far. Internally we have overridden all public methods. In these methods we call #activate() and then delegate to the parent methods. Please note that we have only added support for JDK 5 methods, because of backward compatibility issues of our build system. If you use JDK 6 methods you will run into problems.

In your code you do not have to use our derived collection classes. Our Ant enhancer task com.db4o.enhance.Db4oEnhancerAntTask will exchange all collection constructor calls in your code to create Activatable collections instead.

Let's create a small sample project that uses Transparent Persistence to check things out:

package com.db4o.sample;

import java.util.*;

public class Team {
	
	private String name;
	
	private List pilots;
	
	public Team(String name){
		this.name = name;
	}
	
	public void addPilot(Pilot pilot){
		if(pilots == null){
			pilots = new ArrayList();
		}
		pilots.add(pilot);
	}
	
	@Override
	public String toString() {
		String str = "Team " + name;
		if(pilots != null){
			str += " with pilots: ";
			for (Pilot p : pilots){
				str += p + " ";
			}
		}
		return str;
	}
	
	public String getName(){
		return name;
	}

}


package com.db4o.sample;

public class Pilot {
	
	private String name;
	
	public Pilot(String name){
		this.name = name;
	}
	
	@Override
	public String toString() {
		return name;
	}
	
}

package com.db4o.sample;

import java.io.*;

import com.db4o.*;
import com.db4o.config.*;
import com.db4o.cs.*;
import com.db4o.cs.config.*;
import com.db4o.query.*;
import com.db4o.ta.*;

public class Program {
	
	private static String FILE = "tp.db4o";
	
	private static int PORT = 4444;
	
	private ObjectContainer objectContainer;
	
	private ObjectServer objectServer;
	
	public static void main(String[] args) {
		new Program().run(); 
	}
	
	public void run() {
		deleteDatabaseFile();  // to make the sample rerunnable
		openObjectContainer();
		storeTeam();
		close();
		openObjectContainer();
		addPilots();
		close();
		openObjectContainer();
		printTeams();
		close();
	}

	private void deleteDatabaseFile() {
		new File(FILE).delete();
	}
	
	private void printTeams() {
		ObjectSet objectSet = objectContainer.query(Team.class);
		while(objectSet.hasNext()){
			System.out.println(objectSet.next());
		}
	}

	private void openObjectContainer(){
		openEmbeddedObjectContainer();
		// openClientServerObjectContainer();  
	}
	
	private void openEmbeddedObjectContainer(){
		EmbeddedConfiguration embeddedConfiguration = Db4oEmbedded.newConfiguration();
		embeddedConfiguration.common().add(new TransparentPersistenceSupport());
		objectContainer = Db4oEmbedded.openFile(embeddedConfiguration, FILE);
	}
	
	private void openClientServerObjectContainer(){
		ServerConfiguration serverConfiguration = Db4oClientServer.newServerConfiguration();
		serverConfiguration.common().add(new TransparentPersistenceSupport());
		objectServer = Db4oClientServer.openServer(serverConfiguration, FILE, PORT);
		objectServer.grantAccess("db4o", "db4o");
		
		ClientConfiguration clientConfiguration = Db4oClientServer.newClientConfiguration();
		clientConfiguration.common().add(new TransparentPersistenceSupport());
		objectContainer = Db4oClientServer.openClient(clientConfiguration, "localhost", PORT, "db4o", "db4o");
	}
	
	private void storeTeam(){
		Team team = new Team("Ferrari");
		objectContainer.store(team);
	}
	
	private void addPilots() {
		Team team = queryTeamByName("Ferrari");
		team.addPilot(new Pilot("Massa"));
		team.addPilot(new Pilot("Raikkonen"));
	}
	
	private Team queryTeamByName(final String name){
		ObjectSet objectSet = objectContainer.query(new Predicate() {
			@Override
			public boolean match(Team team) {
				return team.getName().equals(name);
			}
		});
		return objectSet.next();
	}

	private void close(){
		objectContainer.close();
		if(objectServer != null){
			objectServer.close();
		}
	}
	
}

We also need an Ant script that enhances our program for Transparent Persistence:

<project name="db4o.TP.sample" basedir="." default="run">
   <property name="dir.src" value="${basedir}/src" />
   <property name="dir.bin" value="${basedir}/bin" />

   <path id="project.classpath">
      <fileset dir="lib">
         <include name="**/*.jar"/>
      </fileset>
   </path>

   <taskdef name="db4o-enhance" 
      classname="com.db4o.enhance.Db4oEnhancerAntTask"
      classpathref="project.classpath" />

   <target name="compile">
      <mkdir dir="${dir.bin}" />
      <javac srcdir="${dir.src}" destdir="${dir.bin}" failonerror="true">
         <classpath refid="project.classpath" />
      </javac>
   </target>

   <target name="enhance" depends="compile">
      <db4o-enhance classtargetdir="${dir.bin}" ta="true" nq="true" collections="true">
         <classpath refid="project.classpath" />
         <sources dir="${dir.bin}">
            <include name="**/*.class" />
         </sources>
      </db4o-enhance>
   </target>

   <target name="run" depends="enhance">
      <java classname="com.db4o.sample.Program" failonerror="true">
         <classpath>
            <pathelement path="${dir.bin}" />
            <path refid="project.classpath" />
         </classpath>
      </java>
   </target>

If you want to try this project out yourself all you need is the db4o-7.[version].[iteration].[build]-all-java5.jar in the lib folder of your project. For your convenience I have attached a full Eclipse project with the above sources and with the db4o-all.jar to this posting. If you now run the Ant script, it prints out:
Team Ferrari with pilots: Massa Raikkonen

Can you see the interesting part of this small project?
The #addPilots() method never has to call ObjectContainer#store(). Because the ArrayList in Team.pilots is exchanged with an ActivatableArrayList and because TransparentPersistenceSupport is added to the db4o configuration, it is sufficient to add the pilots to the list and they will be stored in the automatic commit of the close() operation. Isn't that nice and easy?

You are invited to experiment with db4o Transparent Persistence and to tell us about your experiences in the db4o forum.

Enjoy!

Hi!

It's a time for a new release again and here is a list of changes, bug fixes and improvements that we are putting into it.

There were quite a lot of fixes for Object Manager Enterprise (Visual Studio version):

And in db4o core we also did lots of improvements, but we also accomplished another great job - cleaned our Jira and closed the issues that were already fixed long time ago, but were still "hanging" open in Jira. Hope it will help our users to see a clearer picture of the db4o current and future state.

Note, that the report below combines the issues actually fixed for 7.9 and those that were closed as they were fixed before:

Hi.

Have you ever used these funny types with a question mark in DotNet?

Well, they are called Nullable types and were introduced to provide the ability to set value types to "null", enabling a more natural integration between some datasets (such relational databases) and DotNet world. For instance, relational databases supports the concept of null fields even for integers. By the other hand .Net integers are value types which doesn't have such concept (someone said object/relational type mismatch?). (Just for the records, the syntax type? identifier is a C# shorthand for the full declaration: Nullable<type> identifier).

As I commented out in this post, at Db4o, we are continually pursuing better platform compliance (be it related to idioms, type system support, etc.) so some time ago we improved support for nullable types like in the following example

class TestNullableContainer
{
    private int? _value;
}

The next logical step was to extend this support to arrays, allowing Db4o to store arrays of nullables. In previous versions (< 7.9) we stored the array successfully but there were no way to retrieve it (instead an exception would be thrown). The problem was that in prior versions we loose the information about "nullableness", i.e, arrays such as int? []items were stored as if they were the "non nullable" counterpart (int []items) and even though we would successfully retrieve them from database .Net would just throw when trying to assign the just read object to the respective field (something like: int? []items = new int[] {1, 2, 3}).

Unfortunately extending nullable support to arrays of nullables proved to be trickier than we expected but after some code rewrite, we finally got it working (please, note that we still have some work to do regarding some not so common array usages scenarios).

To make it easier to spot the differences I've included a sample application that allows you to select a specific Db4o version (starting with 7.1) and run a piece of code against it (it is an adaptation of my last post sample app). You can test it with any number of different Db4o versions and compare the outcome of each one. To use this application simply compile the 2 projects (Db4oTestRunner and NullableArraysTest), then fire Db4oTestRunner and select NullableArraysTest assembly and finally click on "run" button (of course, you can write your own tests :).

Since our goal is to continue improving platform compliance, we'd like to know which improvements regarding platform compliance you'd like to see.

Please, go, download the test application (either  from the attached TestNullableArrays.zip file or get it through svn: svn co https://source.db4o.com/db4o/trunk/sandbox/Adriano/Blogs/), and play with it.

Best

Adriano

The following Jira bugs and issues were fixed for db4o-7.8 Development Release:

Looking for ways to make db4o faster, we recently invested some work into better caching. We found a nice paper here and decided to implement a LRU/2Q cache for db4o. Our new cache implementations can now be found in the classes LRU2QCache, LRU2QXCache and LRUCache. Feel free to use them anywhere the GPL allows you to.

As the best testing ground for the new caches we identified our IoAdapters. Driven by the requirement to make the caching layer pluggable, we first refactored IoAdapters from their old factory-and-implementation-in-a-single-class approach to a clean factory concept. The roots of the new class hierarchy:
Storage - a factory class that knows how to open a Bin
Bin - a representation of a container for storage of data
If you are interested in details of the implementation, it's probably easiest if you browse the sources and open the type hierarchy for Storage and Bin from your favourite IDE. The existing old IoAdapter interface still continues to work, wrapped by an IoAdapterStorage.

After we had the caching and a corrected Storage interface, we wrote a new CachingStorage that makes use of the caches. Benchmarking the new cache implementations against our old specialized CachingIoAdapter we found that the old cache was already very good. It took us quite a bit of tuning work to reach the same speed again, but we managed. Accordingly we could safely replace the old CachingIoAdapter with the new CachingStorage in the default db4o configuration.

Here are three of the most common configuration setups that you may be interested in for everyday use of db4o:

(1) Setting up db4o with a different cache page size and page count

final int cachePageCount = 128;
final int cachePageSize = 4096;
Storage cachedStorage = new CachingStorage(new FileStorage(), cachePageCount, cachePageSize){
// By overriding the newCache method you can plug in a different cache
    @Override
    protected Cache4 newCache() {
        return CacheFactory.new2QXCache(cachePageCount);
    }
};

// opening an embedded session
EmbeddedConfiguration embeddedConfiguration = Db4oEmbedded.newConfiguration();
FileConfiguration fileConfiguration = embeddedConfiguration.file();
fileConfiguration.storage(cachedStorage);
Db4oEmbedded.openFile(embeddedConfiguration, "myEmbeddedDb.db4o");

// opening a server for a client/server session
ServerConfiguration serverConfiguration = Db4oClientServer.newServerConfiguration();
FileConfiguration fileConfiguration = serverConfiguration.file();
fileConfiguration.storage(cachedStorage);
Db4oClientServer.openServer(serverConfiguration, "myServerDb.db4o", PORT);

(2) Using db4o as a fast in-memory database

//opening an embedded session
EmbeddedConfiguration embeddedConfiguration = Db4oEmbedded.newConfiguration();
FileConfiguration fileConfiguration = embeddedConfiguration.file();
fileConfiguration.storage(new MemoryStorage());
Db4oEmbedded.openFile(embeddedConfiguration, "myEmbeddedDb.db4o");

// opening a server for a client/server session
ServerConfiguration serverConfiguration = Db4oClientServer.newServerConfiguration();
FileConfiguration fileConfiguration = serverConfiguration.file();
fileConfiguration.storage(new MemoryStorage());
Db4oClientServer.openServer(serverConfiguration, "myServerDb.db4o", PORT);

(3) Using the NonFlushingStorage for highly improved db4o speed at the risk of corrupted database files in case of abnormal system failures

Storage nonFlushingStorage = new NonFlushingStorage(new CachingStorage(new FileStorage()));

// opening an embedded session
EmbeddedConfiguration embeddedConfiguration = Db4oEmbedded.newConfiguration();
FileConfiguration fileConfiguration = embeddedConfiguration.file();
fileConfiguration.storage(nonFlushingStorage);
Db4oEmbedded.openFile(embeddedConfiguration, "myEmbeddedDb.db4o");

// opening a server for a client/server session
ServerConfiguration serverConfiguration = Db4oClientServer.newServerConfiguration();
FileConfiguration fileConfiguration = serverConfiguration.file();
fileConfiguration.storage(nonFlushingStorage);
Db4oClientServer.openServer(serverConfiguration, "myServerDb.db4o", PORT);

In case you have missed Adriano's article, the above also demonstrates how the new configuration interface is intended to be used.

Now that we had a new good working caching architecture, we thought about further places where to use it. We came up with the idea to cache BTree nodes directly. Since these nodes are used for class indexes and for field indexes we expected improved performance for repetitive queries.

To get caching of BTree nodes to work, we implemented a new CacheablePersistentBase class to override some of the default behaviour of PersistentBase and derived BTreeNode from this new class.

After we had all up and running smoothely we compared the performance of different setups with the Poleposition benchmark. We could indeed see much faster queries if queries were executed in the same way a second time. Therefore we decided to leave this new second level cache in db4o as the default with a size of 30.

The size of the new node cache is also configurable as follows:

// opening an embedded session
EmbeddedConfiguration embeddedConfiguration = Db4oEmbedded.newConfiguration();
embeddedConfiguration.cache().slotCacheSize(64);
Db4oEmbedded.openFile(embeddedConfiguration, "myEmbeddedDb.db4o");

//opening a server for a client/server session
ServerConfiguration serverConfiguration = Db4oClientServer.newServerConfiguration();
serverConfiguration.cache().slotCacheSize(64);
Db4oClientServer.openServer(serverConfiguration, "myServerDb.db4o", PORT);

Do not expect wonders from using the new caches but please do play with all parameters, if you want to get the maximum speed out of db4o. The best cache settings will depend a lot on your individual usecase. We would be very happy if you would tell us about your experiences from testing out the new caches. Thanks in advance!

Enjoy!