db4o Developer Community

From looking at the diagnostic messages produced by Db4o it would appear a lot of my native queries are not being optimised.  After searching through the forum I believe the reason is native query optimisation cannot optimise queries which use methods defined in an interface.  Am I correct about this and if so has an issue been created to fix this?

 Below is an example of a native query which cannot be optimised. As you can see it is very simple but both the methods 'isOn' and 'isCancelled' are defined as part of an interface.  Unfortunately the design of my application makes this interface a necessity, so it cannot easily be removed.

public static final class GetAppointments extends Predicate<Appointment>
   {
        private Date date;
       
        public GetAppointments(Date date)
        {
            this.date = (Date) date.clone();
        }
       
        public boolean match(Appointment appointment)
        {
            return appointment.isOn(date) &&
                    !appointment.isCancelled();
        }
   } 

Regards
Graham

Hi Graham,

#isOn() and #isCancelled() seem to be method-calls, which do a little bit more than to access a simple property, aren't they?

A Native Query can only be optimized if it can be translated into a low-level SODA-Query. SODA-Queries can operate on fields, not on methods.

So a a rule of thumb: If there's no possibility to provide an appropriate SODA-Query, a Native Query cannot be optimized.

Cheers, Maik 

---

http://db4o.blogspot.com/

Thanks Maik, I will modify my queries to access the fields instead.  The reason for using methods was to make the code more meaningfull, but this small sacrifice will be worth it for the performance benefits.

I wonder whether as an enhancement it would be possible to provide hints to Db4o on how to optimise a native query e.g. by suggesting which index to use, thus allowing this type of query to be written.

Thanks again for your reply.

Graham

I have simplified my native query so it references the fields directly, but still the query is not being optimised.  I have tried this with the latest development version 6.3.201 and I have db4o-6.3-nqopt.jar and bloat-1.0.jar in the classpath.  The revised query is shown below including the runtime diagnostic messages.

I have checked the reference documentation for Native Query Optimisation and it seems to suggest optimisation only works for primitive types and String.  Could it be that Db4o cannot yet optimise queries which reference java.util.Date fields?

 public static final class DailyAppointmentsQuery
            extends Predicate<Appointment>
    {
        private Date date;
       
        public DailyAppointmentsQuery(Date date)
        {
            this.date = (Date) date.clone();
        }
       
        public boolean match(Appointment appointment)
        {
            return appointment.appointmentDate.equals(date);
        }
    }

  es.database.Appointment$DailyAppointmentsQuery@159ea8e :: Native Query Predicate could not be run optimized
    This Native Query was run by instantiating all objects of the candidate class. Consider simplifying the expression in the Native Query method. If you feel that the Native Query processor should understand your code better, you are invited to post yout query code to db4o forums at http://developer.db4o.com/forums
:: db4o 6.3.201 Diagnostics ::
  es.database.Appointment :: Query candidate set could not be loaded from a field index
    Consider indexing fields that you query for: Db4o.configure().objectClass([class]).objectField([fieldName]).indexed(true)
:: db4o 6.3.201 Diagnostics ::

As an aside, it took my laptop over 2.5 hours to extract the 6.3 download file.  Is this normal?  My laptop has an AMD 64bit 1.8GHz processor with 1.25GB memory and is running Windows Vista Home Premium 32 bit.  The 6.1 download also usually takes over an hour to extract.  This seems to me to be a very long time just to extract a download file but I wondered what other users experiences are.

Regards
Graham

graphi wrote:
> As an aside, it took my laptop over 2.5 hours to extract the 6.3 download
> file.
> Is this normal? My laptop has an AMD 64bit 1.8GHz processor with 1.25GB
> memory
> and is running Windows Vista Home Premium 32 bit. The 6.1 download also
> usually
> takes over an hour to extract. This seems to me to be a very long time
> just to
> extract a download file but I wondered what other users experiences are.

Interesting.
Does anyone have similar experiences?

With Winzip extracting takes exactly 10 seconds on my machine.

Do you maybe have a very safe virus scanner on your machine?

I rather suspected I should be using an unzip utility!  I have downloaded ALZip and can now extract Db4o in a matter of seconds.

I'm still struggling with query optimisation though!  Does anyone know if it can optimise Date fields?

Graham

Responding to graphi:

> From looking at the diagnostic messages produced by Db4o it would appear
> a lot of my native queries are not being optimised. After searching
> through the forum I believe the reason is native query optimisation
> cannot optimise queries which use methods defined in an interface. Am I
> correct about this and if so has an issue been created to fix this?

Yes, you're right.

This constraint is not restricted to interfaces, it basically applies to
all methods with multiple (re-)redefinitions in an inheritance
hierarchy. In the worst case, as of now, NQ may even return incorrect
results for overridden methods, because the "parent" version of the
method is taken into account for NQ optimization where the overridden
"child" version is intended.

To fix this issue, db4o would have to create an internal "vtable",
mapping the overridden variants of all methods used in NQ predicates to
their pre-calculated flow graphs. An NQ predicate referring to such a
"virtual" method then would have to be optimized to a disjunction over
all "most specific" types as the extents and the corresponding flow graph.

I'm not sure whether a Jira issue exists for that. I'll check later,
create if not existent, and post the link here. However, I fear this one
won't be tackled short-term.

Best regards,
Patrick

Thanks for your response Patrick.  I understand the issues with methods and interfaces and don't expect this to be fixed soon.  However, if you read my later post, I have re-written a native query to reference the date field of my object directly but the query is still not being optimised.  I read in the reference section under Native Query Optimisation that only primitive types and Strings can be optimised.  So my question now is can Date fields be optimised and if not will this issue be tackled soon.  The reason I ask is because my application uses dates quite heavily so for me this is very important.  If I have to re-write my queries in SODA then I will but I would prefer to stick with Native Queries if possible.

Regards
Graham

Responding to graphi:

> I have checked the reference documentation for Native Query Optimisation
> and it seems to suggest optimisation only works for primitive types and
> String. Could it be that Db4o cannot yet optimise queries which
> reference java.util.Date fields?

Dates are considered primitives from the db4o POV. However, there was a
bug in the Java NQ optimizer that didn't handle Dates as primitives, so
the fallback behavior for equals() would kick in - which is unoptimized
execution, since equals() is the perfect example for method with
multiple polymorphic definitions that would require a "vtable" for
optimization.

http://tracker.db4o.com/browse/COR-760

Fixed in repository, should go into the next continuous build.

Best regards,
Patrick

Responding to myself:

> I'm not sure whether a Jira issue exists for that. I'll check later,
> create if not existent, and post the link here. However, I fear this one
> won't be tackled short-term.

http://tracker.db4o.com/browse/COR-128

Best regards,
Patrick

Patrick,

Thanks for fixing the bug.  I will download the continuous build over the weekend and test.

Regards
Graham

I downloaded the java continuous build 6.4 on Monday afternoon but I still receive the same query not optimised diagnostic message as shown below.  I guess I must be doing something wrong but I don't know what this might be.  The query is designed to select all Appointment objects which occur on the giveln date and the appointment date field is indexed.  I also have bloat-1.0.jar and db4o-6.4-nqopt.jar in the classpath.  Does it matter which order these jar's are specified?  The reason I ask is because if the nqopt.jar is specified before db4o-6.4-java5.jar my code does not compile with errors 'com.db4o.query.Predicate does not take parameters'.

 :: db4o 6.4.000 Diagnostics ::
  es.database.company.Appointment$DailyAppointmentsQuery@190b72c :: Native Query Predicate could not be run optimized
    This Native Query was run by instantiating all objects of the candidate class. Consider simplifying the expression in the Native Query method. If you feel that the Native Query processor should understand your code better, you are invited to post yout query code to db4o forums at http://developer.db4o.com/forums
:: db4o 6.4.000 Diagnostics ::
  es.database.company.Appointment :: Query candidate set could not be loaded from a field index
    Consider indexing fields that you query for: Db4o.configure().objectClass([class]).objectField([fieldName]).indexed(true)

Regards
Graham