db4o Developer Community

db4o open source object database, native to Java and .NET
Welcome to db4o Developer Community Sign in | Join
in Search
More Search Options

what can i do to optimize?

Last post 08-03-2008, 04:07 PM by quqtalk. 15 replies.
Page 1 of 2 (16 items)   1 2 Next >
Sort Posts: Previous Next
  •  07-31-2008, 12:45 PM 50419

    what can i do to optimize?

    I store FPValue into OODB and retrieve FPValue from OODB,but when Object amount >1000000,store and retrieve performance become very low.

    query execute spend almost 40ms,insert commit spend almost 1000ms.

    My system is real time,request very high performance.I need OODB can deal 10 FPValues per second.what can i do to optimize?

    Thank you!

    See my code:

    package com.boco.service;

    import java.util.ArrayList;
    import java.util.HashMap;
    import java.util.Iterator;
    import java.util.List;

    import javax.management.MBeanServer;
    import javax.management.MalformedObjectNameException;

    import org.apache.commons.logging.Log;
    import org.apache.commons.logging.LogFactory;
    import org.jboss.mx.util.MBeanProxyExt;
    import org.jboss.mx.util.MBeanServerLocator;

    import com.boco.alarm.common.utils.Property;
    import com.boco.alarm.interfaces.service.HandleService;
    import com.boco.dnms.common.alarm.model.AlarmObject;
    import com.boco.model.FPValue;
    import com.boco.service.db4o.mbean.Db4oToolServiceMBean;
    import com.db4o.Db4o;
    import com.db4o.ObjectContainer;
    import com.db4o.ObjectSet;
    import com.db4o.query.Candidate;
    import com.db4o.query.Evaluation;
    import com.db4o.query.Predicate;
    import com.db4o.query.Query;

    /**
     *
     * @date 2008-07-15
     * @author QSH
     * @use 对所有进入系统的告警进行判断,如果相同FP的告警曾经进如果系统,那么不在对此告警进行处理
     */
    public class AlarmCheckService implements HandleService {
     private static Log log = LogFactory.getLog(AlarmCheckService.class);
     private static final long FP_SEASON = Long.parseLong(Property.getProperty("FP_SEASON","7").trim())* 24 * 60 * 60 * 1000;
     private static  ObjectContainer db;
     //init OODB
     static {
      try {
       //对FPValue的属性加索引
       Db4o.configure().objectClass(FPValue.class).objectField("fp0").indexed(true);
       Db4o.configure().objectClass(FPValue.class).objectField("fp1").indexed(true);
       Db4o.configure().objectClass(FPValue.class).objectField("fp2").indexed(true);
       Db4o.configure().objectClass(FPValue.class).objectField("fp3").indexed(true);
       Db4o.configure().objectClass(FPValue.class).objectField("currentTimeMillis").indexed(true);
       Db4o.configure().optimizeNativeQueries(true);
       //使用单独的数据文件
       db = Db4o.openFile(System.getenv("ALARM_HOME")+"/cfg/common_service/NEWHISFP.yap");
       
      } catch (Exception e) {
       log.error("init OODB ERROR:");
       e.printStackTrace();
      }
     }

     public Object run(Object object) {
      long start = System.currentTimeMillis();
      try {
       log.info("AlarmCheckService run");
       log.info("FP_SEASON: "+FP_SEASON);
       final long currentTimeMillis = System.currentTimeMillis();
       log.info("currentTimeMillis: "+currentTimeMillis);
       long t1 = System.currentTimeMillis();
       // 删除过时告警FP
       clearOverSeasonFP(currentTimeMillis);
       long t2 = System.currentTimeMillis();
       log.info("delete over season alarms spend : "+(t2-t1)+" ms");
       if (object instanceof List) {
        List objectList = (List) object;
        log.info("before check objectList size: "+objectList.size());
        long t3 = System.currentTimeMillis();
        objectList = (abandon(treat(objectList),currentTimeMillis));
        log.info("after check objectList size:"+objectList.size());
        long t4 = System.currentTimeMillis();
        log.info("abandon alarms spend : "+(t4-t3)+" ms");
        log.info("AlarmCheck total spend: "+(t4 - start)+" ms");
        return objectList;
       } else
        log.warn("AlarmCheckService receive must be List");
       return object;
      } catch (Exception e) {
       log.error("AlarmCheckService occur ERROR:");
       e.printStackTrace();
       return object;
      }
     }


     List abandon(List<AlarmObject> alarmList, long currentTimeMillis) {
      int index = 0;
      for (Iterator iterator = alarmList.iterator(); iterator.hasNext();) {
       AlarmObject alarmObject = (AlarmObject) iterator.next();
       String alarmId = alarmObject.getAlarmId();
       final Long[] fp = treatAlarmId(alarmId);
       Query query = db.query();
       Query pointQuery0 = query.descend("fp0");
       Query pointQuery1 = query.descend("fp1");
       Query pointQuery2 = query.descend("fp2");
       Query pointQuery3 = query.descend("fp3");
       query.constrain(FPValue.class).and(
         pointQuery0.constrain(fp[0]).equal()).and(
         pointQuery1.constrain(fp[1]).equal()).and(
         pointQuery2.constrain(fp[2]).equal()).and(
         pointQuery3.constrain(fp[3]).equal());
       long t1 = System.currentTimeMillis();
       ObjectSet result = query.execute();
       long t2 = System.currentTimeMillis();
       log.info("query execute: "+(t2-t1));
       if (result.size() > 0) {
        log.warn("系统已经接收到过此告警FP对应的告警:" + alarmObject);
        iterator.remove();
       } else {
        index++;
        FPValue fpValue = new FPValue(alarmId, currentTimeMillis);
        db.set(fpValue);
       }
       query= null;
       result= null;
      }
      long t3 = System.currentTimeMillis();
      db.commit();
      long t4 = System.currentTimeMillis();
      log.info("insert commit: "+(t4-t3));
      return alarmList;
     }
     
     List<AlarmObject> treat(List<AlarmObject> list) {
      long tt = System.currentTimeMillis();
      log.info("before put into map : "+list.size());
      HashMap<String, AlarmObject> map = new HashMap<String, AlarmObject>();
      for (Iterator iterator = list.iterator(); iterator.hasNext();) {
       AlarmObject alarmObject = (AlarmObject) iterator.next();
       String fp = alarmObject.getAlarmId();
       map.put(fp, alarmObject);
      }
      log.info("after put into map : "+map.size());
      list= null;
      long tt2 = System.currentTimeMillis();
      log.info("treat "+ " use "+(tt2-tt)+" ms");
      return new ArrayList<AlarmObject>(map.values());
     }
     
     Long [] treatAlarmId(String alarmId){
      String []s = alarmId.split("_");
      Long fp0 = Long.valueOf(s[0]);
      Long fp1 = Long.valueOf(s[1]);
      Long fp2 = Long.valueOf(s[2]);
      Long fp3 = Long.valueOf(s[3]);
      return new Long []{fp0,fp1,fp2,fp3};
     }
     
     void clearOverSeasonFP(long currentTimeMillis) {
      long r =currentTimeMillis - FP_SEASON;
      Query query = db.query();
      Query pointQuery = query.descend("currentTimeMillis");
      query.constrain(FPValue.class).and(
        pointQuery.constrain(new Long(r))
          .smaller());
      long t1 =System.currentTimeMillis();
      ObjectSet result = query.execute();
      long t2 =System.currentTimeMillis();
      log.info("query over season alarms spend: "+(t2-t1));
      log.info("OverSeasonFP size: "+result.size());
      for (Iterator iterator = result.iterator(); iterator.hasNext();) {
       FPValue fpValue = (FPValue) iterator.next();
       db.delete(fpValue);
      }
      db.commit();
      result=null;
     }
    }

  •  07-31-2008, 01:21 PM 50420 in reply to 50419

    Re: what can i do to optimize?

    Sorry I have few time.

    Look here: maybe it could help

    http://developer.db4o.com/Resources/view.aspx/Reference/Tuning/Performance_Hints/Reserving_Storage_Space 

    It is important that you commit every object? It´s better if you commit not every object, try commit only after 10.000 or 20.000 objects.

    Or please look here.

    http://developer.db4o.com/Resources/view.aspx/Reference/Tuning/Performance_Hints

    http://developer.db4o.com/Resources/view.aspx/Reference/Tuning


     


    Germany
  •  07-31-2008, 03:33 PM 50428 in reply to 50420

    Re: what can i do to optimize?

    Yes,commit once for 10000 or 2000 objects can save some time.but if there are 1,000,000 objects in OODB, commit once like this will spend much time too.

    Ask questions:

    1.My index use is right?

       Db4o.configure().objectClass(FPValue.class).objectField("fp0").indexed(true);
       Db4o.configure().objectClass(FPValue.class).objectField("fp1").indexed(true);
       Db4o.configure().objectClass(FPValue.class).objectField("fp2").indexed(true);
       Db4o.configure().objectClass(FPValue.class).objectField("fp3").indexed(true);
       Db4o.configure().objectClass(FPValue.class).objectField("currentTimeMillis").indexed(true);

    2.My query's performance is highest?

       Query query = db.query();
       Query pointQuery0 = query.descend("fp0");
       Query pointQuery1 = query.descend("fp1");
       Query pointQuery2 = query.descend("fp2");
       Query pointQuery3 = query.descend("fp3");
       query.constrain(FPValue.class).and(
         pointQuery0.constrain(fp[0]).equal()).and(
         pointQuery1.constrain(fp[1]).equal()).and(
         pointQuery2.constrain(fp[2]).equal()).and(
         pointQuery3.constrain(fp[3]).equal());
       long t1 = System.currentTimeMillis();
       ObjectSet result = query.execute();
       long t2 = System.currentTimeMillis();
       System.out.println("query execute: "+(t2-t1));

    com.boco.model.FPValue:

    package com.boco.model;

    import java.io.Serializable;

    /**
     *
     * @date 2008--07-15
     * @author QSH
     * @use 到达系统告警的FP
     *
     */
    public class FPValue{

     /**
      *
      */
     private static final long serialVersionUID = 5744685275674558328L;
     private long fp0;
     private long fp1;
     private long fp2;
     private long fp3;
     private long currentTimeMillis;

     public FPValue(String fp, long currentTimeMillis) {
      String fpArray[] = fp.split("_");
      this.fp0 = Long.parseLong(fpArray[0]);
      this.fp1 = Long.parseLong(fpArray[1]);
      this.fp2 = Long.parseLong(fpArray[2]);
      this.fp3 = Long.parseLong(fpArray[3]);
      this.currentTimeMillis = currentTimeMillis;
     }

     public long getFp0() {
      return fp0;
     }

     public long getFp1() {
      return fp1;
     }

     public long getFp2() {
      return fp2;
     }

     public long getFp3() {
      return fp3;
     }

     public void setFp0(long fp0) {
      this.fp0 = fp0;
     }

     public void setFp1(long fp1) {
      this.fp1 = fp1;
     }

     public void setFp2(long fp2) {
      this.fp2 = fp2;
     }

     public void setFp3(long fp3) {
      this.fp3 = fp3;
     }

     public long getCurrentTimeMillis() {
      return currentTimeMillis;
     }

     public void setCurrentTimeMillis(long currentTimeMillis) {
      this.currentTimeMillis = currentTimeMillis;
     }

     public String getFp() {
      return fp0 + "_" + fp1 + "_" + fp2 + "_" + fp3;
     }

     @Override
     public String toString() {
      return "<"+this.getClass().getName() + ">" + fp0 + "_" + fp1 + "_" + fp2
        + "_" + fp3;
     }

     @Override
     public int hashCode() {
      final int prime = 31;
      int result = 1;
      result = prime * result + (int) (fp0 ^ (fp0 >>> 32));
      result = prime * result + (int) (fp1 ^ (fp1 >>> 32));
      result = prime * result + (int) (fp2 ^ (fp2 >>> 32));
      result = prime * result + (int) (fp3 ^ (fp3 >>> 32));
      return result;
     }

     @Override
     public boolean equals(Object obj) {
      if (this == obj)
       return true;
      if (obj == null)
       return false;
      if (getClass() != obj.getClass())
       return false;
      final FPValue other = (FPValue) obj;
      if (fp0 != other.fp0)
       return false;
      if (fp1 != other.fp1)
       return false;
      if (fp2 != other.fp2)
       return false;
      if (fp3 != other.fp3)
       return false;
      return true;
     }

    }

     

  •  07-31-2008, 04:05 PM 50429 in reply to 50428

    Re: what can i do to optimize?

     Why so much com.db4o.internal.btree.FieldIndexKey instance?

    And,i use Jpfofiler watch the memory,find there are 963,241 com.db4o.internal.btree.FieldIndexKey instance.What FieldIndexKey effect here?

  •  08-01-2008, 06:45 AM 50448 in reply to 50428

    Re: what can i do to optimize?

    quqtalk:

    1.My index use is right?

       Db4o.configure().objectClass(FPValue.class).objectField("fp0").indexed(true);
       Db4o.configure().objectClass(FPValue.class).objectField("fp1").indexed(true);
       Db4o.configure().objectClass(FPValue.class).objectField("fp2").indexed(true);
       Db4o.configure().objectClass(FPValue.class).objectField("fp3").indexed(true);
       Db4o.configure().objectClass(FPValue.class).objectField("currentTimeMillis").indexed(true);

    Yes and No! Because :

        For creating and dropping indexes on large amounts of objects there are two possible strategies:

    1. Import all objects with indexing off, configure the index and reopen the ObjectContainer/ObjectServer.
    2. Import all objects with indexing turned on and commit regularly for a fixed amount of objects (~10,000).
    1. will be faster.
    2. will keep memory consumption lower.
    If you don´t query the "currentTimeMillis" you need no index.

    Do you need to query between the inserts?

    Have you read the chapter about "Reserving Storage Space"? 

    quqtalk:

    2.My query's performance is highest?

       Query query = db.query();
       Query pointQuery0 = query.descend("fp0");
       Query pointQuery1 = query.descend("fp1");
       Query pointQuery2 = query.descend("fp2");
       Query pointQuery3 = query.descend("fp3");
       query.constrain(FPValue.class).and(
         pointQuery0.constrain(fp[0]).equal()).and(
         pointQuery1.constrain(fp[1]).equal()).and(
         pointQuery2.constrain(fp[2]).equal()).and(
         pointQuery3.constrain(fp[3]).equal());
       long t1 = System.currentTimeMillis();
       ObjectSet result = query.execute();
       long t2 = System.currentTimeMillis();
       System.out.println("query execute: "+(t2-t1));

    You can take QBE (Query By Exsample). It's faster and in my oppinion more easy to use in your usecase.

    quqtalk:

     public FPValue(String fp, long currentTimeMillis) {
      String fpArray[] = fp.split("_");
      this.fp0 = Long.parseLong(fpArray[0]);
      this.fp1 = Long.parseLong(fpArray[1]);
      this.fp2 = Long.parseLong(fpArray[2]);
      this.fp3 = Long.parseLong(fpArray[3]);
      this.currentTimeMillis = currentTimeMillis;
     }

    Where is the logic of fp.split implemented?

    Why you make this transformation within your class?

    I think it´s better to make a constructor like this:

     public FPValue(Array fp, long currentTimeMillis) {
      this.fp0 = fp[0];
      this.fp1 = fp[1];
      this.fp2 = fp[2];
      this.fp3 = fp[3];
      this.currentTimeMillis = currentTimeMillis;
     }

    and transform the string befor you to get it constructer.
     


    Germany
  •  08-01-2008, 06:55 AM 50449 in reply to 50429

    Re: what can i do to optimize?

    quqtalk:

     Why so much com.db4o.internal.btree.FieldIndexKey instance?

    And,i use Jpfofiler watch the memory,find there are 963,241 com.db4o.internal.btree.FieldIndexKey instance.What FieldIndexKey effect here?

    You can read about effect and tuning here: http://developer.db4o.com/Resources/view.aspx/Reference/Tuning/Performance_Hints/B-Tree_Tuning

     

    Another Idea:

    Maybe it´s possible to split your DB in several databases?  

     


    Germany
  •  08-01-2008, 11:13 AM 50456 in reply to 50448

    Re: what can i do to optimize?

    Joined:

    If you don´t query the "currentTimeMillis" you need no index. 

    I use "currentTimeMillis"  index to quicken delete FPValue  in method " void clearOverSeasonFP";

    Joined:

    Do you need to query between the inserts?
    Have you read the chapter about "
    Reserving Storage Space"?

    What does " query between the inserts" mean? means do query when insert?

     "Reserving Storage Space" means use "MemoryIoAdapter"?Can this improve performance a lot?Can this bring non-stability memory?This is very important for my system.

    Joined:

    You can take QBE (Query By Exsample). It's faster and in my oppinion more easy to use in your usecase.

    Now,i know QBE is as useful as advanced example which i used.

    Joined:

    transform the string befor you to get it constructer.

    Yes,i should amend the constructer.

     

  •  08-01-2008, 11:23 AM 50457 in reply to 50449

    Re: what can i do to optimize?

    Joined:

    You can read about effect and tuning here: http://developer.db4o.com/Resources/view.aspx/Reference/Tuning/Performance_Hints/B-Tree_Tuning

    Another Idea:

    Maybe it´s possible to split your DB in several databases?  

    When i close ObjectContainer(db) after each run(Object object),amount of com.db4o.internal.btree.FieldIndexKey can keep an stable state.But another class "com.db4o.inside.freespace.FreeSlotNode" keep high amount.And can't be GC(garbage collection) immediately.

    What does FreeSlotNode use?How can i keep away from it?

     

  •  08-01-2008, 11:34 AM 50459 in reply to 50457

    Re: what can i do to optimize?

    And,i write a stand alone programe to test the performance and stability of memory,up to now,i still can't arrive gain my ends.

    In future,this module will run in Jboss 3.

    Who can give me some advice and help,thank you!

    And,master-hand,you can leave your e-mail,i will post the stand alone programe to you.thank you!

  •  08-01-2008, 11:37 AM 50460 in reply to 50449

    Re: what can i do to optimize?

    Joined:

    Another Idea:

    Maybe it´s possible to split your DB in several databases?  

    Do this can improve the performance and keep memory stable?

    And how to split DB?

    Thank you very much!

  •  08-01-2008, 12:07 PM 50464 in reply to 50456

    Re: what can i do to optimize?

    quqtalk:

    What does " query between the inserts" mean? means do query when insert?

    I want to say: When or how often do you query your stored objects? If you query seldom it is a good strategie to disable the index´s to improve your insert performance. Can you tell me more about your usecase? As I understood your usecase you save all second 10 objects like a log system, but when do you need the stored objects.

    Have you try out to run your application without indexing?
     

    quqtalk:

    "Reserving Storage Space" means use "MemoryIoAdapter"?Can this improve performance a lot?Can this bring non-stability memory?This is very important for my system.

    No MemoryIoAdapter is another technique. MemoryIoAdapter hold complete DB in memory. It increase the performance but I think your DB is to big to hold complete in Memory. Try it. The memory should be stable (possible that occure "out of heap space") but if your system crashes the database in memory is dead.(You have to save the DB from time to time to disk to avoid it) What do you mean excactly with "bring non-stability"? 

    Reserving Storage Space:

    Advantage:
    The allocation of a fixed number of bytes at one time makes it more likely that the database will be stored in one chunk on the mass storage. Less read/write head movement can result in improved performance. 

    quqtalk:

    Now,i know QBE is as useful as advanced example which i used.

    Have you try QBE? It is Faster than S.O.D.A. It should increase your Performance.


     


    Germany
  •  08-01-2008, 12:22 PM 50465 in reply to 50460

    Re: what can i do to optimize?

    quqtalk:

    Do this can improve the performance and keep memory stable?

    And how to split DB?

    Thank you very much!

    Your problem is the insert performance at a amount of more than 1mio objects. Right? OK!  I think it should be possible to build a method(manager) which administrate your objects in several DB´s. Imagine you have a table where are a entry for your object and another one where (in which db) it is.

    So, imagine your object count reach the 500,000 then your manager detect "db is full" then it create a new db and store the next objects in this new db. Problems occours if you want query objects, then you have to look at all db´s or you have a table or something where you had store in which db your object should be.

    I don´t know exactly what is stable for you but in my oppinion it is stable because it´s the same in principle like before. 

    Sorry for my bad language skills but I hope you understand my idea.  


    Germany
  •  08-01-2008, 12:33 PM 50466 in reply to 50457

    Re: what can i do to optimize?

    quqtalk:

    When i close ObjectContainer(db) after each run(Object object),amount of com.db4o.internal.btree.FieldIndexKey can keep an stable state.But another class "com.db4o.inside.freespace.FreeSlotNode" keep high amount.And can't be GC(garbage collection) immediately.

    What does FreeSlotNode use?How can i keep away from it?

    Before you close the objectcontainer after each run you had so much com.db4o.internal.btree.FieldIndexKey because the objects were hold in cache.

    Now when you close your object container you have afterthat less objects in cache. Therefore you have less com.db4o.internal.btree.FieldIndexKey. 

    Freespace management: The default in-memory free space manager keeps free slot infomration in memory as trees. It consumes memory proportional to the number of free slots. Usually, the bigger db file is, the more free slots. With BTree free space manager, you can get static memory consumption.

    BTree nodes: ClassIndex, FieldIndex, and BTree free space manager are implemented with BTree.

     


    Germany
  •  08-03-2008, 07:49 AM 50489 in reply to 50466

    Re: what can i do to optimize?

    Germany:

    My usecase:


    My system is a net-management system for a mobile telecom company.System's primary action is deal with the data which is producedby switch equipment(we call this data "alarm").We create a model com.xxx.model.AlarmObject,which is a JavaBean.One of its field is "Fp".Fp is primary key of AlarmObject.AlarmObjects enter system must ensure Fp is different.But sometimes these AlarmObjects enter maybehave same Fp.So i must persistent all Fp,at present,for small YAP file,i only use a new object "test.model.FPValue" replace Fp.I store 7days's Fp into OODB and delete the overtime(over 7 days) Fp.In 7 days if i can find AlarmObject's Fp in OODB,i will remove thisAlarmObject.


    There will be 5-10 AlarmObjects enter system per second,in other words,i must ensure OODB can work on high performance whenthere are over 1,500,000-2,000,000 objects in DB,otherwise,AlarmObjects that can't be treat will become more and more,till mysystem become invalid...


    But luckily,after communitate with customer,i only need persistent 3 days's AlarmObject.So,now the performance is not the first business.I can use QBE get high performance,however,another thing,"memory stability" become intractability.Memory stability is if memory whichprograme use can keep at stable size.In my test use JProfiler,i find the memory size is grow higer and higer along with execute time.This is the agonisingest,my system must service for a long time,can't be shut down because outofmemory or other analogue.


    Now,result in memory consumption grow all along is two Object,com.db4o.internal.btree.FieldIndexKey & com.db4o.inside.freespace.FreeSlotNode.Especially "com.db4o.inside.freespace.FreeSlotNode",can't be GC immediately.I usedb4o-6.x and db4o-7.x respectively,Objects in memory are different.But 7.x has more high query performance.My ststem now isusing 6.x jar file.So,i have two way,one is improve performance use 6.x,another is update to 7.x,but must keep memroy stable.On db4o download page,6.x is declared stable version,but 7.x don't.How can i keep stable amount of these two object,especially"com.db4o.inside.freespace.FreeSlotNode"?


    In any case,Germany,thank you for your advice.In addition,your explain is very clear.Relative to you,my english is too poor,hehe.

  •  08-03-2008, 03:24 PM 50491 in reply to 50489

    Re: what can i do to optimize?

    quqtalk wrote:
    > I store 7days's Fp into OODB and delete the overtime(over 7 days) Fp.
    > In 7 days if i can find AlarmObject's Fp in OODB,i will remove
    > thisAlarmObject.

    We have a couple of realtime customers with similar requirements.

    For many of them it has worked very good to create a database
    for a certain timeframe, for instance for one day.

    With this approach you create one new database every day and
    you delete (or archive) one database each day.

    Using the com.db4o.cluster.Cluster class, you can query over
    all databases with one query.

    Using this approach our customers have found that it's much
    more efficient to delete (or archive) a database than to
    delete individual objects.
    First of all it is faster.
    Second, you avoid fragmentation.

    You can also choose the timeframe per database to be longer or
    shorter and you can even load-balance queries and keep multiple
    copies of the old databases.
Page 1 of 2 (16 items)   1 2 Next >
View as RSS news feed in XML