forum.vdsworld.com

X-Tools · Posted: Fri Aug 15, 2003 3:37 pm Post subject: Speed of Lists

Hi,

i have one problem.
I have to compare around 50000 to 100000 entries in 2 lists.
Adding them to the list is really fast, but comparing (using @match)
is slow.

Are Ascii Lists faster then displayed lists ?

Bye, Fabian

LiquidCode · Posted: Fri Aug 15, 2003 3:57 pm Post subject:

Are your lists on a dialog or are they hidden lists (Lists 1 - 12). If the lists are on a dialog, then yes it will be slow because it has to update the list graphically as well as search for the item. If they are in a hidden list, it should be pretty fast.

Dr. Dread · Posted: Fri Aug 15, 2003 4:33 pm Post subject:

If you compare an entire list of 50,000 items with another holding 100K then it will be quite slow
cause even though each @match() cycle does not take long then things add up when you multiply by
50K...

A word of caution for you: VDS lists get unstable around 100,000 items, so when you do a LIST LOADFILE then
check the result with @ok(), otherwise you may end up with a list that does not hold the last items in your
file.

If you wanna manipulate really large files of items, then take a look at the VDSLIST.DLL in Tommy's
VDSDLL3 package. It's very much faster when loading files and is consequently ideal if you do
random access to single items in a big list (i.e. load file - find a couple of items - extract those - exit).

Greetz
Dr. Dread

jwfv · Posted: Fri Aug 15, 2003 5:00 pm Post subject:

Does VDSLIST.DLL have a limit on the size of lists like VDS does?

Tommy · Posted: Fri Aug 15, 2003 10:54 pm Post subject:

VDSLIST.DLL will have a more relaxed limit than VDS itself, though probably still
limit, but I couldn't even be sure myself about any exact limits.

For comparing two texts however, only to determine whether they are equal or not,
it could be a solution to use VDSMD5.DLL and compare the checksums of the two lists.
This doesn't provide 100% certainty, but should provide a very good indication.

Garrett · Posted: Fri Aug 15, 2003 11:21 pm Post subject:

You might also try a simple

X-Tools · Posted: Sat Aug 16, 2003 6:56 am Post subject:

Hi,

thanks for all your answers. I will try the ascii lists and the vdslists.dll.

I cannot compare the whole file:
I am reading records from a mysql database and comparing
them for a synchronisation.
So List1 holds records of database1 (only a datetime field) and database2 the datetime of records of database2.
Then I compare both lists to find out whats newer.

At this time I have visible lists.
adding the 50000 records of the mysql database to the visible
lists is ready in around 10 seconds. comparing with the other list needs multiple minutes. I will try ascii lists (1-12)

Bye, Fabian

Sanjuro · Posted: Sat Aug 16, 2003 8:41 am Post subject:

Would it be possible with VDSDB.dll to create a connection to both databases and then create a query that compares date/time functions from DB1 to DB2? Question

The connection is the easy part i think, but the SQL query string for 2 dbases is the tricky part. Crying or Very sad

Anyone with SQL expereince know if you can query 2 dbases like this? Question

Just my thoughts, I am in no way a database expert. Wink

Cheers
Sanjuro

X-Tools · Posted: Mon Aug 18, 2003 9:23 am Post subject:

Hi everyone.

Sanjuro, i am already doing 2 connections with the new mysql.dll, but comparing local is better than making thousands of single querys.

To all:

I have tested it running ascii lists:
on a firstsync (without entries in one database) the new sync was
up to 85% faster !!!

on a normalsync (entries in both databases with comparing) the
new one was up to 53 % faster Big Smile

This is really great. Thank you all.

Bye, Fabian