forum.vdsworld.com

Dr. Dread · Posted: Thu Jul 03, 2003 1:35 pm Post subject: Big lists

Like the previous versions VDS5 has problems with big lists. I can create a list and fill it with for instance
150,000 items and save it to a file. But if I want to load the list back in from the file, then it breaks at
about 100K items.

Test:

Garrett · Posted: Thu Jul 03, 2003 9:02 pm Post subject:

Isn't this related to how much memory is installed and limitations of the
Windows OS itself??

-Garrett

Dr. Dread · Posted: Thu Jul 03, 2003 9:57 pm Post subject:

I dunno what the reason is. I'm pretty sure that this has nothing to do with available memory coz even though
this list has a lot of entries it's only about 1MB in size.

And the thing that makes me wonder is that I can create and save a list with 150,000 items by cycling a "list add"
command - no problems. But after clearing the list, I cannot get the exact same data into the very same list by
"list loadfile" - it breaks somewhere around 100,000 recs (but not at specific count - it will vary each time)...

Greetz
Dread

DanTheMan · Contributor Joined: 15 Mar 2002 Posts: 56 Location: Sweden

Tryed your script Dr. ,but when running I've got this :

Although the file "vdslist_multiK.lst" contains 150000 posts, things
are not going well here ...

/Dan

DanTheMan · Contributor Joined: 15 Mar 2002 Posts: 56 Location: Sweden

Some problem with the included "popup" image. Sad

Should be:

Oops!

VDS has caused an exeption of class "EThread"
It may be wise to save your work, close VDS and then restart it.

/Dan

Garrett · Posted: Fri Jul 04, 2003 7:54 pm Post subject:

Decided to give this a try to see if I could also get the same result... I also
received the error mentioned above. After clicking the button, twice, the
script continued and eventually got the popup message showing the list
counts:

101280 items loaded

Then, thought I'd give this a try in VDS 4:

102160 items loaded

Ok, let's try VDS 3 now.....

102144 items loaded

Ohhh, let's try VDS 2 just for the fancy of it:

102080 items loaded

Well, that's all the versions I have for testing this. This was done on
Windows ME. Anyone try this on 2k or XP?

BTW, the resulting file is about 1.03 Mb

-Garrett

jules · Posted: Thu Jul 31, 2003 6:36 pm Post subject:

It's the same on XP. The problem is in the version of Delphi that the VDS runtime is built with. A Delphi test program that tried to load the file produced an "Out of memory" error. Unfortunately, that means the problem isn't fixable in the short term.

FreezingFire · **Admin Team** Joined: 23 Jun 2002 Posts: 3508

Please don't take this the wrong way, I'm not trying to start any sort of
fight or anything. Just wondering. Smile

Will we ever see VDS compiled in a newer version of Delphi? It seems that
VDS has a lot of limits that could be fixed by compiling with a new version
of Delphi... I think the standard version only costs about $99, I may be
wrong. I know that it invloves a lot of work to update the code, but I think
it would be more than beneficial to VDS to upgrade it to a newer version
of Delphi. Smile

PGWARE · **Web Host** Joined: 29 Dec 2001 Posts: 1563

The new version of Delphi costs about $399 if you are updating from previous versions, $999 for first time buyers. The personal edition $99 is only for freeware and for applications for personal use, you cannot distribute commercial/shareware applications with the personal edition of Delphi.

As to recompiling VDS in Delphi:

It wouldn't require just opening the vds project in Delphi and hitting compile, I suspect alot of changes would be required and any components VDS uses would also need to be Delphi 7 compatible. This would be a major update to VDS though and would probably be a whole new version of VDS. I don't know if customers would appreciate seeing (speculation) vds 6 with little to no new features but just recompiled in Delphi 7 to fix some outstanding bugs.

FreezingFire · **Admin Team** Joined: 23 Jun 2002 Posts: 3508

CodeScript · Posted: Fri Aug 01, 2003 1:37 am Post subject:

Additionally the size of exe and especially runtime would grow very much I think Question

Skit3000 · Posted: Fri Aug 01, 2003 8:32 am Post subject:

When compiled in a new version of Delphi, VDS could 'create' new bugs... Rolling Eyes

jules · Posted: Fri Aug 01, 2003 9:08 am Post subject:

When we began work on VDS 5, we had to make a decision whether to continue using Delphi 2 for the runtime or not. (The new IDE was developed using Delphi 6.) When VDS began, Delphi produced good, lean code making it a good choice of tool for this type of development. The latest versions of Delphi create much larger files because the VCL includes support for enterprise database stuff. If compiled in Delphi 6, the run time would be more than twice the size and much of this extra code would be not be used by VDS.
We took the decision that a small run-time was a good selling point for VDS. Most of the bugs or limitations of Delphi 2 have been worked around in VDS. Unfortunately, this particular issue is caused by limitations of the Delphi memory manager which is so integral to the Delphi 2 VCL that it would not be possible to work around it using Delphi 2 except by rewriting the VDS runtime so it didn't use the Delphi VCL at all. This would introduce even more incompatibilities than using a newer version of Delphi.
I think what happens in the future depends on what VDS users see as the main uses and benefits of VDS. Is a much bigger run-time a price worth paying for getting rid of these limitations and adding new features, bearing in mind that many users may only be developing simple scripts, and may be put off by this?
As some have said, it is not a matter of just recompiling the code with a newer version of Delphi. Some bugs and incompatibilities would inevitably occur, requiring more extensive testing. Therefore this is an issue to consider for the next major release, rather than 5.01 or whatever the next bug-fix update will be called.

Dr. Dread · Posted: Fri Aug 01, 2003 10:38 am Post subject:

Well, as this issue is not gonna be fixed in foreseeable future, I'd suggest that others using VDS for
handling bulk lists (above 100,000 items) use the VDSLIST.dll from Tommy's VDSDLL3 collection.

Now a number of bugs have been fixed, making this DLL stable and much much faster for loading
large files into lists. If you wanna cycle through the lines in big lists, you may wanna use plain VDS lists:
with VDSLIST the file will be loaded much quicker but because getting each line involves a call to the
external DLL, it may be quicker in total time to use a normal VDS list. If you want to pick out single lines
from the file (random access) then VDSLIST will outspeed plain VDS lists by multiple factors.

Greetz
Dr. Dread

jules · Posted: Fri Aug 01, 2003 10:52 am Post subject:

I think this illustrates the great thing about VDS extensibility. It's possible to create a solution to a problem that affects only a few people without bloating the runtime with code that most people won't use, just by using a DLL instead. It's worth bearing in mind that VDS lists were also intended to be compatible with list type window elements, and many of these have even more severe capacity limitations.