forum.vdsworld.com

marty · Professional Member Joined: 10 May 2001 Posts: 789

Ok guys I need advice and ideas over here...

Here is what I am trying to acheive and what I have done so far.

Application info - URL Filtering (300,000 urls)

Code done so far:

- VDS app loads the TXT that contains the 300,000 Urls into a internal list created by VDSLIST.DLL (more efficient than the original VDS one)
- VDS app monitors every 0.9 sec IE and Firefox for bad URLS
If IE used : I capture IE url first (from the comboEX class) than scan my internal list.. if found I send by DDE another webpage to block site.
If FireFox : I capture the URL using DDE, than check in the internal list.. if ound I send by DDE another webpage to block it

Now this seems the most efficient way I found so far.. BUT its slow on an older machine (PII 400).. it takes up to 10 seconds sometimes my app sends back the block page... Sad

Here is what I tried also so far:

- Used String.dll instead of VDSList.dll. A bit slower
- Used the sqllite.. also slow..

Thanks in advance...

Skit3000 · Posted: Mon Apr 24, 2006 3:41 pm Post subject:

How about creating multiple list of urls? Smile

Lets say one list first all urls which start with "a", one for "b", et cetera.

marty · Professional Member Joined: 10 May 2001 Posts: 789

Yes thats an idea Prakash gave me this morning.. and I will certain see if I can do that.

Problem is the list is one big file and I would have to manually do that every time there is an update from the guys that maintain that list.

Good suggestion Smile

Skit3000 · Posted: Mon Apr 24, 2006 3:55 pm Post subject:

Another way which might be a good solution, is to load the list into VDS (make sure it is sorted by the alphabet) and then create "indexes". Just create a second list, and put the 1000th, 2000th, 3000th, etc. item from the first one into the second. Now, when searching, look in the second list between which indexed items the current url belongs, and start searching from that item.

Edit: little example added Smile

marty · Professional Member Joined: 10 May 2001 Posts: 789

Thanks Skit Smile

Will look into that.

Will see if I can do a proxy with VDS.. the HTTPX extension is not stable.. so will look at other solution

Serge · Posted: Tue Apr 25, 2006 3:19 am Post subject:

marty,

i would use that idea

marty · Professional Member Joined: 10 May 2001 Posts: 789

Thanks Serge for the suggestions! Smile

vdsalchemist · Posted: Tue Apr 25, 2006 4:50 pm Post subject:

Marty,
I agree with Serge's suggestions since I understand how the list commands and functions work. If the list has several values that have the same prefixes it would cause the list commands and functions longer to find the item of interest since they work off of binary trees. The prefixes cause the tree to be 1 sided since they use a string compare function that compare the string <, >, or = to the new string being added. If the string is < the previous string added the tree will grow to the right if the string is > the previous string the tree will grow to the left. If the string is equal most will return an error that is handled in different ways depending on the binary tree package that is being used and the programmer that is implementing the package.

With this said I am implementing a Dictionary command and function in GadgetX that will allow you to do really fast name = value look ups. I will send you a demo ASAP.