Advanced Threading – Thread Docta Framework
Tab 3 – Commands ad Functions
Append Item to File – define
Parameters – #ITEM TO SAVE, #THREAD INDEX
This custom command adds the passed item through the #ITEM TO SAVE variable to a dynamic list or appends to dynamic file name using variable to denote what thread the data came from. Thread 1 will always save its thread data to thread_1.txt and same with the list. This saving style is not a must for this framework, it is just a couple of ideas to show you saving dynamically.
“Add item to large list” is a plugin command from the “large Data” plugin and is very useful for handling a lot of data fast.
“Append to file” you can append to the top or bottom/end of the file but be sure to add the $new line or it wont look the way you want it. Unless you want your data to be a continuous string. Like item item2item3 instead of like:
Main Thread – define
This spawns your threads on each loop passing #INDEX_thread_ID to give it an ID so everything in the thread will correspond to that ID or thread name like a list or file.
Everything is in a “thread” command which you will use every time you wish to thread something and everything is in an “in new browser” command only to be used when using Ubot’s native browser. So using Socket, HTTP Plugin or Ex-Browser Plugin’s you just want the “thread” command.
Inside the “in new browser” command
“wait” command – this helps when spawning threads to keep them out of sync, most helpful with more threads.
“set” command – sets the index variable #C_Thread_Index to zero. Each thread gets its own list and this is the index to loop that list.
“large List from text” command – is from the Large Data Plugin and this is creating a dynamic list with the name “user_data_#INDEX_thread_ID” which looks like user_data_1 or whatever the thread is. In the “text” field we are getting a list item from another list called “C_split_list” that was created while splitting the starting list. This list could be from a file or something downloaded. I will get to that in a minute. “C_split_list” is essentially a list of list and we are using one of those list here.
“loop” – now we will loop the list and pass the “#user_data_NLI” next list item and the thread indx into the custom command “CUSTOM SCRIPT” that was discussed in the last post.
“Current Line Total” – I will discuss more in a bit. It just reads the number of lines in a file for the stat monitor.
“increment” – adds 1 to the “#C_Thread_Index” variable. This is local to the thread. What happens in the thread stays in the thread!
“Add item to large list” – has two uses one is to make sure all the threads are done and if you wish to log the data you can do that too. I will explain more in the Main Bot command.
“Debug STOP” – is a debug stop script I will explain more soon.
Main Bot define – runs everything and can be used in the scheduler.
Parameter – #DATA FILE PATH
You will notice that the variables here are global as they are not in a thread.
The #DATA FILE PATH just takes the file path to the data you wish to loop.
“Create Folders” – creates folders/directories if they don’t exist.
“Clear all Large List” – clears out all the large list created for a fresh start.
Inside the “if” is where the “DL fake data for httpbin” custom command is. In this case we are only downloading new data if it does not exist. This is for demo reasons only. Take it out of the if command to download fresh data for each cycle or download large quantities in another bot.
“set” #file_in line total – reads the number of lines in a file with a function “file Number of lines” from the Large Data Plugin. It is getting the file path from the #DATA FILE PATH variable.
“if” – this is dividing the initial data list by the number of threads the user wants and if there is a remainder to add one more. Adding one more gets the remaining data into a list to not truncate it.
“Big File Splitter” – is a custom command to split up a list into smaller list. In this case it is for each thread required by the user and is explained more later. It is creating dynamic list for our threads to loop.
“set” #main_index – an index to loop the “C_split_list” that is incremented at the end of the loop.
You can look at what this list looks like by using $large list return and it in this case should look like this.
Item 1 (thread 1)
mr john smith,(456)212-0456,firstname.lastname@example.org
mr jane smith,(456)212-0456,email@example.com
Item 2 (thread 2)
mr jack foo,(654)212-0456,firstname.lastname@example.org
mr jane doe,(777)212-0456,email@example.com
and so on for each thread.
Now we loop the list which will be the same as the number of threads. Inside the loop we hit the “Main Thread” custom command(discussed above) passing the index #main_index then incremented. Now the
“Main Thread” knows which list item to use and take its share of the load(list). So each thread is working equally the same except for that remainder list.
What makes this so cool is traditionally we spawn a thread for every list item which is resource intensive or I like to say expensive. Thread Docta is more efficient by spawning less threads additionally we don’t have to worry about incrementing and decrementing issues with race conditions from threads fighting over the same things. Another benefit we don’t need to worry about locks either. Too many locks will just make your threading useless anyway. It just becomes linear in nature anyway.
“loop while” – blocks until all threads are complete. When each thread is done it adds to the
“completed_threads” list. When that list total is equal to the main index(threads created) the script will move on.
“Combine Results List” – combines or joins all the list into a results list for saving.
“if” – if we are in a continuous loop we save our data to unique files with random text appended to the mane else: save as usual file name.
There are so many ways to append or save data, please use the saving data as a lose guideline for your scripts.
Big List Append Results To File – define
Parameters #FILE TO APPEND TO, #BIG LIST NAME
I am using it to append results here. But you can see it can be used for any large list #BIG LIST NAME. It loops the named list and appends to named file.
Combine Results List define – loops all the thread lists and combines them into one big list.
Delete Session Data define – deletes the “Data” directory and all its contents if the #UI_CB_session delete variable is not true. It also checks to see if the directory exists first. Then using the “Advanced Shell” Plugin command “shell batch hidden” to remove the directory silently(with out pop up warning) it does remove it permanently so be careful with that baby.
Current Line Total define – loops all the thread files to get a total line count for completed tasks. It is threaded however probably not needed to be. It is a good example how to thread a sub routine to speed up your software.
Big File Splitter define
Parameters #FILE PATH IN, #LINES PER
This baby splits up a large file into smaller lists. Give it a file path #FILE PATH IN and how many lines per list #LINES PER you want. It then adds the list as an item to C_split_list. In this case each item is for each thread created.
Create Foldersdefine – creates the required folders/directories that are in the define if they don’t exist.
Debug STOP define – used for debugging your script if you come across an issue.
Debug Wait define – if you need to temporarily slow down your script in various places to see what is happening. You can use something like this or even “pause script”
UI Drop Down – you should only need to use this in your custom script to change the scope on multiple nodes. It is optional but handy if you have a large bot to debug.
Tab 4 – Functions
Random User Agents define
Parameter #USER AGENT LIST
Takes a given list passed through #USER AGENT LIST. This can be any list delimited by a new line. Whether you read it form a file into a variable or a UBot %list with an onload(Dev) or at the start of a script just pass it and the function will return a random User Agent.
Random Referrer define
Parameter #REFERRER LIST
Sane as the User Agent above accept for referrers you may want.
Data file IN file path define
This you will need to change with your project.
If #UI_OF_file IN path is empty, then save to bot folder\Data\ru-DL-httpbin-#UI_TB_Data in quantitiy-Data.txt. So if you wanted 10 users your file name would be ru-DL-httpbin-10-Data.txt. “ru DL” is random user download.
Data file OUT file path define
Same concept as above accept it is not used in this bot.
Wait for Element timeout define
Parameter #TIME TO WAIT
#TIME TO WAIT is used to set a specific time. The default is 15 seconds.
SUM list of integers define
Parameters #LIST, #DELIMITER
Takes a list of integers(numbers) and adds them all up to get the sum.