UBot Boot Camp
Thread Docta Framework
Tab 2 – Custom Script
I am going to skip over some of these commands that were covered in previous “POSTER” tutorial and just concentrate on the mechanics of defines in the “Thread Docta Framework”. If I skip anything it is probably covered in a previous tutorial.
The extreme use of these commands is to show the use of passing parameters and modulating your code. This bot is too small to appreciate the concept and is better utilized in a larger software.
I will first explain as if you don’t have any of your custom code in this tab. So this first part is assuming you downloaded the empty framework that doesn’t have a working bot inside. The primary define “CUSTOM SCRIPT” is a must and all the rest are a template that you can decide for your self whether to use them or not. This is designed to make it simple to just drop you non threaded code in and go! It is also designed to be flexible and take most styles of software ie. posting and scraping/downloading/site testing types of programs.
The “CUSTOM SCRIPT” define takes two parameters #NEXT LIST ITEM and #INDEX THREAD ID. The
“Thread Docta” framework takes care of the looping and the next list item is passed in to be parsed if required. For example if you are passing nick,(111)-222-3333,firstname.lastname@example.org you may need to parse and separate them to fill in a form with their respective fields. Another example would be just a URL http://ubotdocta.com would not need further processing. In this case we’re passing both variables to the “Page Load” custom command.
If you ever make a huge bot you will/can appreciate separating your code into chunks/modules it is easier to find and debug.
“Page Load” – Takes care of loading a clearing cookies, setting the User Agent, Referrer, Proxies(if supported), Navigation and anything else that may fit into this category.
Before “page load” we have the “set headless browser” command. This can be manually set of as you can see you can put a variable in there. #UI_DD_headless is from the UI drop down element in the first tab. When I tried it in “page load” it did not work for me. After “page load” is where you might put the rest of your script. At the bottom you will see a “stop script” command to not run the “Append item to file” custom command. It is placed there out of convenience in case you would like to use it in your script.
Okay, now we will assume you are in the “poster” example provided in the “Advanced Threading” store item.
It is recommended to build your script outside the “framework” first couple of times. However, you can do it almost as easy just use one thread at first and once you have it all working increase the threads. If you use any lists or set command the scope needs to be set to local so they are not contaminated by the other threads. As your bot/software grows I would separate the defines out into their own tabs to decrease tab load times. This example is a simple one so it is not needed here.
Continuing on from the Boot Camp series, I used this poster except for now we’re downloading some data from the web using an API from “randomuser.me”. I made a custom command “DL Fake Data for httpin” to handle that process. It is easily adapted for other uses but I made it specifically for this use case. It downloads user data name, email and phone to a CSV file. It also takes two parameters #QUANTITY TO DOWNLOAD and #DIRECTORY.
The #DIRECTORY is for where to save the file to. See how it is generic enough to be used somewhere else or in another software? This also use the Large Data Plugin commands “Large list from file” to load the file into a list and “Large list remove item” to remove the column headers from the file. Then it saves the contents of the list to the directory path given and in this case we are overwriting the same file we loaded. $Large list return gives you the entire list. Also if you have not guessed the name of list is “c_remove”, the “c_” stands for being inside a define command. If it were in a define function I would’ve used “f_”. I normally do that for any variable, list or table that is inside a define. I have not done it so much in the training series to not confuse you. You will see it consistently in the source code that I sell tho. Finally, this command is placed inside the “Main Bot” command as commented in the tab above its define. The reason why the command is placed there is so that it is outside the loop we only need to download once per cycle. Please not that this demo bot reuses the downloaded data on every cycle intentionally.
Inside the “Page Load” define again we see code more specific to this bot. For the navigation we have an “if” for remote or local, If I lose you here just use “Remote” in the respective drop down UI command and the bot will run fine. Remote just means to use the remote server at httpbin.org and Local means you have to use the httpbin local server on your machine. This involves installing Python and httpbin package on your machine. I will have a post on that soon if I don’t already. Why would you use a local server? You can pound it with browser or HTTP requests without interfering with anyone else. There is more than just a form to play with on httpbin.org. More on that in future tutorials.
Next we have a $Wait for element timeout with a default setting of 15 seconds. You can change that if you like in the field provided.
The next define “httpbin form commands” takes three parameters #CUSTOMER NAME, #PHONE and #EMAIL. This command is called in the “CUSTOM SCRIPT” which ultimately we are passing a parsed name that is passed to some functions to process the text. The full name is pulled from the list by position 0, then camel cased from mr joe smilth to Mr Joe Smith. The phone and name are just parsed by position in the virtual list. So $list from text creates a nameless list and $list item pareses by position which saves you from a “clear list” and “add list to list” commands. Which is possibly not using the memory block that is allocated to create a list. When creating a data structure such as a list it has to have a predetermined chunk of memory whether you have one item or 5,000. Each time you add something to a list it is not efficient to reallocate memory slots. So keep this in mind when you are creating those #variables, %list and &tables!! Yes, variables to work the same way whether it is one character or large chunk. Each one takes a chunk of memory in RAM. What do you think of functions/commands now? They are huge memory savers!! Another case to set your scope to local because that memory is free after it is used. Some people use commands and functions in non threaded bots but don’t change the scope to local wherever possible and use up a lot of memory.
The next two custom commands “Random Toppings” and “Type Delivery Time” are containers for a sub process(kinda) that run a lot of commands. What I mean by kinda is that I think of a sub process as one that is threaded out and non blocking. It is beneficial to speed up your program to stick all the commands in your define into a thread. However you do not want the rest of you code to finish before the threaded code does. A sub precess/routine usually runs in the back ground on another thread and does not block the flow of the program. Otherwise it is the same as any other linear process. So these two commands are blocking and do not move on until they are complete. They are in defines to make the code more readable and if there is an issue you don’t have to scroll through a ton of nodes to get to it. So when someone says “It is not doing the toppings right.” 2 months later, you cruz through your code and see “Random Toppings” you can just got to its define in no time at all.
At the end of “httpbin form commands” define I have a simple check after the submission. It verifies that the email I submitted is the same as the current email in this thread and that the submission took place. Typically you will get a success page after a submission and if it you do then you save that user to file or a list. In this case we are going to append it to a file.
Okay, back to the “CUSTOM SCRIPT” define, I created a custom command “Append item to File”. The define for this is located under the “Commands” tab which I will get to shortly. In the “item to save” field I chose to use the $scrape attribute in this case. All it is doing is saving the success page to file for processing later which is a very good idea to save memory. Additionally, it does not take much time to do this at the end of a script or have another program do it while the data comes in. I mean do you really need to license a program that parses JSON?(in this case). Just have your main program call the “JSON” parser .exe pass it the file or folder and it will use its own memory and wont bog down the main program. In the next field it takes the thread index which was passed from Main Bot, this is used in the file name. Each thread has its own file to save/append to.
Finally we have “close page” it may be optional for your case. This case does not require it. I just added to bring awareness.