https://sergeyzhuk.me/assets/images/posts/fast-webscraping-reactphp/logo.jpg

Hi ,
my name Joe (a.k.a  @rxhector on twitter/steemit )

and this is the result of 15 years data-scraping using php w/multi-curl.

I have just recently solved one of the biggest problems I've had in scraping using php w/multi-curl,
the problem where you have to post to first page and use that data to get to yet another page of results (without having the never-ending unweildy cascading if/then/else url check bullshit)


it took 15 years because I started as a carpenter by trade (15 years) and just 
kind of accidentally discovered php/mysql,
but we will save that story for another time.

I have a working example to check out so you wont be left flying blind like i was while i learned this shit.

https://github.com/rxhector/ultimate-multicurl


And I have tried to comment the code (still needs a ton of better / prettier format)



The first caveat from the code is this little xml beauty
load any web page into xml without it breaking the shit out of php simplexml_import_dom


~~~
if (!function_exists('load_simplexml_page')) {
    /*

        this will 'force' xml to load a web page (pure html)
        sometimes simplexml_import_dom breaks when trying to import html with bad mark up (i.e - old crappy coding / scripts)
        DOMDocument will auto-magically fix shitty html

        then we can simplexml-ize it !!!

        NOTE : 
            when using php file_get_contents or dom document or simplexml
            those functions use the php.ini user_agent

            most web sites will not return a web request to empty user_agent
            or user_agent "PHP"
            i ALWAYS set php.ini user_agent to a valid browser string

			php.ini
            user_agent="Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0"

    */
    //load from url(default) or string
    function load_simplexml_page($page , $type='url'){
        /*
            static DOMDocument::loadHTMLFile() errors on malformed documents
            $dom = new DOMDocument('1.0', 'UTF-8'); //force doctype/utf - get rid of warnings
            prepend @$dom = tell php to ignore any warnings/errors
        */
        $dom = new DOMDocument('1.0', 'UTF-8');
        $type === 'url'
        ? @$dom->loadHTMLFile(trim($page))
        : @$dom->loadHTML($page);
        return simplexml_import_dom($dom);
    }//end function
}//end function check
~~~


here's a quick and dirty script that lets you test your php.ini user_agent from curl request

~~~

<?php	
	// save this on your  web server so you can hit it as web page ex. /www/test_user_agent.php
	// then run from cmd line ex. php /www/test_user_agent.php

	//require( somefile with load_simplexml_page )
	
    if(isset($_SERVER['REQUEST_METHOD'])){
        //this is a webpage hit 
        $return = print_r($_SERVER , true) ;
        echo $return;   //return something to browser/curl request
    }
    else
    {
        //no server request - this is running as cmd line script
        
        //returns default php user_agent "PHP" (if set)
        //$text = file_get_contents("http://localhost/test_user_agent.php");
        //echo $text;

        //returns default php user_agent "PHP" (if set)
        $xml = load_simplexml_page("http://localhost/test_user_agent.php");
        echo $xml->asXML();

    }  

~~~


i tried regex and substr and other normal php string processing functions
but i learned early on that this can become unweildy and complex
thats when i discovered simplexml - what an awesome tool for html !!!
it is super easy to use xpath expressions to get to any element on a web page to get data !!!


my first few scraping jobs i was a NOOB and was using good old file_get_contents and getting one slow page at a time,
but that has it limitations...
when you get a little more advanced you have to learn to make a query string and post data..(pagination is a bitch)

that's when i discovered curl - wow it took me about a week to learn how to set cookies so i could log in to sites and get backend data.

so my load_simplexml_page and curl tools came in handy - but man was it slow doing 'synchronous' page loading - one slow page at a time.
when you get a client that wants 10,000 pages at a time (instead of 100) - you better figure out how to do it AS FAST AS POSSIBLE !!!

then i stumbled into multi-curl - HOLY SHIT , talk about a learning curve..
i know there are a few 'wizards' out there that probably picked it up right away
but you gotta remember - I was a 15 year carpenter/guitar player/stoner - it took me a bit
to grab the 'asynchronous' concept - let's load 1000 pages all at once - and process them.



so now we can move on to the main multi-curl object itself
i wont post the code in its entirety here - you can find that posted at PUT GITHUB PROJECT LINK HERE

some of my favorite code tricks i've discovered along the way

~~~

	//notice the &(reference) here
	if ($x = &$this->result_callback($this->rs[$i]['options'])) {
	
	//$x is now a string and can be called as a function
		
		$x($this->rs[$i] , $i);	//old school would be something like mixed call_user_func_array ( callable $callback , array $param_arr )
		//call_user_func_array($x , [$this->rs[$i] , $i]);
		
~~~



another pretty cool use for references



~~~

	public function start_callback(&$options , $set = false){ 
	//$options can now be changed from with the callback function (trust me - it comes in handy for passing variables in multi-curl)
	
		$options['result_callback'] = $set;		//now $options is changed in the main(calling) flow 
		
~~~


the next really cool caveat you get is a proxy scraper - you know - get a list of 300 proxies
then do a quick check on the target domain to make sure you have a good proxy ;)



so - the whole point for me was to get a better understanding of how multi-curl works.
if you want to know how something works - you gotta break it and try to rebuild it.



looking forward
this code really needs some more clean up and better comments/formatting.

i would really like to add some of the functionality from zebra-curl (i didnt need all the bells right away - so i just built this for quick get/post json requests)

I hope you guys like it


https://github.com/rxhector/ultimate-multicurl


this is running in production for mid-market cap company - we are scraping about 30k records @ 1000/hr (25pages/second) - not bad !!!

tipping is allowed
the old slow way - paypal rxhector2k5@yahoo.com
the super fast ~3 second way twitter @xrptipbot https://twitter.com/@rxhector
	twitter trx bot @goseedit