Thread: PHP bug 43038 reported as Bogus

Well after I sent the package which reproduced the error that I was getting, Dmitry sent me an email about my bug and his findings:

Hi Mike,

I've looked into your example and I don't see any PHP bug here.
Your application tries to construct huge result set and as result it outs of memory.
May be some old versions required less memory overhead and was able to pass the example with the same memory but it is not a bug.

I suggest re-implement the application a little bit to improve memory usage.

In function get_records() you collect all records into one array and then insert each of them into database.

But you don't need to collect all of them.

You are able to retrieve a portion of data then insert each record from this portion and then select the next portion...

Using this approach you'll need memory only for one portion of data.

While I do not agree that we should have to change our applications based on new overhead by a scripting language, I do agree that I needed to find a way to fix this by not processing so much data into an array. So I started today by changing my process of capturing everything into one LARGE php array and then processing the data/loading the data into my local database, I split it up in chunks of 500 to 1000 records a time and then move onto the next set of data. Doing it this way REALLY speeds up the script.

If I process 1000 datasets at a time, the runtime of my Product2 data replication script decreases from 45 minutes to 6 minutes, on average
So I consider this issues closed!

Once I fine tune the scripts and make sure that I am not running into any other problems, I will show you some examples that can vastly improve on your data processing against the API of Salesforce.com