Jump to content


Photo

Exporting Reports to CSV

Reports CSV Export

  • Please log in to reply
6 replies to this topic

#1 khetrick

khetrick

    Member

  • Members
  • PipPip
  • 10 posts

Posted 28 March 2014 - 03:52 PM

When we try to export a report over about 1000 records we get a message saying that we will get an email notification with a download link. We never get the email.

 

1. The chron job for Export has been set up.

 

2. When I try to manually run the job it gets stuck and I have to reset it.

 

What can I do to fix this problem?



#2 riwern

riwern

    Member

  • Members
  • PipPip
  • 16 posts

Posted 01 April 2014 - 08:22 AM

I've run into more or less the same problem.
I still haven't found a solution for my problem although I've narrowed it down to what's causing the situation for me: execution time.
 
Check your (apache) logs for errors similar to "PHP Fatal error:  Maximum execution time of 300 seconds exceeded in...". It's due to that the jobManager has a 300 second execution limit regardless of your php settings on the server.
 
If you supply an argument to the jobManager you can manually try a larger value like:

php ./zurmoc jobManager <super user> Export 3600

Which in this case changes the execution time limit to 3600 seconds instead of the default 300.
 
If it works for you, you may also try adding the larger value to your cron job as well. For some reason the jobs keep piling up after a while for me but you may have better luck finding values that work. This might be a separate issue so I'll be creating my own thread for that.
 
--igu



#3 jackbauer86

jackbauer86

    Member

  • Members
  • PipPip
  • 25 posts

Posted 01 April 2014 - 04:42 PM

We're running 2.6.5 and I was finally able to get the export functionality working. However, we never received the email message indicating the export was ready to download. As far as we can tell the email was never generated by Zurmo (theoretically ruling out delivery issues).

A message appeared in my Inbox after some time and I was able to download the file.

Our instance has access to 8GB of RAM at this point so raising the PHP memory limit and execution time limit is a good start, especially if Zurmo doesn't generate a message in your Inbox after a reasonable time period has elapsed after your export request.

#4 khetrick

khetrick

    Member

  • Members
  • PipPip
  • 10 posts

Posted 01 April 2014 - 05:49 PM

I've run into more or less the same problem.
I still haven't found a solution for my problem although I've narrowed it down to what's causing the situation for me: execution time.
 
Check your (apache) logs for errors similar to "PHP Fatal error:  Maximum execution time of 300 seconds exceeded in...". It's due to that the jobManager has a 300 second execution limit regardless of your php settings on the server.
 
If you supply an argument to the jobManager you can manually try a larger value like:

php ./zurmoc jobManager <super user> Export 3600

Which in this case changes the execution time limit to 3600 seconds instead of the default 300.
 
If it works for you, you may also try adding the larger value to your cron job as well. For some reason the jobs keep piling up after a while for me but you may have better luck finding values that work. This might be a separate issue so I'll be creating my own thread for that.
 
--igu

 

This worked. Thanks!



#5 khetrick

khetrick

    Member

  • Members
  • PipPip
  • 10 posts

Posted 04 April 2014 - 03:02 PM

This worked. Thanks!

 

This worked to a point. We are able to export anything less than about 300 records or so now, but I am getting this error in phperror_log for anything over that amount.

 

Class 'FailedToSaveFileModelException' not found in 



#6 khetrick

khetrick

    Member

  • Members
  • PipPip
  • 10 posts

Posted 08 April 2014 - 09:36 AM

Ok. So I have moved to a VPS and I am still seeing unusually long delays and performance issues when exporting a report over 1000 records. I don't get it. In the demo I can export 3k records in a matter of seconds, but in 3 different environments one 3 different servers I can't get it to work like that. my php limit is set at 1GB now. I am at a complete loss.

 

If we need a server with 8GB of RAM to export 1300 records, then that sounds like a complete failure in programming. We have been using Vtiger for the better part of 5 years and I never had any issue exporting records. In fact I exported almost 80k at one time and it took a few minutes just yesterday from it.



#7 riwern

riwern

    Member

  • Members
  • PipPip
  • 16 posts

Posted 13 May 2014 - 01:53 PM

You should check this out: https://bitbucket.or...-large-datasets
Worked wonders on my exports.
 
Also for a shared host edit in /app/protected/modules/export/ExportModule.php these 3 variables:

 public static $asynchronousThreshold = 1000;

 public static $asynchronousPageSize  = 500;

 public static $asynchronousMaximumModelsToProcess = 2500;

 

First one is "how many records before background export".

The last one is maybe more important as it defines how many records to process in each batch. It looks like you never finish a batch, so lower this value to 250 or something as a start and work your way up.

 

On a shared environment I'd lower all of these values + the background process is single threaded so you won't gain anything by adding vCPU's either.

 

Good luck!







Also tagged with one or more of these keywords: Reports, CSV, Export

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users