Speeding up Magento Dataflow (Import/Export)

Importing and Exporting data in Magento using the data-flow profiles has never been particularly quick, neither does it always work perfectly every time around. This article covers how to clear up unsuccessful imports and tips to speed the process up. The technical skill level required for this task would be basic Magento development knowledge, shell access and MySQL access. If you are unable to meet these requirements, we would strongly advise getting your Magento Development company or Magento Web Host to do this for you.

Set up the Magento Cron Job

First of all we will need to setup the cron job in Magento, so we can make use of it’s automated features. If this is already done on your Magento store you can skip this step. You will need SSH access and a shell client such as Putty, speak to your web host about this if you have any problems or don’t have access.

1. Log-in to your server with SSH and navigation to your web root directory for Magento. (e.g. “cd public_html”)
2. Now type “pwd” and copy the current path it returns (select the text and press enter in Putty).
3. Then you need to type “crontab -e”, to open the cron list for editing in SSH.
4. After this line “# m h dom mon dow command”, on a new line, enter the following (replacing the path_to_cron with the current path you copied earlier from the “pwd” command:

   [pyg language=”php”]
   @hourly /bin/bash path_to_cron/cron.sh
   [/pyg]

5. Now to save the file, press “esc” key, followed by “:wq” to save and quit.
6. To confirm your cron job has been saved, you can use the “crontab -l” command.

Enable Log Cleaning

If you turn on log cleaning in your Magento installation, this will reduce the overall size of the database and the amount of data that is read in logs. Effectively speeding up the whole Magento system, by cleaning out the old data logs. Here’s how you do it:

1. Log-in to your Magento Admin Panel and Navigation to (System > Configuration).
2. Then if you go into Advanced > System > Log Cleaning (from the left hand side-menu).
3. If you change the “Save log, days” value to 14 days, this is more than enough days to store and view logs for.
4. And change “Enable Log Cleaning” to Yes.
5. Then you can enter your email address for “Error Email Recipient”, so you can monitor for any issues/errors.
6. Now save your configuration and if it’s all setup correctly, by default your logs will automatically cleaned at midnight every day.

Checking the status of an Export

You can check the status of an export currently running, by running the following SQL command in your Magento database:

[pyg language=”sql”]
SELECT COUNT(batch_export_id) FROM dataflow_batch_export
[/pyg]

This will show the current count of the records exported, this process shouldn’t take more than 5 minutes. The count will return zero once the export has been completed.

Clearing out unsuccessful imports from the database

If you run the following SQL truncate commands on the import and export tables, this will clear the database and speed up future imports and exports using Magento’s data-flow. If you run these commands during an import/export, this will break the operation, so be advised!

[pyg language=”sql”]
TRUNCATE `dataflow_batch_export`;
TRUNCATE `dataflow_batch_import`;
[/pyg]

Hopefully you’ve got the basics on setting up the Magento Cron, clearing your logs and speeding up your data-flow operations in Magento. If your looking for a Magento Specialist, you can contact us with your enquiry here, and we would be happy to help.