Important

You are browsing the documentation for version 4.1 of OroCommerce, OroCRM and OroPlatform, which is no longer maintained. Read version 5.1 (the latest LTS version) of the Oro documentation to get up-to-date information.

See our Release Process documentation for more information on the currently supported and upcoming releases.

Accelerate Import

This article contains several recommendation about import process acceleration, explained below.

Make Sure Xdebug is Disabled

Xdebug is a very useful debug tool for PHP, but at the same time it adds lots of overhead, especially for heavy and long operations. Xdebug status can be checked with php -m command:

1# xdebug is enabled
2$ php -m | grep xdebug
3xdebug
4
5# xdebug is disabled (no result)
6$ php -m | grep xdebug

To disable it, a developer has to remove or comment inclusion of Xdebug library (usually this should be done in php.ini).

Run Import Operation from the Command Line

Import from the UI is good for relatively small amount of data (up to 1000 entities), but if you need to import thousands or millions of entities the command line is your best choice. OroPlatform provides the CLI command oro:import:csv that allows to import records from the specified CSV file.

 1$ php bin/console oro:import:csv --help
 2Usage:
 3 oro:import:csv [--validation-processor="..."] [--processor="..."] file
 4
 5Arguments:
 6 file                    File name, to import CSV data from
 7
 8Options:
 9 --validation-processor  Name of the processor for the entity data validation contained in the CSV
10 --processor             Name of the processor for the entity data contained in the CSV

Here is a small example of its usage:

 1$ php bin/console oro:import:csv ~/Contact_2000.csv
 2Choose Processor:
 3  [0 ] orocrm_contact.add_or_replace
 4  [1 ] orocrm_contact.add
 5  [2 ] orocrm_account.add_or_replace
 6  [3 ] oro_tracking.processor.data
 7  [4 ] orocrm_sales_lead.add_or_replace
 8  [5 ] orocrm_sales_opportunity.add_or_replace
 9  [6 ] orocrm_sales_b2bcustomer
10  [7 ] orocrm_magento.add_or_update_newsletter_subscriber
11  [8 ] orocrm_magento.add_or_update_customer
12  [9 ] orocrm_magento.import_guest_customer
13  [10] orocrm_magento.add_or_update_customer_address
14  [11] orocrm_magento.add_or_update_region
15  [12] orocrm_magento.add_or_update_cart
16  [13] orocrm_magento.add_or_update_order
17  [14] orocrm_magento.store
18  [15] orocrm_magento.website
19  [16] orocrm_magento.customer_group
20> 1
21Choose Validation Processor:
22  [0] orocrm_contact.add_or_replace
23  [1] orocrm_contact.add
24  [2] orocrm_account.add_or_replace
25  [3] orocrm_sales_lead.add_or_replace
26  [4] orocrm_sales_opportunity.add_or_replace
27  [5] orocrm_sales_b2bcustomer
28> 1
29+---------------+-------+
30| Results       | Count |
31+---------------+-------+
32| errors        | 0     |
33| process       | 2000  |
34| read          | 2000  |
35| add           | 2000  |
36| replace       | 0     |
37| update        | 0     |
38| delete        | 0     |
39| error_entries | 0     |
40+---------------+-------+
41Do you want to proceed [yes]?
42File was successfully imported.

Perform Import in the Prod Environment

The default environment for CLI is dev. In dev environment the application stores lots of data generally not required for real-life usage. Therefore, it is recommended to run import in prod environment so it would finish much faster. To do so you should add the --env=prod option to your import command:

1$ php bin/console oro:import:csv ~/Contact_2000.csv --env=prod

Skip Import File Validation

During regular import operation, the validation process is performed twice: first, during the validation itself and then before saving imported entities (invalid entities will not be saved to the DB). Initial validation can be skipped and import can be performed without it. To do so, start the import command in no interaction mode with the --no-interaction option:

 1$ php bin/console oro:import:csv ~/Contact_2000.csv --processor=orocrm_contact.add --no-interaction --env=prod
 2+---------------+-------+
 3| Results       | Count |
 4+---------------+-------+
 5| errors        | 0     |
 6| process       | 2000  |
 7| read          | 2000  |
 8| add           | 2000  |
 9| replace       | 0     |
10| update        | 0     |
11| delete        | 0     |
12| error_entries | 0     |
13+---------------+-------+
14File was successfully imported.

Hint

This trick can be very useful if you need to perform import on regular basis (e.g. by cron using external source).

Disable Optional Listeners

With OroPlatform you can disable some event listeners for the command execution. The oro:platform:optional-listeners command shows the list of all such listeners:

1$ bin/console oro:platform:optional-listeners
2List of optional doctrine listeners:
3  > oro_dataaudit.listener.send_changed_entities_to_message_queue
4  > oro_notification.docrine.event.listener
5  > oro_search.index_listener
6  > oro_workflow.listener.event_trigger_collector

To disable these listeners the --disabled-listeners option can be used. Also this option can receive value “all” - it will disable all optional listeners. Here is an example:

1$ bin/console oro:import:csv ~/Contact_2000.csv --processor=orocrm_contact.add --disabled-listeners=all --no-interaction --env=prod

Caution

Remember that disabling listeners actually disables a part of backend functionality, so before using it make sure this part is not required. E.g. if the oro_search.index_listener listener is disabled then imported entities will not be found by the search engine (however, this may be fixed by manual search reindex using the oro:search:reindex command).

Write Custom Import Strategy

OroPlatform provides Oro\Bundle\ImportExportBundle\Strategy\Import\ConfigurableAddOrReplaceStrategy to be used as the default one. This strategy automatically handles field types, relations etc. However, all this functionality significantly slows down the import process and might perform operations and requests that are not required for some specific cases.

To solve this issue, a developer can implement a custom strategy to perform required actions only. The following example shows services that should be created to add a new import strategy:

 1# Custom strategy
 2orocrm_contact.importexport.strategy.contact.add:
 3    class: Oro\Bundle\ContactBundle\ImportExport\Strategy\ContactAddOrUpadteOrDeleteStrategy
 4    parent: oro_importexport.strategy.add
 5
 6# Processor for custom strategy
 7orocrm_contact.importexport.processor.import.add:
 8    parent: oro_importexport.processor.import_abstract
 9    calls:
10        - [setStrategy, ['@orocrm_contact.importexport.strategy.contact.add']]
11    tags:
12        - { name: oro_importexport.processor, type: import, entity: 'Oro\Bundle\ContactBundle\Entity\Contact', alias: orocrm_contact.add }
13        - { name: oro_importexport.processor, type: import_validation, entity: 'Oro\Bundle\ContactBundle\Entity\Contact', alias: orocrm_contact.add }