Important

You are browsing the documentation for version 3.1 of OroCommerce, OroCRM and OroPlatform, which is no longer maintained. Read version 5.1 (the latest LTS version) of the Oro documentation to get up-to-date information.

See our Release Process documentation for more information on the currently supported and upcoming releases.

Import and Export Entities

You have to create some services and add some configuration to make OroPlatform capable to export your custom entities as CSV files and to be able to load data from CSV files for your entities.

All the configuration described below is added to the importexport.yml file in the Resources/config directory of your application bundle. Make sure that you have a container extension class in your bundle that loads the configuration file:

 1// src/AppBundle/DependencyInjection/AppExtension.php
 2namespace AppBundle\DependencyInjection;
 3
 4use Symfony\Component\Config\FileLocator;
 5use Symfony\Component\DependencyInjection\ContainerBuilder;
 6use Symfony\Component\DependencyInjection\Extension\Extension;
 7use Symfony\Component\DependencyInjection\Loader\YamlFileLoader;
 8
 9class AppExtension extends Extension
10{
11    public function load(array $configs, ContainerBuilder $container)
12    {
13        $loader = new YamlFileLoader($container, new FileLocator(__DIR__.'/Resources/config'));
14        $loader->load('importexport.yml');
15    }
16}

Set Up the Import and Export Processor

Import and export are handled by so-called processors which transform the imported data into actual entities and vice versa. The easiest way to quickly set up import and export processors for your entities is to reuse the Oro\Bundle\ImportExportBundle\Processor\ImportProcessor and Oro\Bundle\ImportExportBundle\Processor\ExportProcessor classes that ship with the OroImportExportBundle. All you need to do is creating services that are based on abstract services from the OroImportExportBundle and let them know which entity class they have to handle:

 1# src/AppBundle/Resources/config/importexport.yml
 2services:
 3    app.importexport.data_converter:
 4        parent: oro_importexport.data_converter.configurable
 5
 6    app.importexport.processor.export:
 7        parent: oro_importexport.processor.export_abstract
 8        calls:
 9            - [setDataConverter, ['@app.importexport.data_converter']]
10        tags:
11            - name: oro_importexport.processor
12              type: export
13              entity: AppBundle\Entity\Task
14              alias: app_task
15    app.importexport.processor.import:
16        parent: oro_importexport.processor.import_abstract
17        calls:
18            - [setDataConverter, ['@app.importexport.data_converter']]
19        tags:
20            - name: oro_importexport.processor
21              type: import
22              entity: AppBundle\Entity\Task
23              alias: app_task

Provide Sample Data

To make it easier for your users to understand the format in which they need to enter the data to be imported, you can provide them with an example file that will be created based on some template fixtures:

 1// src/AppBundle/ImportExport/TemplateFixture;
 2namespace AppBundle\ImportExport\TemplateFixture;
 3
 4use AppBundle\Entity\Task;
 5use Oro\Bundle\ImportExportBundle\TemplateFixture\AbstractTemplateRepository;
 6use Oro\Bundle\ImportExportBundle\TemplateFixture\TemplateFixtureInterface;
 7
 8class TaskFixture extends AbstractTemplateRepository implements TemplateFixtureInterface
 9{
10    public function getEntityClass()
11    {
12        return 'AppBundle\Entity\Task';
13    }
14
15    public function getData()
16    {
17        return $this->getEntityData('example-task');
18    }
19
20    public function fillEntityData($key, $entity)
21    {
22        $entity->setId(1);
23        $entity->setSubject('Call customer');
24        $entity->setDescription('Please call the customer to talk about their future plans.');
25        $entity->setDueDate(new \DateTime('+3 days'));
26    }
27
28    protected function createEntity($key)
29    {
30        return new Task();
31    }
32}

Then, register your fixtures class as a service:

1# src/AppBundle/Resources/config/importexport.yml
2services:
3    # ...
4
5    app.importexport.template_fixture.task:
6        class: AppBundle\ImportExport\TemplateFixture\TaskFixture
7        tags:
8            - { name: oro_importexport.template_fixture }

Add Import and Export Actions to UI

Finally, you need to add control elements to the UI to let your users export existing data and add new entities by uploading a CSV file. You can include the buttons.html.twig template from the OroImportExportBundle while passing it the names of the needed services (see the configuration above) to do so:

 1{# src/AppBundle/Resources/views/Task/index.html.twig #}
 2{% extends 'OroUIBundle:actions:index.html.twig' %}
 3
 4{% set gridName = 'app-tasks-grid' %}
 5{% set pageTitle = 'Task' %}
 6
 7{% block navButtons %}
 8    {% include 'OroImportExportBundle:ImportExport:buttons.html.twig' with {
 9        entity_class: 'AppBundle\\Entity\\Task',
10        exportProcessor: 'app_task',
11        exportTitle: 'Export',
12        importProcessor: 'app_task',
13        importTitle: 'Import',
14        datagridName: gridName
15    } %}
16
17    {# ... #}
18{% endblock %}

Import and Export Entity Data

The OroImportExportBundle is intended to import entities into or export them out of OroPlatform. The bundle uses the OroBatchBundle to organize the execution of import/export operations. Any import/export operation is a job.

A job itself is abstract. It doesn’t know any specific details of what is happening during its execution. A job consists of steps which can be configured to run in an execution context and are executed by the client.

Each step aggregates three crucial components which are not aware of each other:

  • Reader

  • Processor

  • Writer

A step uses the reader to read data from the source. Once the reader has run, the data is passed to the processor. The processor can modify the data before it is forwarded to the writer. Finally, the writer saves data to its final destination.

See also

You can take a look at the code in the OroCRM ContactBundle for a real-world example. It extends base classes from the ImportExportBundle (see classes in the ImportExport namespace) to implement contact specific behavior. The configuration is located in the Resources/config/importexport.yml file.

Import and Export Configuration

Import is a basic operation for any entity. The import operation is one step. See the following example configuration:

 1# Oro/Bundle/ImportExportBundle/Resources/config/batch_jobs.yml
 2connector:
 3    name: oro_importexport
 4    jobs:
 5        entity_import_from_csv:
 6            title: "Entity Import from CSV"
 7            type: import
 8            steps:
 9                import:
10                    title:     import
11                    class:     Oro\Bundle\BatchBundle\Step\ItemStep
12                    services:
13                        reader:    oro_importexport.reader.csv
14                        processor: oro_importexport.processor.import_delegate
15                        writer:    oro_importexport.writer.entity
16                    parameters: ~

The import algorithm being performed is (in pseudocode):

1Process job:
2  - Process step 1:
3    - loop
4      - read item from source
5      - if source is empty exit from loop
6      - process item
7      - save processed item to array of entities
8    - end loop
9    - save array of prepared entities to DB

The OroBatchBundle provides the Oro\Bundle\BatchBundle\Step\ItemStep class that executes each step of a job. In its doExecute() method, it creates a Oro\Bundle\BatchBundle\Step\StepExecutor instance, passes a Oro\Bundle\ImportExportBundle\Reader\ReaderInterface, a Oro\Bundle\ImportExportBundle\Processor\ProcessorInterface and a writer to it and executes it in the StepExecutor through the execute() method. After this step is done, all imported items are written to the destination.

Import Process in Detail

For example, here is what happens in detail when you import contact data from a CSV file:

  1. The Oro\Bundle\ImportExportBundle\Reader\CsvFileReader reads one row from the CSV file in its read() method and transforms it to an array representing the columns of that row.

  2. The data being read is then passed to the process() method of the Oro\Bundle\ImportExportBundle\Processor\ImportProcessor class which converts the item to a complex array using the convertToImportFormat() method of the Oro\Bundle\ImportExportBundle\Converter\ConfigurableTableDataConverter data converter class.

  3. The processor deserializes the item from the converted array using the Oro\Bundle\ImportExportBundle\Serializer\Serializer class.

  4. Optionally, the deserialized object can then be modified by the Oro\Bundle\ImportExportBundle\Strategy\Import\ConfigurableAddOrReplaceStrategy class.

  5. Finally, the processed entity is returned by the processor and then passed to the Oro\Bundle\ImportExportBundle\Writer\EntityWriter class. This writer stores the data when its write() method is executed.

Export Process in Detail

The export process is essentially the import process in reverse, except that it doesn’t use a strategy:

  1. First, the Oro\Bundle\ImportExportBundle\Reader\EntityReader class reads an object.

  2. Then, the Oro\Bundle\ImportExportBundle\Processor\ExportProcessor class serializes and converts the object into an associative array with property names as keys and the property values as values of the array.

  3. The Oro\Bundle\ImportExportBundle\Serializer\Serializer class normalizes each field and converts objects to complex arrays.

  4. A Oro\Bundle\ImportExportBundle\Converter\ConfigurableTableDataConverter converts the associative array into a dimensional array.

  5. Finally, all array entries are written to a CSV file by the Oro\Bundle\ImportExportBundle\Writer\CsvFileWriter class.

The export algorithm being performed is (in pseudocode):

1Process job:
2  - Process step 1:
3    - loop
4      - read entity from DB
5      - if source is empty exit from loop
6      - process entity
7      - save plain array to array of items for save
8    - end loop
9    - save array of prepared items to DB

Serializer and Normalizer

One very important concept to know is how we normalize/denormalize relations between entities and other complex data.

The Serializer class extends the standard serializer of the Symfony Serializer component and has its own normalizers and denormalizers. Each entity that you want to export/import should be supported by the serializer. This means that you should add normalizers and denormalizers that will take care of converting your entity to the array/scalar representation (normalization during serialization) and vice versa, converting arrays to the entity object representation (denormalization during deserialization).

The platform converts entities to complex arrays for which it uses the normalize() method from the ConfigurableEntityNormalizer class. This method uses the field helper to process the fields:

  • If the configuration excludes the field, it will be skipped during normalization.

  • If the field is an object, another entity or a collection, the normalize() method for this type of object will be called.

  • If the field is a scalar value, the field will be added with this value to the array of normalized values.

You can configure your fields in the UI under System / Entities / Entity Management. Alternatively, you can describe the field configuration in your entity directly using Oro\Bundle\EntityConfigBundle\Metadata\Annotation\ConfigField:

1 /**
2  * @ConfigField(
3  *      defaultValues={
4  *          "importexport"={
5  *              "order"=200,
6  *              "full"=true
7  *          }
8  *      }
9  */

You can use the following options:

Option

Description

identity

If true, the field is part of the key used to identify an instance of the entity. It is required to configure the object identity to support imports.

order

The position of the property in the export.

excluded

The skip is field during export if excluded is true.

full

If false, the normalize() method returns only identity fields of associated entities during exports. If true, all fields of the related entity are exported. Fields with Excluded option are skipped.This option cannot be configured in the user interface, but can only be set using annotations.

Import One-to-Many Relations

If you want to import one-to-many relations from a CSV file, you should use the following field name rules for the header columns: “RelationFieldName NumberOfInstance FieldName” where these strings have the following meaning:

  • RelationFieldName (string): entity relation name;

  • NumberOfInstance (integer): for example 1;

  • FieldName (string): The name of the referenced field name.

For example:

1"Addresses 1 First name"

FieldName may be a field label or a column name from a configuration field. You can look it into UI System/Entities/Entity Management. You should import all identity fields for the related entity.

Import Many-to-One Relations

If you want to import many-to-one relations, you should use the following rule: “RelationFieldName IdentityFieldName” where these placeholders have the following meaning:

  • RelationFieldName (string): entity relation name;

  • IdentityFieldName (string): identity field of the related entity. If the related entity has two or more identity fields, you should import all identity fields of the related entity.

For example:

1"Owner Username"

Extension of Import/Export Contacts

Add a New Provider to Support Different Formats

To write your own provider for import operations, you should create a class that extends the Oro\Bundle\ImportExportBundle\Reader\AbstractReader class. To support custom export formats, you just need to create a new class that implements the ItemWriterInterface from the Akeneo BatchBundle. The new classes must be declared as services:

1services:
2    oro_importexport.reader.csv:
3        class: Acme\DemoBundle\ImportExport\Reader\ExcelFileReader
4
5    oro_importexport.writer.csv:
6        class: Oro\Bundle\ImportExportBundle\Writer\CsvFileWriter

Change Import Strategy

OroPlatform provides a basic “add or substitute” import strategy. The basic process is implemented in the ConfigurableAddOrReplaceStrategy class. To create your own import strategy you can extend this class and override the following methods:

  • process()

  • processEntity()

  • updateRelations()

  • findExistingEntity()

See also

You can see an example of an adapted strategy in the ContactAddOrReplaceStrategy from the OroCRM ContactBundle.

Learn more

More information is available in the ImportExportBundle documentation.