Import-Export is built on top of OroBatchBundle Jobs. Every job consists of steps, and each step aggregates three critical components:
Processor itself can aggregate three data processing components: Data Converter, Serializer, Strategy
Each component is independent, with its own area of responsibility.
First, a step uses the reader to read the data from the source and passes it to the processor.
Then data is routed to the Processor. The processor can consist of three data manipulation layers: Data Converter, Serializer, and Strategy. In the processor, the data received from the reader is passed to the data converter, where it is transformed into a format that can be used by the serializer. The serializer converts the data into a format suitable for the writer. In addition, data processed by the serializer can be prepared for the writer by the strategy, for example, the strategy can be used to resolve entity relationships or to handle data conflicts.
In the final third stage, the Step passes the processed and write-ready data to the Writer.
Import Export Components¶
The reader reads the data from a source. In terms of import, it can be a CSV file with imported data. In terms of export, the source is a Doctrine entity, its repository, or another query builder.
The processor is at the forefront of the job execution. The main logic of the specific job is concentrated here. The import processor converts array data to the entity object. The export processor does the opposite, it converts the entity object into an array representation.
The writer is responsible for saving the results at a specific destination. In terms of import, it is a storage encapsulated with Doctrine. In terms of export, it is a plain CSV file.
The data converter converts the data from data structure returned by the reader to a format applicable for the serializer.
The serializer namespace contains a dummy encoder (encoding/decoding is not needed for csv import), normalizers (collection, datetime, and entity), and required interfaces. It also contains the Serializer class extended from
Symfony\Component\Serializer\Serializer to use both the extended
The strategy namespace contains a strategy helper with generic import entities and ConfigurableAddOrReplaceStrategy that manages the entity import. StrategyInterface defines an interface for custom strategies.
When implementing import/export, it is very important that the data follows the expected format to give the user an example of how the data should look. For this purpose, the TemplateFixture has been introduced. This fixture is used to represent an exportable record that is used to create a downloadable data template. The TemplateFixture namespace contains a fixture functionality template. TemplateFixtureInterface is an interface used to create fixtures. TemplateManager is a storage for the template fixtures import.
Batch Bundle Job Components¶
OroImportExportBundle uses OroBatchBundle to organize the execution of the import/export operations. OroBatchBundle implements a job which is configured with execution context and is run by a client. The job is abstract by itself, it does not know specific details of what happens during its execution.
Stores step elements (reader, processor and writer), responsible for Step Executor initialization and execution.
Step executor is responsible for data flow. In the step executor, the data returned from the reader is passed to the processor. Then the processed data can be accumulated and finally passed to the writer.
This configuration is used by OroBatchBundle and encapsulates three jobs for importing the entity from a CSV file, validating the imported data and exporting the entity to a CSV file.
connector: name: oro_importexport jobs: entity_export_to_csv: title: "Entity Export to CSV" type: export steps: export: title: export reader: oro_importexport.reader.entity processor: oro_importexport.processor.export_delegate writer: oro_importexport.writer.csv entity_import_validation_from_csv: title: "Entity Import Validation from CSV" type: import_validation steps: import_validation: title: import_validation reader: oro_importexport.reader.csv processor: oro_importexport.processor.import_validation_delegate writer: oro_importexport.writer.doctrine_clear entity_import_from_csv: title: "Entity Import from CSV" type: import steps: import: title: import reader: oro_importexport.reader.csv processor: oro_importexport.processor.import_delegate writer: oro_importexport.writer.entity
Out-of-the-box Import/Export is bundled with Readers that support CSV and XLSx file formats, and can also read data from Doctrine entities. There are writer implementations that support CSV and XLSx file formats, Doctrine entities, and direct writing to DB with InsertFromSelectWriter.
As was mentioned previously, OroBatchBundle is a major dependency of this bundle. OroBatchBundle is used to execute the import/export batch operations. But when a client bundle uses OroImportExportBundle, it does not depend directly on any classes, interfaces, or configuration files of OroBatchBundle. OroImportExportBundle provides its own interfaces and domain models for the client bundle to interact with. From the client bundle’s perspective, it is not necessary to create any job configurations to support the import/export of an entity.