Introduction to the Data Import service

The Data Import service provides a method of building, populating and updating Q_Panel child databases with panellist data and outcome codes.

A Q_Panel database needs updating on a regular basis, this may be:

• adding new variables
• updating existing variables
• adding new panellist data
• updating existing panellist data
• activating or deactivating panellists (unsubscribing)
• updating sample outcome codes (backfeed)

Data Import requires separate configurations for the panellist data and backfeed functions.  An overview of each type of configuration is described in the following sections.

Panellist Import

For variables and panellist data, Data Import uses the Triple-S XML format for survey data and variables.

A Triple-S survey is described using two text files – a schema definition file and a data file. The schema file contains the definition of the survey variables which is used to interpret the contents of the data file.  The data file can be either fixed-length data files (.txt or .dat) or CSV data files.

Note that currently the Data Import does not support the use of Triple-S hierarchy defined data.

There are two methods of using Data Import to load panellist data.  The standard method is using Triple-S schema and data files dropped into pre-defined 'drop' folders, the second method, for clients using Q_Survey interviewing software, is to automatically capture responses from live questionnaires, rather than providing data files.  The schema definition is still required, but can be generated by Q_Survey.

It is possible to use both these methods in one installation with the correct configuration.

Standard Triple-S with data files

The Data Import service supports four different panellist import types which can be used to add new variables, update existing variables, add new panellists and update existing panellist data (including activation or deactivation of panellists).

For each type of import, the name of the schema and data files are specified, along with a designated drop folder. The service waits for the schema and/or data files to be dropped into the drop folders and processes them according to the configuration specified.

As the files are processed, they may be moved to other sub-folders.  When they are moved they will be renamed with the addition of a UTC date-time stamp.

• Success - on successful load of a schema or data file, the file will be moved to the Success sub-folder (with the exception of the schema file in a single-type import which will remain, waiting for further drop files). 
• Failure - if a schema or data file fails to load it will be moved to the Failure sub-folder.
• Unmatched - if a schema or data file is found which does not match the filename expected, it will be moved to the Unmatched sub-folder.

If the schema or data file is moved to the Failure sub-folder, the log file will contain an error message which will help to identify the cause of the problem, so that it can be rectified before dropping in another file.  If the error is within the Data Import configuration file you will need to restart the service after correcting the error.

There is no need to create the drop folder structure, the Data Import service will create the folders as required.

The results of a panellist import can be viewed in the Import Manager in Q_Panel.

New variables added by the import will be added to a sub-folder of the folder marked as the Import Drop Folder within Hierarchy Manager on Q_Panel.  All new variables will by default have all the availability options (apart from Anonymise) set to true, therefore they will be immediately available for use in samples.

Using Q_Survey

The Data Import service polls the Q_Survey API every minute for new survey data.  Instead of loading data from a file, data is pulled directly by an API call.  The data is matched to a Triple-S file which must be located in the schema drop folder.

Depending on how frequently Data Import is scheduled to run, any survey responses since the last request processed by Data Import will be retrieved.

Note that when configuring the Data Import service to use the Q_Survey plugin, a Triple-S file should be exported from Q_Survey and placed in the schema drop folder. We recommend that you contact QuenchTec support for assistance with this process, since some small changes may need to be made to this Triple-S file.

The schema file will stay in the schema drop folder unless an error is encountered, in which case it will be moved to the Failure sub-folder.

The results of a Q_Survey import can be viewed in the Import Manager in Q_Panel.

Backfeed

For backfeed, Data Import uses CSV data files or XML formatted data.  Both can be used in the same installation, but need to be configured separately.

For each backfeed import, the name of the backfeed file is specified, along with a designated drop folder. The service waits for the backfeed file(s) to be dropped into the drop folder and processes them according to the configuration specified.

As the files are processed, they are moved to other sub-folders and renamed with the addition of a UTC date-time stamp.

• Success - on successful load of a backfeed file, the file will be moved to the Success sub-folder.
• Failure - if a backfeed file fails to load it will be moved to the Failure sub-folder.
• Unmatched - if a backfeed is found which does not match the filename expected, it will be moved to the Unmatched sub-folder.

If the backfeed file is moved to the Failure sub-folder, the log file will contain an error message which will help to identify the cause of the problem, so that it can be rectified before dropping in another file.  If the error is within the Data Import configuration file you will need to restart the service after correcting the error.

There is no need to create the drop folder structure, the Data Import service will create the folders as required.

Although the Data Import service loads the backfeed records into the database, the outcome codes will not be updated in Q_Panel until the Backfeed Service has processed them.  The Backfeed Service is a separate service which runs periodically, depending on the configuration, and updates the contact history with the outcome codes.