Batch Pipeline Processor

The Batch Pipeline Processor allows using an Excel table to define what image files to open and which pipelines with what parameters to run on them, and where to export results.. Then row-by-row the same instructions are repeated, with arbitrarily varied data, pipelines or parameters.

How to process multiple recordings automatically?

The Batch Pipeline Processor (Tools/Batch Pipeline Processor) can be programmed using a simple Excel table to open different (multi-file) recordings and process them with one or more pipelines.

In the Batch Pipeline Processor dialog press 'Open blank batch worksheet in Excel'.
The second row of the worksheet defines a series of commands, that will be performed for each following row. Use the pull-down lists in cells of the second row to select from the available commands. E.g. In B2 set 'Open Images'
Each row from the third and below defines one cycle of batch operation. Within each cycle, commands are executed from left to the right as defined in row 2. Select B3 and with using the 'Add image folder and file names' in the dialog fill in the image file names, similarly to opening them.
- Alternatively, filenames from a Multi-Dimensional dialog can be also transferred to the cell selected in Excel using the 'Add list of files or condition names from Multi-Dimensional Open dialog'.
- Hint: selecting 'Set image folder' or 'Set image subfolder' in row 2, in a column left of the 'Open Images' command (e.g. A2 in this case) will separate the folder name from the filenames. Use the button to fill in folder names if needed. Folder names do not have to be redefined in the following rows, those cells can be left empty.
In row 2 in a column right of 'Open Images' select 'Pipeline' as command (e.g. in C2).
Select the cell after the image names, in the column of the 'Pipeline' as command (e.g. C3).
Select a Pipeline in the Pipelines main menu and press the 'Add Pipeline' button of the dialog. If no run command is defined in row 2, at the end of the row this pipeline will be automatically performed for each position within the opened recording. Alternatively specify a 'Positions to run' or 'Part plate run' command at the end of row 2 to constrain processing to specific wells.
Add multiple lines as above described to process different recordings.
Save the batch using the 'Save batch in Excel Data Window and transfer to here'.
Press 'Run all' or 'Run selected row'. Image Analyst MKII will first verify if all entries and filenames are correct in the batch and then perform all operations.

How to process recordings with multiple pipelines or parameter settings?

The Batch Pipeline Processor (Tools/Batch Pipeline Processor) can be programmed using a simple Excel table to open different recordings and process them with one or more pipelines, with optional adjustments to pipeline parameters.

Set up a basic batch as explained in the first section above.
To use a different pipeline in each batch cycle, simply enter a different pipeline name into each row in the column where the 'Pipeline' command was set in row 2.
- To enter pipelines, select a Pipeline in the Pipelines main menu and press the 'Add Pipeline' button of the dialog. Alternatively pipeline files with full path can be also given.
To change a pipeline parameter in a batch cycle:
- Define a 'Set pipeline parameter' in row 2, right from the 'Pipeline' command.
- Select and edit the desired pipeline parameter in the Parameter Bar of the main window.
- Go to the row of the batch cycle in the same column, and in the Batch Pipeline Processor dialog press the ' Add pipeline parameter' button. The generic format is the ordinal number (starting at 1) of the parameter = new value.
- If no run command is defined in row 2, at the end of each row this pipeline will be automatically performed for each position within the opened recording after the parameter has been set.
- Hint: use multiple 'Set pipeline parameter' columns to change multiple parameters. Cells may be left empty in particular cycles to use values set in rows above.
To run multiple pipelines in the same recording you may use the 'Run pipeline' command in row 2. In contrast to the 'Pipeline' command that loads a pipeline and executes at the end of the row, 'Run pipeline' will activate and immediately execute the given pipeline. This allows running other pipelines in columns further to the right, or opening other recordings followed by running other pipelines.
- Hint: To load a Pipeline, change a parameter and execute it before doing something else, use the 'Pipeline', 'Set pipeline parameter' and 'Run pipeline' commands, where the cell corresponding to the 'Run pipeline' is left empty.

How to process multiple recordings into one or multiple output files?

The Batch Pipeline Processor (Tools/Batch Pipeline Processor) can be programmed by a simple Excel table to open different (multi-file) recordings and process them with one or more pipelines, end to export data to specified Excel or Prism files.

Set up a basic batch as explained in the first section.
Prism output:
- Insert a new first column, and in A2 set 'Open Prism file' command.
- Enter a file name with full path, or use the 'Add Prism file' button of the Batch Pipeline Processor dialog.
- The prism file does not have to be closed at the end of the operation, but can be closed from the File main menu.
- Hint: to record the whole batch operation in to a single Prism file, it is sufficient to define the filename in the first batch cycle, and cells below in that column can be left empty.
Excel output:
- Use a 'Save Excel data as' as a last command in row 2.
- In each batch cycle, enter a file name with full path, or use the 'Add Excel file' button of the Batch Pipeline Processor dialog.
- During batch operation, in each cycle a new Excel Data Window will be opened, and at the end of the cycle saved in the provided name.
- Hint: you may use Excel formulas in the batch to automatically generate save filenames.

Note: Prism files are defined before data processing and then automatically saved when data is added. In contrast, Excel workbooks are opened automatically as blank, and have to be explicitly saved at the end of the operation. Both Prism and Excel data can be manually saved, if the whole batch works into the same file.

How to use the Batch Pipeline Processor to keep track of condition names?

Each entry of the position name list in the Multi-Dimensional Open dialog

can be renamed. Typically, this is used to associate condition names with positions. One condition name list can be saved in the Multi-Dimensional Open dialog. This list is automatically loaded when the same image file(s) are opened. This condition list can be transferred to a batch allowing using multiple condition lists with the same recording.

To do this:

Define a column with 'Rename conditions' in the batch.
Use the 'Add list of files or condition names from Multi-Dimensional Open dialog' button to transfer the names.

How to use the Batch Pipeline Processor to organize data?: You can use Excel-based batch files built by the above functions of the Batch Pipeline Processor to keep track of the location and organization of image files, output files, and which pipelines or settings you used to generate the output. Simply using the Batch Pipeline Processor for routine image processing, adding new experiments as new rows to existing batches will document how the analysis was performed. This not only speeds up retrieving and reanalyzing old data but also makes freshly analyzing new data with an established paradigm fast and straightforward.