Combining Operations Using Pipelining

In a DSS workflow, a SQL pipeline is a procedure that mixes numerous subsequent recipes (each using the same SQL engine). Then, a single job activity containing these integrated recipes?which may be both visual and "SQL query" recipes?can be executed.

Typically, a SQL query will be converted into an expression in relational algebra, which is a series of relational operations. If we do one operation at a time, we incur too much cost because we need to create temporary files on a disc to store the outcomes of these temporary operations. Large temporary files must be created and stored on a disc, but this takes time and is frequently unneeded because the following process will use these files as input right away. It is usual practice to produce query execution code that corresponds to algorithms for particular combinations of operations in a query in order to decrease the number of temporary files.

Example

One method with two input files and one output file can, for instance, perform a JOIN along with two SELECT operations on the input files and a final PROJECT operation on the resultant file rather than each of these operations being implemented individually. Instead of making four temporary files, we just apply the method and obtain just one result file. Pipelining or stream-based processing is the term used for this.

Conclusion

To perform numerous operations, it is usual practice to generate the query execution code dynamically. The query is produced by the created code, which incorporates numerous algorithms that correspond to different processes. Following an operation, the result tuples are generated and used as input for other operations. For instance, if two select operations on base relations are followed by a join operation, the tuples created by each SELECT operation are fed into a stream or pipeline and used as input for the join process.