Add on_line callback #454
Description
Summary
Add an on_line event which is called for every line. This would allow for custom processing of lines even if they are skipped, comments, or used as a column headings.
Motivation
I have a file which starts with the text "Generated Date: MM/DD/YYYY" on the first line. Using the Stream API, I can skip this line with from_line:1 and everything works great. However, I would like to be able to read this line and parse out the date.
Alternative
I tried using on_record to parse the date I need and then return null for the first row. However, setting columns:true seems to try and use the first row as column names without first passing it through on_record and skipping it.
Draft
Option on_line
The on_line option provides an option to alter and filter lines. It expects a function which receives the line and a context as arguments and which returns the new altered line or nothing if the line is to be filtered.
Type: function
Optional
Default: undefined
Related: on_record, info — see Available Options
This option works at the line level. It complements the on_record option which is adapted to record-level transformations. Also, the stream-transform package provides more advanced control on the record and stream of records with asynchronous execution and concurrent control.
Use cases
Use this option to filter, enrich, and apply any transformations on a line.
Usage
The option takes a function which is called with two arguments: the input line and the context. The return value is the new line or is filtered if null or undefined are returned.
Altering lines
In the [alter line example], Member# is replaced with Member Number in the header row.
parser.on('line', function(line, {lines}){
if(lines === 1) {
return line.replace("Member#", "Member Number")
}
});
parser.write('Member#, First Name, Last Name\n');
parser.write('12345, Bill, Waterson\n');
Filtering lines
In the [filter lines example], the function returns null for the first line, filtering it from the result, but allows us to parse the Generated Date.
parser.on('line', function(line, {lines}){
if(lines === 1) {
return line.replace("Member#", "Member Number")
}
});
parser.write('Date Generated: March 5, 2000\n');
parser.write('Member#, First Name, Last Name\n');
parser.write('12345, Bill, Waterson\n');
Additional context
Has some similarities to #292, though I feel this is a more generic solution.