Add new tail -n +N, head -n -N options (#2071)

Inspired by GNU head & tail, they match their behavior while supporting
the usual grouping operations.

Co-authored-by: John Kerl <kerl.john.r@gmail.com>

docs/src/reference-verbs.md

@@ -1592,6 +1592,8 @@ Without -g, ceases consuming more input (i.e. is fast) when n records have been
 Options:
 -g {a,b,c} Optional group-by-field names for head counts, e.g. a,b,c.
 -n {n} Head-count to print. Default 10.
+           A negative count, e.g. -n -2, passes through all but the last n records,
+           optionally by category.
 -h|--help Show this message.
 </pre>
 
@@ -1620,6 +1622,23 @@ zee eks 17 0.29081949506712723 0.054478717073354166
 wye hat 24 0.7286126830627567  0.19441962592638418
 </pre>
 
+A negative count passes through all but the last *n* records, like GNU `head -n -n`:
+
+<pre class="pre-highlight-in-pair">
+<b>mlr --c2p head -n -2 example.csv</b>
+</pre>
+<pre class="pre-non-highlight-in-pair">
+color  shape    flag  k index quantity rate
+yellow triangle true  1 11    43.6498  9.8870
+red    square   true  2 15    79.2778  0.0130
+red    circle   true  3 16    13.8103  2.9010
+red    square   false 4 48    77.5542  7.4670
+purple triangle false 5 51    81.2290  8.5910
+red    square   false 6 64    77.1991  9.5310
+purple triangle false 7 65    80.1405  5.8240
+yellow circle   true  8 73    63.9785  4.2370
+</pre>
+
 ## histogram
 
 <pre class="pre-highlight-in-pair">
@@ -3974,8 +3993,11 @@ a b c filename
 Usage: mlr tail [options]
 Passes through the last n records, optionally by category.
 Options:
--g {a,b,c} Optional group-by-field names for head counts, e.g. a,b,c.
--n {n} Head-count to print. Default 10.
+-g {a,b,c} Optional group-by-field names for tail counts, e.g. a,b,c.
+-n {n} Tail-count to print. Default 10.
+           A leading '+' means start at the nth record rather than print
+           the last n: e.g. -n +3 passes through all but the first 2
+           records, optionally by category.
 -h|--help Show this message.
 </pre>
 
@@ -4002,6 +4024,23 @@ blue   square   1    499872 0.618906 0.263796 0.531147 6.210738
 yellow circle   1    499974 0.764951 0.252842 0.499699 5.013810
 </pre>
 
+As with GNU `tail`, a leading `+` starts at the *n*th record rather than printing the last *n* -- so `-n +3` passes through all but the first 2 records:
+
+<pre class="pre-highlight-in-pair">
+<b>mlr --c2p tail -n +3 example.csv</b>
+</pre>
+<pre class="pre-non-highlight-in-pair">
+color  shape    flag  k  index quantity rate
+red    circle   true  3  16    13.8103  2.9010
+red    square   false 4  48    77.5542  7.4670
+purple triangle false 5  51    81.2290  8.5910
+red    square   false 6  64    77.1991  9.5310
+purple triangle false 7  65    80.1405  5.8240
+yellow circle   true  8  73    63.9785  4.2370
+yellow circle   true  9  87    63.5058  8.3350
+purple square   false 10 91    72.3735  8.2430
+</pre>
+
 ## tee
 
 <pre class="pre-highlight-in-pair">

docs/src/reference-verbs.md.in

@@ -543,6 +543,12 @@ GENMD-RUN-COMMAND
 mlr --opprint head -n 1 -g b data/medium
 GENMD-EOF
 
+A negative count passes through all but the last *n* records, like GNU `head -n -n`:
+
+GENMD-RUN-COMMAND
+mlr --c2p head -n -2 example.csv
+GENMD-EOF
+
 ## histogram
 
 GENMD-RUN-COMMAND
@@ -1213,6 +1219,12 @@ GENMD-RUN-COMMAND
 mlr --c2p tail -n 1 -g shape data/colored-shapes.csv
 GENMD-EOF
 
+As with GNU `tail`, a leading `+` starts at the *n*th record rather than printing the last *n* -- so `-n +3` passes through all but the first 2 records:
+
+GENMD-RUN-COMMAND
+mlr --c2p tail -n +3 example.csv
+GENMD-EOF
+
 ## tee
 
 GENMD-RUN-COMMAND

docs/src/streaming-and-memory.md

@@ -93,7 +93,7 @@ They are memory-friendly, and they don't wait for end of input to produce their
 * [gap](reference-verbs.md#gap)
 * [grep](reference-verbs.md#grep)
 * [having-fields](reference-verbs.md#having-fields)
-* [head](reference-verbs.md#head)
+* [head](reference-verbs.md#head) -- if -n is non-negative
 * [json-parse](reference-verbs.md#json-parse)
 * [json-stringify](reference-verbs.md#json-stringify)
 * [label](reference-verbs.md#label)
@@ -111,6 +111,7 @@ They are memory-friendly, and they don't wait for end of input to produce their
 * [skip-trivial-records](reference-verbs.md#skip-trivial-records)
 * [sort-within-records](reference-verbs.md#sort-within-records)
 * [step](reference-verbs.md#step)
+* [tail](reference-verbs.md#tail) -- if invoked with -n +N (start-at-record-n form)
 * [tee](reference-verbs.md#tee)
 * [template](reference-verbs.md#template)
 * [unflatten](reference-verbs.md#unflatten)
@@ -144,7 +145,8 @@ They are memory-unfriendly, and they wait for end of input to produce their outp
 These retain a bounded number of records from one record to the next.
 They are memory-friendly, but they wait for end of input to produce their output.
 
-* [tail](reference-verbs.md#tail)
+* [head](reference-verbs.md#head) -- if -n is negative
+* [tail](reference-verbs.md#tail) -- if not invoked with -n +N
 * [top](reference-verbs.md#top)
 
 ## Non-streaming, retaining some state

docs/src/streaming-and-memory.md.in

@@ -77,7 +77,7 @@ They are memory-friendly, and they don't wait for end of input to produce their
 * [gap](reference-verbs.md#gap)
 * [grep](reference-verbs.md#grep)
 * [having-fields](reference-verbs.md#having-fields)
-* [head](reference-verbs.md#head)
+* [head](reference-verbs.md#head) -- if -n is non-negative
 * [json-parse](reference-verbs.md#json-parse)
 * [json-stringify](reference-verbs.md#json-stringify)
 * [label](reference-verbs.md#label)
@@ -95,6 +95,7 @@ They are memory-friendly, and they don't wait for end of input to produce their
 * [skip-trivial-records](reference-verbs.md#skip-trivial-records)
 * [sort-within-records](reference-verbs.md#sort-within-records)
 * [step](reference-verbs.md#step)
+* [tail](reference-verbs.md#tail) -- if invoked with -n +N (start-at-record-n form)
 * [tee](reference-verbs.md#tee)
 * [template](reference-verbs.md#template)
 * [unflatten](reference-verbs.md#unflatten)
@@ -128,7 +129,8 @@ They are memory-unfriendly, and they wait for end of input to produce their outp
 These retain a bounded number of records from one record to the next.
 They are memory-friendly, but they wait for end of input to produce their output.
 
-* [tail](reference-verbs.md#tail)
+* [head](reference-verbs.md#head) -- if -n is negative
+* [tail](reference-verbs.md#tail) -- if not invoked with -n +N
 * [top](reference-verbs.md#top)
 
 ## Non-streaming, retaining some state

pkg/lib/getoptify.go

@@ -26,8 +26,15 @@ func Getoptify(inargs []string) []string {
 				// Example: 'mlr unsparsify --fill-with -99999'.
 				outargs = append(outargs, inarg)
 			} else {
-				for _, c := range inarg[1:] {
-					outargs = append(outargs, "-"+string(c))
+				rest := inarg[1:]
+				for i := 0; i < len(rest); i++ {
+					// Pass integers without a leading dash, so that negative integers can be represented.
+					// Example: `head -n 4` and head -n4` can be differentiated from `head -n -4`.
+					if rest[i] >= '0' && rest[i] <= '9' {
+						outargs = append(outargs, rest[i:])
+						break
+					}
+					outargs = append(outargs, "-"+string(rest[i]))
 				}
 			}
 		} else if splitRegex.MatchString(inarg) {

pkg/transformers/head.go

@@ -28,6 +28,8 @@ func transformerHeadUsage(
 	fmt.Fprintf(o, "Options:\n")
 	fmt.Fprintf(o, "-g {a,b,c} Optional group-by-field names for head counts, e.g. a,b,c.\n")
 	fmt.Fprintf(o, "-n {n} Head-count to print. Default 10.\n")
+	fmt.Fprintf(o, "           A negative count, e.g. -n -2, passes through all but the last n records,\n")
+	fmt.Fprintf(o, "           optionally by category.\n")
 	fmt.Fprintf(o, "-h|--help Show this message.\n")
 }
 
@@ -68,16 +70,6 @@ func transformerHeadParseCLI(
 			}
 			headCount = n
 
-			// This is a bit of a hack. In our Getoptify routine we preprocess
-			// the command line sending '-xyz' to '-x -y -z', but leaving
-			// '--xyz' as-is. Also, Unix-like tools often support 'head -n4'
-			// and 'tail -n4' in addition to 'head -n 4' and 'tail -n 4'.  Our
-			// getoptify paradigm, combined with syntax familiar to users,
-			// means we get '-n -4' here. So, take the absolute value to handle this.
-			if headCount < 0 {
-				headCount = -headCount
-			}
-
 		} else if opt == "-g" {
 			names, err := cli.VerbGetStringArrayArg(verb, opt, args, &argi, argc)
 			if err != nil {
@@ -116,6 +108,7 @@ type TransformerHead struct {
 	recordTransformerFunc RecordTransformerFunc
 	unkeyedRecordCount    int64
 	keyedRecordCounts     map[string]int64
+	recordListsByGroup    map[string]*[]*types.RecordAndContext
 
 	// See ChainTransformer
 	wroteDownstreamDone bool
@@ -126,15 +119,23 @@ func NewTransformerHead(
 	groupByFieldNames []string,
 ) (*TransformerHead, error) {
 
+	allButLast := headCount < 0
+	if allButLast {
+		headCount = -headCount
+	}
+
 	tr := &TransformerHead{
 		headCount:           headCount,
 		groupByFieldNames:   groupByFieldNames,
 		unkeyedRecordCount:  0,
 		keyedRecordCounts:   make(map[string]int64),
+		recordListsByGroup:  make(map[string]*[]*types.RecordAndContext),
 		wroteDownstreamDone: false,
 	}
 
-	if groupByFieldNames == nil {
+	if allButLast {
+		tr.recordTransformerFunc = tr.transformAllButLast
+	} else if groupByFieldNames == nil {
 		tr.recordTransformerFunc = tr.transformUnkeyed
 	} else {
 		tr.recordTransformerFunc = tr.transformKeyed
@@ -207,3 +208,36 @@ func (tr *TransformerHead) transformKeyed(
 		*outputRecordsAndContexts = append(*outputRecordsAndContexts, inrecAndContext)
 	}
 }
+
+func (tr *TransformerHead) transformAllButLast(
+	inrecAndContext *types.RecordAndContext,
+	outputRecordsAndContexts *[]*types.RecordAndContext, // list of *types.RecordAndContext
+	inputDownstreamDoneChannel <-chan bool,
+	outputDownstreamDoneChannel chan<- bool,
+) {
+	if !inrecAndContext.EndOfStream {
+		inrec := inrecAndContext.Record
+
+		groupingKey, ok := inrec.GetSelectedValuesJoined(tr.groupByFieldNames)
+		if !ok {
+			return
+		}
+
+		recordListForGroup := tr.recordListsByGroup[groupingKey]
+		if recordListForGroup == nil { // first time
+			recordListForGroup = &[]*types.RecordAndContext{}
+			tr.recordListsByGroup[groupingKey] = recordListForGroup
+		}
+
+		*recordListForGroup = append(*recordListForGroup, inrecAndContext)
+		for int64(len(*recordListForGroup)) > tr.headCount {
+			// Emit records that have fallen out of the window.
+			*outputRecordsAndContexts = append(*outputRecordsAndContexts, (*recordListForGroup)[0])
+			(*recordListForGroup)[0] = nil // release the backing-array slot's reference
+			*recordListForGroup = (*recordListForGroup)[1:]
+		}
+
+	} else {
+		*outputRecordsAndContexts = append(*outputRecordsAndContexts, inrecAndContext)
+	}
+}

pkg/transformers/tail.go

@@ -26,8 +26,11 @@ func transformerTailUsage(
 	fmt.Fprintln(o, "Passes through the last n records, optionally by category.")
 
 	fmt.Fprintf(o, "Options:\n")
-	fmt.Fprintf(o, "-g {a,b,c} Optional group-by-field names for head counts, e.g. a,b,c.\n")
-	fmt.Fprintf(o, "-n {n} Head-count to print. Default 10.\n")
+	fmt.Fprintf(o, "-g {a,b,c} Optional group-by-field names for tail counts, e.g. a,b,c.\n")
+	fmt.Fprintf(o, "-n {n} Tail-count to print. Default 10.\n")
+	fmt.Fprintf(o, "           A leading '+' means start at the nth record rather than print\n")
+	fmt.Fprintf(o, "           the last n: e.g. -n +3 passes through all but the first 2\n")
+	fmt.Fprintf(o, "           records, optionally by category.\n")
 	fmt.Fprintf(o, "-h|--help Show this message.\n")
 }
 
@@ -45,6 +48,7 @@ func transformerTailParseCLI(
 	argi++
 
 	tailCount := int64(10)
+	fromStart := false
 	var groupByFieldNames []string = nil
 
 	for argi < argc /* variable increment: 1 or 2 depending on flag */ {
@@ -62,22 +66,15 @@ func transformerTailParseCLI(
 			return nil, cli.ErrHelpRequested
 
 		} else if opt == "-n" {
+			if argi < argc && strings.HasPrefix(args[argi], "+") {
+				fromStart = true
+			}
 			n, err := cli.VerbGetIntArg(verb, opt, args, &argi, argc)
 			if err != nil {
 				return nil, err
 			}
 			tailCount = n
 
-			// This is a bit of a hack. In our Getoptify routine we preprocess
-			// the command line sending '-xyz' to '-x -y -z', but leaving
-			// '--xyz' as-is. Also, Unix-like tools often support 'head -n4'
-			// and 'tail -n4' in addition to 'head -n 4' and 'tail -n 4'.  Our
-			// getoptify paradigm, combined with syntax familiar to users,
-			// means we get '-n -4' here. So, take the absolute value to handle this.
-			if tailCount < 0 {
-				tailCount = -tailCount
-			}
-
 		} else if opt == "-g" {
 			names, err := cli.VerbGetStringArrayArg(verb, opt, args, &argi, argc)
 			if err != nil {
@@ -98,6 +95,7 @@ func transformerTailParseCLI(
 
 	transformer, err := NewTransformerTail(
 		tailCount,
+		fromStart,
 		groupByFieldNames,
 	)
 	if err != nil {
@@ -109,24 +107,43 @@ func transformerTailParseCLI(
 
 type TransformerTail struct {
 	// input
-	tailCount         int64
+	count             int64 // last-n count, or skip-count in from-start mode
 	groupByFieldNames []string
 
 	// state
+	recordTransformerFunc RecordTransformerFunc
 	// map from string to record slices
 	recordListsByGroup *lib.OrderedMap[*[]*types.RecordAndContext]
+	// for the from-start mode: per-group counts of records seen so far
+	countsByGroup map[string]int64
 }
 
 func NewTransformerTail(
 	tailCount int64,
+	fromStart bool,
 	groupByFieldNames []string,
 ) (*TransformerTail, error) {
 
 	tr := &TransformerTail{
-		tailCount:         tailCount,
 		groupByFieldNames: groupByFieldNames,
 
 		recordListsByGroup: lib.NewOrderedMap[*[]*types.RecordAndContext](),
+		countsByGroup:      make(map[string]int64),
+	}
+
+	if fromStart {
+		// '-n +N' is a 1-based record index, i.e. skip the first n-1.
+		tr.count = tailCount - 1
+		if tr.count < 0 {
+			tr.count = 0
+		}
+		tr.recordTransformerFunc = tr.transformFromStart
+	} else {
+		if tailCount < 0 {
+			tailCount = -tailCount
+		}
+		tr.count = tailCount
+		tr.recordTransformerFunc = tr.transformLastN
 	}
 
 	return tr, nil
@@ -139,6 +156,15 @@ func (tr *TransformerTail) Transform(
 	outputDownstreamDoneChannel chan<- bool,
 ) {
 	HandleDefaultDownstreamDone(inputDownstreamDoneChannel, outputDownstreamDoneChannel)
+	tr.recordTransformerFunc(inrecAndContext, outputRecordsAndContexts, inputDownstreamDoneChannel, outputDownstreamDoneChannel)
+}
+
+func (tr *TransformerTail) transformLastN(
+	inrecAndContext *types.RecordAndContext,
+	outputRecordsAndContexts *[]*types.RecordAndContext, // list of *types.RecordAndContext
+	inputDownstreamDoneChannel <-chan bool,
+	outputDownstreamDoneChannel chan<- bool,
+) {
 	if !inrecAndContext.EndOfStream {
 		inrec := inrecAndContext.Record
 
@@ -155,7 +181,8 @@ func (tr *TransformerTail) Transform(
 		}
 
 		*recordListForGroup = append(*recordListForGroup, inrecAndContext)
-		for int64(len(*recordListForGroup)) > tr.tailCount {
+		for int64(len(*recordListForGroup)) > tr.count {
+			(*recordListForGroup)[0] = nil // release the backing-array slot's reference
 			*recordListForGroup = (*recordListForGroup)[1:]
 		}
 
@@ -166,3 +193,28 @@ func (tr *TransformerTail) Transform(
 		*outputRecordsAndContexts = append(*outputRecordsAndContexts, inrecAndContext) // end-of-stream marker
 	}
 }
+
+func (tr *TransformerTail) transformFromStart(
+	inrecAndContext *types.RecordAndContext,
+	outputRecordsAndContexts *[]*types.RecordAndContext, // list of *types.RecordAndContext
+	inputDownstreamDoneChannel <-chan bool,
+	outputDownstreamDoneChannel chan<- bool,
+) {
+	if !inrecAndContext.EndOfStream {
+		inrec := inrecAndContext.Record
+
+		groupingKey, ok := inrec.GetSelectedValuesJoined(tr.groupByFieldNames)
+		if !ok {
+			return
+		}
+
+		tr.countsByGroup[groupingKey]++
+		if tr.countsByGroup[groupingKey] > tr.count {
+			// Emit records now that we skipped the requested number of them.
+			*outputRecordsAndContexts = append(*outputRecordsAndContexts, inrecAndContext)
+		}
+
+	} else {
+		*outputRecordsAndContexts = append(*outputRecordsAndContexts, inrecAndContext) // end-of-stream marker
+	}
+}

test/cases/cli-help/0001/expout

@@ -423,6 +423,8 @@ Without -g, ceases consuming more input (i.e. is fast) when n records have been
 Options:
 -g {a,b,c} Optional group-by-field names for head counts, e.g. a,b,c.
 -n {n} Head-count to print. Default 10.
+           A negative count, e.g. -n -2, passes through all but the last n records,
+           optionally by category.
 -h|--help Show this message.
 
 ================================================================
@@ -1305,8 +1307,11 @@ tail
 Usage: mlr tail [options]
 Passes through the last n records, optionally by category.
 Options:
--g {a,b,c} Optional group-by-field names for head counts, e.g. a,b,c.
--n {n} Head-count to print. Default 10.
+-g {a,b,c} Optional group-by-field names for tail counts, e.g. a,b,c.
+-n {n} Tail-count to print. Default 10.
+           A leading '+' means start at the nth record rather than print
+           the last n: e.g. -n +3 passes through all but the first 2
+           records, optionally by category.
 -h|--help Show this message.
 
 ================================================================

test/cases/verb-head/0004/cmd

@@ -0,0 +1 @@
+mlr head -n -2 test/input/abixy-het

test/cases/verb-head/0004/expout

@@ -0,0 +1,8 @@
+a=pan,b=pan,i=1,x=0.34679014,y=0.72680286
+a=eks,b=pan,i=2,x=0.75867996,y=0.52215111
+aaa=wye,b=wye,i=3,x=0.20460331,y=0.33831853
+a=eks,bbb=wye,i=4,x=0.38139939,y=0.13418874
+a=wye,b=pan,i=5,xxx=0.57328892,y=0.86362447
+a=zee,b=pan,i=6,x=0.52712616,y=0.49322129
+a=eks,b=zee,iii=7,x=0.61178406,y=0.18788492
+a=zee,b=wye,i=8,x=0.59855401,yyy=0.97618139

test/cases/verb-head/0005/cmd

@@ -0,0 +1 @@
+mlr head -n -1 -g a test/input/abixy-het

test/cases/verb-head/0005/expout

@@ -0,0 +1,4 @@
+a=eks,b=pan,i=2,x=0.75867996,y=0.52215111
+a=eks,bbb=wye,i=4,x=0.38139939,y=0.13418874
+a=zee,b=pan,i=6,x=0.52712616,y=0.49322129
+a=pan,b=pan,i=1,x=0.34679014,y=0.72680286

test/cases/verb-head/0006/cmd

@@ -0,0 +1 @@
+mlr head -n3 test/input/abixy-het

test/cases/verb-head/0006/expout

@@ -0,0 +1,3 @@
+a=pan,b=pan,i=1,x=0.34679014,y=0.72680286
+a=eks,b=pan,i=2,x=0.75867996,y=0.52215111
+aaa=wye,b=wye,i=3,x=0.20460331,y=0.33831853

test/cases/verb-tail/0004/cmd

@@ -0,0 +1 @@
+mlr tail -n +3 test/input/abixy-het

test/cases/verb-tail/0004/expout

@@ -0,0 +1,8 @@
+aaa=wye,b=wye,i=3,x=0.20460331,y=0.33831853
+a=eks,bbb=wye,i=4,x=0.38139939,y=0.13418874
+a=wye,b=pan,i=5,xxx=0.57328892,y=0.86362447
+a=zee,b=pan,i=6,x=0.52712616,y=0.49322129
+a=eks,b=zee,iii=7,x=0.61178406,y=0.18788492
+a=zee,b=wye,i=8,x=0.59855401,yyy=0.97618139
+aaa=hat,bbb=wye,i=9,x=0.03144188,y=0.74955076
+a=pan,b=wye,i=10,x=0.50262601,y=0.95261836

test/cases/verb-tail/0005/cmd

@@ -0,0 +1 @@
+mlr tail -n +2 -g a test/input/abixy-het

test/cases/verb-tail/0005/expout

@@ -0,0 +1,4 @@
+a=eks,bbb=wye,i=4,x=0.38139939,y=0.13418874
+a=eks,b=zee,iii=7,x=0.61178406,y=0.18788492
+a=zee,b=wye,i=8,x=0.59855401,yyy=0.97618139
+a=pan,b=wye,i=10,x=0.50262601,y=0.95261836

test/cases/verb-tail/0006/cmd

@@ -0,0 +1 @@
+mlr tail -n3 test/input/abixy-het

test/cases/verb-tail/0006/expout

@@ -0,0 +1,3 @@
+a=zee,b=wye,i=8,x=0.59855401,yyy=0.97618139
+aaa=hat,bbb=wye,i=9,x=0.03144188,y=0.74955076
+a=pan,b=wye,i=10,x=0.50262601,y=0.95261836

test/cases/verb-tail/0007/cmd

@@ -0,0 +1 @@
+mlr tail -n -2 test/input/abixy-het

test/cases/verb-tail/0007/expout

@@ -0,0 +1,2 @@
+aaa=hat,bbb=wye,i=9,x=0.03144188,y=0.74955076
+a=pan,b=wye,i=10,x=0.50262601,y=0.95261836