Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-18 00:45:47 +00:00
Code Issues Projects Releases Wiki Activity

Fix exponential memory use when chaining seqgen (#2072) (#2094)

Browse source 
seqgen.Transform was generating its full sequence on every call,
including once per upstream record. When chaining two seqgens of
N records each, the second produced N+1 copies of N records in
memory (O(N²)) before writing only the first N. Fix: return
immediately for non-EOS input; generate the sequence only when
the upstream end-of-stream arrives, which is the correct
semantics for a verb that discards its input record stream.

Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
John Kerl 2026-06-19 18:09:54 -04:00 • committed by GitHub
parent 988a396344
commit 5976b43d04
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 5 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

5
pkg/transformers/seqgen.go
View file

@ -183,6 +183,11 @@ func (tr *TransformerSeqgen) Transform(
inputDownstreamDoneChannel <-chan bool,
outputDownstreamDoneChannel chan<- bool,
) {
if !inrecAndContext.EndOfStream {
// Discard upstream records; generate output only when upstream is done.
return
}
counter := tr.start
context := types.NewNilContext()
context.UpdateForStartOfFile("seqgen")