dbsort - sort rows based on the the specified columns
dbsort [-M MemLimit] [-T TemporaryDirectory] [-nNrR] column [column...]
Sort all input rows as specified by the numeric or lexical columns.
Dbsort consumes a fixed amount of memory regardless of input size.
(It reverts to temporary files on disk if necessary, based on the -M
and -T options.)
The sort should be stable, but this has not yet been verified.
For large inputs (those that spill to disk),
dbsort will do some of the merging in parallel, if possible.
The --parallel option can control the degree of parallelism,
if desired.
General option:
- -M MaxMemBytes
-
Specify an approximate limit on memory usage (in bytes).
Larger values allow faster sorting because more operations
happen in-memory, provided you have enough memory.
- -T TmpDir
-
where to put tmp files.
Also uses environment variable TMPDIR, if -T is
not specified.
Default is /tmp.
- --parallelism N or -j N
-
Allow up to N merges to happen in parallel.
Default is the number of CPUs in the machine.
Sort specification options (can be interspersed with column names):
- -r or --descending
-
sort in reverse order (high to low)
- -R or --ascending
-
sort in normal order (low to high)
- -t or --type-inferred-sorting
-
sort fields by type (numeric or leicographic), automatically
- -T or --no-type-inferred-sorting
-
sort fields only as specified based on
-n or -N
- -n or --numeric
-
sort numerically
- -N or --lexical
-
sort lexicographically
This module also supports the standard fsdb options:
- -d
-
Enable debugging output.
- -i or --input InputSource
-
Read from InputSource, typically a file name, or
- for standard input,
or (if in Perl) a IO::Handle, Fsdb::IO or Fsdb::BoundedQueue objects.
- -o or --output OutputDestination
-
Write to OutputDestination, typically a file name, or
- for standard output,
or (if in Perl) a IO::Handle, Fsdb::IO or Fsdb::BoundedQueue objects.
- --autorun or --noautorun
-
By default, programs process automatically,
but Fsdb::Filter objects in Perl do not run until you invoke
the
run() method.
The --(no)autorun option controls that behavior within Perl.
- --header H
-
Use H as the full Fsdb header, rather than reading a header from
then input.
- --help
-
Show help.
- --man
-
Show full manual.
#fsdb cid cname
10 pascal
11 numanal
12 os
cat data.fsdb | dbsort cname
#fsdb cid cname
11 numanal
12 os
10 pascal
# | dbsort cname
dbmerge(1),
dbmapreduce(1),
Fsdb(3)
$filter = new Fsdb::Filter::dbsort(@arguments);
Create a new object, taking command-line arguments.
$filter->set_defaults();
Internal: set up defaults.
$filter->parse_options(@ARGV);
Internal: parse command-line arguments.
$filter->setup();
Internal: setup, parse headers.
$self->segment_start(\@rows);
Sorting happens internally,
to handle large things in pieces if necessary.
call $self-segment_start> to init things and to restart after an overflow
$self-segment_overflow> to close one segment and start the next,
and $self-segment_merge_finish> to put them back together again.
Note that we don't invoke the merge code unless the data
exceeds some threshold size, so small sorts happen completely
in memory.
Once we give up on memory, all the merging happens by making
passes over the disk files.
$out = $self->segment_next_output($input_finished)
Internal: return a Fsdb::IO::Writer as $OUT
that either points to our output or a temporary file,
depending on how things are going.
$self->segment_overflow(\@rows, $input_finished)
Called to sort @ROWS, writing them to the appropriate place.
$INPUT_FINISHED is set if all input has been read.
$self->segment_merge_start($fn);
Start merging on file $FN.
Fork off a merge thread, if necessary.
$self->segment_merge_finish();
Merge queued files, if any.
Just call dbmerge(1) to do all the real work.
$filter->run();
Internal: run over each rows.
Copyright (C) 1991-2018 by John Heidemann <johnh@isi.edu>
This program is distributed under terms of the GNU general
public license, version 2. See the file COPYING
with the distribution for details.
|