Skip to content

Backport patches fixing binary data piping

Emanuele Aina requested to merge wip/em/fix-binary-data-pipin into apertis/v2022dev1

When invoking a command, the sh module checks some tty-related stuff and while doing so it tries to figure out if a custom stdin is being passed in.

To do so, it check for its truthiness: https://github.com/amoffat/sh/blob/1.12.14/sh.py#L1738

But we pass a RunningCommand as stdin, which has a __len__() method that does something like len(str(self.stdout)): https://github.com/amoffat/sh/blob/1.12.14/sh.py#L880.

This means that the truthiness of stdin is evaluated by fully converting it to a unicode string.

Unfortunately with pristine-lfs we always deal with binary data and we got quite lucky as we only recently hit this error:

$ cd aalib && pristine-lfs --debug checkout -o _build --auto
Traceback (most recent call last):
  File "/usr/bin/pristine-lfs", line 11, in <module>
    load_entry_point('pristine-lfs==20190910.0', 'console_scripts', 'pristine-lfs')()
  File "/usr/lib/python3/dist-packages/pristine_lfs/main.py", line 183, in main
    args.func(args)
  File "/usr/lib/python3/dist-packages/pristine_lfs/main.py", line 89, in do_checkout
    checkout_package(package, version, branch, args.outdir)
  File "/usr/lib/python3/dist-packages/pristine_lfs/main.py", line 106, in checkout_package
    checkout_lfs_file(branch, f, outdir)
  File "/usr/lib/python3/dist-packages/pristine_lfs/util.py", line 271, in checkout_lfs_file
    git.lfs.smudge(filename, _out=tarball, _in=metadata)
  File "/usr/lib/python3/dist-packages/sh.py", line 1427, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/usr/lib/python3/dist-packages/sh.py", line 767, in __init__
    self.call_args, pipe, process_assign_lock)
  File "/usr/lib/python3/dist-packages/sh.py", line 1738, in __init__
    custom_in_out_err = stdin or stdout or stderr
  File "/usr/lib/python3/dist-packages/sh.py", line 835, in __len__
    return len(str(self))
  File "/usr/lib/python3/dist-packages/sh.py", line 884, in __str__
    return self.__unicode__()
  File "/usr/lib/python3/dist-packages/sh.py", line 893, in __unicode__
    self.call_args["decode_errors"])
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Task: T7771

Edited by Emanuele Aina

Merge request reports