byte[] transformed to List[int]

Currently byte[] is mapped to List[int]. In general this is done because we couldn't map to numpy types and preserve the contained type. This limitation is beginning to be lifted with numpy 1.21 (where you can write things like npt.NDArray[np.int8]).

Perhaps we should move to the numpy standard form already, but if not, then we should map byte[] to bytearray rather than List[int]. I suggest bytearray rather than bytes because the former is mutable (like a Java array). The complexity in all of this is that byte is a signed integer, whereas bytes in Python are unsigned (https://stackoverflow.com/a/40951288/741316). For this reason, perhaps we should proceed with the npt.NDArray[np.int8] option (and require numpy 1.21 for array types).

mentioned in merge request pjlsa!15 (closed)

Fixed partially (for INPUT arguments) in 4ee40af8

Indeed, JPype implicitly converts both bytes and bytearray to byte[] when passed as method arguments. This is now reflected by offering Union[List[int], bytes, bytearray] for arguments.

The problem for return types is that we can not really decide how to map them - in reality they will be always jpype.JArray, we can just decide what we claim them to be. In principle, we should use jpype.JArray, but this is not generic, so we would lose the information about the element type. Since JArray (almost) behaves like a list, I advertise a generic List[...].

As a small note, the typing module explicitly states that the annotation bytes is equivalent to typing.ByteString, which also includes bytearray and memoryview. The following type-checks in MyPy:

def access(arg: bytes) -> int:
    return arg[0]
def main() -> None:
    access(b"hello")
    access(bytearray(b"hello"))
    access(memoryview(b"hello"))
if __name__ == "__main__":
    main()

Thanks, valid point!

I will change to make the union a bit simpler - Union[List[int], bytes].

Alright, I've run a small test on what types exactly are accepted by Java methods with annotation byte[]. It seems to accept pretty much any Sequence[int] as long as the integers are in range [−128; 127]. It also accepts bytes and its two siblings and handles their bytes correctly. Union[List[int], bytes] might still be the more informative annotation, mind you.

However, everything except jpype.JArray is passed by copy instead of reference and so any modifications to those arguments are lost. Passing a bytearray is possible but almost certainly be useless.

As for return types, I propose MutableSequence[int], since it does not have the necessary list methods. One small issue is that issubclass(JArray(JByte)(b""), Sequence) fails even though it satisfies all requirements.

---

Test program:

#!/usr/bin/env python3

import array
import ctypes  # pylint: disable=unused-import
import io
import typing as t

import jpype.imports  # type: ignore
import numpy


def run_experiment(source: t.Any, dest: t.Any) -> None:
    # pylint: disable = broad-except, import-error, import-outside-toplevel
    from java.nio import ByteBuffer

    try:
        buffer = ByteBuffer.wrap(source)
        print("    Source:", bytes(buffer.array()))
        print("    Dest:", bytes(dest))
        print("    Copying …")
        buffer.get(dest)
        print("    Dest:", bytes(dest))
    except Exception as exc:
        print("    ", exc, sep="")


def main() -> None:
    """Main function."""
    print("List[int]")
    run_experiment(list(b"hello"), list(b"world"))
    print("Tuple[int, ...]")
    run_experiment(tuple(b"hello"), tuple(b"world"))
    print("bytes")
    run_experiment(b"hello", b"world")
    print("bytearray")
    run_experiment(bytearray(b"hello"), bytearray(b"world"))
    print("memoryview")
    run_experiment(memoryview(bytearray(b"hello")), memoryview(bytearray(b"world")))
    print("np.ndarray[np.int8]")
    run_experiment(
        numpy.array(list(b"hello"), dtype=numpy.byte),
        numpy.array(list(b"world"), dtype=numpy.byte),
    )
    print("array.array['b']")
    run_experiment(array.array("b", b"hello"), array.array("b", b"world"))
    print("C byte[]")
    print("    segfaults")
    # run_experiment((ctypes.c_byte * 5)(*b"hello"), (ctypes.c_byte * 6)(*b"world"))
    print("io.BytesIO")
    run_experiment(io.BytesIO(b"hello"), io.BytesIO(b"world"))
    print("Java byte[]")
    run_experiment(
        jpype.JArray(jpype.JByte)(b"hello"),
        jpype.JArray(jpype.JByte)(b"world"),
    )


if __name__ == "__main__":
    jpype.startJVM()
    main()

Output:

List[int]
    Source: b'hello'
    Dest: b'world'
    Copying …
    Dest: b'world'
Tuple[int, ...]
    Source: b'hello'
    Dest: b'world'
    Copying …
    Dest: b'world'
bytes
    Source: b'hello'
    Dest: b'world'
    Copying …
    Dest: b'world'
bytearray
    Source: b'hello'
    Dest: b'world'
    Copying …
    Dest: b'world'
memoryview
    Source: b'hello'
    Dest: b'world'
    Copying …
    Dest: b'world'
np.ndarray[np.int8]
    Source: b'hello'
    Dest: b'world'
    Copying …
    Ambiguous overloads found for java.nio.HeapByteBuffer.get(numpy.ndarray) between:
	public byte java.nio.HeapByteBuffer.get(int)
	public java.nio.ByteBuffer java.nio.ByteBuffer.get(byte[])


array.array['b']
    Source: b'hello'
    Dest: b'world'
    Copying …
    Dest: b'world'
C byte[]
    segfaults
io.BytesIO
    No matching overloads found for *static* java.nio.ByteBuffer.wrap(_io.BytesIO), options are:
	public static java.nio.ByteBuffer java.nio.ByteBuffer.wrap(byte[],int,int)
	public static java.nio.ByteBuffer java.nio.ByteBuffer.wrap(byte[])


Java byte[]
    Source: b'hello'
    Dest: b'world'
    Copying …
    Dest: b'hello'

Thanks Nico for the test, very useful.

I agree with the MutableSequence for return types, that's a bit closer to the truth than List. For arguments I would keep List as it is more informative. Also, we should probably add JArray to the union for arguments, it does not cost anything.

Therefore, I would propose this mapping:

def translateJavaArrayType(javaType: Any, typeVars: Optional[List[TypeVarStr]], isArgument: bool) -> TypeStr:
    """
    Translate a Java array type to python type.

    Java arrays returned by JPype are of type "JArray", but JArray is not generic. To conserve the type
    information of the elements, we map them to typing.MutableSequence for the time being.

    For arguments, JPype accepts JArray, and it does implicit conversions for Sequences/Lists for all arrays
    and bytes -> byte[] specifically (these conversions happen by copy).
    """
    elementType = javaArrayComponentType(javaType)
    pythonElementType = pythonType(elementType, typeVars)
    if isArgument:
        union = [TypeStr('typing.List', [pythonElementType]), TypeStr('jpype.JArray')]
        if str(elementType) == 'byte':
            # hack: JPype supports converting bytes/bytearray to byte[] but this is not advertised in hints...
            union.append(TypeStr('bytes'))
        return TypeStr('typing.Union', union)
    else:
        # actually JArray, but it is not generic
        return TypeStr('typing.MutableSequence', [pythonElementType])

What do you think?

Looks good to me!

Thanks!

Released as 0.2.8

closed