n-dimensional slice-shell [ready for comments]

DUB package at dip80-ndslice at code.dlang.org.
Requests:
projection template
Remove range properties
Iteration over N-1 dimension
Implement byDim property
shape rework

Alternatively, the existing shape could be renamed structure and shape would just return the lengths, like it does in numpy.

Iterating over elements
_1. byElement
foreach(elem; tensor.byElement) {...}
foreach(ref elem; tensor.byElement) {...}

_2. byIndexedElement
foreach(i, j, k, elem; tensor3D.byIndexedElement) {...}

_3. opApply for indexed  foreach with ref
foreach(i, j, k, ref elem; tensor3D) {...}

Lazy functionality like deep map and deep zip
View from core team is wanted!
Implemented requests

shape property https://github.com/9il/phobos/commit/eacc511398326e28297b9fdf2d69e6adf2af9e7b
Numpy like shape property, structure property https://github.com/9il/phobos/commit/b5fb8956a4b5c60d3820cf00a4d1f4684bd220e0

n-dimensional slices
Slicing
    auto a = 1000000.iota.sliced(10, 20, 30, 40);
    auto b = a[0..$, 10, 4..27, 4];
    auto c = b[2..9, 5..10];
    auto d = b[3..$, $-2];
    assert(b[4, 17] == a[4, 10, 21, 4]);
    assert(c[1, 2] == a[3, 10, 11, 4]);
    assert(d[3] == a[6, 10, 25, 4]);

Operator overloading. # 1
    auto fun(ref int x) { x *= 3; }

    auto tensor = 1000
        .iota
        .array
        .sliced(8, 9, 10);

    ++tensor[];
    fun(tensor[0, 0, 0]);

    assert(tensor[0, 0, 0] == 3);

    tensor[0, 0, 0] *= 4;
    tensor[0, 0, 0]--;
    assert(tensor[0, 0, 0] == 11);

Operator overloading. # 2
    auto matrix = 100
        .iota
        .map!(i => BigInt(i))
        .array
        .sliced(8, 9);

    matrix[3..6, 2] += 100;
    foreach(i; 0..8)
        foreach(j; 0..9)
            if(i >= 3 && i < 6 && j == 2)
                assert(matrix[i, j] >= 100);
            else
                assert(matrix[i, j] < 100);

Operator overloading. # 3
    auto matrix = 100
        .iota
        .array
        .sliced(8, 9);
    assert(matrix.equal(matrix));

    matrix[] = matrix;
    matrix[] += matrix;
    assert(matrix[2, 3] == (2 * 9 + 3) * 2);

    auto vec = iota(100, 200).sliced(9);
    matrix[] = vec;
    foreach(v; matrix)
        assert(v.equal(vec));

    matrix[] += vec;
    foreach(vector; matrix)
        foreach(elem; vector)
            assert(elem >= 200);

Properties and methods
    auto tensor = 100.iota.sliced(3, 4, 5);
    static assert(isRandomAccessRange!(typeof(tensor)));

    // `save` method
    // Calls `range.save`
    auto a = tensor.save;
    static assert(is(typeof(a) == typeof(tensor)));

    // `length` property
    assert(tensor.length   == 3);
    assert(tensor.length!0 == 3);
    assert(tensor.length!1 == 4);
    assert(tensor.length!2 == 5);

    // `front` and `back` properties
    // and `popFront`, `popBack`,
    // `popFrontN` and `popBackN` methods
    auto matrix = tensor.back;
    matrix.popBack!1;
    auto column = matrix.back!1;
    column.popFrontN(3);
    auto elem = column.front!0;
    assert(elem == tensor[$-1, 3, $-2]);

    // `stride` property
    assert(tensor.stride   == 20);
    assert(tensor.stride!0 == 20);
    assert(tensor.stride!1 ==  5);
    assert(tensor.stride!2 ==  1);

    assert(matrix.stride   ==  5);
    assert(matrix.stride!1 ==  1);

    matrix = tensor.back!2;
    assert(matrix.stride   == 20);
    assert(matrix.stride!1 ==  5);

    // `structure` property
    auto structure = tensor.structure;
    assert(tensor.length!2 == structure.lengths[2]);
    assert(tensor.stride!1 == structure.strides[1]);
    import std.typecons: Tuple;
    alias Structure = Tuple!(size_t[3], "lengths", size_t[3], "strides");
    static assert(is(typeof(structure) == Structure));

    // `shape` property
    assert(tensor.shape == structure.lengths);

    // `range` method
    // Calls `range.save`.
    auto theIota0 = tensor.range;
    assert(theIota0.front == 0);
    tensor.popFront;
    auto theIota1 = tensor.range;
    assert(theIota1.front == 20);

    // `ptr` property
    import std.array: array;
    auto ar = 100.iota.array;
    auto ts = ar.sliced(3, 4, 5)[1..$, 2..$, 3..$];
    assert(ts.ptr is ar.ptr + 1*20+2*5+3*1);

Type conversion
    foreach(T; TypeTuple!(int, const int, immutable int))
        static assert(is(typeof((T[]).init.sliced(3, 4)) == Slice!(2, T*)));

    Array!int ar;
    static assert(is(typeof(ar[].sliced(3, 4)) == Slice!(2, typeof(ar[]))));

    auto      i0 = 100.iota;
    const     i1 = 100.iota;
    immutable i2 = 100.iota;
    alias S = Slice!(3, typeof(iota(0))); //unqualified range
    // const/immutable Iota is implicitly convertable to unqualified type
    foreach(i; TypeTuple!(i0, i1, i2))
        static assert(is(typeof(i.sliced(3, 4, 5)) == S));

N-dimenstional transpose operator
    auto t0 = 1000.iota.sliced(3, 4, 5);
    auto t1 = t0.transposed!(2, 0, 1); //CTFE - recommended
    auto t2 = t0.transposed (2, 0, 1); //Runtime
    assert(t0[1, 2, 3] == t1[3, 1, 2]);
    assert(t0[1, 2, 3] == t2[3, 1, 2]);
    static assert(is(typeof(t0) == typeof(t1)));
    static assert(is(typeof(t0) == typeof(t2)));

2-dimenstional transpose operator
    auto t0 = 1000.iota.sliced(3, 4);
    auto t1 = t0.transposed();      //CTFE
    auto t2 = t0.transposed!(1, 0); //CTFE
    assert(t0[1, 2] == t1[2, 1]);
    assert(t0[1, 2] == t2[2, 1]);
    static assert(is(typeof(t0) == typeof(t1)));
    static assert(is(typeof(t0) == typeof(t2)));

References:

DIP80 Phobos additions (BLAS support)
Discussion about DIP80/BLAS at forum.

ping @John-Colvin @kyllingstad

put Example 0 in the unittests
the is also some spacing and brace placement issues

I'd love to get stuck in to this immediately, but I'm completely swamped with other work.
I don't think we should be targeting phobos so quickly. Whatever the design, it needs to get some proper use before it gets set in stone. I would love to have it as the heart of https://github.com/DlangScience (see my DConf talk) for a start and then see how well it works out.
I imagine a struct NDArray(R, uint nDims) if(isRandomAccessRange!R) { R r; /*...*/ } that defines the full set of operators for indexing and slicing, iteration by element, along any dimension etc. along with element-wise operations (lazily, via map).

I completely agree.

First release http://code.dlang.org/packages/dip80-ndslice.
See also examples at PR's description.

There is problem to build Matrix on top of NDArray/Slice:
Because generalised slicing NDArray requires N strides, but BLASMatrix only 1 (stride between elements in row always equals 1). 2DArray/Slice!2 is generalisation of BLASMatrix that requires runtime check of strides before calling C BLAS functions. We need std.container.matrix, thought.

Why can’t you get away with N-1 strides? In terms of the underlying range, one of the N strides is always 1, no?

On 13 Jun 2015, at 20:50, Ilya Yaroshenko<EMAIL_ADDRESS>wrote:
There is problem to build Matrix on top of NDArray/Slice:
Because generalised slicing NDArray requires N strides, but BLASMatrix only 1 (stride between elements in row always equals 1). 2DArray/Slice!2 is generalisation of BLASMatrix that requires runtime check of strides before calling C BLAS functions. We need std.container.matrix, thought.
—
Reply to this email directly or view it on GitHub https://github.com/D-Programming-Language/phobos/pull/3397#issuecomment-111743126.

E.g. NxN has 2 strides, (1,N), but you only need store one of them, (N). The transpose of NxN has two strides, (N,1), but again you only need one plus a flag to say "this is transposed".

In unittest Properties and methods, subsection stride property:
    auto tensor = 100.iota.sliced(3, 4, 5);

    assert(tensor.stride   == 20);
    assert(tensor.stride!0 == 20);
    assert(tensor.stride!1 ==  5);
    assert(tensor.stride!2 ==  1);

    //`matrix` can be casted to BLASMatrix
    // strides: (1, 5)
    auto matrix = tensor.back;
    assert(matrix.stride   ==  5);
    assert(matrix.stride!1 ==  1);

    //Runtime error if casting `matrix` to BLASMatrix:
    // strides: (20, 5)
    matrix = tensor.back!2;
    assert(matrix.stride   == 20);
    assert(matrix.stride!1 ==  5);

Ah, yes, I see now, my mistake.

This is awesome.
I would like a way to iterate a matrix in a foreach loop, that visits each element at a time, rather than entire rows at once.
Perhaps there is some template that implements multi-dimensional iteration logic? matrix.elementsByRow or matrix.elementsByColumn... you get the idea, except it obviously doesn't map to n-dimensions. I wonder if this sort of API would be possible/convenient for n-dimensions?

I would like a way to iterate a matrix in a foreach loop, that visits each element at a time, rather than entire rows at once.

I can add opApply. However delegates can be slow. Other option is deepEach  property like each from std.algorithm. What do you think would be better?

Perhaps there is some template that implements multi-dimensional iteration logic? matrix.elementsByRow or matrix.elementsByColumn... you get the idea, except it obviously doesn't map to n-dimensions. I wonder if this sort of API would be possible/convenient for n-dimensions?

The order of dimension can be changed with generalized transpose (last 2 examples).
    auto t0 = 1000.iota.sliced(3, 4, 5);
    auto t1 = t0.transposed!(2, 0, 1); //CTFE - recommended
    auto t2 = t0.transposed (2, 0, 1); //Runtime
    assert(t0[1, 2, 3] == t1[3, 1, 2]);
    assert(t0[1, 2, 3] == t2[3, 1, 2]);
    static assert(is(typeof(t0) == typeof(t1)));
    static assert(is(typeof(t0) == typeof(t2)));

@9il I think what @TurkeyMan wants is a byElement method. Essentially it's just a byRow.joiner (generalised for n-dims obviously), but joiner doesn't know about rectangular-ness so you only get a forward range.

Instead of offering length at all, I would prefer shape that returns a static array of the dimensions, like numpys ndarray.
All the pointer specialisation is just premature optimisation that obscures the code.
Lazy matrix operations could really take this work to the next level. The design I had come up with is as follows:
a = b; //shallow copy
a[] = b; //deep copy

auto c = <op>a;
//auto c = Slice!(N, typeof(a._range.map!"<op>a"))(a.lengths, a.strides, a._range.map!"<op>a");

auto d = c <op> b;
//auto d = Slice!(N, typeof(zip(c, b).map!"a[0]<op>a[1]"))(a.lengths, a.strides, zip(c, b).map!"a[0]<op>a[1]");

a <op>= b; //error
a[] <op>= b;
//(<op>b).byElement.copy(a.byElement);

The comments are just for showing the semantics, obviously they could be implemented in a more specialised way.
Also, any operands could be of ElementType!Range as well, e.g. adding a number to every element.
The reason why this is so great, is that it saves hugely on temporaries. Temporary allocations and copies are the bane of my life when working with traditional array-based scientific tools like numpy and matlab. This is very in keeping with Walter's "don't allocate" message.

@9il I think what @TurkeyMan wants is a byElement method. Essentially it's just a byRow.joiner (generalised for n-dims obviously), but joiner doesn't know about rectangular-ness so you sadly only get a forward range.

OK, we can add both:

Fast byElem (along side with trsansposed and slices we can do anything)

foreach(elem; tensor.byElement) {...}

Slow opApply, but with indexes.

foreach(i, j, k, elem; tensor3D) {...}

Slow opApply, but with indexes.
Why not implement an n-dimensional version of std.range.enumerate and then you can have tensor.byElement.enumerate and everyone wins, no?

Slow opApply, but with indexes.

Why not implement an n-dimensional version of std.range.enumerate and then you can have tensor.byElement.enumerate and everyone wins, no?

tensor.byElement.enumerate would not work because lengths is not defined for enumerate.
But Tuple usage is very good! Thank you for this note.
Looks like we need three options
_1. byElement
foreach(elem; tensor.byElement) {...}
foreach(ref elem; tensor.byElement) {...}

_2. byIndexedElement
foreach(i, j, k, elem; tensor3D.byIndexedElement) {...}

_3. opApply for  foreach with ref
foreach(i, j, k, ref elem; tensor3D) {...}

or this...
foreach(i, j, k, pointer; tensor3D.byIndexedPointer) {...}

I'll just state my use cases, and the most appropriate solution may follow.
I can imagine I will use this mostly to deal with images, and matrices.
In image processing, you want to pass over all pixels. I frequently want to perform some operation akin to a pixel shader, in which case the order is not particularly important, just that the loop must visit each pixel individually. That said, some control over the order is desirable in some cases, so iterating pixels by-row or by-column for images may be useful (effectively a transpose!).
You have obviously considered matrices fairly comprehensively; transpose is the only critical operation there. When considering transpose though, I wonder if other axis mirrors are also useful? Images rarely want to transpose, but they do want to mirror axiis, and they also want to rotate... if you're declaring transpose, which is effectively a tweaked iteration pattern, then perhaps mirrors and rotations should work in the same way?
I think 1, 2, and 3 are all useful.  Why is that?

Instead of offering length at all, I would prefer shape that returns a static array of the dimensions, like numpys ndarray.

length!d looks pretty along side with front!d, popBack!d, empty!d and other stuff.
But shape property can be added as well:
TypeTuple!(size_t[N], "lengths", size_t[N], "strides") shape() @property @safe pure nothrow @nogc

All the pointer specialisation is just premature optimisation that obscures the code.
Lazy matrix operations could really take this work to the next level. The bare-bones of the design I had come up with is as follows:
a = b; //shallow copy

Already works.

a[] = b; //deep copy

Already works.

auto c = a;
//auto c = Slice!(N, typeof(a._range.map!"a"))(a.lengths, a.strides, a._range.map!"a");

It is very ambiguous syntax.
Current state (with shape) for ranges (not for Sliced(size_t N, T*)):
auto c = a.range.map!"<fun>".sliced(a.shape.lengths);

What about slicedMap template?

auto d = c  b;
//auto d = Slice!(N, typeof(zip(c, b).map!"a[0]a[1]"))(a.lengths, a.strides, zip(c, b).map!"a[0]a[1]");

ditto
slicedZip?

a = b; //error

Already works (error).

a[] = b;
//(b).byElement.copy(a.byElement);

Already works. (Implemented in more specialised way)

The comments are just for showing the semantics, obviously they could be implemented in a more specialised way.

--

Also, any operands could be of ElementType!Range as well, e.g. adding a number to every element.

I don't understand this bit.

The reason why this is so great, is that it saves hugely on temporaries. Temporary allocations and copies are the bane of my life when working with traditional array-based scientific tools like numpy and matlab. This is very in keeping with Walter's "don't allocate" message.

Agreed. However your syntax looks like "I allocate" instead of "I don't allocate".

EDIT: auto sliced(...)(Range, Tuple(...) shape) should be added, thought
auto c = a.range.map!"<fun>".sliced(a.shape);

I think 1, 2, and 3 are all useful.  Why is that?

Because it calls delegate. For example: if you want to set up some integer slice to simple funciton from indexes it would be approximately two times slower then naive chain of foreach loops.
What do you think about 'byIndexedPointer'?
foreach(i, j, k, pointer; tensor3D.byIndexedPointer)
{
    pointer* = i*j+k;
}

R.e. slicedMap, slicedZip, ambiguous syntax:
It isn't super-explicit, but it is very usable and consistent.
Consider this sort of function:
auto imageAdjust(S0, S1)(S s, float gamma, float preOffset, postOffset, S1 mask)
{
    return (postOffset + (s + preOffset)^^gamma) & mask;
}```

The maths is directly visible, no allocations needed.

Compare to the same without the syntax sugar:
```D
auto imageAdjust(S0, S1)(S s, float gamma, float preOffset, postOffset, S1 mask)
{
    return slicedZip(s.range, mask.range)!(x => (postOffset + (x[0] + preOffset)^^gamma) & x[1]);
}

And then consider how it scales are more Slices are involved. Also, note that the lambda in the non-sugar version would have to be allocated as a closure.
I'm not sure how I would sell D to a normal data scientist if I told them that they can't write simple multidimensional expressions without using template lambda functions.
The other big win of supporting these basic algebraic operations is that it allows some code to be written once and support everything from scalars through to 1000-dim arrays. The imageAdjust function works just as well if you pass it two Slices or two floats, or even (if D's simd/arrayOp support was better), float4 or float[32].

What's the advantage of the pointer api vs ref? I don't think foreach syntax and pointers should mingle.

Another thought; Walter dropped the ideas of some lowering for array syntax to range functions by the compiler... consider how those ideas interact with multi-dimensional arrays?

What's the advantage of the pointer api vs ref? I don't think foreach syntax and pointers should mingle.

Nothing in terms of syntax. Only speed (Tuple vs delegates).
OK, lets droop byIndexedPointer . We can use foreach chain for speed or opApply.
It is fine because foreach without indexes can be both ref and not  (case 1).

Another thought; Walter dropped the ideas of some lowering for array syntax to range functions by the compiler... consider how those ideas interact with multi-dimensional arrays?

@TurkeyMan Please describe what do you mean.

Just a heads-up that I've edited my previous post to clarify some things, I'll delete this message shortly.

Just a heads-up that I've edited my previous post to clarify some things, I'll delete this message shortly.

Nitpick: just slicedZip(s, mask).
Generally I like your syntax! But I have doubts that it would be accepted.
What about ping someone from D core team?

Nitpick: just slicedZip(s, mask).

fixed

Generally I like your syntax! But I have doubts that it would be accepted.

ping @andralex @WalterBright any opinions on having operator overloading such that ndSlice0 + ndSlice1 is a lazy operation? See https://github.com/D-Programming-Language/phobos/pull/3397#issuecomment-112013506 and https://github.com/D-Programming-Language/phobos/pull/3397#issuecomment-112032238 for more details.
Having easy syntax is critical to compete the incumbents in this space, especially when attracting newcomers, and having it lazy by default will dramatically reduce the number of allocations in peoples code, which is pretty much guaranteed to show nice speedups and allow handling of larger datasets.

auto imageAdjust(S0, S1)(S0 s, float gamma, float preOffset, float postOffset, S1 mask)
{
    return (postOffset + (s + preOffset)^^gamma) & mask;
}

@John-Colvin What about this way:
slicedZip(s, mask).slicedMap!(a => imageAdjust(a[0], gamma, preOffset, postOffset, mask));

Is not so bad)

Other point to reject lazy syntax like ndSlice0 + ndSlice1  syntax is a huge template bloat.
Note: Curently this syntax ndSlice0 + ndSlice1 is not supported at all, only opOpAssign syntax like ndSlice0 += ndSlice1 is supported.

That would have to be

slicedZip(s, mask).slicedMap!(a => imageAdjust(a[0], gamma, preOffset, postOffset, a[1]));

which of course you would wrap in an overload of imageAdjust. While that works for smoothing over the API of a given function, it doesn't really help writing ndArray expressions, it directly avoids writing them at all! People definitely want to write their expressions inline.
R.e. template bloat.
Mostly these should be inlined and hopefully some co-operation between compiler and linker would remove them (ldc already does some of this by default IIRC). They could even be forced inline, they're not going to be big code in-and-of-themselves. I'm not convinced template bloat is a real problem in this case, and if it is then people always have the option of doing eager manipulation instead. At least in the sort of work I do, binary size and template bloat disappear way down the list of priority problems compared to memory allocation.

Inlining can help with size of executable, but not with compilation time and memory usage.
imageAdjust  contains 4 operation, witch would produce 4 template instantiations Slice.
Can you assess compilation time and compiler memory usage for program with 10_000...1_000_000 of operations. Of corse number of installations would lower, but not significantly.
Another point that D array does not have such syntax.
Current Slice have not too, so user would be enforced to used foreach chain or slicedMap/slicedZip templates. I think there is no problem with memory allocations, only with syntax simplicity.

Anyway we need slicedMap and slicedZip for features like lazy array, because other math functions like sin and user functions can not be expressed through operator syntax.

Inlining can help with size of executable, but not with compilation time and memory usage.
imageAdjust contains 4 operation, witch would produce 4 template instantiations Slice.

Can you assess compilation time and compiler memory usage for program with 10_000...1_000_000 of operations. Of corse number of installations would lower, but not significantly.

Considering that std.algorithm and std.range are pretty central to modern idiomatic D, i'd say that a) it's too late and we're already doomed by this problem or b) a solution will be found the fixes the problem, both for here and for range/algorithm

Another point that D array does not have such syntax.
This is the sort of the thing @WalterBright was interested in adding, as mentioned by Manu(@TurkeyMan). I think the Slice type is a good place to introduce this syntax. Who knows, perhaps in time all of this Slice semantics will become part of the builtin arrays anyway!

Anyway we need slicedMap and slicedZip for features like lazy array, because other math functions like sin and user functions can not be expressed through operator syntax.

We definitely do need these anyway, or an equivalent. Are there any inherent advantages to having slicedMap as opposed to byElement.map using std.algorithm.map?

We definitely do need these anyway, or an equivalent. Are there any inherent advantages to having slicedMap as opposed to byElement.map using std.algorithm.map?

*.byElement is a range over the elements without Slice interface.
If we implement this interface through some getSlice it would be hidden by std.algorithm.map.
But naming is good. What is better slicedMap or byElementMap or something else?
P.S. shape was implemented.

*.byElement is a range over the elements without Slice interface.

Of course, my mistake.

But naming is good. What is better slicedMap or byElementMap or something else?

why not just map?

P.S. shape was implemented.

:+1:

why not just map?

Slice has Range interface. So it should work with std.algorithm.map like an common range of Slices (N > 1) or elements N == 1.

Slice has Range interface. So it should work with std.algorithm.map like an common range of Slices (N > 1) or elements N == 1.

I'm not 100% sold on how useful that is. I'd just as happily call something like byDim(size_t pos)() to get a range over N-1 dimensional chunks.

I'm not 100% sold on how useful that is. I'd just as happily call something like byDim(size_t pos)() to get a range over N-1 dimensional chunks.

We can't remove this syntax because foreach loops is based on range concept.
Other option is that we can extend implementation of std.algorithm.map and std.range.zip with Slice concept.
byDim(size_t pos)() should be byDim(size_t pos)(size_t index). Current implementation allows front!pos/back!pos or simple  nd-slicing.
The idea looks good.
What about this generalisation:
template projection(Dimensions...)
{
     auto projection(Indexes)(Indexes indexes)
         if (Indexes.length == Dimensions.length)
     {
           ...
     }
}

?
Also runtime version of projection can be added (like CTFE and Runtime transposed).

Hmm... seems extend implementation of std.algorithm.map and std.range.zip with Slice concept is bad idea because users may want to use like for common ranges.

We can't remove this syntax because foreach loops is based on range concept.

I'm suggesting that Slice wouldn't itself be iterable, you'd have to use byDim to explicitly say what iteration you wanted. It doesn't really matter though, having the iteration built-in is probably worth it for convenience.
I think I'm missing something, what would index be for? Could you provide a small test showing projection in use?

shape returning something other than a size_t[N] of lengths seems odd to me. If it is useful to have to have the strides there too, perhaps it could return this:
struct Shape(size_t N)
{
    size_t[N] lengths;
    size_t[N] strides;
    alias lengths this;
}

From the outside one shouldn't have to care about the strides, it's the lengths that are interesting.
Alternatively, the existing shape could be renamed structure and shape would just return the lengths, like it does in numpy.

We can't remove this syntax because foreach loops is based on range concept.

I'm suggesting that Slice wouldn't itself be iterable, you'd have to use byDim to explicitly say what iteration you wanted. It doesn't really matter though, having the iteration built-in is probably worth it for convenience.
I think I'm missing something, what would index be for? Could you provide a small test showing projection in use?

I misunderstood you, my mistake.
OK, byDim looks fine! So what I need is replace (size_t pos = 0) to (size_t pos).
With byDim we can surely extend current map and zip.
Thanks!
Projection example:
auto tensor = 100.iota.sliced(3, 4, 5);
size_t index = 3;
auto matrix0 = tensor.projection!1(index);
auto matrix1 = tensor.back!1;
assert(matrix0.equals(matrix1));

Alternatively, the existing shape could be renamed structure and shape would just return the lengths, like it does in numpy.

OK

A not on the lazy range operations: The way I currently see them, they aren't a blocker for getting this done, they're are a totally additive change that would be built on top of map and zip. They could even by added by a wrapper type.

I was going to ask "What value do the front, back, popFront etc. functions add over indexing and assignment?", then i realised that they are much more usable when the number of dimensions is high. It might be worth thinking about what operations would be hard with the current API when there are lots of dimensions. I guess in that case you could manually build the slicing and then pass it to opIndex, but there'd often be a lot of redundant work in there, specifying [0..$] or [0] or [$-1] for a large number of indices. Hopefully the optimiser would clear it up, but it's not ideal.

I was going to ask "What value do the front, back, popFront etc. functions add over indexing and assignment?", then i realised that they are much more usable when the number of dimensions is high. It might be worth thinking about what operations would be hard with the current API when there are lots of dimensions. I guess in that case you could manually build the slicing and then pass it to opIndex, but there'd often be a lot of redundant work in there, specifying [0..$] or [0] or [$-1] for a large number of indices. Hopefully it wouldn't require much manual writing and the optimiser would clear it up, but it's not ideal.

Combination of

projection described above and generalised to multiple indexes
generalized transpose (implemented)
generalized byDim described here

can help to remove all [0..$] or [0] or [$-1] or [i] from slice!
byDim is very useful stuff!
auto tensor = 1000.iota.sliced(5, 6, 7, 8);
//foreach(matrix; tensor.byDim (1, 3)) // Runtime
  foreach(matrix; tensor.byDim!(1, 3)) // CTFE
{
    assert(matrix.shape == [5, 7]);
}

///indexed
//foreach(i, j, matrix; tensor.byDimIndexed (1, 3)) // Runtime
  foreach(i, j, matrix; tensor.byDimIndexed!(1, 3)) // CTFE
{
    assert(matrix.shape == [5, 7]);
}

Also you can use byDim in combination with existing std.range!

... and what we will do with front, back, popFront?
The idea:

Retain them in Slice
Extend current map and zip with 'by element' style for Slice
Cowboy map and zip with generalized byDim!

What do you think?

EDIT: more generalisation:
Generalised byDim should be splitte into two steps splitDim!(...).byElement

splitDim -- creates ndslice over ndslice ( without any allocations, very fast).
byElement -- iterates over ndslice where elements are ndslices

Whooohoo

Extend current map and zip with 'by element' style for Slice
Cowboy map and zip with generalized byDim!

I don't totally understand what you mean by these two

I have a suggestion for future work. The following code does not compile:
void main()
{
int[][] x;
x.length = 2;
x[0] = [0, 1];
x[1] = [2, 3];
auto y = x.dup;
y[][] += 4; //this fails, y[] += 4 doesn't work either
}
Looping through y[0][] and y[1][] (and so on more generally) solves the problem. It's like the array operations are only 1-dimensional. The simplest things I can think of is to overload the binary operators with the foreach by element or alternately specializing the operators for 2+ dimensional arrays to recursively loop through each dimension. It might also be that more substantive changes may be required for speed (giving the compiler the ability to make some optimizations, I don't know).
Even if that is improvement is made, I would be sort of worried about y[][] instead of y[]. I haven't thought it through thoroughly, but I think I would prefer y[] applying the operation to every member of the array. It becomes hard to write generic code for y[][] and y[][][]. One alternative might be a kind of special operator that will expand it, like where y[:] is equivalent to y[] for 1-d, y[][] for 2-d, y[][][] for 3-d, etc. I have no idea how to implement that though.

@jmh530  Current state y[] += 4; compiles for nd-Slices.

@9il Awesome! Now that I think about it, I think I had seen this before you moved the documentation to the other page, but then when I was looking for it last night I had forgotten (i.e., I'm an idiot).

Has somebody cooked up any linear algebra types on top of this?

Has somebody cooked up any linear algebra types on top of this? I want to use and recommend usage of this work :)

Answer:

There is problem to build Matrix on top of NDArray/Slice:
Because generalised slicing NDArray requires N strides, but BLASMatrix only 1 (stride between elements in row always equals 1). 2DArray/Slice!2 is generalisation of BLASMatrix that requires runtime check of strides before calling C BLAS functions. We need std.container.matrix, thought.

Why can’t you get away with N-1 strides? In terms of the underlying range, one of the N strides is always 1, no?

E.g. NxN has 2 strides, (1,N), but you only need store one of them, (N). The transpose of NxN has two strides, (N,1), but again you only need one plus a flag to say "this is transposed".

In unittest Properties and methods, subsection stride property:

    auto tensor = 100.iota.array.sliced(3, 4, 5);

    assert(tensor.stride   == 20);
    assert(tensor.stride!0 == 20);
    assert(tensor.stride!1 ==  5);
    assert(tensor.stride!2 ==  1);

    //`matrix` can be casted to BLASMatrix
    // strides: (1, 5)
    auto matrix = tensor.back;
    assert(matrix.stride   ==  5);
    assert(matrix.stride!1 ==  1);

    //Runtime error if casting `matrix` to BLASMatrix:
    // strides: (20, 5)
    matrix = tensor.back!2;
    assert(matrix.stride   == 20);
    assert(matrix.stride!1 ==  5);

So, we can implement cast between Slice and future Matrix in the future.

However Level 1 and Level 2 BLAS functions can be easily simulated with current Slice with other Phobos.

A lot of stuff was added: everted, partial transpose (aka generalised byDim), packed, unpacked, packEverted and byElement. byElement can be used in combination with packed.
opApply postponed until the relaxation of foreach overloading.
See the last commit for documentation (CyberShadow/DAutoTest -> Details -> Phobos -> menu -> experimental -> range -> ndslice).
Ping @John-Colvin @denis-sh @TurkeyMan @nordlow for the next review iteration.
Thanks!

Maybe I'm a bit late to the party, but my code extensively uses N (mostly 2-3) dimensional arrays based on @denis-sh implementation. My arrays represent points in space and as I need to often check values of neighbouring sites (also of other arrays) I've since hacked my fork to allow arr[p] and foreach(immutable p, ref pop; arr) { ... } where p is now size_t[N]. Is this something that could be done using your implementation (or is there a reason I shouldn't want this at all)?

@SFrijters
Current implementation allows code like
import std.range: enumerate;
...
foreach(i, matrix; tensor3D.enumerate)
    foreach(j, row; matrix.enumerate)
        foreach(k, ref elem; row.enumerate)
        {...}

The code like following can be supported after a DMD change:
foreach(i, j, k, ref elem; tensor3D) {...}

Can this implementations replace your fork?

Well, the thing is, my code is currently generic with respect to the dimensionality of my arrays, which is chosen at compile time. So the alternative would be to use N nested foreach loops in the first case (recursive template?), or in the latter (future) case I would have to... do something with TypeTuples? I'm not sure. It would look a lot worse in either case I think. Of course, as long as nothing get broken I can keep using my current code, but I would prefer to do as much as possible with just the standard library.
As an aside, foreach(elem, parallel(arr)) should work, right? As it would only need opApply and opIndex?
If I find some time I will try to look into this some more.

Well, the thing is, my code is currently generic with respect to the dimensionality of my arrays, which is chosen at compile time. So the alternative would be to use N nested foreach loops in the first case (recursive template?), or in the latter (future) case I would have to... do something with TypeTuples? I'm not sure. It would look a lot worse in either case I think. Of course, as long as nothing get broken I can keep using my current code, but I would prefer to do as much as possible with just the standard library.

OK, looks like we need your variant of iteration too.

As an aside, foreach(elem, parallel(arr)) should work, right? As it would only need opApply and opIndex?

Yes is should work. Slice is Random Access Range without opApply.

If I find some time I will try to look into this some more.

merge conflicts...

Rebased.
@John-Colvin  What common name would be better for opApply iterators? apply?
foreach(size[3] p, ref elem; tensor3D.apply) { ... }

foreach(i, j, k, ref elem; tensor3D.apply) {...}
foreach(i, j, k, elem; tensor3D.apply) {...}
foreach(i, j, vector; tensor3D.apply) {...}
foreach(i, matrix; tensor3D.apply) {...}

@9il why can't we define opApply directly in Slice? Is this the dmd/language restriction you were talking about?

Yes, opApple overloads front, empty, popFront. However if opApple can be inlined then this is not a problem thought...

Do you mean foreach selects opApply instead of the range interface? Hmm...
A few brief tests suggest that gdc and ldc can successfully inline opApply and - more importantly - the delegate it's passed. @ibuclaw @klickverbot @redstar Are there any particular situations you know of where the inlining will fail and make performance worse than it would be using a range?

Yes.
Great! Slice slicing and indexing would be very optimised after static foreach, so I can add opApply thought :)

@John-Colvin Type deduction doesn't work for opApply template, but not template opApply should not to be @nogc nothrow and etc :(

Is there a bug filed on issues.dlang.org for the type deduction failure?

Yes. https://issues.dlang.org/show_bug.cgi?id=14196
Looks like we need ideas for new DIP.

How about this as a workaround:

Make a templated implementation of opApply (call it opApplyImpl)
Create an overload set of non-templated opApplys as aliases to instantiations of opApplyImpl with all the different combinations of attributes for the delegate.

5 templates X nogc X safe X pure X nothrow = 20 functions per only 1 dimension. Looks b

@9il well it would all be auto-generated code, you wouldn't have them written out manually. It doesn't look great, but it gets the job done and there's no big practical disadvantage to it that I can think of. Perhaps in the future there will be a better way, but for now...

I ran into the attribute problem using Denis' multidimensional arrays. Since I already have them wrapped in a struct anyway (with array being alias this'ed), I handled it by implementing a second opApply at struct level, so I could either foreach(struct) -- which is @nogc etc, or foreach(struct.arr), which isn't. It would be lovely if this problem was handled properly, but any implementation is bound to be an upgrade to my current one...

Essentially this pull defines the generic interface for multidimensional homogeneous or pseudo-homogenous iteration in D. It should be easy for people to create their own implementation that defines the same primitives. There are a lot of other paths to a multidimensional range other than the 1-D -> N-D that is implemented here.

byElement should support slicing, indexing and opIndexAssign, opIndexOpAssign etc.

@9il In numpy one can write a = data[::-1] to reverse the order of iteration. How would I, for example, translate the numpy expression data[3:5, 1:7:-1, 3:] to D using Slice?

@9il In numpy one can write a = data[::-1] to reverse the order of iteration. How would I, for example, translate the numpy expression data[3:5, 1:7:-1, 3:] to D using Slice?

data[3:5, 1:7:-1, 3:] would be data[3..4, 1..7, 3..$].map!retro. However the result would not be a Slice. Thank you for the good example! I will try to add negative shifts :)

@9il In numpy one can write a = data[::-1] to reverse the order of iteration. How would I, for example, translate the numpy expression data[3:5, 1:7:-1, 3:] to D using Slice?

I have added corresponding functionality https://github.com/9il/phobos/commit/fe189143bcc050c0127880e88c2f6dae4ff842b9
No Python's code data[3:5, 1:7:-1, 3:]  can be translated to
//CTFE optimized
auto a = data[3..5, 1..7, 3..$].reversed!1;
//Runtime
auto a = data[3..5, 1..7, 3..$].reversed(1);

EDIT: However reversed works only for Slices over arrays, because D ranges has no support of negative indexes.

@9il can you push these changes to the dub repo so people can test them?

@9il I've always thought negative striding (as part of an API) is a questionable pattern*, so I like the explicit use of reversed. :+1:
* Because negative strides don't combine multiplicatively, which is very easy to miss when it's so easy e.g. for 1-D data: data[::5] == data[::-1][::-5] == data[::-5][::-1] is only true in special cases (data happens to match up or (data.size - 1) % 5 == 0). If strides are always positive, they are much easier to reason about.
I can't see any (public) way of doing striding like in the following numpy expression data[::2, ::3, ::42]. I think we need a multi-dimensional generalisation of std.range.stride.
I don't understand why reverse can't be done for (bidirectional?) ranges in general, could you explain?

@JackStouffer Pushed. DUB will add new (0.0.14) version soon.

@JackStouffer I forgot to add v. v0.0.15 is available now.

@9il Thanks!

Because negative strides don't combine multiplicatively, which is very easy to miss; e.g. for 1-D data: data[::5] == data[::-1][::-5] == data[::-5][::-1] is only true in special cases (data happens to match up or (data.size - 1) % 5 == 0). If strides are always positive, they are much easier to reason about.

Negative strides are used in reversed internally. So it is a reason why they are available only for arrays, but not available for ranges. Implementation details:
// _range is a pointer
 _range += _strides[index] * (_lengths[index] - 1);
 _strides[index] = -_strides[index];

However, it works like it should work.

I don't understand why reverse can't be done for (bidirectional?) ranges in general, could you explain?

It can be done for random access ranges with additional parameter (shift). OK, I will do it :)

I can't see any (public) way of doing striding like in the following numpy expression data[::2, ::3, ::42]. I think we need a multi-dimensional generalisation of std.range.stride.

Good point! There are public way, but it is very complicated.
strided was added :
// a = data[::2, ::3, ::42]
auto a = data.stride(0, 2).sritde(1, 3).stride(2, 42);

The name is strided (with d) because stride property exists.

I don't understand why reverse can't be done for (bidirectional?) ranges in general, could you explain?

It can be done for random access ranges with additional parameter (shift). OK, I will do it :)

Implemented  v0.0.17. Works for any random access range. Probably Slice should work for any struct/object with only defined auto opIndex(size_t index).

Optimisation with LDC and latest v0.0.18 looks amazing!
void fun(Slice!(4, double*) slice)
{
	slice
		.transposed!(3, 2)
		.packed!(2)
		.packEverted
		.unpacked
		.swapped!(1, 3)
		.strided(2, 3)
		.reversed!(0)
		.everted
		[8, 0..$, 7, 2..$] *= 10;
		;
}

LDC optimisation:
	push	r14
	push	rbx
	mov	rdi, qword ptr [rsp + 24]
	mov	rcx, qword ptr [rsp + 48]
	mov	r8, qword ptr [rsp + 80]
	mov	r9, qword ptr [rsp + 72]
	mov	rsi, qword ptr [rsp + 56]
	mov	r11, qword ptr [rsp + 64]
	mov	r10, qword ptr [rsp + 88]
	movabs	rdx, -6148914691236517205
	mov	rax, rcx
	mul	rdx
	shr	rdx
	lea	rax, [rdx + 2*rdx]
	cmp	rcx, rax
	je	LBB0_2
	inc	rdx
	jmp	LBB0_3
LBB0_2:
	cmp	rcx, 3
	jb	LBB0_8
LBB0_3:
	cmp	rdi, 2
	je	LBB0_8
	mov	r14d, 2
	sub	r14, rdi
	add	rdi, -3
	imul	rdi, rsi
	lea	rax, [rdi + 8*r11]
	imul	rcx, r9, 7
	add	rcx, rax
	lea	rax, [r10 + 8*rcx]
	shl	r8, 3
	lea	rdi, [r8 + 2*r8]
	shl	rsi, 3
	neg	rsi
	movsd	xmm0, qword ptr [rip + LCPI0_0]
	.align	4, 0x90
LBB0_5:
	mov	rcx, rax
	mov	rbx, r14
	.align	4, 0x90
LBB0_6:
	movsd	xmm1, qword ptr [rcx]
	mulsd	xmm1, xmm0
	movsd	qword ptr [rcx], xmm1
	add	rcx, rsi
	inc	rbx
	jne	LBB0_6
	add	rax, rdi
	dec	rdx
	jne	LBB0_5
LBB0_8:
	pop	rbx
	pop	r14
	ret

Looks great. This is a critical building block for numerical work in D, thank-you so much for putting in so much time here.

Almost all the functions need more detailed descriptions
The module itself needs documenting with a full description of the general behaviour and some motivating examples

@John-Colvin Thank you for many good advices for this PR :)
Agreed, hope to add docs during this weekend.

@John-Colvin Could you please review updated docs?
http://dtest.thecybershadow.net/artifact/website-7a646fdea76569e009844cdee5c93edab10980ca-cdcb694c40ec47acf88b0aa878e1623e/web/phobos-prerelease/std_experimental_range_ndslice.html

Wanted: Review Manager
<EMAIL_ADDRESS>

@9il Did you want to start the review process right away?

@9il Did you want to start the review process right away?

@JackStouffer Yes, if you have not additional remarks, then review can be started.

@9il I'm very busy right now, but I'll have a proper look later this week.

@John-Colvin  Thanks!

It seems that the main file is large enough that it won't let me make comments on individual lines, so I guess I have to write them here. I have only played with the API so far and haven't yet studied the code, so here are some things based on what I've seen:

For the member functions of Slice that are annotated with @property, you are specifying @safe nothrow pure @nogc when this would be automatically inferred
does alias N = _N in Slice need to be documented?
why does the parameter _length for createSlice have a leading underscore?
missing Params and Returns throughout
The docs for the packed family of functions doesn't really do a good job of explaining what they do IMO. For example, could you be more specific on what "Packs a slice into the composed slice" means?

@JackStouffer Thank you for remarks!

For the member functions of Slice that are annotated with @property, you are specifying @safe nothrow pure @nogc when this would be automatically inferred

fixed.

does alias N = _N in Slice need to be documented?

fixed.
N and Range are private now. User can use slice.shape.length and is instead.

why does the parameter _length for createSlice have a leading underscore?

fixed.

missing Params and Returns throughout
The docs for the packed family of functions doesn't really do a good job of explaining what they do IMO. For example, could you be more specific on what "Packs a slice into the composed slice" means?

I will detailed descriptions tomorrow.

for<EMAIL_ADDRESS>auto makeSlice(T, Allocator, Lengths ...)(auto ref Allocator alloc, Lengths lengths)
{
    import std.experimental.allocator : makeArray;
    return alloc.makeArray!T(lengths);
}

@John-Colvin @burner
Constructor for allocators:
/++
Allocates array and n-dimensional slice over it.
Params:
    alloc = allocator, see also $(LINK2 std_experimental_allocator.html, std.experimental.allocator)
    lengths = list of lengths for dimensions
Returns: `array` created with `alloc` and `slice` over it
See_also: $(LREF sliced)
+/
auto makeSlice(T, Allocator, Lengths...)(auto ref Allocator alloc, Lengths lengths)
{
    //Can not use `Tuple` because `slice` is name of the `slice` method in `Tuple`.
    struct Result { T[] array; Slice!(Lengths.length, int*) slice; }
    size_t length = 1;
    foreach(len; lengths)
        length *= len;
    import std.experimental.allocator: makeArray;
    T[] a = alloc.makeArray!T(length);
    return Result(a, a.sliced(lengths));
}

///
unittest {
    import std.experimental.allocator;
    auto tup = theAllocator.makeSlice!int(2, 3, 4);

    static assert(is(typeof(tup.array) == int[]));
    static assert(is(typeof(tup.slice) == Slice!(3, int*)));

    assert(tup.array.length           == 24);
    assert(tup.slice.elementsCount    == 24);
    assert(tup.array.ptr == &tup.slice[0, 0, 0]);

    theAllocator.dispose(tup.array);
}

@John-Colvin @burner
Constructor for allocators:

Allocators cause linking error on Win32 :-(
 Error 42: Symbol Undefined _D3std12experimental9allocator12theAllocatorFNdZC3std12experimental9allocator10IAllocator
unittest2.obj(unittest2)
 Error 42: Symbol Undefined _D3std12experimental9allocator12__ModuleInfoZ

https://issues.dlang.org/show_bug.cgi?id=15281

What about immutable?

@lablanu static arrays are not ranges, so they can't, and shouldn't, be supported.

@9il What's the unit test coverage percent? Is it possible to get it to 100%?

@lablanu static arrays are not ranges, so they can't, and shouldn't, be supported.

@lablanu
However you can get common slice from static array and then make Slice over it.
size_t[100] data;
auto slice = data[].sliced(2, 3, 4);

@lablanu static arrays are not ranges, so they can't, and shouldn't, be supported.

sliced works with immutable arrays. I have added checks: https://github.com/D-Programming-Language/phobos/commit/3cb0c0d6de942d240e1f283ad664c675157937c3 and https://github.com/9il/phobos/commit/390b96a1b0d42cdb442c8e881a3690d638ef878d

@9il What about assigning from a Slice shell, like immutable int[5][2] data = 10.iota.sliced(2, 5);?

@9il What about assigning from a Slice shell, like immutable int[5][2] data = 10.iota.sliced(2, 5);?

You can get common a dynamic array using transposed and then ndarray/ cast(int[][]).

Maybe I ran into a bug, as immutable int[5][2] data = 10.iota.sliced(2, 5).ndarray; gives the error
Error: cannot implicitly convert expression (ndarray(sliced(iota(10), 2, 5))) of type int[][] to immutable(int)[]

When pragma(msg, typeof(data)) gives immutable(int[5][2])

Maybe I ran into a bug, as immutable int[5][2] data = 10.iota.sliced(2, 5).ndarray; gives the error
Error: cannot implicitly convert expression (ndarray(sliced(iota(10), 2, 5))) of type int[][] to immutable(int)[]
When pragma(msg, typeof(data)) gives immutable(int[5][2])

Yes, looks like bug.
BTW: sliced(5, 2), not sliced(2, 5).

https://issues.dlang.org/show_bug.cgi?id=15359

@9il What's the unit test coverage percent? Is it possible to get it to 100%?

Yes, it is already near 100% :-)

Nice

I created this function:
import std.conv : to;
enum Zero(T) = 0.to!T;
enum One(T) = 1.to!T;

import std.meta : allSatisfy;
import std.traits : isNumeric;
auto identity(T = int, X, Y)(X x, Y y) if (allSatisfy!(isNumeric, X, Y))
{
	import std.range : iota, sequence, take;
	import std.algorithm : map;
	return x.iota.map!(i => sequence!((a, n) => n == i ? One!T : Zero!T).take(y));
}

writeln(identity!int(2, 2)); prints [[1, 0], [0, 1]]. Could it be possible to use ndslice functions with identity?

Could it be possible to use ndslice functions with identity?

ndslice constructors require an entity with random access like [1, 0, 0, 1]. So it is possible to rewrite identity to return a plain range.

@9il Could an ndarray overload that requires the N template parameter be made that converts ranges of arbitrary dimensionality to the builtin array types?

@9il Could an ndarray overload that requires the N template parameter be made that converts ranges of arbitrary dimensionality to the builtin array types?

Yes, you can use IsInputRange to write template.

I'm resisting the urge to recommend more functions to add, a pretty large subset of numpy's functions are useful but they are easy to add later.
The ones that should be done now are the ones that need to make use of private data or implementation details to be efficient. E.g. perhaps reshape.

The ones that should be done now are the ones that need to make use of private data or implementation details to be efficient. E.g. perhaps reshape.

@John-Colvin Could you please list this functions?
So I can add them to ndslice or add corresponding example for users with numpy experience.
Does this code cover reshape functionality?
import std.range: iota, array;
auto slice = 6.iota.sliced(2, 3);
// this should be pretty efficient
auto newShape = slice.byElement.array.sliced(3, 2);

Hmm.. yes, like separate function reshape would be a little bit faster.

So I can add elementsArray

Returns: reshaped_array : ndarray
This will be a new view object if possible; otherwise, it will be a copy.

This is not very good for Slice because it can be used for static data (slice of static array) and arrays allocated with malloc. So reshape should always allocate additional memory, thought.

In many pieces of code I come across (scientific NumPy/Julia), the fact that reshape just creates a view is rather essential.

OK, would be implemented.

In the past when I've written identity functions, it was always easier to
do it with an existing matrix since you can just loop by the number of
dimensions. I'm finding it rather clumsy to do it with ranges since you
need to index the range by something like
auto index = iota(n).map!(a => ((a + 1) * (a  + 1)) - 1);
I would be interested to see how you end up doing it.
It might be easier if it were possible to move through a range based on
some index (I tried something with enumerate, but it increasingly became
complicated). That would make it easier to do things like write diag and
trace functions as well as deal with triagonal arrays.
On Wed, Nov 18, 2015 at 5:06 PM, Ilya Yaroshenko<EMAIL_ADDRESS>wrote:

Could it be possible to use ndslice functions with identity?
ndslice constructors require an entity with random access like [1, 0, 0,
1]. So it is possible to rewrite identity to return a plain range.
—
Reply to this email directly or view it on GitHub
https://github.com/D-Programming-Language/phobos/pull/3397#issuecomment-157879823
.

Slice/Matrix/BLAS dev guide<EMAIL_ADDRESS>Destroy!

Adding a code example in the docs that shows how to take an already existing multi-dimensional array and wrap it in a slice would be useful.

Adding a code example in the docs that shows how to take an already existing multi-dimensional array and wrap it in a slice would be useful.

OK, but it would be convert operation with reallocation,  because Slice requires uniform strides.

OK, but it would be convert operation with reallocation, because Slice requires uniform strides.

Makes sense, but judging by the number of times I see np.asarray used on already existing multi-dimensional arrays, it's needed.

I am going to add examples from numpy reshape based on reshaped,diag, diagflat, eye, fliplr, flipud, rot90, tri, triu, tril, vander, histogram2d, mask_indices, tril_indices,
tril_indices_from, triu_indices, triu_indices_from.

@9il if you format your TODO list like this
- [ ] reshaped

Then you get this nice checkbox and people can see the number of items to complete from the PR list page.

Looks like module should be split to:
std.ndslice.package
std.ndslice.slice
std.ndslice.structure
std.ndslice.primitives
std.ndslice.operators
std.ndslice.iterators or std.ndslice.ranges - 1D iterators and nd-triangular masks over Slice
std.ndslice.allocators

A lot of these commits are rather small and can be squished.

I will squish them all :-)

I believe it would be more intuitive if the function names would be in a verb form (e.g. rotated -> rotate, everted->evert) and if byElementwereflatten`.
Structs should remain in the adjective form.

Mutating algorithms (ref T) should be in verb form (rotate)
Non-mutating algorithms (in T) should be in adjective form (rotated)

@nordlow @lablanu I agree with nordlow, although phobos is somewhat inconsistent about this.

All ndslice are non-mutation except (op)= overloads.

Type-mutation (not data) are in verb (reshape) or noun (blocks) forms. They all are located in ndslice.selection.
Algorithm that save type are in adjective form (rotated) except drop family. They all are located in  ndslice.iteration.

@John-Colvin I have added zip-like functionality:
Declaration example:
import std.algorithm.comparison: equal;
import std.experimental.ndslice.selection: byElement;
import std.range: iota;

auto alpha = 12.iota;
auto beta = new int[12];

auto m = sliced!("a", "b")(alpha, beta, 4, 3);
foreach(r; m)
    foreach(e; r)
        e.b = e.a;
assert(equal(alpha, beta));

beta[] = 0;
foreach(e; m.byElement)
    e.b = e.a;
assert(equal(alpha, beta));

Zip example with assumeSameStructure:
    import std.algorithm.comparison: equal;
    import std.experimental.ndslice.selection: byElement;
    import std.range: iota;

    auto alpha = 12.iota   .sliced(4, 3);
    auto beta = new int[12].sliced(4, 3);

    auto m = assumeSameStructure!("a", "b")(alpha, beta);
    foreach(r; m)
        foreach(e; r)
            e.b = e.a;
    assert(equal(alpha, beta));

    beta[] = 0;
    foreach(e; m.byElement)
        e.b = e.a;
    assert(equal(alpha, beta));

@kyllingstad @John-Colvin I will ask @JackStouffer to open voting after 16 hours (I need to check English first). If you have some notes, please let me know. Code size was grown from 1.5 LOC to 4.7 LOC. Please use the latest CyberShadow/DAutoTest docs.

Having ReplaceArrayWithPointer is a pity. Is it transitive? The most basic example would be static a = [1].sliced!(ReplaceArrayWithPointer.no)(1).sliced(1);

in the docs for ndslice.slice.sliced:
size_t shift	number of the first element in a range; first shift elements are ignored
doesn't read well. I suggest
size_t shift       the first shift elements of range are ignored.

"slice tuples" need more explanation.

please drop opCast and use a function instead. cast should be a warning sign for user-verified type-system circumvention, not a commonplace tool for rearranging data.
Note also that what you're doing here isn't in fact a cast in any way I'm familiar with: you're chasing pointers, copying data and evaluating arbitrary code (range API could be doing anything).
It's great to have this functionality, but please don't call it cast

ndslice.selection still has russian in it ("Селекторы создают новый тип итератора для тех же данных. Сами данные остаются неизменными (to translate).")

In general, reading the documented unittests is difficult. Many rely on already knowing how many other functions in this package work. It would be good if the first example in each function was a minimal example using array literals.

evert is a cool word, but it needs explanation. I'd never heard of it until you used it in this PR.

internal is listed in the sidebar but is 404

Having ReplaceArrayWithPointer is a pity. Is it transitive? The most basic example would be static a = [1].sliced!(ReplaceArrayWithPointer.no)(1).sliced(1);

Yes, it is transitive.
    static a = [1].sliced!(ReplaceArrayWithPointer.no)(1).sliced(1);
    static assert (is(typeof(a) == Slice!(1, Slice!(1, int[]))));
    static assert (is(typeof(a.unpack) == Slice!(1, int[])));

    static b = [1].sliced!(ReplaceArrayWithPointer.no)(1, 1).sliced(1, 1);
    static assert (is(typeof(b) == Slice!(2, Slice!(2, int[]))));
    static assert (is(typeof(b.unpack) == Slice!(3, int[])));

Perhaps we could have the default as ReplaceArrayWithPointer.no until CTFE support improves? It's good to keep things as small as possible, but it's only 8 extra bytes and it won't make any difference to most programs, considering the already quite large size of the struct.

Because iteration operators like reversed and transposed, Slice can works only with types that have behaviour like pointer. Pointer can be iterated both forward and backward multiple times. Simple example:
       0123 - data, sequential (byElement) access:
step 1: ^
step 2:^
step 3:   ^
step 4:  ^

So, any random access range or array simulates pointer:
struct PtrShell(Range)
{
    size_t _shift; // + 8 extra bytes
    Range range; // + 8 extra bytes
    ...
}

It is larger by 16 bytes. However, the real problem is performance. Processor needs to store _shift in register and add it to a pointer every time. Furthermore, ndslice would be be very fast with LDC, probably faster then any other similar product, and performance by default is more important than CTFE by default for users, which work with data.

please drop opCast and use a function instead. cast should be a warning sign for user-verified type-system circumvention, not a commonplace tool for rearranging data.
Note also that what you're doing here isn't in fact a cast in any way I'm familiar with: you're chasing pointers, copying data and evaluating arbitrary code (range API could be doing anything).
It's great to have this functionality, but please don't call it cast
edit: This is the only blocker for me.

Agreed. It would be completely removed. Any code that allocates data would be added only after allocators support for std.array: array. User can copy-past ndarray example for now.

@JackStouffer @9il:
Just in case: please don't start a vote while the auto-tester is failing.

@JackStouffer Please start official voting!
The latest docs http://dtest.thecybershadow.net/artifact/website-13cbdcf17d84fc31328c3f517a56bea783c418d6-dd2292a424959b594956eeeba64d391f/web/phobos-prerelease/std_experimental_ndslice.html
@John-Colvin Thank you for review! I have added simplified tests for strided and reversed; description for slice; and fixed a lot of docs with translator.

@9il<EMAIL_ADDRESS>

@kyllingstad @klickverbot @ibuclaw @DmitryOlshansky Could you please participate in the voting about multidimensional ranges? From D core team only @burner has voted. I hope to have at least two votes from D core team. ndslice package would be the basis for the future D implementation of generic BLAS.
Voting<EMAIL_ADDRESS>Updated docs: http://dtest.thecybershadow.net/artifact/website-76234ca0eab431527327d5ce1ec0ad74c6421533-904569dd4c4451a4514dc4b456c7b395/web/phobos-prerelease/std_experimental_ndslice.html
The latest docs are always available at CyberShadow/DAutoTest -> Details -> Phobos -> std -> experimental -> ndslice.

I've been following this development, and I'm very excited for it, but I have a question about codegen. Has anyone done a set of comprehensive disassemblies and proven that the codegen is as expected? This code will result in many nested loops, hidden among a complex network of template instantiations; we need to know the optimiser(s) are able to sort this out properly.
I would like to see evidence that it all flattens out properly, and I'd also be curious to know what GCC/Clang do with it in terms of auto-vectorisation.
In my (extensive) past experience with this sort of thing, I've observed no end to cases where minor syntactic or API tweaks can strongly influence the codegen, and so it should be very rigorously scrutinised before it is committed. This should include at least ARM/NEON.

I've been following this development, and I'm very excited for it, but I have a question about codegen. Has anyone done a set of comprehensive disassemblies and proven that the codegen is as expected? This code will result in many nested loops, hidden among a complex network of template instantiations; we need to know the optimiser(s) are able to sort this out properly.
I would like to see evidence that it all flattens out properly, and I'd also be curious to know what GCC/Clang do with it in terms of auto-vectorisation.
In my (extensive) past experience with this sort of thing, I've observed no end to cases where minor syntactic or API tweaks can strongly influence the codegen, and so it should be very rigorously scrutinised before it is committed. This should include at least ARM/NEON.

I have updated dip80-ndslice, so version v0.8.6 can be compiled with LDC 0.17.0 alpha1. I will post few assembler codes for DMD and LDC. One assembler example already available: https://github.com/D-Programming-Language/phobos/pull/3397#issuecomment-156355329.

and I'd also be curious to know what GCC/Clang do with it in terms of auto-vectorisation.

We need to check if 1D strides equals to 1 and create special code case to allow auto-vectorisation.

Has anyone done a set of comprehensive disassemblies and proven that the codegen is as expected? This code will result in many nested loops, hidden among a complex network of template instantiations; we need to know the optimiser(s) are able to sort this out properly.
I would like to see evidence that it all flattens out properly

Optimisation check
Source: https://gist.github.com/9il/bc7966823d96557c566c
LDC disassembled: https://gist.github.com/9il/47aea1621a9fba609869 (all functions inlined!)
DMD disassembled: https://gist.github.com/9il/a5c0ee9bdb4ddd25c4d6

Has anyone done a set of comprehensive disassemblies and proven that the codegen is as expected? This code will result in many nested loops, hidden among a complex network of template instantiations; we need to know the optimiser(s) are able to sort this out properly.
I would like to see evidence that it all flattens out properly
Optimisation check
Source: https://gist.github.com/9il/bc7966823d96557c566c
LDC disassembled: https://gist.github.com/9il/47aea1621a9fba609869 (all functions inlined!)
DMD disassembled: https://gist.github.com/9il/a5c0ee9bdb4ddd25c4d6

@TurkeyMan Please vote<EMAIL_ADDRESS>:-)

@kyllingstad @klickverbot @ibuclaw @DmitryOlshansky Could you please participate in the voting about multidimensional ranges? From D core team only @burner has voted.

I have been on holidays since the 10th, and suspect others are taking much needed time out too.  I'll try to have a look today.
I'd request for an extension to vote (or allow core team to vote in their own time :-)

@kyllingstad @klickverbot @ibuclaw @DmitryOlshansky Could you please participate in the voting about multidimensional ranges? From D core team only @burner has voted.

I have been on holidays since the 10th, and suspect others are taking much needed time out too. I'll try to have a look today.

Thanks!

I'd request for an extension to vote (or allow core team to vote in their own time :-)

@JackStouffer please extend voting for all up to January 31. Probably new announce would be fine.

please extend voting for all up to January 31. Probably new announce would be fine.

The 31st seems a little much, that would mean the voting will last for six weeks. Not to mention that extending voting threads is a bit above my pay grade as no other voting thread has been extended. For example, voting threads like std.testing didn't have many core maintainers review either, but it wasn't extended.
I understand people not being able to review over the holidays, but the announcement for the review thread was made 40 days ago.

please extend voting for all up to January 31. Probably new announce would be fine.

The 31st seems a little much, that would mean the voting will last for six weeks. Not to mention that extending voting threads is a bit above my pay grade as no other voting thread has been extended. For example, voting threads like std.testing didn't have many core maintainers review either, but it wasn't extended.
I understand people not being able to review over the holidays, but the announcement for the review thread was made 40 days ago.

OK, I am fine to finish voting)

Just to clarify, I'd be fine extending it a week if a prominent Phobos maintainer says it's ok.

Just to clarify, I'd be fine extending it a week if a prominent Phobos maintainer says it's ok.

Yeah, I'm not sure what is the process if a module gets rejected by core maintainers after community voting.  But then again, this is std.experimental, I guess it doesn't get as brutal a treatment as module in the main namespace.

Voting is over with the final result of 12-0. If there are no major issues found by the core maintainers then this can be merged!

Voting is over with the final result of 12-0. If there are no major issues found by the core maintainers then this can be merged!

@JackStouffer Thank you for review management!

I have removed compiler version check. It can be merged IMO.

DMD 2.070 is going to be branched 1st Jan. @andralex @MartinNowak @klickverbot @kyllingstad  @burner please make a decision about ndslice package for 2.070 release ASAP.
Notes:

ndslice pass all auto-tester tests (I have rebased and checked), CyberShadow/DAutoTest merge failed. So it is CyberShadow/DAutoTest  bug.
Documentation: link
Final votes

11Yes
1 Conditional Yes by @burner (because explanation style in documentation)
0 No

Thanks!

DMD 2.070 is going to be branched 1st Jan. @andralex @MartinNowak @klickverbot @kyllingstad  @burner please make a decision about ndslice package for 2.070 release ASAP.
Notes:

ndslice pass all auto-tester tests (I have rebased and checked), CyberShadow/DAutoTest merge failed. So it is CyberShadow/DAutoTest  bug.
Documentation: link
Final votes

11Yes
1 Conditional Yes by @burner (because explanation style in documentation)
0 No

Thanks!

Merge failed for DAutoTest. In the same time auto-tester works well .

Fixed now (and one step closer to fixing this permanently), thanks for letting me know. Happy new year.

Merge failed for DAutoTest. In the same time auto-tester works well .

Fixed now (and one step closer to fixing this permanently), thanks for letting me know. Happy new year.

@CyberShadow Thanks! Happy new year)

@CyberShadow Thanks! Happy new year)

I still think that the docs need improvements

I still think that the docs need improvements

I still think that the docs need improvements

OK, if this is hard constraint, then a decision maker from Core team can close this PR.

I still think that the docs need improvements

OK, if this is hard constraint, then a decision maker from Core team can close this PR.

@CyberShadow DAutoTest failed again .

@CyberShadow DAutoTest failed again .

I still think that the docs need improvements

I am not much involved in this project but I think docs can be imploved post release. As long as the design is sound and backwards-incompatible changes are unlikely, it could be merged.

@CyberShadow DAutoTest failed again .

Fixed

I still think that the docs need improvements

I am not much involved in this project but I think docs can be imploved post release. As long as the design is sound and backwards-incompatible changes are unlikely, it could be merged.

@CyberShadow DAutoTest failed again .

Fixed

I still think that the docs need improvements

I think @9il will have a hard time to improve the docs on it's own, and it's something that's better done collectively while working the module from std.experimental to std.

That said the documentation doesn't need a few improvements but a complete restructure. There is no clear introduction/overview, just the table for the submodules. When I try to understand the example (which seems a bit too hard for an intro) I'm already stuck trying to understand what pack does.
And it's documentation isn't too helpful.
http://dtest.thecybershadow.net/artifact/website-8566711a7833f9dcdf044cac5c48bfe666251245-1b1f44a3f35545b790eac434df7d7835/web/phobos-prerelease/std_experimental_ndslice_selection.html#pack

Auto-merge toggled on

Let's move on with this, there has hardly been a module w/ more support by the community, and as there are many users (and already articles in preparation), I'm pretty confident, we can get the documentations in shape soon.

Whoohoooo!
Many thanks for all reviewers and @JackStouffer, ndslice review manager.
@John-Colvin I have mentioned you in Acknowledgment section in package description.

#3896 upgrade win*.mak & uncomment unittest with dummyranges

According to http://digger.k3.1azy.net/trend/, this caused a 85KiB increase in the hello world binary!? Is DMD really including symbols from this when it's never imported?

According to http://digger.k3.1azy.net/trend/, this caused a 85KiB increase in the hello world binary!?

@JackStouffer wasn't it https://github.com/dlang/druntime/pull/1453 ?


Wrong results with .text()

demo: https://runkit.com/caub/59c69df49c22730011431e6f
code:
const express = require('express');
const bodyParser = require('body-parser');
const fetch = require('node-fetch');

const app = express();

app.use((req, res, next) => { // CORS middleware, for testing in browser
	res.setHeader('Access-Control-Allow-Origin', '*');
	if (req.method.toUpperCase()=='OPTIONS') {
		res.setHeader('Access-Control-Allow-Methods', 'GET,HEAD,PUT,POST,DELETE');
		res.setHeader('Access-Control-Allow-Headers', req.headers['access-control-request-headers']);
		res.setHeader('Access-Control-Max-Age', '86400');
		return res.status(204).end();
	}
	next();
});
app.use(bodyParser.text());
app.post('/', (req, res) => {
	res.send(req.body);
});

const httpServer = app.listen(3000, '<IP_ADDRESS>');

fetch(`http://localhost:3000`, {
	method: 'POST',
	body: `test`
}).then(r => r.text())
.then(text => {
	console.log('received', text);
	httpServer.close();
});

outputs: received {}
instead of received test
if you comment out the last lines with fetch, and try it from devtools, it outputs the right result

update, it seems working on the latest node-fetch, from github
please update npm :)

pretty much this is completely unrelated, but there is a new release alpha.9 out.


export * from "./base-types"; SyntaxError: Unexpected token 'export'

Describe the bug 📝
I'm using bim-fragment in Nodejs, it throw error
Reproduction ▶
https://github.com/RyugaRyuzaki/bim-fragment-server.git
Steps to reproduce 🔢

Run "npm install"
Run "npm run dev"

System Info 💻
Nodejs 20.0.5

Used Package Manager 📦
npm
Error Trace/Logs 📃
No response
Validations ✅

[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a IFC.js fragment issue and not a framework-specific issue. For example, if it's a THREE.js related bug, it should likely be reported to mrdoob/threejs instead.
[X] Check that this is a concrete bug. For Q&A join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

You can fix this by using bundler for your backend app i ve solved this issue by using vite.

I think this is already solved, as we are bundling our libraries. Let me know otherwise!


Version 7 exclusive changes

The compatibility breaking features should be kept in this PR.

Pushed the effects and conditions code and rebased with the buffering implemented in master.


Intellisense: wrong suggetions for node

From @nobil007 on September 17, 2016 8:57

VSCode Version: Code 1.5.2 (66f37fd2a99eb9d628dd374d81d78835b410c39b, 2016-09-12T13:09:51.625Z)
OS Version: Linux ia32 4.4.0-36-generic

Steps to Reproduce:
1.In my code I used __dirname. [Only 2 '' as prefix]
2.When I type '' next time, it will show suggestion. But the suggested string is __dirname.[3 '' as prefix]
Copied from original issue: Microsoft/vscode#12182

From @jrieken on September 19, 2016 8:59
Iff so its from TypeScript/JavaScript as it's not a word based suggestion but a symbol suggestion from them

From @jrieken on September 19, 2016 8:58
@joaomoreno Unsure what your screen shot shows. But you mean this?

From @joaomoreno on September 19, 2016 8:50
Can totally reproduce. Very strange. @jrieken any idea?

From @nobil007 on September 19, 2016 9:17
Ya, My issue is same as @jrieken reported.

From @joaomoreno on September 19, 2016 9:7
Yeah. Moving to @waderyan then.

From @nobil007 on September 19, 2016 9:29
The issue in vs code is fine. But the tool [GitHub] which you provided to report the issue have the same issue. One underscore is eliminated from all places in my comment. I had put a underscore in between the single quotes in " [Only 2 '' as prefix] ". But is not showing in the view. But it shows in the editing view.

Thanks for opening this issue. As this falls under the TS language service moving to them.

Can't repro in latest.
@waderyan is there any way we could have you guys running on latest bits of VS Code + TypeScript when doing bug triage on your side? I can help you get set up with it.

@RyanCavanaugh 1.5.2 are the latest VS Code bits. What TypeScript would you like me to use?

Preferably a live build from the release-2.0.5 branch of TypeScript

Would it be npm install typescript@latest?

@RyanCavanaugh  release-2.0.5 seems like a lot of work for the vscode folks to test against, unless we publish a nightly from release-2.0.5.
@waderyan typescript@next would be sufficient. there are some changes in release-2.0.5 that are not in typescript@next, but hopefully we get the two in sync in the next few weeks.


Add Blackjack app example

Description
Added new Blackjack app example
Test Plan
Local

PR set to draft, pending the Blackjack repo being made public.
cc @teohaik

Applied @ronny-mysten 's suggestions in a single review.
Also made repo public.
However, I have to state that the code is due for refactoring to:

support working mode in devnet/testnet
comply with latest sui.js library


Python3.13: ModuleNotFoundError: No module named 'xdrlib'

python-vxi11 uses xdrlib:
https://github.com/python-ivi/python-vxi11/blob/cc4671da699f1f379137dc40ffc4a302d72e6f55/vxi11/rpc.py#L22
https://github.com/python-ivi/python-vxi11/blob/cc4671da699f1f379137dc40ffc4a302d72e6f55/vxi11/rpc.py#L65
https://github.com/python-ivi/python-vxi11/blob/cc4671da699f1f379137dc40ffc4a302d72e6f55/vxi11/rpc.py#L100
The xdrlib module documentation in the Python standard library reads:

Deprecated since version 3.11, will be removed in version 3.13: The xdrlib module is deprecated (see PEP 594 for details).

Python 3.13 is expected on 2024-10-07.

Importing the vxi11 now indeed fails with ModuleNotFoundError over the xdrlib. Could this be updated to use e.g. the https://pypi.org/project/xdrlib3/ is a fork that could be used, and is for Python versions >=3.7. Only change needed would be in the import to change it import xdrlib3 as xdrlib. The second fork that is also available, https://pypi.org/project/py-xdrlib/, can also be used but there the Python version requirement is set for some reason to be >=3.11. This gives in our case compatibility problems with our packages.

The easiest solution might be to use https://github.com/youknowone/python-deadlib.
After pip install standard-xdrlib you can do:
>>> import xdrlib
<python-input-0>:1: DeprecationWarning: xdrlib was removed in Python 3.13. Please be aware that you are currently NOT using standard 'xdrlib', but instead a separately installed 'standard-xdrlib'.
  import xdrlib

And the version is currently 3.13.0, so version requirement should even be fine as is.


v2ray-core日志中会有proxy/vmess/encoding: Critical Warning: potentially invalid user: a non VMessAEAD connection is received. From 2022 Jan 1st, this kind of connection will be rejected by default. You should update or replace your client software now. This message will not be shown for further violation on this inbound.信息。请问这是什么

如果你遇到的问题不是 V2Ray 的 bug，比如你不清楚要如何配置，请使用Discussion进行讨论。
此 Issue 会被立即关闭。
If you are not sure if your question is truely a bug in V2Ray, please discuss it here first.
This issue will be closed immediately.

预期行为


fix: Allow tokenUrl to be relative

Looks like the change before:
https://github.com/openapi-generators/openapi-python-client/commit/15c2fd5db2dd8afa6947cbf5a852a6f895f054db#diff-1a1a2865f336a376229eaffe0f56caac01c7a3a77602e07fe4a38fd550c6c318
Has been reverted:
https://github.com/openapi-generators/openapi-python-client/commit/0101a039c4f709977e136b3c0f5d186a63a7c6c5#diff-6676609c9c37705f6953ffcc9eb6d33164df50f241d4844c5665cf5ff4305995R16
As mentioned, relative URLs are supported in OpenAPI 3.0+:
https://github.com/swagger-api/swagger-ui/issues/5243

Just ran into this issue with fastapi-users usage . Will this be added to an official release soon? Thanks. @dbanty

All set 👍


how to process UAV_VisLoc_dataset for the proposed framework

the work is so interesting, could you please indicate how to process the UAV_VisLoc_dataset for the proposed framework, it is hard to implement with the code in scripts folder and visloc.py, thanks!!!

This part of the code is not fully organized yet and might be missing some preprocessing steps (such as splitting satellite images into tiles). Or you might prefer having the preprocessed dataset directly? If feasible, we also plan to release the processed UAV-VisLoc dataset in a structure similar to GTA-UAV.

This part of the code is not fully organized yet and might be missing some preprocessing steps (such as splitting satellite images into tiles). Or you might prefer having the preprocessed dataset directly? If feasible, we also plan to release the processed UAV-VisLoc dataset in a structure similar to GTA-UAV.

We don't have a preprocessed dataset, so we look forward to your finished fully organized code. Do you have any plans to update it? Thank you very much

I would say expected in 2-3 weeks, and I will try my best. :)

This part of the code for processing UAV-VisLoc should work now. You can try it now. If you want to extend it to other datasets, some modifications will be needed.

This part of the code for processing UAV-VisLoc should work now. You can try it now. If you want to extend it to other datasets, some modifications will be needed.

Many thanks ：）


Task: Create GET vehicleID method

Create vehicleID method in server.js file
Use responses from model ID and makeID to retrieve vehicleID from the API interface
User vehicleID as an input for the vehicleCarbonReport route.
Display body (contents) of the vehicleCarbonReport route onto the Carbon Report Page

https://docs.carboninterface.com/#/ <-----  LOOK AT THIS our api calls have a beginning but they are currently looking up to json instead of carbon interface. We need api calls converted back to talk directly to carbon API to get this information BUT LIMIT THE TIMES YOU DO THIS as we get like 200 api calls a month. Eventually we will store the car's vehicle ID in the sql database so that each time they drive it doesn't need 3 api calls, only 1.


Pointer is missing a nullability type specifier (_Nonnull, _Nullable, or _Null_unspecified)

Pointer is missing a nullability type specifier (_Nonnull, _Nullable, or _Null_unspecified)

This appears to be a bug in Xcode. http://www.openradar.me/22917151
I don't know of any work around until Xcode is fixed.

Just to add to this, I've noticed it in my project too. I've got warnings-as-errors turned on, so it was actually causing the build to fail. I tried a clang diangnostic pragma around my @import statement, but that doesn't seem to work like it does on #import or #include.
The only workaround I found was to add -Wno-nullability-completeness to my host project's Other Warning Flags section. Of course, this turns off that warning for the whole app, which is unfortunate, but it's better than nothing.

Update: This issue continues in the the final 7.2. Haven't discovered any way to address it yet.

Showing up for me as well:

To reiterate for future commenters: is a known bug in Swift (radar://22805286). Apple has not responded with a fix or workaround. If you discover a workaround or fix, please post. If you just want to keep track of it, you can click "Subscribe" in the right-hand pane.

FWIW, marking the file (or even just that line) as an audited region will prevent the warning:
NS_ASSUME_NONNULL_BEGIN
static NSString * const RNCryptorErrorDomain = @"RNCryptor.RNCryptorError";
NS_ASSUME_NONNULL_END

This should now be resolved in 5.0. I've moved ObjC bridging to RNCryptor/RNCryptor-objc.

Remove all headerfiles from /usr/local/include that have shortcut icons on them.
Worked as permanent fix for me.


Add Turtle.Format.makePrinter function

I was trying to implement some abstraction on top of Turtle to redirect the default output (stdout and/or stderr) to a logger function withing some monad where the logger is available. While I was able to implement most of the Turtle function in one way or another it turned out it’s not that trivial to implement custom printf and eprintf since (>>-) of Turtle.Format.Format is not exposed.
This change adds makePrinter function that just adds extra argument for printf that is a printer function ending with m () as Format result. The implementation is trivial (flip (>>-)), printf and eprintf are reimplemented using this function.

@Gabriel439 I have some doubts about the name of the function. Maybe it’s better to explicitly say that it’s about Format? Like name the function not just makePrinter but makeFormatPrinter. What do you think?

@unclechu: Another option would be to just expose the Format constructor so that people can use (>>-) directly.  There's no real reason that it needs to be hidden

@Gabriel439 Yeah, I would like to just get it exposed in the first place so I wouldn’t have this kind of issue.

I put  up a PR to expose the Format constructor here: https://github.com/Gabriel439/turtle/pull/417

@Gabriel439 I did the same meanwhile https://github.com/Gabriel439/turtle/pull/418 :sweat_smile: Close either one. Anyway, closing this one.


Idea: Improve typeof

Question:
Why does typeof! [1,2,3] return 'Number'?
Idea:
Can we interchange typeof and typeof! so that the operator we know and hate can become the operator we know and love? The lesser of the two should be hidden behind alt syntax.
My only concern is not being able to typeof arrays as questioned above.

After (the still undocumented, alas) #863, it doesn't seem like there are any outstanding needs here, so I'll reclose.

it's not well documented what typeof! returns


[Feature] Flink Doris Connector ,Can tables without primary keys be automatically created as detailed tables

Search before asking

[X] I had searched in the issues and found no similar issues.

Description
Flink Doris Connector ,Can tables without primary keys be automatically created as detailed tables
Use case
Flink Doris Connector
Synchronizing MySQL tables without primary keys will result in errors
Synchronize all MySQL tables
Related issues
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

doris 2.0.0

doris version：2.0.0

Flink-doris-connector-{flink.version}-1.5.0 will support Synchronize all MySQL tables without primary keys.


Error: Cannot find module './build/Release/analytics.node'

I just installed Zenbot on my new Ubuntu 16.04 VM, but what ever I do, I just get errors such as:
xxxx@Ubuntu-16:~/zenbot$ zenbot list-selectors
module.js:471
throw err;
^
Error: Cannot find module './build/Release/analytics.node'
at Function.Module._resolveFilename (module.js:469:15)
at Function.Module._load (module.js:417:25)
at Module.require (module.js:497:17)
at require (internal/module.js:20:19)
at Object. (/home/xxxx/zenbot/node_modules/forex.analytics/index.js:1:79)
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
xxxx@Ubuntu-16:~/zenbot$ zenbot sim --help
module.js:471
throw err;
^
Error: Cannot find module './build/Release/analytics.node'
at Function.Module._resolveFilename (module.js:469:15)
at Function.Module._load (module.js:417:25)
at Module.require (module.js:497:17)
at require (internal/module.js:20:19)
at Object. (/home/xxxx/zenbot/node_modules/forex.analytics/index.js:1:79)
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
What's wrong with this?? :-o

Hey. At me something similar.
server_1        | Error: Cannot find module './conf-sample'
server_1        |     at Function.Module._resolveFilename (module.js:527:15)
server_1        |     at Function.Module._load (module.js:476:23)
server_1        |     at Module.require (module.js:568:17)
server_1        |     at require (internal/module.js:11:18)
server_1        |     at module.exports (/app/boot.js:12:18)
server_1        |     at Object. (/app/zenbot.js:20:1)
server_1        |     at Module._compile (module.js:624:30)
server_1        |     at Object.Module._extensions..js (module.js:635:10)
server_1        |     at Module.load (module.js:545:32)
server_1        |     at tryModuleLoad (module.js:508:12)
server_1        | module.js:529
server_1        |     throw err;
server_1        |     ^
server_1        |

Duplicated: see #432 (already solved)
"Guys, please upgrade to nodejs 8, that should solve the build problem.
Please make sure to rm -rf node_modules && npm install to reinstall everything."

node -v v8.7.0
npm -v v5.4.2
the error is repeated

Ok, I too installed the newer node.js, same versions that zemelax just listed. Now when I do "npm install", I get the following:
xxxx@Ubuntu-16:~/zenbot/zenbot$ sudo npm install
npm ERR! code 1
npm ERR! Command failed: /usr/bin/git clone -q git://github.com/mkmarek/forex.analytics.git /home/xxxx/.npm/_cacache/tmp/git-clone-a1afc987
npm ERR! /home/xxxx/.npm/_cacache/tmp/git-clone-a1afc987/.git: Permission denied
npm ERR!
npm ERR! A complete log of this run can be found in:
npm ERR!     /home/xxxx/.npm/_logs/2017-10-19T18_36_32_331Z-debug.log
Tried to Google all around about that, did not get any hits, can someone help?

I tried with "yarn" (do a npm install yarn before) instead of "npm install" and it finally worked. Also, had to remove the previous zenbot instance. But yes, the installation could be improved (avoiding sudo npm would be great).
Also, yesterday Github had some issues (I wasn't able to clone using git, just with https...) https://status.github.com/messages .

gyp ERR! configure error
gyp ERR! stack Error: EACCES: permission denied, mkdir '/home/masternode/Documents/zenbot/node_modules/forex.analytics/build'
gyp ERR! System Linux 4.10.0-37-generic
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure"
gyp ERR! cwd /home/masternode/Documents/zenbot/node_modules/forex.analytics
gyp ERR! node -v v8.7.0
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>postinstall: node-gyp configure && node-gyp build
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /home/masternode/.npm/_logs/2017-10-21T00_52_40_441Z-debug.log
got this too node version 8.7, npm version 5.5.1 running VM Ubuntu 16.0.4

Ubuntu 16.04 Tested
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install build-essential mongodb

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs

git clone https://github.com/carlos8f/zenbot.git
cd zenbot
npm install

./zenbot.sh trade --paper

Sniper7kills, I don't know why, but when I copy-pasted your lines on my Ubuntu, it just started working!  \o/
I surely did try all those command previously also, but it might be that used "sudo" in front of curl-, git- and npm install- commands also..? I'm not sure though, and don't know if would matter anyway?
Thank you, you´re the best! :=)

@jasurakk Glad to hear, it was probably the lack of the build-essentials being installed; just and fyi don't use sudo if you don't need to (I.E. Curl, Git, npm), it runs commands as root which can cause you major issues if your not careful.
If this solved your issue you might want to close it  ;)


Update vfnet_head.py

fix bug:#4172 vfnet can training with fp16

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Please fix the lint, you can know where is the error here: https://github.com/open-mmlab/mmdetection/pull/4173/checks?check_run_id=1451120339

Done in https://github.com/open-mmlab/mmdetection/pull/4198.

Hi
@carry-xz
We are grateful for your efforts in helping improve this open-source project during your personal time.
Welcome to join OpenMMLab Special Interest Group (SIG) private channel on Discord, where you can share your experiences, ideas, and build connections with like-minded peers. To join the SIG channel, simply message moderator— OpenMMLab on Discord or briefly share your open-source contributions in the #introductions channel and we will assist you. Look forward to seeing you there! Join us ：https://discord.gg/UjgXkPWNqA
If you have a WeChat account，welcome to join our community on WeChat. You can add our assistant ：openmmlabwx. Please add "mmsig + Github ID" as a remark when adding friends：）
Thank you again for your contribution❤


Script editor no longer scrolls when you move line with Alt+Up/Down

Tested versions
Reproducible in 4.3.beta2
System information
Godot v4.3.beta2 - Pop!_OS 22.04 LTS - X11 - GLES3 (Compatibility) - Mesa Intel(R) Graphics (ADL GT2) - 12th Gen Intel(R) Core(TM) i5-1235U (12 Threads)
Issue description
Script editor doesn't scroll to the caret when you press Alt+Up/Down. Works on any GDScript file opened in Godot. Used to function correctly in 4.2.2
Steps to reproduce
N/A
Minimal reproduction project (MRP)
N/A

Haven't bisected but might be a regression from:

https://github.com/godotengine/godot/pull/86978

As it touched the relevant code
CC @kitbdev


isSelected and common parents

added IsSelected to ModInput and ModButton
added common parents where it made sense (Not sure if I should also implement class ModInputBase, common interface sounds enough for storage purposes)

IsSelected is tested only for ModInputCombinationElement

Although it kinda implies it would also work for all TwoButtonToggle-based things
Not yet sure if UIStyleApplier-based IsSelected works flawlessly

Super cool stuff! :)
Not sure we have the best naming of all of these UI elements tho ;)
Example: ModMenuInput, ModInputField - can we find more descriptive names?

maybe ModFieldInput
Also ModPopupInput intead of ModMenuInput might be better

Awesome.
You have a couple unused usings in interfaces (IModInput and IModInputBase)


added What is X-Forwarded-For All About?

This one is a BEAST but I think it's exactly the sort of thing you'll like.
I used mermaid-js in here using a iaWriter extension, which was useful for writing but might need a different approach for Gatsby.
There are plugins for this: https://www.gatsbyjs.com/plugins/gatsby-remark-mermaid/
Or I can make them into images but where's the challenge in that.

Thanks Phil! All looks good now :+1:
I'll do final tweaks & merge to publish this soon, probably next week. Thanks for your help!

Brilliant, thank you!


incompatibility with bobby?

https://pastebin.com/2fddh525

(Edited to put the crashlog in a pastebin)
At a glance it looks like Bobby doesn't like that the mod creates a dummy world to render the player character in UIs. I'm not sure how easy this will be to fix, but I'll look into it.

Fixed in 1.1.1.


Changes for Issue #14 - Requirements API - Implement YAML requirements generation from a chat conversation

Resolves #14

I implemented the requirements API to requirements/requirements.go and created a Cobra command to test it on production credentials. It now generates acceptable requirements that we can use during the outlining, coding, and debugging processes.


Fix GoTo Definition issues

This fixes
https://github.com/Microsoft/visualfsharp/issues/1706
https://github.com/Microsoft/visualfsharp/issues/1752
and also an issue where GTD didn't work when the cursor was placed at the end of a word.
We already have a lot of unit testing for the information coming from the language service, this just wasn't wiring things up the way that's needed.

Awesome


Second Contribution by Ufuoma Ejite

Name: Ufuoma Ejite
Projects Analyzed: CVXPY | OpenRefine
Links to the Governance Model of the Projects: CVXPY governance model can be accessed here, and that of OpenRefine can be accessed here
A Written Analysis: Similarities and Differences
Description of the Governance Models
CVXPY: CVXPY
OpenRefine: OpenRefine
The Project Roles
CVXPY: The project roles that exist within CVXPY are

Benevolent Dictator For Life (BDFL): This is the project creator, and he retains the final say regarding disputes and arguments in the community.
Project maintainers: They are the ultimate authority regarding the CVXPY project.
Emeritus project maintainers: They are the ex-project maintainers, and perform a less active role.
Steering committee: They handle the legal and administrative tasks within CVXPY.

OpenRefine: The project roles that exist within OpenRefine are

Users: They are community members who have a need for the project.
Contributors: They are users getting more involved with the project.
Committers: They make regular contributions to the project. They can review and merge pull requests.
Release manager: They coordinate new releases, issues, and pull requests.
Steering committee: They oversee the general direction of the project, and perform advocacy duties.
Advisory committee: They run the administrative duties of the project.
Project director: They help to improve OpenRefine's governance and community diversity, and other day-to-day operations.

The Decision-making Process
The decision-making process in CVXPY is based on the consensus of the project maintainers and steering committee; however, the BDFL is the final decision-maker.
The decision-making process in OpenRefine is user-based.
Conclusion
OpenRefine operates a

@joy-commits . Check out your Conclusion

@joy-commits . Check out your Conclusion

Thank you @maryamgbemisola. I was just already editing it

Hello @arliss-NF I've made my second contribution.
Thank you

@joy-commits - this is great - I support your choice that OpenRefine is a better model.  BDFL model can stall a projects progress when only 1 person makes the final choice.  Thanks so much for completing Tasks 1 and 2.

Thank you @arliss-NF for the review


Pin OpenBLAS to 0.2.20

Bumps the OpenBLAS pin from 0.2.19 to 0.2.20.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


feat: reject badly formatted DOIs

Reviewers
Functional:
@MillenniumFalconMechanic
Readability:

Changes
Rejects DOIs that aren't formatted properly.
Note: this uses the regex validation proposed in this article, which is quite strict but should work well for any newly published paper. This will also not allow using test DOIs anymore, which might be a concern (~even for smoke tests~ smoke tests are unaffected). Alternatively, we could use a weaker regex that only validates the 123/456 form. @brianraymor thoughts?

@ebezzi - That's s nice find and crossref approved. And since the post is from 2015, I'd expect that all DOI(s) that the portal sees will be "modern".
I support requiring a real DOI in all cases instead of something DOI-ish like 123/456.