RFC: Add a standard trait for getting many &mut to places

Rendered: https://github.com/Nokel81/rfcs/blob/get-mut-refs/text/0000-get-mut-refs.md
Previous IRLO discussion: https://internals.rust-lang.org/t/add-as-mut-ref-for-slice-or-array/14199/
Previous Zulip discussion: https://rust-lang.zulipchat.com/#narrow/stream/219381-t-libs/topic/GetMutRefs

If used without unsafe on slices, then isn't this less flexible than using split_at_mut in a loop or tree-like recursion?  I suppose multi_split_at_mut! macro sounds handy.  If used with unsafe on slices, then this seems redundant, no?
I agree we should explore adding arrayref-like functionality to core, meaning convenient conversion from slices or arrays into multiple arrays and slices, but with an interface resembling arrayref or slice indexing probably.
Index/IndexMut<ConstRange> where rustc builds pub struct ConstRange<const LOWER: ops::Bound, const UPPER: ops::Bound>; from .. literals maybe?

seems to me that Value should be an associated type since it's uniquely determined by Key and Self -- like Index::Output

seems to me that Value should be an associated type since it's uniquely determined by Key and Self -- like Index::Output

I do have Value as an associated type though ...

@burdges I don't quite understand what you mean by "If used without unsafe on slices, then isn't this less flexible than using split_at_mut in a loop or tree-like recursion?".
This isn't a way to get multiple &mut [T] out of a single &mut [T] but to get multiple &mut T out of a single &mut [T] (or any other collection type).
I guess an alternative could be to have this be an operator as well, ie IndexMutMany, where it de-sugars from let [x1, x2] = &mut source[index_1, index_2]; to let [x1, x2] = source.get_mut_many([index_1, index_2]).expect("non-overlapping"); or something, but I see that as an extension in the future. Though GetMutMany might be a better name.

@adlerd Can you give an example as to how those traits might look? I am having trouble seeing how you would be able to use an unsafe "raw" trait. Even with the current proposal you could just make RawEntry = () if you wanted to implement the safe trait by itself for some reason.
I can change what the example implementation is, though I don't see how using [T; N] is any less motivating than [T].

Finally, is there sufficient motivation to put this in the standard library instead of a crate? I believe the rust features which started in a crate nursery generally end up with a better design and implementation than otherwise, although this is not a universally held opinion.

I think that there is. For instance, this sort of construct needs specific knowledge of the inner workings of types. So the trait impl's really need to be in the same "crate" as the types. Meaning that having a 3rd party crate wouldn't really work very well since the for types (like BTreeMap) their inner workings are completely private.

There is a handy interface for this type of thing in https://github.com/burdges/lake/blob/master/Xolotl/src/sphinx/slice.rs that avoids annoying the borrow checkers:
/// Returns an initial segment of a `mut &[T]` replacing the inner
/// `&[T]` with the remainder.  In effect, this executes the command
/// `(return,heap) = heap.split_at(len)` without annoying the borrow
/// checker.
pub fn reserve<'heap, T>(heap: &mut &'heap [T], len: usize) -> &'heap [T] {
    let tmp: &'heap [T] = ::core::mem::replace(&mut *heap, &[]);
    let (reserved, tmp) = tmp.split_at(len);
    *heap = tmp;
    reserved
}

/// A version of `reserve` for fixed length arrays.
macro_rules! reserve_fixed { ($heap:expr, $len:expr) => {
    array_ref![reserve($heap,$len),0,$len]
} }

/// Returns an initial segment of a `mut &mut [T]` replacing the inner
/// `&mut [T]` with the remainder.  In effect, this executes the command
/// `(return,heap) = heap.split_at_mut(len)` without annoying the borrow
/// checker.
pub fn reserve_mut<'heap, T>(heap: &mut &'heap mut [T], len: usize) -> &'heap mut [T] {
    let tmp: &'heap mut [T] = ::core::mem::replace(&mut *heap, &mut []);
    let (reserved, tmp) = tmp.split_at_mut(len);
    *heap = tmp;
    reserved
}

/// A version of `reserve_mut` for fixed length arrays.
macro_rules! reserve_fixed_mut { ($heap:expr, $len:expr) => {
    array_mut_ref![reserve_mut($heap,$len),0,$len]
} }

We then implement an iterator for your interface using using Interator::scan and reserve_mut like
impl [T] {
    /// Iterator yielding `&mut T` for prescribed elements
    fn get_muts<'a>(&'a mut self, offsets: impl IntoIterator<Item=usize>)
     -> impl Iterator<Item = &'a mut T>+FusedIterator
    {
        let mut offset = None;
        let newoff = |o| {
            if let Some(offset) = offset {
                assert!(o > offset);
                let n = o-offset;
                offset = Some(o);
                n
            } else {
                offset = Some(o);
                o
            }
        };
        offsets.into_iter().scan(&mut self, |selfy,o| reserve_mut(selfy,newoff(o)).first_mut() ).fuse()
    }
}

As with iter_mut,  ourget_muts borrows the slice only once and returns references with that lifetime, but it iterates over the slice using efficient skips.  Remove newoff if you want relative offsets.
We'll gain impl<T,const N: usize> FromIterator<T> for [T; N] from min_const_generics so then this works:
let [a,b,c] = slice.get_muts([7,13,17]).collect::<[&mut T; 3]>();

And some macro could save you specifying the [&mut T; 3] type here.
There is some case for all these tools to live in core, including reserve_mut, get_muts, some cleaner macro, and even array_ref or whatever replaces it with min_const_generics, and then its reserve_fixed_mut.  They're not as fundamental as split_at_mut or mem::replace, but all seem pretty competitive with mem::take.

@adlerd Can you give an example as to how those traits might look? I am having trouble seeing how you would be able to use an unsafe "raw" trait. Even with the current proposal you could just make RawEntry = () if you wanted to implement the safe trait by itself for some reason.

Here's what I was thinking to remove RawEntry from GetMutRefs; it also gives a little more flexibility in RawEntry impl I think by providing a reference or a pointer back to the container:
pub unsafe trait GetMutRefsRaw<Key, Value> {
    type RawEntry;

    fn would_overlap(&self, e1: &Self::RawEntry, e2: &Self::RawEntry) -> bool;

    // NB. we provide *mut Self instead of &'_ self because we can't soundly
    // hold a shared reference to self and a unique reference to some values at
    // the same time, even though borrowck won't notice the relationship.
    unsafe fn entry_to_ref<'a>(this: *mut Self, entry: Self::RawEntry) -> &'a mut Value;

    fn get_entry<'a>(&mut self, key: Key) -> Option<Self::RawEntry>;
}

pub trait GetMutRefs<Key, Value> {
    fn get_mut_refs<'a, const N: usize>(&'a mut self, keys: [Key; N])
        -> Option<[&'a mut Value; N]>;
}

This is extracted from a proposed implementation at this gist. NB this isn't meant to be suitable for RFC reference implementation; it's more like a proposed crate implementation.

reserve and reserve_mut were designed for parsing binary data without allocations, btw.

@burdges Thanks for the explanation. Though I don't see how it generalizes to non-slice types (like BTreeMap for instance). I also don't think that the requirement that all the indexes are monotonic in the input either makes sense from an API perspective or from a usability perspective.
Finally, I don't see how it solves the general question for user defined types since you cannot rely on Eq or PartialEq being correct for a general unsafe impl.

@adlerd Thank you.
I am not convinced that one should need a reference back to the container for overlap checking. Furthermore, in the general case I am pretty sure that a RawEntry should be able to compute the reference itself without the need of a pointer to the collection being passed in. Take BTreeMap for instance, (which I am in the process of adding to the RFC doc) the "natural" RawEntry type, imo, would be Handle<NodeRef<marker::Owned, K, V, marker::LeafOrInternal>> which already has an into_val_mut() function. As containers get more complex the pointer to the root of the collection would probably become less and less useful. And implementers might be lead astray by that argument being there.
Finally, to the root of your suggestion. Now that I see what you meant I think that I like it better. Though I don't see how the argument about trait bounds applies since there would only really be one trait that you would want to make a bound on in either case.

I'd expect any safe interface you provide for slices reduces to being monotonic.  And split_at_mut with perhaps a replace trick works for access patterns in which sequencing happens differently, like a recursion maintaining the same lifetime.
There is no split_at_mut analog for BTreeMap currently, but again inherent methods exposing this functionality would prove more universal than the interface proposed here.
impl<K, V> BTreeMap<K, V> {
    fn as_mut_view(&mut self) -> BTreeMapViewMut<'_,K,V>
}
pub struct BTreeMapMutView<'a,K,V> { .. }
impl<'a,K, V> BTreeMapMutView<'a,K, V> {
    fn expand_mut_view(self) -> &'a [BTreeMapMutView<'a,K,V>]
}

Finally, I don't see how it solves the general question for user defined types

Rust's std never provided traits for user defined types previously.  I'd think new traits set a higher bar than inherent methods.
I do not rule out that perhaps these traits reach this bar, but I've serious doubts given the long struggle over even the entry interface for hashmap, in https://github.com/rust-lang/rfcs/pull/1769 and https://internals.rust-lang.org/t/pre-rfc-abandonning-morals-in-the-name-of-performance-the-raw-entry-api/7043  I suppose the entries interface has a harder goal of modifying keys, while you only permit modifying values here, which sounds fair if limited.
Interestingly, cuckoo tables admit an entry interface with local indexing similar to or better than slices, meaning it admits modifying keys while indexes remain tied to particular cuckoo tables, either via a runtime equality test of randomness, or else via a compile-time GhostCell-like invariant lifetime trick.  In short, this interface is looks pointless for cuckoo table.
In this vein, we might explore GhostCell based or inspired entry-like tooling too, likely some fn frozen(&mut self) -> FrozenHashMap<'a,K,V> which permits modifying values but not keys.

I'd expect any safe interface you provide for slices reduces to being monotonic.

Why though, my first example within the RFC doc is a clear example as to why you wouldn't want something like that.

@scottmcm and @adlerd care to take another look?

I meant safety reduces to the way slices store data.  Yes, the newoff closure I wrote turns out rather simple.  Yes, it's plausible optimizer might encounter trouble with more branches that swap sides, not sure.
There is a larger issue however, split_at_mut enables safe accesses to arbitrarily places in programmatic ways that avoids allocation, perhaps passing multiple subslices in recursive ways, or storing them in temporaries, while get_mut_many requires a one-off array construction, which is far more limiting
You prohibit slices in get_mut_many presumably because allocation kills any efficiency gain, but [usize]::sort_unstable only requires O(n log n) time and your n stays very small without VLAs.
In short, it's unclear whether this interfaces provides any speedup for slices, but it's clear this interfaces avoids teaching users about the much more powerful split_at_mut interface.

As written, these traits imply an entry-like interface, but if I understand they can only deliver a mutable value interface for slices, BTreeMap, HashMap, etc., meaning no conditional insertions or deletions.  I'd suggest reserving the term entry for interfaces with entry-like conditional insertion and deletion, ala https://github.com/rust-lang/rfcs/pull/1769 and https://internals.rust-lang.org/t/pre-rfc-abandonning-morals-in-the-name-of-performance-the-raw-entry-api/7043
I think BTreeMapMutView could provide a restricted entry-like interface, making it strictly more powerful than the interface provided here.  I think some non-growable tree-like structures could provide a restricted entry-like behavior with the interface provided here, maybe even simple heaps.
I'm no stickler for traits having a rigorously consistent meaning, but traits do benefit from some consistency in their meaning, especially traits in std.  I'm saying it's really unclear how much consistency of meaning this trait interface provides.

It seems like I'm currently proposing something
similar: https://github.com/rust-lang/rust/pull/83608

@Kimundi Yes it does seem that you have. I have been working on this for over a month now (on IRLO and Zulip) so any feedback would be appreciated.
I think that a more general solution than just for slices would be best.
Finally, you don't need non-mut versions of these since you can already get multiple &T out of a single &[T]: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=a1ad5ccc54a1634492a7b389c9c69799

Shouldn't the GetMutMany trait be also unsafe? The GetMutMany.get_mut_many method is responsible for checking the places don't overlap, otherwise safety invariants are violated, so implementing the trait (because someone might still implement it in other way than using the GetMutManyRaw) should be unsafe, no?

@jan-hudec That I am unsure about, my reaction after reading the book on unsafe traits would imply that no it doesn't need to be. For the same reason that a function that uses unsafe doesn't necessarily need to be declared as unsafe.

@jan-hudec The rule for unsafe trait is "could an implementation of this trait which used no unsafe blocks cause UB? If so, then the trait must be unsafe". The way the GetMutMany trait is written currently, this doesn't hold, because a safe impl of get_mut_many cannot cause UB: borrowck prevents it already. In other words, for a safe impl, the impl itself is not responsible for checking the pieces don't overlap, borrowck is (or other existing unsafe like split_at_mut - which is outside the scope of this trait).

@Nokel81 I don't think the functions actually need receivers per se; I put them in just as a little something extra I thought I would improve the trait. I agree that in most cases they won't be needed, but it seems counterproductive to leave them out when doing so provides no benefit other than being potentially a tiny bit misleading, and it's possible they may be useful. In a situation where they are actually useful, the only alternative would be to embed a *mut Container in each handle. Probably the receiver in would_overlap is much more useful than the other one, since the only thing entry_to_ref should conceivably need is an element pointer. After all, entry_to_ref is just there to attach the borrowck imprimatur to that pointer and nothing more, so if there's a question of being misleading, the *mut self could be removed.
That said, I think the doc comment on entry_to_ref could use a bit of work. In order for this API to be used soundly, it should be phrased in terms of, "Here are the exact conditions under which the get_mut_many implementation will call this function*; the trait implementation must not cause UB under those conditions. A likely source of UB is &mut overlaps from different calls to this function. Another possible source of UB is if RawEntry contains & to the container." *And these conditions are: Whenever entry_to_ref<'output> is called with a RawEntry returned from a call to get_raw_entry<'input>, then there exists a &'origin mut Self from which both 'input and 'output are derived; and moreover, no other references will be created deriving from 'origin (until the expiration of 'output) except via calls to get_raw_entry for nonoverlapping RawEntry.

Bevy (an ECS game engine) would very much like this sort of functionality. We implemented something similar by hand for one of the most common use cases: grabbing combinations of items in a list in https://github.com/bevyengine/bevy/pull/1763.
I'm not sure what the solution would look like, but hopefully seeing more concrete use cases might be helpful.

This was discussed in the libs meeting today. We feel that this shouldn't be implemented as a trait: we already have inherent get methods on our collection types, so it doesn't really make sense to special case get_mut_many.
Apart than that, we are interested in a simpler version of this proposal which doesn't use a trait.

@Amanieu would the libs team be more in favour of a proposal for the types in the std-lib that should implement such methods?

Yes. As a general rule, I feel that all types that have a get_mut(index/key) method should probably have an equivalent get_mut_many method. So slices, VecDeque, HashMap and BTreeMap.

Okay thanks. Final question, should that be a direct PR then?

I think it should still be an RFC, because even if it's not a trait, it's very clearly setting a precedent.
(As in, it'll be an API that people will expect all the types in std, and even types in crates, to start providing, so it's worth the extra oversight of an RFC.  It's not just a one-off thing that'd be on one std type but nowhere else.)

Sounds good. Shall this one be closed and I open a new one then?

@Nokel81 I think that's mostly up to you.  If you think that the discussion in this thread is valuable for people reading an updated RFC, then updating this one is probably fine.  If you think that the discussion here has too much about the trait details and would thus be more of a distraction from the new one, then feel free to close this and open a new one.

Without an unsafe trait SafeOrd : Ord {} or alike, it is not possible to implement BTreeMap with this RFC.
That said, it is not strictly necessary to compare equality of keys to check equality of resolved pointers, is it? Since the safe interface of this RFC performs an O(n^2) check either way, why not check these using the references returned by the unsafe implementation? In other words, the unsafe implementation only returns an array of *muts, and the safe trait is responsible for checking that these *muts do not overlap, and converting them into &mut afterwards.
A bonus effect is that it improves performance by comparing pointers of usize size instead of arbitrary types (e.g. long strings), and the O(n^2) could probably be reduced to O(n log n) by numerically sorting the pointers (or maybe there are faster algorithms that I did not notice).

Without an unsafe trait SafeOrd : Ord {} or alike, it is not possible to implement this RFC for BTreeMap.

If you look at my proposal, then you should be able to tell that such a trait is not necessary even for types that have keys which are user defined types.

Would it make sense to implement this as a more exotic implementation of the SliceIndex trait, i.e. something like impl<T, const N: usize> SliceIndex<[T]> for [usize; N]?

@skmendez That can't work, because SliceIndex returns &mut Self::Output from index_mut, whereas this wants to be able to return [&mut _; N] -- there's no array anywhere in memory with the necessary things in it to be able to point at.

You're right; I think this could work with GATs so that Output could be generic on the lifetime.

The would_overlap approach here seems to inherently require a quadratic implementation, checking every pair of indexes. I think most practical data structures will hopefully be able to do better than that. What would a trait design look like that avoids the inherent quadratic complexity?

The would_overlap approach here seems to inherently require a quadratic implementation,

maybe use this instead, which allows using a temporary array (no allocator required if the array length is const since you can just use an array on the stack) to sort RawEntrys and guarantees that checking adjacent RawEntrys in sorted order is sufficient to check for overlap. this makes the worst case O(N log N):
unsafe trait GetMutManyRaw<Key> {
    type Value;
    type RawEntry;

    /// must be usable to sort `RawEntry`s such that for any list of `RawEntry`s
    /// there must be at least one pair of overlapping `RawEntry`s that are next
    /// to each other in the sorted list if any `RawEntry`s overlap.
    fn raw_entry_sort_cmp(&self, a: &Self::RawEntry, b: &Self::RawEntry) -> Ordering;
    fn would_overlap(&self, e1: &Self::RawEntry, e2: &Self::RawEntry) -> bool;
    unsafe fn entry_to_ref<'a>(*mut self, entry: Self::RawEntry) -> &'a mut Self::Value;
    fn get_raw_entry<'a>(&'a mut self, key: Key) -> Option<Self::RawEntry>;
}

    /// must be usable to sort `RawEntry`s such that for any list of `RawEntry`s
    /// there must be at least one pair of overlapping `RawEntry`s that are next
    /// to each other in the sorted list if any `RawEntry`s overlap.
    fn raw_entry_sort_cmp(&self, a: &Self::RawEntry, b: &Self::RawEntry) -> Ordering;

for BTreeMap, you can just compare the pointer to the node and index in the node, forHashMap and [T] you can just compare the index. for Range on slice indexes, you can compare lexicographicly.


[Admin Cabinet] The dates on the 'Статус замовлення" column are not sorted in the “Я”-”A” order after clicking on the arrow icon

Environment: macOS Big Sur 11.3,Google Chrome 105.0.5195.125
Reproducible:always
Build found: 29.01.2023
Preconditions
User is logged in as administrator.
User navigated to the Orders tab
There is at least two order on the 'Order' table with different  statuses
Steps to reproduce

Click on the arrow icon on the 'Статус замовлення' column
Pay attention on the statuses sorting

Actual result
The statuses of order are not sorted in the  “Я”-”A” format on the 'Статус замовлення" column after clicking on the arrow icon
Expected result
The statuses of order are sorted in the  “Я”-”A” format and icon points upwards  on the 'Статус замовлення" column after clicking on the arrow icon
User story and test case links
E.g.: "User story #2537
Labels to be added
"Bug", Priority ("pri: "), Severity ("severity:"), Type ("UI, "Functional"), "API" (for back-end bugs).

The statuses of order are not sorted in the “Я”-”A” format on the 'Статус замовлення" column after clicking on the arrow icon

Sorting works only in English localization, since property used for sorting - orderStatus, comes in English only. So when we make a request to fetch data in asc or desc order, system does it accordingly to English alphabet, not depending on the language chosen.
The way to resolve the bug is to add orderStatus in both languages in the endpoint  https://greencity-ubs.greencity.social/ubs/management/bigOrderTable, so the frontend can get sorted tables depending on the user's language.

Dublicate #5527


question: antispam logic is strange

Logic in LogRequests method https://github.com/Geta/404handler/blob/6b0c6a09db559c47f4353585c7f8bbd975925091/src/Core/Logging/RequestLogger.cs#L42 looks really strange to me.
In current state it tries to postpone message saving to database until after average requests per second becomes lower than threshold, BUT if it starts saving, it will save whole queue at once regardless number of messages in queue and buffer size - while still locking all the threads here https://github.com/Geta/404handler/blob/6b0c6a09db559c47f4353585c7f8bbd975925091/src/Core/Logging/RequestLogger.cs#L24
So being under heavy (spam?) load, we sooner or later will lock all the threads (because logic just depends on time, as soon as number of seconds > than buffer size, we'll start saving), and start saving all the spam messages in database.
Is it supposed to work in this way?
Or should it instead respect bufferSize during saving to save in batches?
Or maybe it supposed to cleanup queue in warning branch?

Looks like this has changed a bit since it's inception, so I won't assume too much about the current implementation.
However, the original idea was to handle spam or excessive 404 gracefully, preferably lossy, since one could speculate in filling the database over time, taking the site down. With locking, this seems just as important, if not more.


Running with fixed embeddings only

Hi and thanks for putting this code up! Is there a way to run the model with only fixed word embeddings, like glove, fasttext etc., but without bert?

If you comment out line 197-207 in biaffine_ner_model.py and replace line 23 with self.lm_file=None it should work without Bert

Thanks for the quick reply! This runs successfully now, but despite playing around with hyperparameters I can't get it to predict anything but the 'O' category for anything. To give some context, I'm trying to get baseline numbers for various neural systems for nested NER in a low resource setting for span detection. I'm not trying to get high numbers, but ideally non-zero :)
I have:

Fixed 50d word embeddings from a low resource language
~40K train tokens, 10K dev, 10K test, divided into a few documents (about 80 documents total)
~6K nested entity spans within these, in 10 categories, which I collapsed into just one non-'O' category

Loss is decreasing throughout training, but predictions on dev are always O. Oddly, even if I feed train as eval_path again, it still only predicts O in candidate_ner_scores in evaluate. Any ideas would be appreciated!
Here is my config (I tried making the model really small here to get a non-zero result, but I've played with various values):
test = ${base}{
  train_path = train.jsonlines
  lm_path = xyz
  eval_path = dev.jsonlines
  test_path = test.jsonlines
  ner_types = ["thing"]
  char_vocab_path = "char_vocab.txt"
  context_embeddings = ${w2v_50d}
  lm_size = 1
  lm_layers = 1
  flat_ner = false
  contextualization_size = 40
  contextualization_layers = 1

  eval_frequency = 150
  report_frequency = 50
  log_root = logs
  max_step = 8000

  lstm_dropout_rate = 0.2
  lexical_dropout_rate = 0.2
  dropout_rate = 0.2
  learning_rate = 0.001
  ffnn_size = 30
  ffnn_depth = 1
  char_embedding_size = 4

}

And here is some training output with decreasing loss, but 0 f-score on dev:
[50] loss=5979.44, steps/s=25.96
[100] loss=6474.12, steps/s=28.04
[150] loss=5528.80, steps/s=29.18
Loaded 13 eval examples.
Evaluated 1/13 examples.
Evaluated 11/13 examples.
Time used: 0 second, 28924.54 w/s
Mention F1: 0.00%
Mention recall: 0.00%
Mention precision: 0.00%
[150] evaL_f1=0.00, max_f1=0.00 at step 0
[200] loss=4291.26, steps/s=27.66
[250] loss=4213.31, steps/s=28.87
[300] loss=4266.32, steps/s=29.62
Evaluated 1/13 examples.
Evaluated 11/13 examples.
Time used: 0 second, 32010.57 w/s
Mention F1: 0.00%
Mention recall: 0.00%
Mention precision: 0.00%
[300] evaL_f1=0.00, max_f1=0.00 at step 0
[350] loss=4609.43, steps/s=28.40
[400] loss=3626.02, steps/s=28.74
[450] loss=3158.52, steps/s=29.28
Evaluated 1/13 examples.
Evaluated 11/13 examples.
Time used: 0 second, 33300.01 w/s
Mention F1: 0.00%
Mention recall: 0.00%
Mention precision: 0.00%
[450] evaL_f1=0.00, max_f1=0.00 at step 0
[500] loss=3116.23, steps/s=28.55
[550] loss=2190.09, steps/s=29.40
[600] loss=3685.80, steps/s=28.79
Evaluated 1/13 examples.
Evaluated 11/13 examples.
Time used: 0 second, 33556.00 w/s
Mention F1: 0.00%
Mention recall: 0.00%
Mention precision: 0.00%
[600] evaL_f1=0.00, max_f1=0.00 at step 0
[650] loss=1716.10, steps/s=28.78
[700] loss=2771.52, steps/s=28.65
[750] loss=1920.65, steps/s=28.89
Evaluated 1/13 examples.
Evaluated 11/13 examples.
Time used: 0 second, 32705.53 w/s
Mention F1: 0.00%
Mention recall: 0.00%
Mention precision: 0.00%
[750] evaL_f1=0.00, max_f1=0.00 at step 0
[800] loss=1655.52, steps/s=28.60
[850] loss=1689.42, steps/s=28.73
[900] loss=1852.03, steps/s=28.62
Evaluated 1/13 examples.
Evaluated 11/13 examples.
Time used: 0 second, 32728.62 w/s
Mention F1: 0.00%
Mention recall: 0.00%
Mention precision: 0.00%
[900] evaL_f1=0.00, max_f1=0.00 at step 0
[950] loss=1338.44, steps/s=28.34
[1000] loss=1473.17, steps/s=28.32
[1050] loss=1163.35, steps/s=28.45
Evaluated 1/13 examples.
Evaluated 11/13 examples.
Time used: 0 second, 30699.79 w/s
Mention F1: 0.00%
Mention recall: 0.00%
Mention precision: 0.00%

Never mind, I figured it out - BTW I ported this to TF2.2 with tf.compat.v1 and Python 3.X, I can push it to a fork/PR if you're interested

Hi, sorry for late reply:) may I ask how did you solve the all ‘O’ problem? I haven’t this problem in all my experiments. I assume it might because of the size of the corpus. Did you do undersampling?
For the updated code feel free to push them and if you could attach here the address to your repository, so people want to use them can find it easily:) thanks.

OK, my changes are now in the PR in #9
You can also see the low resource parameters I used in the experiments.conf, I'm getting F1=0.757 for span detection (no entity type classification). I'm comparing it to using syntax tree based spans (predicting what should be a candidate for type classification using a normal dependency parser), which currently gets 82.3 for predicted POS tags and parses, and 87 for gold parses.

Thanks a lot, Amir, I've include a link in the readme for people to find your tf2.0 ready code. For the span detection in the under-resourced case, you might want to use undersampling by masking out some of a large portion of negative during the training say give T (e.g. 5) negative examples per positive example. You can do this by simply add a new boolean placeholder (us_masks) same shape as gold_labels modify your code 135-145 as:
   us_ratio = config['under_sampling_ratio'] #can be calculated by T * num_positive_example/num_negative_example
   gold_labels = []
   us_masks = []
   for sid, sent in enumerate(sentences):
       ner = {(s,e):self.ner_maps[t] for s,e,t in ners[sid]}
       for s in range(len(sent)):
         for e in range(s,len(sent)):
           label = ner.get((s,e),0)  if is_training else 0
           gold_labels.append(label)
           mask = (random.rand() < us_ratio if label == 0 else True) if is_traning else True
           us_masks.append(mask)
   us_masks = np.array(us_masks)
   gold_labels = np.array(gold_labels)

    example_tensors = (tokens, context_word_emb,lm_emb, char_index, text_len, is_training, gold_labels,us_masks)

And before compute the loss in line 246:
candidate_ner_scores = tf.boolean_mask(candidate_ner_scores, us_masks)
gold_labels = tf.boolean_mask(gold_labels,us_masks)

I find this method very helpful when dealing with under-resourced cases, (I did for other task using similar architecture)


Add voter info requirement to CHANGES.md

This PR alters the GTFS change process to require some basic information from people who vote on a GTFS proposal. The information required is listed below.

Voter name
Organizational affiliation
Software where the change has been implement (if applicable)
Voter category (producer or consumer)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.giocorti seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

I think this should be in https://github.com/google/transit/blob/master/gtfs/CHANGES.md and https://github.com/google/transit/blob/master/gtfs-realtime/CHANGES.md ?


How do you disable shipping/billing options in card payment?

I tried to use the default option and also give null value for buyer but it doesnt work

can you give me you version you use and reproduce you want?


Support us

General Checks

[ ] the branch is up to date with main/master
[ ] the code works when pulled and run locally
[ ] All CI checks pass
[ ] all conflicts are resolved (if any)
[ ] PR has a descriptive title
[ ] PR has appropriate labels and milestones for easy identification
[ ] PR it is assigned to the owner
[ ] reviewers are assigned
[ ] the PR contributes only one focused change
[ ] It is in the appropriate column in the project board (if necessary)
[ ] has short and clear description
[ ] is linked to an issue (if it is related)
[ ] feedback is addressed (if any and if it is appropriate feedback.)

Markdown

[ ] the markdown source is formatted
[ ] spelling and grammar is correct in all text
[ ] The markdown looks correct when you preview the file
[ ] all links and images work

HTML

[ ] pages besides /index.html are stored in /pages
[ ] the code is well-formatted
[ ] the HTML is valid
[ ] there are no inline styles (example: style='color: red;')
[ ] there are no <style> tags with CSS, all styles are hrefs
[ ] there is no inline JavaScript (example: onclick='doSomething()')
[ ] there are no <script> tags with JS, all JS is in an separate file
[ ] ids are used for JavaScript only, not for CSS
[ ] semantic tags are used
[ ] spelling and grammar is correct in all site content

CSS

[ ] the code is well-formatted
[ ] passes all the linting checks

You can link it this issue to the project board and you can tick the checkboxes.

Check out the layout of the web site.
https://user-images.githubusercontent.com/979362/164925341-9f80de42-93cc-47b6-b0d9-ad6798ae3deb.mp4

Check out the layout of the web site. https://user-images.githubusercontent.com/979362/164925341-9f80de42-93cc-47b6-b0d9-ad6798ae3deb.mp4

Check out the layout of the web site. https://user-images.githubusercontent.com/979362/164925341-9f80de42-93cc-47b6-b0d9-ad6798ae3deb.mp4

Thank you for the suggestion! I never checked how it looks on extra large screens. I will set a max-width

you can add this to the section you are using (you can give it a name):

  <section class="contact-form">
  ...

.contact-form {
  max-width: 1320px;
  margin: 0 auto;
}


Installation for torch 2.5.0

Hi,
I noticed that you have upgraded the installation to torch 2.5.0.
But the official flash attention repository hasn't release a whl for torch 2.5.0. And I have trouble in compiling from source.
Is the flash attention 2.6.3 compatible with torch 2.5.0? If so, could you tell me how you installed it?

Oh probably not yet - flash-attn still hasn't released binaries I think


Change WireRenderFile to $files->render

It is the right way to do now looking in the code.

thanks,
I will put a comment in the docs mention it.
Since both methods will work fine.
It just a matter of taste :)

I have a feeling that this is not just a matter of taste and there could be contexts in which the API variable will work and the function will not. Like in RepeaterMatrix field templates, for example. Can't be 100% sure though.

I think is going to work well in Processwire 3.x
since have namespaces. You can call the function like
\Processwire\wireRenderFile() if you are in a different context.
But maybe in 2.8 will have some troubles if you are in a different context,
and maybe its better to use wire('files')->render() function.


[Feature Request] jexl

Checklist

[X] I searched exising issues and no one else requests similar feature.
[X] I think that 25%+ users are positive towards this feature.

Describe the feature
https://github.com/apache/commons-jexl
Additional context
No response

https://blog.csdn.net/weixin_43830468/article/details/125424422

https://blog.csdn.net/weixin_43830468/article/details/125424422


add electrical tape to lens

this will help to avoid collecting dust on the sensor.

is there a finicky procedure for this I can find on the web, or is it just the obvious?

Lens barrel taped!


Problem occured, internal error

My query just return 50000 lines.
PS. Which .log file I should look into?
System information:

Linux Mint 19.3
21.0.3
Additional extensions

Connection specification:

PostgreSQL 13.3

Does this issue still actual?

I do not have that environment at the moment, so I can not test it.

Please let me know if you find that it's still actual. For now I'm closing it.


nixos/qemu-guest-agent: pkgs.{kvm -> qemu}

there is no top-level pkgs.kvm

there is no top-level pkgs.kvm

It's an alias defined here: https://github.com/NixOS/nixpkgs/blob/340c76263d88bf47119c638164093c63d5292f8b/pkgs/top-level/aliases.nix#L150
But it's better to avoid aliases, so this change makes sense.

I have config.allowAliases = false so hit issues like this :)

But if you build locally, qemu.ga would force a lengthy full qemu build just for the small ga output. I would guess most people run NixOS qemu guests with qemu_test (the default setting) or qemu_kvm.

Maybe not.
I wrote qemu instead of qemu_kvm after noticing that locally I have qemu (as default of virtualisation.libvirtd.qemuPackage) but  qemu_kvm has to be compiled
Perhaps those small derivations of qa and disk image utility of qemu-img (it was discussed some time ago) are better to be made independent on qemu/qemu_test/qemu_kvm.

Good point. So let's merge this as is and consider separating ga later.


Add 10 second FTP timeout

Let's see how this goes on Travis!

Pull Request Test Coverage Report for Build 160

0 of 1 (0.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 65.437%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pudl/datastore.py
0
1
0.0%

Totals

Change from base Build 159:
0.0%

Covered Lines:
2376

Relevant Lines:
3631

💛  - Coveralls

We could definitely switch to urllib instead of ftplib. Reading the docs on urllib.request.Request, it sounds like user-agent (or any other header) is only accepted for HTTP requests.

headers should be a dictionary, and will be treated as if add_header() was called with each key and value as arguments. This is often used to “spoof” the User-Agent header value

and

Headers are currently ignored by all handlers except HTTP handlers

Have you been able to reproduce the slooooooooow downloads of FERC & EPA data locally? Do you want to go ahead and edit the .travis.yml script to make it try and pull down the slow data, so we can see how the timeout affects things?
Hmm, well, if FTP ignores all headers then I don't know why it downloads fast from the browser, but slow via the command line. Any thoughts? I thought maybe it was because I was downloading from outside the US, but again, the fast browser doen't fit with that. Hmm.

The command-line version works quickly (enough) on my computer:
$ time python update_datastore.py --clobber --sources epacems --years 2017 --states ID

Downloading epacems data for 2017...

real	0m11.182s
user	0m1.356s
sys	0m0.109s

Adding a timeout didn't solve the problem, so I'm closing this PR.


Unable to detect “Escape from Tarkov Arena” connect

Hello there!
Currently, the connect of "Escape from Tarkov Arena" on Twitch is the same as the connect with "Escape from Tarkov." However, Twitch is not accurately displaying the connection for "Escape from Tarkov Arena," which leading to the software's inability to initiate the drops activity.
I have conducted tests, and even though Twitch does not exhibit the link with "Escape from Tarkov Arena," the Drops & Rewards is still effective.
Thank you for your time and efforts in maintaining this awesome program!

Hello. I'm assuming you've resolved your issue by yourself?

Hello. I'm assuming you've resolved your issue by yourself?

Kind of. I noticed that Twitch fixed the connect for "Escape from Tarkov Arena" after the first day of drops. Thank you for your attention!


Flutter Doctor is throwing error in macOS Mojave

flutter doctor -v

[✓] Flutter (Channel stable, v1.0.0, on Mac OS X 10.14.2 18C54, locale en-IN)
    • Flutter version 1.0.0 at /Users/stenjoll/github/flutter
    • Framework revision 5391447fae (7 days ago), 2018-11-29 19:41:26 -0800
    • Engine revision 7375a0f414
    • Dart version 2.1.0 (build 2.1.0-dev.9.4 f9ebf21297)

⣾Unhandled exception:
RangeError (index): Invalid value: Only valid value is 0: 1
#0      List.[] (dart:core/runtime/libgrowable_array.dart:145:60)
#1      IntelliJValidatorOnMac.pluginsPath (package:flutter_tools/src/doctor.dart:677:31)
#2      IntelliJValidator.validate (package:flutter_tools/src/doctor.dart:514:53)
<asynchronous suspension>
#3      Doctor.startValidatorTasks (package:flutter_tools/src/doctor.dart:111:52)
#4      Doctor.diagnose (package:flutter_tools/src/doctor.dart:184:41)
#5      _AsyncAwaitCompleter.start (dart:async/runtime/libasync_patch.dart:49:6)
#6      Doctor.diagnose (package:flutter_tools/src/doctor.dart:174:24)
#7      DoctorCommand.runCommand (package:flutter_tools/src/commands/doctor.dart:48:39)
#8      _AsyncAwaitCompleter.start (dart:async/runtime/libasync_patch.dart:49:6)
#9      DoctorCommand.runCommand (package:flutter_tools/src/commands/doctor.dart:34:42)
#10     FlutterCommand.verifyThenRunCommand (package:flutter_tools/src/runner/flutter_command.dart:401:18)
#11     _asyncThenWrapperHelper.<anonymous closure> (dart:async/runtime/libasync_patch.dart:77:64)
#12     _rootRunUnary (dart:async/zone.dart:1132:38)
#13     _CustomZone.runUnary (dart:async/zone.dart:1029:19)
#14     _FutureListener.handleValue (dart:async/future_impl.dart:129:18)
#15     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:642:45)
#16     Future._propagateToListeners (dart:async/future_impl.dart:671:32)
#17     Future._complete (dart:async/future_impl.dart:476:7)
#18     _SyncCompleter.complete (dart:async/future_impl.dart:51:12)
#19     _AsyncAwaitCompleter.complete.<anonymous closure> (dart:async/runtime/libasync_patch.dart:33:20)
#20     _rootRun (dart:async/zone.dart:1124:13)
#21     _CustomZone.run (dart:async/zone.dart:1021:19)
#22     _CustomZone.bindCallback.<anonymous closure> (dart:async/zone.dart:947:23)
#23     _microtaskLoop (dart:async/schedule_microtask.dart:41:21)
#24     _startMicrotaskLoop (dart:async/schedule_microtask.dart:50:5)
#25     _runPendingImmediateCallback (dart:isolate/runtime/libisolate_patch.dart:115:13)
#26     _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:172:5)

It's working from me using Mojave , I'm in dev channel :
flutter doctor -v
[✓] Flutter (Channel dev, v1.0.0, on Mac OS X 10.14.1 18B75, locale en-PE)
    • Flutter version 1.0.0 at /Users/diegoveloper/Development/SDKS/flutter
    • Framework revision 5391447fae (9 days ago), 2018-11-29 19:41:26 -0800
    • Engine revision 7375a0f414
    • Dart version 2.1.0 (build 2.1.0-dev.9.4 f9ebf21297)

Does not work on dev channel for me
./app/flutter/bin/flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel dev, v1.1.4, on Mac OS X 10.14.2 18C54, locale en-US)
⣾Unhandled exception:
RangeError (index): Invalid value: Only valid value is 0: 1
#0      List.[] (dart:core/runtime/libgrowable_array.dart:145:60)
#1      IntelliJValidatorOnMac.pluginsPath (package:flutter_tools/src/doctor.dart:666:31)
#2      IntelliJValidator.validate (package:flutter_tools/src/doctor.dart:509:53)
<asynchronous suspension>
#3      Doctor.startValidatorTasks (package:flutter_tools/src/doctor.dart:112:52)
#4      Doctor.diagnose (package:flutter_tools/src/doctor.dart:185:41)
#5      _AsyncAwaitCompleter.start (dart:async/runtime/libasync_patch.dart:49:6)
#6      Doctor.diagnose (package:flutter_tools/src/doctor.dart:175:24)
#7      DoctorCommand.runCommand (package:flutter_tools/src/commands/doctor.dart:48:39)
#8      _AsyncAwaitCompleter.start (dart:async/runtime/libasync_patch.dart:49:6)
#9      DoctorCommand.runCommand (package:flutter_tools/src/commands/doctor.dart:34:42)
#10     FlutterCommand.verifyThenRunCommand (package:flutter_tools/src/runner/flutter_command.dart:526:18)
#11     _asyncThenWrapperHelper.<anonymous closure> (dart:async/runtime/libasync_patch.dart:77:64)
#12     _rootRunUnary (dart:async/zone.dart:1132:38)
#13     _CustomZone.runUnary (dart:async/zone.dart:1029:19)
#14     _FutureListener.handleValue (dart:async/future_impl.dart:129:18)
#15     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:642:45)
#16     Future._propagateToListeners (dart:async/future_impl.dart:671:32)
#17     Future._complete (dart:async/future_impl.dart:476:7)
#18     _SyncCompleter.complete (dart:async/future_impl.dart:51:12)
#19     _AsyncAwaitCompleter.complete.<anonymous closure> (dart:async/runtime/libasync_patch.dart:33:20)
#20     _rootRun (dart:async/zone.dart:1124:13)
#21     _CustomZone.run (dart:async/zone.dart:1021:19)
#22     _CustomZone.bindCallback.<anonymous closure> (dart:async/zone.dart:947:23)
#23     _microtaskLoop (dart:async/schedule_microtask.dart:41:21)
#24     _startMicrotaskLoop (dart:async/schedule_microtask.dart:50:5)
#25     _runPendingImmediateCallback (dart:isolate/runtime/libisolate_patch.dart:115:13)
#26     _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:172:5)


test pull

Coverage remained the same at 27.646% when pulling 20427c1d1e1bcfc62d7b7d1f4502b8d7d0f1f415 on dev into 072e9944db8fe05cbb47f8ea6d1a327c2a8929b1 on master.

Coverage increased (+2.1%) to 29.781% when pulling 4a0524aee904c01d8dac43309eeae6aab826d8ee on dev into 072e9944db8fe05cbb47f8ea6d1a327c2a8929b1 on master.


kCFBoolean casting failed in Swift 5.1

I got errors when swift test on ubuntu 18.04 with official Swift 5.1 docker image.
It seem like they are all related to kCFBoolean.
There are plenty of similar errors, I just paste a few of them.
/NextBookServer/.build/checkouts/GraphQL/Sources/GraphQL/Map/AnyCoder.swift:2066:27: error: cannot convert value of type 'CFBoolean?' to type 'NSNumber' in coercion
            if number === kCFBooleanTrue as NSNumber {
                          ^~~~~~~~~~~~~~
/NextBookServer/.build/checkouts/GraphQL/Sources/GraphQL/Map/AnyCoder.swift:2068:34: error: cannot convert value of type 'CFBoolean?' to type 'NSNumber' in coercion
            } else if number === kCFBooleanFalse as NSNumber {

Fixed by #54!


Feature: DraggableBlockPlugin for multiplayer

I'm trying to implement the title feature, but with the current implementation that manages orders in arrays, it's difficult to move blocks without conflicts.
Is there any motivation to change to a specification such as fractional indexing used by figma? https://www.figma.com/blog/realtime-editing-of-ordered-sequences/#fractional-indexing
Finally, lexical is really cool, thanks for making it OSS.

All sets of sibling block elements are implemented with a doubly linked list. This is something that we implemented fairly recently for performance reasons, changing from a standard Array.
In terms of adding fractional index, we've not had many reports about the linked lists leading to issues with collaborative editing, yet, and I wouldn't want to complicate an implementation which is already quite complex.

Closing this as I don't think this is a change we will implement.


slint-lsp: 1.4.1 -> 1.5.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/project/slint-lsp/versions.
meta.description for slint-lsp is: Language Server Protocol (LSP) for Slint UI language
meta.homepage for slint-lsp is: https://slint-ui.com/
meta.changelog for slint-lsp is: https://github.com/slint-ui/slint/blob/v1.5.0/CHANGELOG.md
Updates performed

Rust version update

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 1.5.0 with grep in /nix/store/z3invkkk213i3pfzazx9jcybzbxjicpz-slint-lsp-1.5.0
found 1.5.0 in filename of file in /nix/store/z3invkkk213i3pfzazx9jcybzbxjicpz-slint-lsp-1.5.0

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
slint-lsp

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/z3invkkk213i3pfzazx9jcybzbxjicpz-slint-lsp-1.5.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A slint-lsp https://github.com/r-ryantm/nixpkgs/archive/614f9b40d226b8e82bacd986fe142bbe3da885e2.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/614f9b40d226b8e82bacd986fe142bbe3da885e2#slint-lsp

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/z3invkkk213i3pfzazx9jcybzbxjicpz-slint-lsp-1.5.0
ls -la /nix/store/z3invkkk213i3pfzazx9jcybzbxjicpz-slint-lsp-1.5.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

slint-lsp

Maintainer pings
cc @xgroleau for testing.

Result of nixpkgs-review pr 296898 run on x86_64-linux 1

1 package built:

slint-lsp


why not try VOC dataset

I've read your paper and I know your task is scene parsing. But why not experimenting on Pascal VOC datasets ?  Will OCNet achive SoTA on VOC too ?

We don't have a good baseline on VOC dataset.

Can you try to explain why ? Did you train VOC like deeplabv3plus does ? It's very hard to finetune the model to get such  high scores on VOC dataset.  And i want to know if OCNet performs better than PSPNet or Deeplabv3 on VOC datasets. I remember the author of Deeplabv3plus remove average global pooling of ASPP and get better results on cityscapes datasets  and prove image pooing is more suitable to VOC dataset. I think if you can also get a better score on pascal voc dataset you can get an oral.

@idealwei Thanks for your advice.
In fact, we also want to validate our method on VOC. But our baseline is far from the current state-of-the-art method.
It requires much more tricks to achieve good performance on VOC.
It would be great if you have access to Pytorch based baseline on VOC dataset. We can conduct the related experiments quickly.
We believe that Object Context is a very general concept towards better segmentation performance.

@PkuRainBow. You can refer to GluonCV and here is its PyTorch implementation. Hope that can help.


Add the storm/rain condition to pokemon that only has one of the two

Some pokemon like Lotad only spawn during Rain, therefore if storm begins, lotad won't spawn. It would be nice to have pokemon that have that conditions fixed to have both.
All pokemon with that condition I know of are:
-Lotad
-Mudkip
I currently can't think of any other pokemon with them. Will edit if so.

Done, thanks for the suggestion! Closing.


cf set-space-role fails with V6.20 against CF 235

High five for submitting a report!
To avoid additional back and forth, please include the following details as relevant:
Command
cf set-space-role    SpaceDeveloper
CLI Version
Fails with v6.20
Works with v6.13 and below.
Error
Server error, status code: 501, error code: 20005, message: The UAA endpoint needed is disabled
Stack Trace
CC API Endpoint Version
API version 2.54.0
CF Trace
Unavailable
Platform Details
Linux
Shell
BASH

Hi @andrew-edgar
Could you give us the output of cf feature-flags?
Cheers,
Dies Koper
CF CLI PM

Also, are you running it as an admin? Or a non-admin with SpaceManager role on that space?
Can you include the trace when running it with cf 6.13.0?

When we investigated where this error comes from, we found that it appears is returned from the CC when it gets a targeting error from it's UAA client while looking up the user.
Do other commands that require a user work, or is it just the set-space-role? Given the code path that this goes through we think it's more likely that this is a problem with the CC->UAA interaction rather than CLI->CC.

Closing. Feel free to reopen or submit a new issue when ready to provide more details.

cli version 6.22.1+6b7af9c-2016-09-24
cf version 235.
user_org_creation was disabled but I just enabled it for this account and it still failed
Current output:
Features                               State
user_org_creation                      enabled
private_domain_creation                enabled
app_bits_upload                        enabled
app_scaling                            enabled
route_creation                         enabled
service_instance_creation              enabled
diego_docker                           disabled
set_roles_by_username                  enabled
unset_roles_by_username                enabled
task_creation                          disabled
space_scoped_private_broker_creation   enabled
space_developer_env_var_visibility     enabled
The part that probably matters most from the trace at the end
Assigning role OrgManager to user<EMAIL_ADDRESS>in org<EMAIL_ADDRESS>...
REQUEST: [2016-09-30T15:47:32-05:00]
PUT /v2/organizations/bf30ce47-1c88-4d76-9011-7db99b3b8778/managers HTTP/1.1
Host: api.customer1.eu-gb.mydomain.net
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Connection: close
Content-Type: application/json
User-Agent: go-cli 6.22.1+6b7af9c / linux
{"username"<EMAIL_ADDRESS>RESPONSE: [2016-09-30T15:47:32-05:00]
HTTP/1.1 501 Not Implemented
Connection: close
Transfer-Encoding: chunked
Content-Type: application/json
Date: Fri, 30 Sep 2016 20:46:31 GMT
Server: nginx
X-Backside-Transport: FAIL FAIL
X-Content-Type-Options: nosniff
X-Global-Transaction-Id: 147218189
76
{
"code": 20005,
"description": "The UAA endpoint needed is disabled",
"error_code": "CF-UaaEndpointDisabled"
}
0
FAILED
Failed assigning org role to user: Server error, status code: 501, error code: 20005, message: The UAA endpoint needed is disabled

Command I was running was cf create-org<EMAIL_ADDRESS>

The only place in UAA code I see this can be thrown is here:
https://github.com/cloudfoundry/uaa/blob/85fbe8ee080c50a86ac2f90fcdc705e3db6e82eb/client-lib/src/main/java/org/cloudfoundry/identity/client/UaaContextFactory.java
Line 185. Looking in to why this would happen in my env.

I get this issue even with 6.13.0 or 6.12.4

Fix for me was --
userids_enabled: true


Unable to overriding column width for checkboxes and radio buttons on horizontal layout

Describe the bug
Unable to overriding column width for checkboxes and radio buttons on horizontal layout when using angular component.
we want to override the label and options as we do for input/select elements.
Irrespective of the col widths specified label. It is always adding "clr-col-md-10 clr-col-12" for the div and forcing the options to next line even we have lot of space

If we delete these col widths on div added by angular component. It is working as expected.

How to reproduce
https://clarity-form-lmezj2.stackblitz.io
Steps to reproduce the behavior:
https://stackblitz.com/edit/clarity-form-lmezj2
Expected behavior
We should be able to override the column width for radio and checkboxes for inline and horizontal forms.
Versions
App

Angular: 8.x
Node:12.x
Clarity: 2.3.2

Device:

Type: [e.g. MacBook]
OS: windows
Browser chrome,firefox

Additional notes
Add any other notes about the problem here.

Any update on this issue?

This is possible. Due to the design decisions for forms in Clarity 2+ it should be handled on the app side when necessary.
I modified the stackblitz to force one of the forms inputs to expand to the full column width: https://stackblitz.com/edit/clarity-form-lmezj2-mx8e9y
I'm going to close this as a question w/ an answer. Please update if I misunderstood what needs to be accomplished.

@hippee-lee Question is about checkboxes and radio button to layout. Please go through the question once again.
Using angular component controller are always wrapping to next even we have of real estate available. There is no way to override this behavior.
Please reopen this issue.

Sorry, I guess I'm not clear on the expected behavior.
In the example, how should Option 1 and Option 2 layout?
I still think this is something that is overridable on the app side but if I can understand better I'll know how to evaluate the options.

Please reopen the issue first.
It  should layout side by side

When we override the label cols the options are going to next line even  lot of space is available.

Any update on this?

This is working as I see it using your demo, what you might be confused by is that at small screens it will wrap (the clr-col-12 is the culprit for that) so that it better supports mobile layouts and narrow screens.
See https://stackblitz.com/edit/clarity-form-lmezj2-rayhfy?file=src/app/app.component.html

Thanks. This looks little odd syntax.
Please update the doc. It will help the library users a lot. :)


burn-tensor does not compile

Describe the bug

I figured out that burn-tensor does not compile (017485b9ea411fd38e7421a579fcb3d2d7200cfc). I could narrow the problem down to being introduced in bd58922784cdbdc10675b318a56c7e5f42ffb99a. The error message is (signal: 7, SIGBUS: access to undefined memory).
To Reproduce

git checkout bd58922784cdbdc10675b318a56c7e5f42ffb99a
cd burn-tensor/
cargo build

Compiling burn-tensor v0.6.0 (/home/temrix/Dokumente/Coding/MachineLearning/burn/burn-tensor)
error: could not compile `burn-tensor` (lib)

Caused by:
  process didn't exit successfully: `rustc --crate-name burn_tensor --edition=2021 burn-tensor/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=88 --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=6ab9f965acd38970 -C extra-filename=-6ab9f965acd38970 --out-dir /home/temrix/Dokumente/Coding/MachineLearning/burn/target/debug/deps -C incremental=/home/temrix/Dokumente/Coding/MachineLearning/burn/target/debug/incremental -L dependency=/home/temrix/Dokumente/Coding/MachineLearning/burn/target/debug/deps --extern derive_new=/home/temrix/Dokumente/Coding/MachineLearning/burn/target/debug/deps/libderive_new-1e714c8bb5df9345.so --extern half=/home/temrix/Dokumente/Coding/MachineLearning/burn/target/debug/deps/libhalf-fb3f9c287b55cfa3.rmeta --extern hashbrown=/home/temrix/Dokumente/Coding/MachineLearning/burn/target/debug/deps/libhashbrown-14afb34d32240458.rmeta --extern libm=/home/temrix/Dokumente/Coding/MachineLearning/burn/target/debug/deps/liblibm-4529f95fe460a643.rmeta --extern num_traits=/home/temrix/Dokumente/Coding/MachineLearning/burn/target/debug/deps/libnum_traits-76557a49a7b225db.rmeta --extern rand=/home/temrix/Dokumente/Coding/MachineLearning/burn/target/debug/deps/librand-8f02bc213b284c94.rmeta --extern rand_distr=/home/temrix/Dokumente/Coding/MachineLearning/burn/target/debug/deps/librand_distr-c9cf08214addd80c.rmeta --extern serde=/home/temrix/Dokumente/Coding/MachineLearning/burn/target/debug/deps/libserde-437ed021fdd15433.rmeta` (signal: 7, SIGBUS: access to undefined memory)

Expected behavior

burn-tensor should compile successfully.
Desktop (please complete the following information):

OS: Devuan 4 chimaera (like Debian 11 bullseye)
Browser N/A
Version commit bd58922784cdbdc10675b318a56c7e5f42ffb99a
rustc 1.70.0 (90c541806 2023-05-31)

It seems the cargo is failing to build properly. I haven't tried building on my end but can you please upgrade your tool chain to the latest with all patches up to date? Could you rebuild with cargo clean first? Also please make sure you're on rust stable.

It seems to be some weird environment issue on my end because in Docker on the same machine, everything works and the same on my laptop. Tool chain is stable-x86_64-unknown-linux-gnu (the latest at the moment) and I have rebuilt with cargo clean.

In that case, I recommend updating your OS. It could be there is a problem with the linked OS libraries. If you're okay removing ~/.cache, then you should remove it and try it again. Some people report it helps. You can also reinstall cargo.

I tried reinstalling cargo and updating to the next release both of which did not work regarding to this problem. It seems I need to reinstall.


Bot API 4.6

🎉 Feature Request
https://core.telegram.org/bots/api#january-23-2020
Summary
Could you please integrate Polls 2.0
I want to be able to save the quiz poll which was just released.
It would require a new Database Table which I think I wont be able to do as I couldn't understand how the update gets saved to the database.
Any help towards this will be much appreciated.

Thanks for opening @HRankit, I'll add the checklist of what needs to be implemented here:
Supported Polls 2.0.

[ ] Add the KeyboardButtonPollType entity and the field request_poll to the KeyboardButton entity.
[ ] Add updates about changes of user answers in non-anonymous polls, represented by the PollAnswer entity and the field poll_answer in the Update entity.
[ ] Add the fields total_voter_count, is_anonymous, type, allows_multiple_answers, correct_option_id to the Poll entity.
[ ] Add more information about the bot in response to the getMe request: added the fields can_join_groups, can_read_all_group_messages and supports_inline_queries to the User entity.
[ ] Add the optional field language to the MessageEntity entity.

I've coded the main bulk of the Bot API 4.6, but some things aren't so clear on how to proceed regarding the polls and how they are saved in the DB.
With the native polls, each vote sent the Poll object, which was then saved in the database.
Using the Polls 2.0, an anonymous poll vote only sends the PollAnswer but not the Poll itself, so there is a lack of that data.
A solution would be to save every anonymous poll sent via sendPoll directly in the DB as it gets sent out. Would need to check the implications on this.
Also, there will be a new table in the database called poll_answers that has all answers to anonymous polls saved.
@php-telegram-bot/developers
Any thoughts on this?

@noplanman That is exactly what I did. My bot sends polls, as soon as the sendPoll is success, I save it to the DB. When someone answers a quiz mode poll which was created by bot, it saves the poll_answer to the db, else just saves the response of the poll to messages.
About anonymous poll, we can give the user an option to save anonymous polls or not and whether to save it database or not.
That's what I think would be useful to everyone.

Right, that's what I had in mind too, thanks for your comment.
I don't really like the prospect of coupling the DB more into the library though :confused:
Especially as it's an idea to extract all DB features to a separate component, to make the core library lighter.
Will see how to implement this nicely, so that it works for all cases :+1:

To keep the foreign key constraints working, it looks like it will need to be added directly to the Request class, so that every poll gets saved to the DB, regardless of anonymous or not. That way, when poll answers come to the bot, there is a poll to reference it with.
We'll deal with the decoupling of the DB things when the time comes...

@HRankit I'd appreciate a look into the PR I just made for Bot API 4.6 :pray:
#1066


Clone "Manage NuGet Packages" behaviour for adding references to projects

If you right click a project, the "Manage NuGet Packages" dialog will

let you find a NuGet package
update the packages.config
add that package to the project
(crucially) add that package's dependencies to the project.

I would dearly love to see Paket.VisualStudio provide something similar:

let me find a package
add that package & its dependencies to the affected project

What's wrong with the current "add package" dialog in Paket.VisualStudio?
On Oct 21, 2015 12:32 PM, "Frank Shearar"<EMAIL_ADDRESS>wrote:

If you right click a project, the "Manage NuGet Packages" dialog will

let you find a NuGet package
update the packages.config
add that package to the project
(crucially) add that package's dependencies to the project.

I would dearly love to see Paket.VisualStudio provide something similar:

let me find a package
add that package & its dependencies to the affected project

—
Reply to this email directly or view it on GitHub
https://github.com/fsprojects/Paket.VisualStudio/issues/77.

"Add package" lets me easily find a package, it's true. But if I want to add a reference to that package to a project I will still have to right-click References, Browse, dig in the file system, add, for that package and its dependencies.
Concrete example: I want to be able to add Chessie to a project, and have the  tags for Chessie and FSharp.Core automatically added, as well as the necessary binding redirects for the project's app.config.
It's the latter part that's more important than the former: if I had to add a package to the solution in one step, and then had to add the package to the project as a second step, that would still work for me.

Right click on references file of that project should also give you paket
add with the stuff you described.
On Oct 21, 2015 12:45 PM, "Frank Shearar"<EMAIL_ADDRESS>wrote:

"Add package" lets me easily find a package, it's true. But if I want to
add a reference to that package to a project I will still have to
right-click References, Browse, dig in the file system, add, for that
package and its dependencies.
Concrete example: I want to be able to add Chessie to a project, and have
the tags for Chessie and FSharp.Core automatically added, as well as the
necessary binding redirects for the project's app.config.
It's the latter part that's more important than the former: if I had to
add a package to the solution in one step, and then had to add the package
to the project as a second step, that would still work for me.
—
Reply to this email directly or view it on GitHub
https://github.com/fsprojects/Paket.VisualStudio/issues/77#issuecomment-149851621
.

Fabulous! I get an error "The path is not of legal form." though - separate bug report time?

yes. that sounds like a real issue. please try to give us repro steps in separate issue.


UnboundLocalError: local variable 'marginal_dict' referenced before assignment with argument -of json

Hello again! I recently added -of json to my DeepPeptide predict.py command and received the following error:
Writing JSON in interactive_testrun_outputs
Traceback (most recent call last):
  File "predict.py", line 218, in <module>
    main()
  File "predict.py", line 212, in main
    json.dump(marginal_dict, open(os.path.join(args.output_dir, 'sequence_outputs.json'), 'w'), indent=1)
UnboundLocalError: local variable 'marginal_dict' referenced before assignment

marginal_dict is defined in line 172 of predict.py, but only if img is the output format.
https://github.com/fteufel/DeepPeptide/blob/main/predictor/predict.py#L172
however, marginal_dict is referenced in line 197 regardless of output format (img or json)
https://github.com/fteufel/DeepPeptide/blob/main/predictor/predict.py#L197

Oops - looks like i've never actually tested the json only mode in this codebase. Made some fixes and just tested it, should work now.


Meta-Data extension fails when document contains dashes

The Meta-Data extension fails to extract metadata when the metadata is wrapped by three dashes. Example:
---
title: why python is great
author: guide von rossum
date: 2014-05-25
---

some **markdown** content

The dashes have become the de-facto standard for enclosing metadata in Markdown, the Markdown2 module already suports this, see: https://github.com/trentm/python-markdown2/wiki/metadata

@kevvvvv, thanks for your suggestion. However, this was recently added in ef6eb03 and will be available in the next release. As such I am closing this as a duplicate.


Missing resource name causes plugin to fail

Hi,
While attempting to use cfn-python-lint (cfn-lint 0.4.1) with aws-cfn-lint-visual-studio-code (vscode-cfn-lint 0.2.0) I found the extension consistently crashing with my sample template:

Settings have been updated...
Settings: {"cfnLint":{"path":"/usr/local/bin/cfn-lint","appendRules":[],"ignoreRules":[],"overrideSpecPath":""}}
Is CFN: true
running............. /usr/local/bin/cfn-lint --format,json,--template,/Users/MYUSER/Test/test.yaml
2018-07-23 08:23:17,194 - cfnlint - ERROR - Tried to process rules on file /Users/MYUSER/Test/test.yaml but got an error: 'str_node' object has no attribute 'get'
child process exited with code 1 and signal null
undefined:1
SyntaxError: Unexpected end of JSON input
at JSON.parse ()
at ChildProcess. (/Users/MYUSER/.vscode/extensions/kddejong.vscode-cfn-lint-0.2.0/server/out/server.js:150:28)
at emitTwo (events.js:106:13)
at ChildProcess.emit (events.js:194:7)
at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)

YAML source:
---
AWSTemplateFormatVersion: "2010-09-09"
Description: "Test description"
Resources:
  Properties:
    BucketName: "bucket_test"
  Type: "AWS::S3::Bucket"

Eventually I realized I didn't have a resource name and resolved the issue, however, it took several application restarts to sort it out. Any chance you could make the extension handle this situation more gracefully?
  I'm on it.

Thanks for the quick turnaround!


[Bug]: Single quotes in Procedure statement are duplicated

Terraform CLI Version
1.9.1
Terraform Provider Version
0.92.0
Terraform Configuration
resource "snowflake_procedure" "mwe" {
  name        = "mwe_test"
  database    = "TEST"
  schema      = "testschema"
  language    = "SQL"
  comment     = "MWE"
  return_type = "VARCHAR"
  execute_as  = "CALLER"
  statement   = <<-EOT
  DECLARE
  c1 CURSOR FOR SELECT "name" FROM ACTUAL_USERS;
  v_username VARCHAR;
  v_sql_statement STRING;
  BEGIN
      FOR row_value IN c1 DO
          v_username := row_value."name";
          v_sql_statement:= 'SHOW GRANTS TO USER "' || :v_username || '";';
          EXECUTE IMMEDIATE :v_sql_statement;
      END FOR;
  END;
  EOT
}

Category
category:resource
Object type(s)
resource:procedure
Expected Behavior
The given procedure should be available in Snowflake with the line
v_sql_statement:= 'SHOW GRANTS TO USER "' || :v_username || '";';
Actual Behavior
The provider adds single quotes to every single quote, rendering the SQL procedure useless.
v_sql_statement:= ''SHOW GRANTS TO USER "'' || :v_username || ''";'';
Steps to Reproduce

Copy the above configuration, create the database and schema in snowflake.
run terraform apply
look at the procedure in snowflake

How much impact is this issue causing?
High
Logs
No response
Additional Information
No response
Would you like to implement a fix?

[ ] Yeah, I'll take it 😎

From testing and the snowflake query log: It appears that all single quotes in the heredoc get converted to \' in the SDK. The snowflake query will then turn around and convert every \' to '' .

Hey @GeMa300. Thanks for reaching out to us.
The procedure resource will be redesigned as part of https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/ROADMAP.md#preparing-essential-ga-objects-for-the-provider-v1. I will link this issue to https://github.com/Snowflake-Labs/terraform-provider-snowflake/blob/main/v1-preparations/ESSENTIAL_GA_OBJECTS.MD. As a workaround, you can use [unsafe_execute](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/unsafe_execute.

Hey @GeMa300.
In v0.100.0 we have introduced new procedure resources. The statements are not wrapped in double dollars ($$) which allows easy usage of commas. Please check it out and let us know.


net: openthread: Init NCP after USB communication is established

The device has sent RESET_POWER_UP message before the communication with the host has been established. It could be observed with pyspinel (with debugging enabled) which displayed Framing error.
This commit fixes the bug by initializing NCP after the host stated is ready to communicate.
For consistency the behavior is also changed for pure serial communication.

CI is not green. The output message is not clear if it is my fault an I should do sth about it or nor. Anyone help?

@carlescufi Could you help with the CI problem?

I rebased the PR few times and always buildkite/zephyr fail, but it seems the problem is in the CI scripts not the PR itself. Can anyone help?

I rebased the PR few times and always buildkite/zephyr fail, but it seems the problem is in the CI scripts not the PR itself. Can anyone help?

The source of CI problem was that the changed code was not covered by any test.


Add platform keywords

Add platform type keywords: HTML5, WINDOWS, ANDROID, IOS, etc.

Please describe the use case?
You can get the platform from both Lua and C++, so I'm uncertain what's needed and for what.

For example, I have a module that is for platforms other than html. If I do a check in the code, then the module still gets into the build. I would like to get rid of it so that the html build is as small as possible.
    if not html5 then
        local mock_api = require("gamepush.mock_api")
        --The mock_api module still gets into the build!
    end


Exclusion Report | Compliance Manager Recommend Acceptance |1|

Story:
As a Compliance Manager, I want to be able to recommend acceptance of an exclusion report.
Scenario:
GIVEN that I am an IDIR user
AND I have the Compliance Manager role
AND I am viewing a reviewed Exclusion Report (i.e. front-end status = Reviewed - Analyst; back-end status = Recommended Acceptance - Analyst or Recommended Rejection - Analyst)
AND I am on the Exclusion Agreements page
WHEN I click on the Recommend Acceptance button
AND I accept the confirmation alert
THEN I am returned to the main compliance reporting page
AND the status of the report in the UI changes to Reviewed - Manager (back-end status should be "Recommended Acceptance - Manager")
AND a report history record is created (see below)
AND a system alert appears on the screen
Status update visibility:

The status change should only be visible to IDIR users. For BCeID users, the status should remain at Submitted.

Report History Record:
A report history record should be created when the Compliance Manager recommends acceptance of the report. The format should be as follows:

Reviewed and <Status> on Month DD, YYYY by <First, Last Name> of <Organization Name>
Example: Reviewed and Recommended Acceptance on March 31, 2019 by Justin Lepitzki of Government of British Columbia

Note: This report history record should only be visible to IDIR users.
Confirmation Alert Messaging:

Are you sure you want to recommend acceptance of the exclusion report?

System Alert Messaging:

Acceptance Recommended

https://trello.com/c/WmzIEyry/1690-exclusion-report-compliance-manager-recommend-acceptance-1

Duplicate. Please refer to #1439


Line numbers in interactive window debugging is incorrect

Looks like the interactive window debugging is broken in main branch.
We need the fixes from cell hash providers.

Not fixed.


Architecture isn't supported: armv8l

LXD 5.0.0 has been uploaded to the Debian archive for the first time, and is hitting the various buildds. Builds for both the armel and armhf architectures compile fine, but several of the tests fail with the error "Architecture isn't supported: armv8l". For example,
=== RUN   TestIntegration_UnixSocket
    testing.go:37:
        	Error Trace:	/<<PKGBUILDDIR>>/_build/src/github.com/lxc/lxd/lxd/testing.go:37
        	            				/<<PKGBUILDDIR>>/_build/src/github.com/lxc/lxd/lxd/daemon_integration_test.go:36
        	            				/<<PKGBUILDDIR>>/_build/src/github.com/lxc/lxd/lxd/daemon_integration_test.go:19
        	Error:      	Received unexpected error:
        	            	Architecture isn't supported: armv8l
        	Test:       	TestIntegration_UnixSocket
--- FAIL: TestIntegration_UnixSocket (0.00s)

It looks like the logic for testing supported platforms is in shared/osarch/architectures.go. I'm not super familiar with all the different ways arm architectures are identified, but given that LXD compiled and lots of other tests pass during each build, I hope it won't be too difficult extend the supported architecture matrix so the tests can be run on these two additional architectures.
I am working on uploading LXD 5.0.1, but I don't think that will have any affect on this issue.
Buildd logs:
https://buildd.debian.org/status/fetch.php?pkg=lxd&arch=armel&ver=5.0.0-1&stamp=1662903087&raw=0
https://buildd.debian.org/status/fetch.php?pkg=lxd&arch=armhf&ver=5.0.0-1&stamp=1662902755&raw=0

Ah yeah, armv8l isn't really a normal architecture, it's the 32bit personally of some arm64 architectures.
We can likely fix it by adding it as an alias for armhf which is true enough.


Add tests for memory validation

This is a top-level issue to capture adding validation tests for memory-related validation cases. The comprehensive list of validation layer checks is contained in vk_validation_layer_details.md. Checks that don't currently have tests identified are below. Tests should be added where missing and the document should be updated.
INVALID_ALIASING
INVALID_LAYOUT
MISSING_MEM_BINDINGS
MEMORY_BINDING_ERROR
MEMORY_LEAK
MEM_OBJ_CLEAR_EMPTY_BINDINGS
INVALID_MAP

#529 took care of a few of the checks, here's the details of remaining work:

INVALID_ALIASING - 2 checks in print_memory_range_error reached via BindBufferMemory and BindImageMemory
MISSING_MEM_BINDINGS - 2 checks, 1 in set_sparse_mem_binding (reached via QueueBindSparse), 1 in set_mem_binding (reached via BindBufferMemory and BindImageMemory)
MEMORY_LEAK - 1 check in deleteRenderPasses called by DestroyDevice
INVALID_MAP - 4 checks in validateMemRange (called by MapMemory), 1 check in deleteMemRanges (called by UnmapMemory), 2 checks in validateMemoryIsMapped (called by FlushMappedMemoryRanges & InvalidateMappedMemoryRanges), 2 checks in validateAndCopyNoncoherentMemoryToDriver (called by FlushMappedMemoryRanges)

#533 finishes up the work on this issue

Fix was cherry-picked and pushed to master


MD5SUM updating at every execution

Environment
Liquibase Version: Community 4.14
Liquibase Integration & Version: CLI
Liquibase Extension(s) & Version: Oracle 4.14, Teradata 4.14, neo4j 4.14, netezza 4.14 (dbeaver extension)
Database Vendor & Version: Oracle <IP_ADDRESS>
Operating System Type & Version: Linux 5.4.0-1084-aws #91~18.04.1-Ubuntu SMP x86_64 GNU/Linux
Infrastructure Type/Provider: AWS
Description
I have a customer who has a large number of changesets that get the MD5SUM updated in databasechangelog at the start of every deployment, even though the changesets have not been modified and the changesets are not actually executed.  The Liquibase output looks like this:
[2022-09-02 15:46:24] INFO [liquibase.executor] UPDATE TST4CIS.DATABASECHANGELOG SET MD5SUM = '8:cf74d98bf9923254d88d77efba90e1ec' WHERE ID = 'ads_cisd015.pks' AND AUTHOR = 'xxx' AND FILENAME = 'Packages/db-changelog-1.xml'
[2022-09-02 15:46:24] INFO [liquibase.executor] 1 row(s) affected
[2022-09-02 15:46:24] INFO [liquibase.executor] UPDATE TST4CIS.DATABASECHANGELOG SET MD5SUM = '8:a1d70adee4b5b830d4b5ab103c371548' WHERE ID = 'ads_cisd109.pks' AND AUTHOR = 'xxx' AND FILENAME = 'Packages/db-changelog-1.xml'
[2022-09-02 15:46:24] INFO [liquibase.executor] 1 row(s) affected
[2022-09-02 15:46:24] INFO [liquibase.executor] UPDATE TST4CIS.DATABASECHANGELOG SET MD5SUM = '8:859cb9b9b4174f5322329ee434232c55' WHERE ID = 'bobfixtca.pkb' AND AUTHOR = 'xxx' AND FILENAME = 'Packages/db-changelog-1.xml'
[2022-09-02 15:46:24] INFO [liquibase.executor] 1 row(s) affected

It's strange because it's not all of his changesets, only the changesets that contain PL/SQL (packages, procedures, trigggers, etc) , executed via sqlFile, and the files have .pkb, .pks, .prc, and .sql extensions.
This seems to have started after two changes we made:

Upgraded to 4.12
Added sql-log-level=INFO

This is only occurring with this one customer (we have 100s) so I'm assuming it's something about his changelogs that is causing this, but I don't see anything obvious.

@doakd Would it be possible for you to share the changelogs that are leading to this behavior? I understand if you prefer to keep those private; you can email them to me at<EMAIL_ADDRESS>(with any private info redacted/removed of course).
Also would be helpful to see one that is not having this issue.
What version did you update from to 4.12 when you started to see this issue?
Thanks!

Files have been sent.
Looks like the issue is only occurring when sqlFile is used in xml (that I'm aware of anyway).  Does not occur when .sql files are used.
We were previously using 4.9.1.

Is there an update on this issue?
This is also preventing runOnChange from working properly, so the changesets have to be converted to runalways to a have a modification to the PL/SQL executed.
Thanks.

Hey Daryl.  Nathan and Fede are looking into it.  We'll post an update as soon as we have one. Thanks for following up!

Hi! we have been looking at this issue and we are unable to find out if this issue is a bug on our code or is an error on your end. Is it possible that your changing MD5SUM on one part of your code or inside of one of your changelogs? The other possible root source can be that your jumping between liquibase versions, but it doesnt make sense just to be in some of the changelogs and not in all of them, maybe the job that its executing these especific changesets has a different liquibase version than the rest?

As a little more background: the only times you should get a UPDATE TST4CIS.DATABASECHANGELOG SET MD5SUM = '8:cf74d98bf9923254d88d77efba90e1ec' WHERE ID ... message is when the md5sum value is either NULL or starting with a different "version" (the number before the colon).
If it was a problem where the checksum keeps being computed differently each time, you'd see the md5sum updated as part of a larger updates statement that also sets the description, marks it as reran, etc.
The only time you'd have a difference in the version is when you are going between liquibase versions where we had a large algorithm change. These are few and far between, but would only be happening if you ran different liquibase versions on different runs. Is that something you are doing?
The only time the the md5sum would be null is if you ran changelogsync or had some other external process that sets the checksum to be null.  Any chance you're doing something like that somewhere?
In either of those two cases, the fact that it's only a few of the checksums that have a problem is interesting. Running changelogsync or running update with an old version of liquibase will update ALL the checksums, not just some. So if there is something outside of each liquibase update changing those, they are being ran against a child changelog instead of master? Or something manually that's just targettting those changesets?

The MD5SUM values are not NULL.
The MD5SUM values are not being updated outside of Liquibase.
changelogsync has not been run.
We are only using version 4.14.
Only files using sqlFile from an xml changelog are impacted.  The .sql files that are using include from xml are fine.

Same issue.
This is the error that I see in the logs:
config/liquibase/changelog/00000000000000_initial_schema.xml::00000000000001::jhipster was: 8:8bacaa5647e7cc5246fc7d5c5d4d7cde but is now: 8:3fa67a5aeb6197bd6ea8fb1df5ee943f
The file<PHONE_NUMBER>0000_initial_schema.xml not has changes since some time ago, and I'm confident that my database is fine.
so I made:
update DATABASECHANGELOG set MD5SUM='8:3fa67a5aeb6197bd6ea8fb1df5ee943f'
where MD5SUM='8:8bacaa5647e7cc5246fc7d5c5d4d7cde';

After restart the server (only restart, without any change) I get a different error.
config/liquibase/changelog/00000000000000_initial_schema.xml::00000000000001::jhipster was: 8:3fa67a5aeb6197bd6ea8fb1df5ee943f but is now: 8:68f03dcc3c4ecfeba73f36c52d3cb4b4
So MD5 changes after some deploys/restart.
I have fixed my problem with
update DATABASECHANGELOG set md5sum=null
But I don't understand the root cause.
My App is running on AWS Elastic Beanstalk with Tomcat 8.5 with Corretto 11 running on 64bit Amazon Linux 2/4.2.16

Are you able to enable fine-level logging which should output something like:
FINE [liquibase.util] Computed checksum for createTable:[
    columns=[
        [
            name="id"
            type="int"
        ]
    ]
    tableName="test_table2"
] as 0c3ee0cfd63d1fc0afeb2418835cba2a

for config/liquibase/changelog/00000000000000_initial_schema.xml::00000000000001::jhipster ?
That will give us the actual string being checksumed and we should be able to see a difference between the two runs in those strings the checksum is based off of.

I'll try to reproduce the error, but is a production environment, it will take some time.

Hi @nvoxland and all,
Thanks for keeping alive this issue.
We have a similar issue. In our case, we moved our environment from 4.4.3 to 4.16.1 and checksums calculated with the new version are different, despite of not doing any change to the changeSet (and regardless the fact that both algorithm versions are the same, version 8).
1 changesets check sum           update/services/messaging/messaging_tables.xml::1559570706::mw185149 was: 8:e1aa84a7a58273fe5908ad86a589612d but is now: 8:dce471b939276f17937a806ee31e276b
We have also noticed that even the calculate-checksum generates a totally new checksum value for the same changeSet. That is, I am having at least 3 different checksum values
liquibase calculate-checksum update/services/messaging/messaging_tables.xml::1559570706::mw185149 --defaultsFile=./cxthcs.onprem.oracle.user.properties --changeLogFile=./update/cxthcs_objects.xml                      ####################################################  LOGO HERE #################################################### Starting Liquibase at 09:43:47 (version 4.16.1 #4594 built at 2022-09-14 15:27+0000) Liquibase Version: 4.16.1 Liquibase Community 4.16.1 by Liquibase 8:f8815ddd856f873e8dd36743b09635ea Liquibase command 'calculate-checksum' was executed successfully.
The issue is not limited to the Oracle environment (this issue was tagged with the DBOracle label) but also in SQL Server. Also, it doesn't seem related to a specific ChangeType but to the Liquibase version change. However, checksums from the calculate-checksums command between version 4.16.1 and 4.17.1 are the same.
As a workaround, we are clearing the checksums during the version upgrade, but a proper solution is truly appreciated.
Looking forward to hearing from you.
Thanks!

We're still having a hard time finding a way to reproduce this to understand why it's happening. @ic185037 are you able to enable fine level logging to get the output like:
FINE [liquibase.util] Computed checksum for createTable:[
    columns=[
        [
            name="id"
            type="int"
        ]
    ]
    tableName="test_table2"
] as 0c3ee0cfd63d1fc0afeb2418835cba2a

for times it ends up computing different checksums? That's what we need to really understand what's going on

Hi @nvoxland
Thanks for your quick reply
Please find below the information you asked me:
Checksum information by the createTable changeSet executed with 4.4.3
[2022-11-11 06:34:49] FINE [liquibase.util] Computed checksum for createTable:[     columns=[         [             [                 nullable="false"                 primaryKey="true"                 primaryKeyName="PK_cs_guaranteed_delivery"             ]             autoIncrement="true"             name="internal_id"             type="bigint"         ],         [             [                 nullable="false"             ]             name="message_header"             type="nclob"         ],         [             [                 nullable="false"          ... [truncated in log] as 2ee3b0b9cebc0f4dc24d5293b30dad26 [2022-11-11 06:34:49] FINE [liquibase.util] Computed checksum for replace:[     applicableDbms=[         oracle,         mssql     ]     applyToRollback="false"     contexts="non-iot"     replace=", CONSTRAINT PK_cs_guaranteed_delivery PRIMARY KEY (internal_id)"     with="" ] as d0e8ec5e3b5380571154dfb483e6080a [2022-11-11 06:34:49] FINE [liquibase.util] Computed checksum for regExpReplace:[     applicableDbms=[         oracle,         mssql     ]     applyToRollback="true"     replace="((?:[A-Z]+)_(?:DATA|INDX)_(?:FG|DF))"     with="CXTHCS_$1" ] as 662da3b53a7d2ec55863e896369220ea [2022-11-11 06:34:49] FINE [liquibase.util] Computed checksum for 8:2ee3b0b9cebc0f4dc24d5293b30dad26:8:d0e8ec5e3b5380571154dfb483e6080a;8:662da3b53a7d2ec55863e896369220ea; as e1aa84a7a58273fe5908ad86a589612d
Checksum information by the createTable changeSet executed with 4.16.1
`[2022-11-11 07:45:28] FINE [liquibase.util] Computed checksum for createTable:[
columns=[
[
[
nullable="false"
primaryKey="true"
primaryKeyName="PK_cs_guaranteed_delivery"
]
autoIncrement="true"
name="internal_id"
type="bigint"
],
[
[
nullable="false"
]
name="message_header"
type="nclob"
],
[
[
nullable="false"
... [truncated in log] as 2ee3b0b9cebc0f4dc24d5293b30dad26
[2022-11-11 07:45:28] FINE [liquibase.util] Computed checksum for replace:[
applicableDbms=[
oracle,
mssql
]
applyToRollback="false"
contextFilter="non-iot"
replace=", CONSTRAINT PK_cs_guaranteed_delivery PRIMARY KEY (internal_id)"
with=""
] as f60940602dcd93a68b2b4a93da27ac35
[2022-11-11 07:45:28] FINE [liquibase.util] Computed checksum for regExpReplace:[
applicableDbms=[
oracle,
mssql
]
applyToRollback="true"
replace="((?:[A-Z]+)(?:DATA|INDX)(?:FG|DF))"
with="CXTHCS_$1"
] as 662da3b53a7d2ec55863e896369220ea
[2022-11-11 07:45:28] FINE [liquibase.util] Computed checksum for 8:2ee3b0b9cebc0f4dc24d5293b30dad26:8:f60940602dcd93a68b2b4a93da27ac35;8:662da3b53a7d2ec55863e896369220ea; as dce471b939276f17937a806ee31e276b
[2022-11-11 07:45:28] FINE [liquibase.executor] Release Database Lock
[2022-11-11 07:45:28] FINE [liquibase.executor] Executing with the 'jdbc' executor
[2022-11-11 07:45:28] INFO [liquibase.lockservice] Successfully released change log lock
[2022-11-11 07:45:28] FINE [liquibase.configuration] No configuration value for liquibase.outputLineSeparator found
[2022-11-11 07:45:28] FINE [liquibase.configuration] Configuration liquibase.outputLineSeparator is using the default value of
[2022-11-11 07:45:28] SEVERE [liquibase.integration] Validation Failed:
1 changesets check sum
update/services/messaging/messaging_tables.xml::1559570706::mw185149 was: 8:e1aa84a7a58273fe5908ad86a589612d but is now: 8:dce471b939276f17937a806ee31e276b
liquibase.exception.CommandExecutionException: liquibase.exception.ValidationFailedException: Validation Failed:
1 changesets check sum
update/services/messaging/messaging_tables.xml::1559570706::mw185149 was: 8:e1aa84a7a58273fe5908ad86a589612d but is now: 8:dce471b939276f17937a806ee31e276b
at liquibase.command.CommandScope.execute(CommandScope.java:179)
at liquibase.integration.commandline.CommandRunner.call(CommandRunner.java:51)
at liquibase.integration.commandline.CommandRunner.call(CommandRunner.java:21)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
at picocli.CommandLine.execute(CommandLine.java:2078)
at liquibase.integration.commandline.LiquibaseCommandLine.lambda$execute$1(LiquibaseCommandLine.java:353)
at liquibase.Scope.child(Scope.java:189)
at liquibase.Scope.child(Scope.java:165)
at liquibase.integration.commandline.LiquibaseCommandLine.execute(LiquibaseCommandLine.java:318)
at liquibase.integration.commandline.LiquibaseCommandLine.main(LiquibaseCommandLine.java:84)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at liquibase.integration.commandline.LiquibaseLauncher.main(LiquibaseLauncher.java:107)

Caused by: liquibase.exception.ValidationFailedException: Validation Failed:
1 changesets check sum
update/services/messaging/messaging_tables.xml::1559570706::mw185149 was: 8:e1aa84a7a58273fe5908ad86a589612d but is now: 8:dce471b939276f17937a806ee31e276b
at liquibase.changelog.DatabaseChangeLog.validate(DatabaseChangeLog.java:339)
at liquibase.Liquibase.lambda$update$1(Liquibase.java:229)
at liquibase.Scope.lambda$child$0(Scope.java:180)
at liquibase.Scope.child(Scope.java:189)
at liquibase.Scope.child(Scope.java:179)
at liquibase.Scope.child(Scope.java:158)
at liquibase.Liquibase.runInScope(Liquibase.java:2414)
at liquibase.Liquibase.update(Liquibase.java:209)
at liquibase.Liquibase.update(Liquibase.java:195)
at liquibase.integration.commandline.Main.doMigration(Main.java:1845)
at liquibase.integration.commandline.Main$1.lambda$run$0(Main.java:398)
at liquibase.Scope.lambda$child$0(Scope.java:180)
at liquibase.Scope.child(Scope.java:189)
at liquibase.Scope.child(Scope.java:179)
at liquibase.Scope.child(Scope.java:158)
at liquibase.integration.commandline.Main$1.run(Main.java:397)
at liquibase.integration.commandline.Main$1.run(Main.java:219)
at liquibase.Scope.child(Scope.java:189)
at liquibase.Scope.child(Scope.java:165)
at liquibase.integration.commandline.Main.run(Main.java:219)
at liquibase.command.AbstractCliWrapperCommandStep.run(AbstractCliWrapperCommandStep.java:33)
at liquibase.command.CommandScope.execute(CommandScope.java:173)
... 20 more

`
As you can see, the issue seems to be related with the computed checksum for the first replace subtag (which are different regardless nothing was changed).
`[2022-11-11 06:34:49] FINE [liquibase.util] Computed checksum for replace:[
applicableDbms=[
oracle,
mssql
]
applyToRollback="false"
contexts="non-iot"
replace=", CONSTRAINT PK_cs_guaranteed_delivery PRIMARY KEY (internal_id)"
with=""
] as d0e8ec5e3b5380571154dfb483e6080a
[2022-11-11 07:45:28] FINE [liquibase.util] Computed checksum for replace:[
applicableDbms=[
oracle,
mssql
]
applyToRollback="false"
contextFilter="non-iot"
replace=", CONSTRAINT PK_cs_guaranteed_delivery PRIMARY KEY (internal_id)"
with=""
] as f60940602dcd93a68b2b4a93da27ac35
`
Thanks for your time and help on this. Looking forward to hearing from you.
Regards,
Italo.

Thanks for the logs, they do point out where the problem is. It's if you have a <modifySql> tag with a context attribute on elements within it. We had changed that attribute from context to contextFilter to improve usability and support backwards compatibility when specifying just context, BUT the attribute name is used in the checksum string and so we get a difference there.
We'll have to find a way to preserve the checksum even with the new canonical attribute name.

It happens everytime "update" is run.  no "clear-checksums" or clear md5sum is being run.  The only examples I have are what I pasted above.

Hello @ic185037 - PR #3530 aims to fix the issue you raised. By reviewing the code it is clear that md5 generation for some sql functions changed on Liquibase 4.16 . When this gets merged it will be back to how it was before - but then MD5 generated from 4.16 until PR release will be broken (probably 4.18.1 or 4.19). In that case, the fix is what you did: clear md5sum before running liquibase or use "clear-checksums" command.

Hi @filipelautert
Thank you for your reply and for your time and effort on this fix. I sincerely appreciate it. Do you have an estimated release date for 4.18.1 or 4.19 releases?
Thanks

@ic185037 we try to have a release each 4~5 weeks, so maybe we may release it around Xmas/new year.

@doakd sorry, I'm not able to reproduce the issue. But @nvoxland found out a different bug that may be related to it: https://github.com/liquibase/liquibase/issues/3549 .

@ic185037 we try to have a release each 4~5 weeks, so maybe we may release it around Xmas/new year.

Hi @filipelautert @nvoxland @XDelphiGrl @MalloD12
Sorry to bother you on this issue again.
I cannot see this issue fixed in the latest release (4.19.0). From the latest Release Notes, I cannot see neither PR #3269 nor PR #3530 but they seems to be Closed. Was this issue fixed as part of a different PR included in 4.19.0?
Thanks and Regards.

Hi @ic185037 - at first we were implementing a fix specific to one of those issues. But as we tried to fix it we found more and more problems, so we decided to work on a bigger Pr that will fix it for once.
As it will be a big change we will need to bump the md5sum algorithm version (that will trigger a md5 recalculation for everyone), so we are trying to add all the possible fixes in this pr. We may merge it to master branch next week and hopefully it will make to next release.
Sorry for the delay on fixing this. The PR including the fixes is https://github.com/liquibase/liquibase/pull/3616 .

Hi @filipelautert - Thanks for your quick ran comprehensive reply. I understand the issues you have faced, you have nothing to apologise for. We will wait for the next release. Thanks again.

Liquibase 4.23.0 includes the checksum upgrade algorithm that fix this issue. Please reopen it if the problem remains.


uds.ugrid.sel() cannot return dataset with new dimension like ds.sel() can do

This issue will be updated with a comparable regularnc case and a public dataset.
import xugrid as xu
import xarray as xr

filename = r'p:\\archivedprojects\\11206044-002ospareutrophication\\final_results_Snellius_latestversion_20220210\\A16b_Numtopsiguniform1_2015_pH\\DFM_OUTPUT_DCSM-FM_0_5nm_waq\\DCSM-FM_0_5nm_waq_0146_map.nc'
all_xu = xu.open_mfdataset(filename, chunks={'time':1})

data_interp = xr.Dataset()
data_interp['plipoint_x'] = xr.DataArray([3.5,3.6,3.7],dims=('plipoints'))
data_interp['plipoint_y'] = xr.DataArray([51.7,51.8,51.9],dims=('plipoints'))

all_xu_sel = all_xu.ugrid.sel(x=data_interp['plipoint_x'], y=data_interp['plipoint_y'])[['mesh2d_s1']]
#TODO: results in dataset with 7 reults, while it should give 3 results

Instead of .ugrid.sel() one should use .ugrid.sel_points() to achieve the desired behaviour. This is deliberately different than in xarray, since (x)ugrid dimensions are explicitly spatial.
all_xu_sel = all_xu.ugrid.sel_points(x=[3.5,3.6,3.7], y=[51.7,51.8,51.9])[['mesh2d_s1']]


Can other ctcdecode be used instead?

Hello,
If parlance/ctcdecode cannot be installed, can I use other ctcdecode instead?
Thank you.

Hi, ctcdecode installation fails using newer PyTorch versions, PyTorch 2.0.0 is working fine. You may be able to use other libraries for beam search decoding but you will need to modify the code.