Segment based addresses need to be interpreted, understood and utilised in further analysis phases correctly.

Is your feature request related to a problem? Please describe.
The interpretation of segmented addresses.
I have determined that although Ghidra does understand segment:offset style of addresses (like that used in protected-mode Windows 1990's) for disassembling, it struggles with the later analysis and turning it into decent C.
There seems to be the following kinds of addresses in this category:
A. Where just the offset is given;
B. Passed as function parameters;
C. References to sub-structures.
Currently there are the following issues:

Numeric values are being misinterpreted as an offset for an address in the data segment (DS is being applied when it shouldn't - default segment usage is not shown in the ListingPanel and presumably not propagated to further analysis phases);
In a function parameter where the address is of thesegment:offset form, an offset to a sub-structure within said parameter is being misunderstood and comes out as either CONCAT2(xxx,yyy+offset) or even worse CONCAT13(xxx,CONCAT12(yyy,zzz+offset));
Passing the full stack address (including the SS) to another function looks like this CString::~CString((CString *)CONCAT13(uVar6,CONCAT12(uVar4,&local_1e)),2) instead of CString::~CString(&local_1e,2) or better still local_1e.~CString(2);
Arrays circum to a similar fate in that ones declared as global (on the DS) look like:

               uVar7 = *(undefined2 *)((int)g_apszProfileStrings_2c1e + 2 + nIdx * 4);
               uVar8 = *(undefined2 *)((int)g_apszProfileStrings_2c1e + nIdx * 4);

instead of psz = g_apszProfileStrings_2c1e[nIdx];
5. I'm sure there's more, but all stem from the same fundamental issue of not interpreting the segment portion of addresses correctly.
Describe the solution you'd like
Correct interpretation of segmented addresses so that all the later great features of Ghidra would be utilised.
Describe alternatives you've considered
Setting the DS to 0 on the individual assembly instruction DOES eliminate issue 1.
Turning pointers into a complex union (see #4469) helps a little with issue 2 and 4 when array is part of a structure whose full address has been passed to another function call.
Others I haven't found anything decent yet!
Additional context
I've been using a modified x86-16.cspec file together with my fork of Ghidra that I sync up regularly!

Here's a better example from where a structure member (in this case a seg:offpointer) is required from within a parent structure (pointed at from the functions first parameter) where at offset 0x46 a pointer is defined as DgnVocabularyList*32:
   FUN_10e8_05c2((DgnVocabularyList *)CONCAT22(param_1._2_2_,(int)param_1 + 0x46),
                 PROFILEINDEX_Systems_Shared_Vocabularies,1 | PROFSECT_IsDirectory);

Here's a better example from where a structure member (in this case a seg:offpointer) is required from within a parent structure (pointed at from the functions first parameter) where at offset 0x46 a pointer is defined as DgnVocabularyList*32:
   FUN_10e8_05c2((DgnVocabularyList *)CONCAT22(param_1._2_2_,(int)param_1 + 0x46),
                 PROFILEINDEX_Systems_Shared_Vocabularies,1 | PROFSECT_IsDirectory);

As you can see the first parameter (above) the segment portion of the address associated with the member at offset 0x46 is the same as the segment of param_1 (bytes 3 & 4, or param_1._2_2_) and the offset is the 0x46th byte of the parent structure, but Ghidra isn't capable of understanding this and consequently misses the glaringly obvious fact it should be finding and using the datatype at that objects offset.

Another problem with the x86-16 sleigh is that it implicitly assumes that the segment register is a multiple of 0x1000 when resolving certain calls/jumps (also see related issue #2892). This would probably need to be fixed as well for the decompiler to have a chance:
currentCS: CS is protectedMode=0 & CS { tmp:4 = (inst_next >> 4) & 0xf000; CS = tmp:2; export CS; } :CALL rm16	    is $(LONGMODE_OFF) & vexMode=0 & addrsize=0 & opsize=0 & byte=0xff & currentCS; rm16 & reg_opcode=2 ...	{ local dest:4 = segment(currentCS,rm16); push22(&:2 inst_next); call [dest]; } :JMP rm16	is vexMode=0 & addrsize=0 & opsize=0 & byte=0xff & currentCS; rm16 & reg_opcode=4 ...	{ target:4 = segment(currentCS,rm16); goto [target]; }

Another problem with the x86-16 sleigh ...

@sad-dev not sure that's exactly the same now, as I think the loaders have changed.  In my case (I'm using protected-mode) my code segments are multiples of 8 and calls seem to have the correct addresses filled in (presumably by the loader).
What gets me is that almost all my pointers are effectively 32-bit (as load/store operations always use a segment register, either implicitly or explicitly) but in x86-16.cspec the pointer size is 16 bits.  (Should it be 32???)  Also, looking through the decompiler code there seems to be support for near pointers, which to me seems to allude that the pointer size should be 32 bits and, where pointers are given as 16-bit, the hiword should be obtained from the appropriate segment register/memory/stack address!
Longs are another issue!!!

I have not touched it in a while, but from memory the pattern was something like this:
1000:F000      e8 00 10       CALL ??
1000:F003      ...

1F00:0000      e8 00 10       CALL ??
1F00:0003      ...

With a segment of 1000, this resolves to a call to 1000:0003 as 0xf003 + 0x1000 becomes 3
With a segment of 1F00, this resolves to a call to 1F00:1003.
It is possible that subsequent improvements have fixed this - I will double check when I can, although my guess is that no, the sleigh is still used to resolve the references.

IS this similar, or different (but maybe related) to an issue I am facing while working on disassembling a game that runs on Sony's PS2/Emotion Engine, in that even though the data structure is defined, and applied correctly to an address, it still shows up in decompiled functions as (var_name.first_field + [offset in hex])?
(perhaps a silly question, perhaps I am doing something wrong if not a related (or the same) issue?

I have not touched it in a while, but from memory the failure pattern I ran into was something like this - both 1000:F000 and 1F00:0000 correspond to the same address but can resolve the CALL operation very differently:
1000:F000      e8 00 10       CALL ??
1000:F003      ...

1F00:0000      e8 00 10       CALL ??
1F00:0003      ...

With a segment of 1000, this resolves to a call to 1000:0003 as 0xf003 + 0x1000 becomes 3 With a segment of 1F00, this resolves to a call to 1F00:1003.
It is possible that subsequent improvements have fixed this - I will double check when I can, although my guess is that no, the sleigh is still used to resolve the references.

I believe what you're describing relates to "real mode" where multiple combinations of seg:off represent the same physical address.

IS this similar, or different (but maybe related) to an issue I am facing while working on disassembling a game that runs on Sony's PS2/Emotion Engine, in that even though the struct is defined, and applied correctly to an address, it still shows up in decompiled functions as (var_name.first_field + [offset in hex])? (where offset points to the offset of a valid member of the struct being utilized, accessed or written to)
(perhaps a silly question, perhaps I am doing something wrong if not a related (or the same) issue?

Looks similar.

Having seen yet another example of this (and engaging brain)
    pnCurrNext = (IDX_ONE_BASED *)((int)this + 0x1a);
from
   11e0:178e 8b 56 08     MOV        DX,word ptr SS:[BP + this+0x2]
   11e0:1791 8b 46 06     MOV        AX,word ptr SS:[BP + this]
   11e0:1794 05 1a 00     ADD        AX,0x1a

and
    *pCurrNext = pCurr$1->nNext_0x10;
from
   11e0:17b8 c4 5e f8     LES        BX,SS:[BP + pCurr$1]
   11e0:17bb 26 8b 5f 10  MOV        BX,word ptr ES:[BX + 0x10]
   11e0:17bf 8e c2        MOV        ES,DX
   11e0:17c1 93           XCHG       AX,BX
   11e0:17c2 26 89 07     MOV        word ptr ES:[BX],AX

I believe it's something to do with mechanism that Ghidra uses to take the address of a structure's member when using a sized version of a pointer to that structure.
For example: if that member is a simple int and something is assigned to it, the decompiler produces
this->nNext_0x1a = nIdx;
from
   11e0:16dc c4 5e 06     LES        BX,SS:[BP + this]
   11e0:16df 26 89 4f 1a  MOV        word ptr ES:[BX + 0x1a],CX

which is correct.

For example: if that member is a simple int and something is assigned to it, the decompiler produces
this->nNext_0x1a = nIdx;
from
   11e0:16dc c4 5e 06     LES        BX,SS:[BP + this]
   11e0:16df 26 89 4f 1a  MOV        word ptr ES:[BX + 0x1a],CX

which is correct.

To get Ghidra to produce the C output above, I modified this
https://github.com/NationalSecurityAgency/ghidra/blob/3e9419d4d793dad0619e51769474be61d6968ef9/Ghidra/Features/Decompiler/src/decompile/cpp/varnode.cc#L1932
to
    if (op2->code() != CPUI_SUBPIECE) {
      if ((Varnode*)0 == op2->getIn(0)) return false;
      op2 = op2->getIn(0)->getDef();
      if (op2->code() != CPUI_SUBPIECE) return false;
    }

and
https://github.com/NationalSecurityAgency/ghidra/blob/3e9419d4d793dad0619e51769474be61d6968ef9/Ghidra/Features/Decompiler/src/decompile/cpp/ruleaction.cc#L8536
to
    PcodeOp* op2 = vn2->getDef();
    if (op2->code() != CPUI_SUBPIECE) {
      data.opInsertInput(op, op2->getIn(1), 1);
      data.opSetOpcode(op, op2->code());
    }
    else {
      data.opSetOpcode(op,CPUI_COPY);
    }

Here is another example of Ghidra's inability to correctly comprehend "sized pointers"[^1] when it comes to working out the consequences of referencing components within them.
The C output is:
int __cdecl16far FUN_1168_09ec(uint numElems,DgnSAData1168_0b3c_0x20_t *param_2,int param_3,int *param_4)
{
   Dgn1170_03c2_0x14_t::LPpfnFUN_1168_1262 *pDVar1;
   int iVar2;
   int iVar3;
   uint i;
   uint nSum;
   long nMultSum;
   undefined2 seg;

   nMultSum = 0;
   nSum = 0;
   for (i = 0; i < numElems; i += 1) {
      seg = *(undefined2 *)
             ((int)((DgnSAData1168_0b3c_0x20_t *)param_2)->field0_0x0 + i * 4 + 2);
      iVar2 = *(int *)(*(int *)(param_2->field0_0x0 + i) + 4);
      if ((param_3 == 0) || (iVar2 != 0)) {
         iVar2 = (*(int *)(*(int *)(param_2->field0_0x0 + i) + 2) + iVar2 / 2) / iVar2 + 1;
         pDVar1 = &param_2->field0_0x0[i]->_vtable->field4_0x10;
         iVar3 = (*pDVar1)(param_2->field0_0x0[i]);
         nSum += iVar2;
         nMultSum += (ulong)(uint)(iVar3 * iVar2);
      }
   }
   *param_4 = nSum;
   return (int)((long)(nMultSum + (ulong)(nSum >> 1)) / (long)(ulong)nSum);
}

The lines of interest are:
      iVar2 = *(int *)(*(int *)(param_2->field0_0x0 + i) + 4);
and
         iVar2 = (*(int *)(*(int *)(param_2->field0_0x0 + i) + 2) + iVar2 / 2) / iVar2 + 1;
where field field0_0x0 (of param_2) is defined as Dgn1170_03c2_0x14_t *32[8] and the fields of Dgn1170_03c2_0x14_t at offsets 2and4 are int types, (offset 0 is a pointer to a vtable - using the default 16-bit size).
These definitions cannot easily have their pointer declarations altered as doing so creates incorrectly sized structures (and offsets within) and issues with calling functions when changing a pointer size.
[^1]: The only way to get pointers of the correct size when the architecture supports multiple (two in this case) sizes.

Is there anyone able to look at this?

Can someone please look at this.  I have examples of this everywhere and its making REing even more difficult!  Thanks.

I'm getting output like
            iVar4 = FindElementIdx((DgnEAClient *)
                                   CONCAT22(*(undefined2 *)
                                             ((int)&((DgnEAVocabData_0x6_t *)
                                                    g_vocabElems_8951.m_pEaData)[local_20].
                                                    pVoc_0x0 + 2),
                                            (DgnEAClient *)
                                            (*(int *)&((DgnEAVocabData_0x6_t *)
                                                       g_vocabElems_8951.m_pEaData + local_20)
                                                      ->pVoc_0x0 + 10)),pClient);

instead of
                     iVar4 = FindElementIdx(g_vocabElems_8951.m_pEaData[local_20].pVoc_0x0->eaClients_0xa, pClient);

Is there anyonw who can look into these???

There's also situations where the CS (code segment) is used (for example, as a switch lookup) and Ghidra knows it's using CS (if some sleigh is modified to show which segment an assembly instruction is using), but insists upon using the value in DS to find the array referenced.  Surely, as Sleigh knows CS is in use, why can't the decompiler use its value?
In the example assembly below
   1028:27fe 2e ff a7     0c8           JMP        CS:word ptr CS:[BX +
             e5 28                                  switchD_1028:27fe::switchdataD_1028:28e5]

without setting the DS to the same value as CS the decompiler produces
      switch((undefined4)
             *(undefined2 *)
              ((int)DgnScriptHolder_1040_25dc.m_szText + 0x307 + pThis->field13_0x824 * 2)) {

whereas after changing the DS register we get that the below assembly
             assume DS = 0x1028
   1028:27fe 2e ff a7     0c8           JMP        CS:word ptr CS:[BX +
             e5 28                                  switchD_1028:27fe::switchdataD_1028:28e5]

now produces
      switch((undefined4)
             ((undefined **)switchD_1028:27fe::switchdataD_1028:28e5)[pThis->field13_0x824]) {

still far from perfect (see #2628), but at least its now referencing the correct memory address!


Hdi ha services

This is a public doc to explain high-availability services provided by HDInsight clusters.

@JasonWHowell - can you review this new content submission? (note that if you want to make this into an article, we'll need to create it in the private repo as well as add a metadata section and have an author, ms.author, etc, etc.) thanks!

@jieyan1269 Thanks for your contribution! Are you working with someone on the self-help or docs team for this article? Maybe @hrasheed-msft ?
Besides adding metadata, we need to review the page in the private repo to check formatting and grammar. And any new article needs a place in the table of contents (TOC) as well.
Note, there is one docs article that is similar to "high availability" https://docs.microsoft.com/azure/hdinsight/hdinsight-high-availability-linux
Thanks, Jason

@JasonWHowell Hi, Jason, I did not work with @hrasheed-msft on this doc yet but intended to assign him as the reviewer. However, I cannot add the reviewer by myself, can you help?
Regarding https://docs.microsoft.com/azure/hdinsight/hdinsight-high-availability-linux, it is related to high-availability of hdinsight hosts, while mine is related to hdinsight services.

#assign:hrasheed-msft

@dagiro could you pull this content into a new branch and transform this into a new conceptual article?

This is now a duplicate of the internal PR https://github.com/MicrosoftDocs/azure-docs-pr/pull/89826
@jieyan1269 there are a few remarks to clarify there before we can publish. Let us know if you can see them, or cannot.

@JasonWHowell I've been in communication with Jie over teams regarding his new article. Yesterday I gave him a heads up that we made some comments and showed him where they were.

@JasonWHowell @hrasheed-msft - if this PR can be closed (as it's now duplicated in the private repo) please enter please-close in a comment. thanks.

#please-close


Cannot define liveness/readiness probes via helm chart

Expected Behavior
It should be possible to define liveness/readiness probe values, now all is hardcoded and for some systems timeout=1s is not acceptable.
Allow to define those probes at values.yaml for helm chart (at least for manager component).
Actual Behavior
Cannot change those values via current chart:
https://github.com/kedacore/charts/blob/9fbc23667f4dd1e37f0b9c9e19dc2ff1a4dbfc45/keda/templates/manager/deployment.yaml#L94
Steps to Reproduce the Problem

use helm chart for keda

Specifications

KEDA Version: chart version 2.11.2
Platform & Version: N/A
Kubernetes Version: N/A
Scaler(s): N/A

Are you willing to adding support for it?

If it is not urgent, I may try to implement as it is defined in bitnami template:
https://github.com/bitnami/charts/blob/main/template/CHART_NAME/values.yaml
I may have time within the month as it is not urgent.

It's not urgent at all. We plan to cut a release at the end of the month, but helm chart can be released also before or after as it has it's own release cycle (basically, on demand xD)

Hi @slawekww , I'm working on a PR. May I know exactly which value you want to parameterise ?

HI @dttung2905, I plan to add values in values.yaml:
operator:
  # Liveness probes
    livenessProbe:
      initialDelaySeconds:  25
      periodSeconds: 10
      timeoutSeconds: 1
      failureThreshold: 3
      successThreshold: 1
  # Readiness probes
    readinessProbe:
      initialDelaySeconds: 20
      periodSeconds: 3
      timeoutSeconds: 1
      failureThreshold: 3
      successThreshold: 1

and use them in charts/keda/templates/manager/deployment.yaml
The default would be like it is now and it allows to define values by users, especially timeoutSeconds.


PHP7 compatibility

Running this on PHP7 results in an exception being thrown:
  [ErrorException]
  Methods with the same name as their class will not be constructors in a future version of PHP; File_X509 has a deprecated constructor

This is probably the case because the used version of phpseclib is a bit outdated:
  - phpseclib/phpseclib (0.3.10) latest is 2.0.2
    Required by spatie/ssl-certificate-chain-resolver (0.3.*)

Seems to work fine now, strangely.

And the issue seems to have resurfaced.
PHP version:
PHP 7.0.8-0ubuntu<IP_ADDRESS> (cli) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.8-0ubuntu<IP_ADDRESS>, Copyright (c) 1999-2016, by Zend Technologies

Support for this had been added in v1.0.6


[Proposal] Support release types

Currently, releases exist.
There is no differentiation between whether a release is a patch, a minor or a major version, something completely different, or none.
Having the ability to mark releases as i.e. one of those mentioned above can make a huge significance when displaying the releases to the user, i.e. by allowing for custom filtering, by adding a different display for releases of different types (e.g. such that major releases will be displayed more prominently than patch releases), or to show releases in a tree-like view where first the major versions comes, then when expanded the underlying minor versions and so on.
Also, when implemented such that releases know their hierarchical super-release (i.e. 1.16 would be the super-release of 1.16.2) and the super-release gets deleted, it could be possible to delete the sub-releases as well, although I do not know whether that is actually needed or wanted.
For now, it should simply be enough if the user can choose the release type himself when creating the repo or updating it (including a nil release type in case this release cannot be identified as belonging to any category). Here, user does not mean any user, but rather those with write-access for the repo or administrators only.
Later on, it could even be possible to autotag releases of repos that follow semantic versioning, i.e. by using a bot, a plugin, or even with built-in support. (I tend rather towards an external bot that simply listens to changes in the releases and updates these releases accordingly)
Originally posted by @delvh in https://github.com/go-gitea/gitea/issues/16585#issuecomment-890403388

I for one would like tags that aren't full releases to be auto-hidden ala Github except for the latest of course.

You can know if it's a minor release via that version number.

I for one would like tags that aren't full releases to be auto-hidden ala Github except for the latest of course.

@mattatobin this should actually the case in 1.14+, see also #16720


Featureset

When I use extended template names for hidden patterns (filename starts with underscore) or states (filename has @[state] appendix) the render viewhelper won’t include the pattern. Is there a way to achieve this behaviour?

This one is a problem from Fluid - it currently has a bug in which a method, which is public API but not used by Fluid itself, did not resolve partial template files recursively.
You can follow the status of that bug on https://github.com/TYPO3/Fluid/pull/348, or use the diff from there to apply the fix in your project until it gets included in a release.


Perform clang-tidy on both cpp and cuda source.

Add `GENERATE_COMPILATION_DATABASE' to CMake.
Rearrange CMakeLists.txt.
Add basic Python clang-tidy script.

Proposed in #4032 .
@hcho3 Could you please help automating this on Jenkins after sorting out other priorities?  I added a simple documentation and a Python script for the new linting,  but I don't know what's happening with Jenkins or Docker.  Thanks! :)
Dependencies:

clang-tidy >= 7
cmake >= 3.8

I removed modernize-use-auto from clang-tidy.  The "use auto everywhere" is based on the assumption that we have a perfect IDE which can display correct hints about variable type.  Such an assumption doesn't hold on CUDA programming. :)  If others can argue the merit can overweight the pitfall then I will add it back.

Codecov Report

Merging #4034 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #4034      +/-   ##
=========================================
+ Coverage   60.79%   60.8%   +<.01%
=========================================
  Files         130     130
  Lines       11699   11699
=========================================
+ Hits         7112    7113       +1
+ Misses       4587    4586       -1

Impacted Files
Coverage Δ

src/objective/multiclass_obj.cu
93.68% <0%> (+1.05%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f368d0d...dfae430. Read the comment docs.

Actually this might be doable just on Jenkins.  Let me take a closer look. :)

@trivialfis In the longer term, I'd like to migrate some CPU tests to Jenkins, to allow for greater customization (such as higher memory limit).

@hcho3 Actually somehow I oddly like the memory limitation.  Those error keeps reminding me that I need to optimize it since all tests use relatively small datasets.

@trivialfis We still need to deal with intermittently failing Travis tests though.

@hcho3 Nope.  At the very least cuda headers are required.  So it seems Jenkins is the right way to go.  Could you please help drafting the Jenkins script after sorting out other priorities?

@trivialfis Clang-tidy error messages now should be visible. Let us merge this after tests pass.

@hcho3 Could you lower down the CUDA version used for clang-tidy to CUDA-9?  10 is not supported by clang-tidy-7

@trivialfis Done. I created a Docker container with CUDA 9.2 and Clang-Tidy 7. Let me know if this is satisfactory.

@hcho3 Thanks for lots of hard work.  This is really helpful.  I will try to correct the remaining errors in later PRs.

@trivialfis I'll go ahead and review #3825 within a day or two, so that you don't get blocked for later re-factoring. Thanks for your effort towards a consistent coding style for GPU code.


Why does job not go from Started Execution to Executed

Hi there,
I have noticed that some jobs never go from Started Execution to Executed for some reason. There is no clear bug, it just happens. Is there a reason for this?

After this happens, jobs supposed to happen later either show up sometimes, or they just don't.

I find these jobs which  never go from Started Execution to Executed  cannot restart again. Can the jobs retry again?

I'm having a similar issue.  Each job seems to run twice, one never makes it out of Started Execution state, the other usually makes it to Executed state. I have Django running behind Gunicorn, so I had to use this hack to get it to only run on the master node and not on the workers.
The scheduler is initialized at the bottom of the wsgi.py file.  Otherwise, it follows the above StackOverflow method to avoid the APScheduler running on every node.

hey @Juanjoorrego @AnthonyBRoberts have you been able to figure out what was the issue ?

hello, I am also having the issue both while using the django runserver in local or using uwsgi in production, my jobs are run as many times there are processes running django.
@jarekwg I have what seems to be a fix by adding a lock in the _get_jobs method to avoid running in a "race" condition when the method get_due_jobs is called multiple times at the same time in different processes/threads. I have add the transaction.atomic and the select_for_update() parts
    def _get_jobs(self, **filters):
        from django.db import transaction
        with transaction.atomic():
            job_states = DjangoJob.objects.filter(**filters).select_for_update().values_list('id', 'job_state')
            jobs = []
            failed_job_ids = set()
            for job_id, job_state in job_states:
                try:
                    jobs.append(self._reconstitute_job(job_state))
                except:
                    self._logger.exception('Unable to restore job "%s" -- removing it', job_id)
                    failed_job_ids.add(job_id)

            # Remove all the jobs we failed to restore
            if failed_job_ids:
                LOGGER.warning("Remove bad jobs: %s", failed_job_ids)
                DjangoJob.objects.filter(id__in=failed_job_ids).delete()

            def map_jobs(job):
                job.next_run_time = deserialize_dt(job.next_run_time)
                return job

            return list(map(map_jobs, jobs))

Could this be a proper fix according to you ?

this could also fix #12

Hi @sdementen,
I don't really have a strong opinion on best approaches in this package anymore.
At a high level, I'd say either:

lock the rows from being read/updated concurrently (as you've done here).
lock the _get_jobs function from being run concurrently (a db-level semaphore would be pretty neat, though idk if that'd limit support to postgres or not..)

Either way, probs won't merge a change like this myself as I'm too far removed at this point. Would you (or anyone else here) be interested in helping maintain this package? Happy to grant write access.

I think locking the _get_jobs is not enough as the update of the jobs is done in a 2nd phase in APScheduler. And between a first call of _get_jobs and a second call of _get_jobs , the update job may not have happened yet leading the second call of _get_jobs to return the same jobs as the first call ==> there is a need to mark the job as "read" in _get_jobs (what I do with setting the job.next_run_time to None.
According to your knowledge, is this enough ?
I can do a PR and update the tests/docs if needed. For the maintenance, I can help if you want (even though I have not a deep knowledge of APScheduler) but I do not feel I can take it over on the LT (not in my core or secondary focus, more my third...)

Hmm, so your idea of somehow marking the DjangoJobs as already queued is sound, however doing this by wiping the next_run_time field feels like a hack:

it overloads the usage of this field.
it could accidentally get broken if next_run_time calculation behaviour were to change in future.
it could break/confuse other things that refer to next_run_time. eg #68 assumes next_run_time being blank implies the Job is paused.

I'm not familiar w the code anymore, but could next_run_time just get bumped to actual next run time here instead? That's stop it getting picked up multiple times.
Would probably be prudent to change the function name as well, since _get_jobs implies it's not writing to the db.

Indeed, it felt to me like a hack too.
I agree also that the name is not right as it has a side effect. Yet, given APScheduler is first getting the jobs and updating afterwards the next run time (as it assumes a single scheduler), I think we can keep this get & updating behavior and rename the function to make it more obvious.
Re marking the job as "retrieved for execution", it maybe better to add a boolean field to be even more explicit than mangling the current field. What do you think?

Most job runners have a status field that includes this, plus other aspects of the Job life cycle. eg. PENDING, QUEUED, RUNNING, SUCCESS, FAILURE
How are the remaining states managed? By APScheduler? Maybe we could pull them all through to a Job.status field? If that doesn't make sense in this case, then yeah, a bool is_queued or a timestamp queued_at (in case knowing when it was queued is useful) might be the way to go.

Fixed via #86.


Improvements around claims

:tada: This PR is included in version 1.4.0 :tada:
Your semantic-release bot :package::rocket:


Queries per DB table in metadata.json

It doesn't appear possible to have separate queries defined per database table.  When I do something like below, my table descriptions show up but not the queries:
 "databases": {       "MYDB": {         "tables": {           "MYFIRSTTABLE": {             "source": "Test",             "source_url": "https://www.google.com",             "queries": {                 "Query 1": {                     "sql": "select * from MYFIRSTTABLE",                     "title": "Query 1",                     "description": "This is the first query"                   },             }           },           "MYSECONDTABLE": {               "source":"Test2",               "source_url":"https://www.google.com",             "queries": {                 "Query 2" : {                     "sql":"select * from MYSECONDTABLE;",                     "title": "Query 2",                     "description":"This is the second query"                 }             }           }         }

I think I see what's gone wrong here. Here's your example re-indented:
{
    "databases": {
        "MYDB": {
            "tables": {
                "MYFIRSTTABLE": {
                    "source": "Test",
                    "source_url": "https://www.google.com",
                    "queries": {
                        "Query 1": {
                            "sql": "select * from MYFIRSTTABLE",
                            "title": "Query 1",
                            "description": "This is the first query"
                        }
                    }
                },
                "MYSECONDTABLE": {
                    "source": "Test2",
                    "source_url": "https://www.google.com",
                    "queries": {
                        "Query 2": {
                            "sql": "select * from MYSECONDTABLE;",
                            "title": "Query 2",
                            "description": "This is the second query"
                        }
                    }
                }
            }
        }
    }
}

The `"queries" section isn't supported at the table level - it only works at the database level. So that example should look like this:
{
    "databases": {
        "MYDB": {
            "tables": {
                "MYFIRSTTABLE": {
                    "source": "Test",
                    "source_url": "https://www.google.com"
                },
                "MYSECONDTABLE": {
                    "source": "Test2",
                    "source_url": "https://www.google.com"
                }
            },
            "queries": {
                "Query 1": {
                    "sql": "select * from MYFIRSTTABLE",
                    "title": "Query 1",
                    "description": "This is the first query"
                },
                "Query 2": {
                    "sql": "select * from MYSECONDTABLE;",
                    "title": "Query 2",
                    "description": "This is the second query"
                }
            }
        }
    }
}

Closing this, but #260 would definitely help here.

Thanks for the reply.  I was hoping queries per table were supported, as I have an application that builds tables depending on the user input to the application.  It will either create one table, or two.. and if one or the other is missing, certain queries will return errors.  Of course I can work around this by labeling the query name and hope users don't click queries they have not created a table for, but ideally the user would see the queries available based on the tables that exist in their database.


Nothing is clickable in very specific circumstances

Discovered 04/12/2017, during normal usage conditions. Nothing special, but not working as usual
In few conditions, the buttons / volume bar / links become fully insensitive to clicks.
Conditions where it happens

Have a triple monitor setup
Windows 10 (fully updated)
Chrome window opened on the right screen (you can have other Chrome windows, and inside them, extension will be working fine)

I'll try to investigate more with that issue, trying to find some circumstances pattern

Ok, so it's not linked to the monitor setup, but actually to the Windows 10 monitor scaling, when going higher than 100%.
You can try this at home, should do the same issue. Also, every other extensions with popup works fine


Feature Request: Generate DockerFile for the project

Hi!
Big fan of the project and the only improvement I could really think of is when we generate the project it also create all the docker files required for it.
Just a suggestion but its overall great and very stable !
thank you :)

there is a guy that already done it in his project check out : https://github.com/cesar3030/Sileau-IoT/blob/master/express-server/Dockerfile


Use packer to build Vagrant boxes

Putting this on "high" priority, since the current Vagrantfile:

takes a long time to provision
sometimes fails - Vagrant doesn't seem to be always fine with rebooting during provisioning, which causes the provisioning to fail (practically randomly) - I tried every solution I could google, but nothing seems to bring us reproducible success

I think the way to go to fix both issues is to use Packer - it will save the time spent on installing packages, configuring GRUB etc. and also will remove the requirement of reboots when provisioning.

I'm giving up on Vagrant. No solution to my problems which I reported:
https://discuss.hashicorp.com/t/building-vagrant-libvirt-box-based-on-the-another-one/26771
https://github.com/hashicorp/packer/issues/11148
I would rather switch to terraform and modify qcow2 images with guestfs-tools: #23


Updating the readme

new UAV models to the table
.bashrc additions are now specific to melodic
check for other old stuff pls

Check https://github.com/ctu-mrs/simulation/pull/22

Check https://github.com/ctu-mrs/simulation/pull/22


FindOne methods should return error when not found

When we execute a FindOne method, we need to "double check" for an error and a not nil object
foo, err := fooStore.FindOneById(id)
if err != nil {
    // do something with err
} else if foo == nil {
    // do something to address that foo was not found
}

It would be nice to have something like NotFoundErr that was returned as the error when foo is not found. This could make error management easier.

This didn't get traction, so I'm closing it.

https://github.com/tyba/storable/blob/master/resultset.go#L42
Was made some weeks ago.


Summary of suggestions

If you're happy using randomizr, you could replace all of Part 2 I-III with the following:
library(randomizr)

## Remove duplicate users
data <- data[!duplicated(data$user_id),]

data <- data |>
  mutate(
	## Stratify by region and assign treatments
    treatment = block_ra(blocks = data$region_id, conditions = c("T1", "T2", "C")),
	## Temporarily assign email to all users using simple randomisation
    temp.assignment = simple_ra(nrow(data), conditions = c("Standard", "Thermostat", "TRV", "Social norm")),
	## For Control group, replace with "Control"
    email = case_when(
      treatment == "C" ~ "Control",
      TRUE ~ as.character(temp.assignment)
    )
  ) |>
  ## Remove temporary assignment column
  select(-temp.assignment)

Awesome - very useful to know. Also pretty amazing that it can be done is so few lines of code!


Boolean kernels: any / all

It would be nice to have fast columnwise any/ all operations.
This has probably some nice optimization options. Maybe reading the bits per u8 or u64 and check if == 0 / == usize::max.

Closed by #739


Error in Creating Collaborative wallet

[x] We do not need that Checkmark option on Collaborative wallet creation flow on mobile key.

[ ] After Creating Collaborative wallet on success modal when I clicked on View wallet , Importing wallet screens appears  this only happens if a user have already imported a tribe wallet before.

[ ] On android after adding 3rd MK it automatically created the Collaborative wallet and it crashed after clicking on View wallet

https://github.com/bithyve/bitcoin-keeper/assets/107177634/5d9fcccd-cac7-4519-a406-470ba88d77ee

Couldn't reproduce 2nd and 3rd point. Please reproduce this in the next release and confirm

Completed

Getting same error

[ ] After Creating Collaborative wallet on success modal when I clicked on View wallet , Importing wallet screens appears  this only happens if a user have already imported a tribe wallet before.
Will create separate for other . Rest 2 is completed


cargo test produces Too many open files on NixOS when using multiple integration tests

Problem
Hi!
Recently I switched to a new machine running NixOS. I've used the same configuration as on the previous machine but with slight modifications. However. When I started working on a project that I've been working on on a previous machine, it turned out, that I cannot run cargo test successfully.
The compilation of integration tests (multiple files) always fails with a Too many open files error on random .rlib or .rmeta file either during compilation or during linking. On my previous machine it compiled just fine.

cargo test: cargo-test.log
strace cargo test: strace-full.log
strace -e open,close,openat cargo test: strace.log

What did help was to change the extension of all integration test files but one, compile, change the extension back to .rs for another file and repeat until all tests are renamed and compiled. But making bigger changes to the codebase means that I have to recompile all of the integration tests again.
I also tried to compile only one crate at the time, but it was taking too much time with all of the dependencies of the project.
Currently the only way to mitigate it for me is to put all the tests into unit tests, as they seem to work fine.
Steps

Clone https://github.com/wvffle/nixos-rust-too-many-open-files/tree/master
enter dev shell with nix develop
run cargo test

Note that the repro, has unused dependencies imported in the tests/common/mod.rs. When I remove them, I can compile just fine. But if they are imported (even if unused) the compilation fails.
Possible Solution(s)
No response
Notes
I've tried rust 1.75, 1.80 and 1.82.0-nightly but all versions fail the same way.
The ulimit -n was initially set to 1024. I've tried to set it higher but that didn't work either. I've even raised the hard and soft limits in PAM using this configuration, but that didn't help also:
  systemd.extraConfig = "DefaultLimitNOFILE=1048576";
  security.pam.loginLimits = [{
    domain = "*";
    item = "nofile";
    type = "-";
    value = "1048576";
  }];

Tried linking using mold, no change.
Version
cargo 1.82.0-nightly (0d8d22f83 2024-08-08)
release: 1.82.0-nightly
commit-hash: 0d8d22f83b066503f6b2b755925197e959e58b4f
commit-date: 2024-08-08
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/3.3.1)
ssl: OpenSSL 3.3.1 4 Jun 2024
os: NixOS 24.11.0 [64-bit]

Your strace calls don't cover child processes, you need to use -ff for that. The error seems to occur in rustc.

The ulimit -n was initially set to 1024. I've tried to set it higher but that didn't work either.

Do you mean that increasing the limit failed or that increasing the limit didn't help?

I'm having a very similar issue but it is not exclusive to tests. I've increased ulimit -n to<PHONE_NUMBER> and it still happens. Importantly, I am using impermanence and mounting / in a tmpfs. Both .cargo and my project (including target/) are within an persistence path which is mounted using fuse. The strace output from a fresh compile contains hundreds of individual logs but the quantity from an incremental build (which still unconditionally fails) is much more manageable.
strace -ffe open,close,openat cargo test: strace.tar.gz

are within an persistence path which is mounted using fuse.

It's possible that the fuse server then is sending this error code and you need to bump its file limit.

That's just what impermanence does. I believe it uses bind mounts for any persisted locations outside of the home directory. I checked and it doesn't seem like there's any built-in way for forcing home directory children from being bindmounted instead of using bindfs. I haven't noticed any performance issues with using fuse besides this error appearing when compiling Rust projects. I'll try to trick impermanence into using a bind mount and see how that goes.

@the8472 here is the  strace with ff:
strace -ffe open,close,openat cargo test: strace-ff.log

Grepping through that log I don't find any EMFILE or many open files match. Use -Z to get only failing syscalls and then see which one is failing with that code.

I was able to fix this issue by using bind mounts instead of bindfs. For anyone who happens to be encountering this issue, move everything from home.persistence."/nix/persist/home/USERNAME".* to environment.persistence."/nix/persist".users.USERNAME.*.

@campbellcole I managed to compile my project when using symlink instead of bindfs:
  home.persistence."/persist/home/${config.home.username}" = {
    allowOther = true;
    directories = [
-      "workspace"
-      ".cargo"
+      {
+        directory = "workspace";
+        method = "symlink";
+      }
+      {
+        directory = ".cargo";
+        method = "symlink";
+      }
    ];
  };

That suggests that the FUSE server process is running into the file limit or emitting the code for some other reason. I.e. it's not a rust process that's exhausting its own limit. If it's the ulimit then you have to raise that of the server. If it's some other limitation then you'll have to file a bug with them. Or don't run Rust on FUSE.

I think we can close this issue in favor of https://github.com/nix-community/impermanence/issues/206


Add parameter to specify which point cloud is used for normal calculation

During M3C2 distance calculation, normals need to be calculated for each core point to derive the surface orientation.
Normals can be computed considering different points in the neighborhood of the core point:
(a) using neighboring points in the core points, i.e. disregarding the input point cloud epochs, or
(b) using neighboring points of an epochs cloud (the first epoch or any other in the available epochs).
Currently, normals are calculated for using the first epoch specified in py4dgeo.M3C2().
Selection by the user could be enabled by implementing an additional parameter cloud_for_normals where the index of the epoch to use is specified, or -1 to use the core points, i.e.:
cloud_for_normals = -1 uses the core points for normals calculation
cloud_for_normals = 0  uses the point cloud of the first specified epoch (default; corresponds to current implementation)
cloud_for_normals = 1 uses the points of the second specified epoch

#59 implements this, but in a slightly more generic way. You may specify any epoch (object of type Epoch) or point cloud (np.ndarray of correct shape) to use for normal calculation. If a point cloud is given, a search tree for that cloud is constructed. If an epoch is given, the search tree is reused. If the parameter is omitted, the first epoch is used.


Support molmo in turbomind

model: allenai/Molmo-7B-D-0924

It seems doesn't support text only input like
pipe('who are you?')

# or
messages = [
    dict(role='user', content=[
        dict(type='text', text='你是谁')
    ])
]
pipe(messages)

It seems doesn't support text only input like
pipe('who are you?')

# or
messages = [
    dict(role='user', content=[
        dict(type='text', text='你是谁')
    ])
]
pipe(messages)

Fixed

When I run the multi-image multi-round conversation example
  File "/home/chenxin/ws3/topk/lmdeploy/vl/model/molmo.py", line 99, in forward
    inputs = self.processor.process(images=message['images'],
  File "/home/chenxin/.cache/huggingface/modules/transformers_modules/preprocessing_molmo.py", line 127, in process
    tokens = self.get_tokens_input(
  File "/home/chenxin/.cache/huggingface/modules/transformers_modules/preprocessing_molmo.py", line 104, in get_tokens_input
    prompt = "User: " + prompt + " Assistant:"
TypeError: can only concatenate str (not "NoneType") to str


autoSelectFirst for v1.8.0

Can you briefly outline the problem you are trying to solve with this PR?

This option allows to select the first item autocomplete list.
For example user entered "Lo" and autocomplete gives five results. Then when user press on submit button on keyboard, the first item('Los Angeles') will be selected.
Actual behaviour - nothing selected.


feat(hierarchy-list): add controlled expandedIds prop

Closes #3037
Summary

Adds the ability to explicitly control which ids are expanded or collapsed in the HierarchyList

Change List (commits, features, bugs, etc)

adds expandedIds prop to control which ids in the list are expanded or collapsed

Acceptance Test (how to verify the PR)

Check https://deploy-preview-3081--carbon-addons-iot-react.netlify.app/?path=/story/1-watson-iot-hierarchylist--with-overflow-menu
confirm Chicago White Sox is expanded
Add another team to the expandedIds array knob (comma separated no space--ie. Chicago White Sox,New York Mets)
Check https://deploy-preview-3081--carbon-addons-iot-react.netlify.app/?path=/story/1-watson-iot-hierarchylist--with-default-expanded-ids
confirm the white sox and yankees are expanded by default
Add a different team in the expandedIds array knob and confirm that team is expanded and the white sox and yankees close

Regression Test (how to make sure this PR doesn't break old functionality)

check for errors with defaultExpandedIds story
check expanding and collapsing on click still works.
expanded state should only change when the expandedIds prop is updated

Things to look for during review

[ ] Make sure all references to iot or bx class prefix is using the prefix variable
[ ] (React) All major areas have a data-testid attribute. New test ids should have test written to ensure they are not changed or removed.
[ ] UI should be checked in RTL mode to ensure the proper handling of layout and text.
[ ] All strings should be translatable.
[ ] The code should pass a11y scans (The storybook a11y knob should show no violations). New components should have a11y test written.
[ ] Unit test should be written and should have a coverage of 90% or higher in all areas.
[ ] All components should be passing visual regression test. For new styles or components either a visual regression test should be written for all permutations or the base image updated.
[ ] Changes or new components should either write new or update existing documentation.
[ ] PR should link and close out an existing issue

Nice work as always. I also tested the deep nesting in https://deploy-preview-3081--carbon-addons-iot-react.netlify.app/?path=/story/1-watson-iot-hierarchylist--with-mixed-hierarchies and it works fine.
Since we are doing this I think it also makes sense to expose an onExpandedChange so that consumers can be informed if the "controlled" state is changed by the user in the HierarchyList itself. Could be useful to determine the enabled state of an external "Expand all" button like in the issue example.

Good idea. I'll add that.


completing test automation for Instances

This PR is closing AAP-21398

It's refactoring existing tests to use getByDataCy instead of get
Adding two component tests for Listener Addresses tab:
listener addresses tab should be hidden for non-k8s system
listener addresses tab should be shown for k8s system

LGTM


Fix inlining and absolute paths on windows

Fixes #244

Don't know why I added the path.resolve in the inlining, but removing that line fixed it :'(


wish: configurable wait time

If the error is 429: Over Quota, you may want to wait 15 minutes before trying again.
Supporting the "Exponential Backoff" algorithm is also useful if you are contacting a server which is trying to recover from high load,  while at the same time it's being hammered by services which retrying their requests as fast they can waiting for it to come back up.

@markstos makes sense to add exponential backoff. Adding it as an enhancement.

@void666 did you close this on purpose? I believe you wanted to just add the label, but not close this request did you?

My bad. Reopened it.

Proposed as https://github.com/void666/request-promise-retry/pull/6

Having number of retries and a wait interval for those retires that you can set by environmental variable would be very helpful.  I am forking this repo and doing just that.  I will PR my branch and you can decide if its worth including.

@mmcguff this feature is already ready and used in real conditions in https://github.com/void666/request-promise-retry/pull/6
Interested to hear feedback from there. Let's build more together instead of another implementation from scratch I guess :). Thanks

Resolved in #6


Fix spread arguments

While trying to simplify React's PooledClass.js and changing it from multiple Poolers to just one that uses a spread operator, I noticed that return new Klass(...arguments); is  wrongly transformed into return new (Function.prototype.bind.apply(Klass, [null].concat(arguments)));. The problem here is that concat only works in the expected way with Arrays, not Array-like Objects, like what arguments is.
This PR fixes that issue and transforms
return new Klass(...arguments);

into
return new (Function.prototype.bind.apply(Klass, [null].concat(Array.prototype.slice.call(arguments))));

Fyi, you can write
function f(...args) {
  otherFunc(...args);
}

In order to manipulate arguments in a safer way

I assume by "safer" you mean the fact that

rest parameters are real Arrays
we don't need Array.prototype.slice which prevents optimizations in JavaScript engines.

Yes, I agree and for PooledClass.js I've planned to propose the following implementation:
var unifiedArgumentPooler = function(...args) {
  var Klass = this;
  if (Klass.instancePool.length) {
    var instance = Klass.instancePool.pop();
    Klass.apply(instance, args);
    return instance;
  } else {
    return new Klass(...args);
  }
};

But that's another Story/PR for React.

Thank you for reporting this issue and appreciate your patience. We've  notified the core team for an update on this issue. We're looking for a response within the next 30 days or the issue may be closed.


Implement Control Port Extensions

Deployed the following code to https://ctrl-kv.alandeploy.com

It provides a read-only access to the key-val storage in public and read-write storage if you have access to the cluster secret:

Demonstrating that control port extensions work. :)

(The cluster has been terminated so that cluster secret is no longer real.)


Observer actor

As an idea, to have Kafka-topic-observer actor.
EXAMPLE
This actor will be scheduled to do its job in the background.

Observe for changes (create, update, delete) Kafka topics. And update their metadata.

This actor allows avoiding manual creating metadata, the only manual update of metadata will be available via interfaces (REST, GraphQL, gRPC, etc. . .)
example rest PUT topics/{topic_name}

I described approach here

@NikitaZhevnitskiy, I like the approach. It can increase usability by reducing steps to document topics.
We can consider this one as part of the initial release.
I will close this as accepted.


feat: integers literal with a base different to 10 now can be typed in literal way

Numeric (integers) typed in a literal way, can now specific the base by type a special syntax before the numeric value.
Binary literals
To declare an integer number using binary syntax. You must type the number using only 1 and 0 numbers
Example to declare a integer variable with value 11: const num = 0b1011
Octal literals
To declare an integer number using octal syntax. You must type the number using numeric values in a range between 0 and 7 (inclusive ranges)
Example to declare an integer variable with value 11: const num = 0o13
Hexadecimal literals
To declare an integer number using hexadecimal syntax. you must type the number using value in a range between 0 and 9.
Since hexadecimal has a base 16, reach this value is possible using letters from A to F
Example to declare an integer variable with value 11: const num = 0xb

Every case that implies to use alphabetics characters, can be used both as lower-case and upper-case characters.

Still needs changes since some cases fails and throws exceptions

Previous feat commit had some errors.

When a numeric value was only a 0 the lexer crash, because it was expecting a numeric literal with a base different to 10
Placing a numeric separator after a hexadecimal digit commit the lexer to crash, becase it was expecting numeric characters instead alphabetic characters like: 'a', 'b', etc


Add BLS support to grub_menuentry

Adds BLS support which should fix issues on EL8 and Fedora 30
Fixed the issue with duplicating kernel options while debugging
Updates both non-EFI and EFI paths

Closes #49
Closes #38
Closes #37

Coverage remained the same at 95.402% when pulling ec748c7cc27000393c80b96e1a54a10e5e454aa2 on trevor-vaughan:add_bls_support into 02ccac55d487d8ad445240dbb0dd8f66476c6817 on hercules-team:master.

@alexjfisher This should be good to go now.


Some frames are missing at the very end of a video

Description
This is the url http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4
If I play the video in web then last frame should be

But while playing using skia the last frame is

Version
"@shopify/react-native-skia": "^1.3.4",
Steps to reproduce
If I check the video duration then it is "duration": 596458
if I check current time after closing the video play then it gives me currentTime.value.toFixed(0)=596421 or 596425 or 596441 . different values most of the time.
Snack, code example, screenshot, or link to a repository
It never produces the last frame. it stops playing before the video actually stops.

both platform has some missing frames. android has more missing frames than ios.

any update regarding this?


Unify syntax for watching providers

This was a thought that came to me recently.
It might be a little controversial, but I wanted to get other opinions / see other sides of the issue. Especially from your viewpoint @rrousselGit
StateNotifierProvider is the only provider that has two different "aspects" of the state. One is the actual .state of the StateNotifier, and one is the StateNotifier itself. One is the primary state and the other is just an interface for mutating/updating the state. The one that is commonly watched is the state, and the one that should be primarily read is the notifier.
Common usage (in my experience) dictates that the:

notifier

should be context.read in callbacks in the widget tree
should also be ref.read in any providers wanting to use the interface (not in the provider create function body, but a ref.read - aka Reader - should be passed into the constructor of the class / repository, to allow it access to the interface).

state

should be watch or ref.watched in most cases.

Also, it seems like there is a lot of confusion in the issues about why StateNotifer is "different" than the other providers when it comes to watching the state in a widget. I know that there really isn't any difference because the StateNotifier subclass is the thing that is being provided. But it is interesting in that it is the ?only? provider that has that different syntax for reading the state. I guess that isn't completely true given the fact that you can read/watch? a FutureProvider's .future. However, there is this disconnect in the fact that the common thing to be watched in a StateNotifierProvider also has a "different" API.
Obviously the current proposed solution is to just document these aspects / differences better. If so, I'm willing to spend some time working on some documentation as requested in #326 and various other issues. However, I'm wondering if there is a better API specifically to unify syntax for watching the state of a StateNotifier (which is the most common thing that should be watched if my experience is correct).
Again, this is based off of my experience which might be different than others' experience. I'm not proposing any specific solution here, just making an observation, and want to understand the viewpoints of others.
Specifically I want to point out that the current API makes sense to me and others who frequently use this library and understand how it works, but to beginners and newcomers, it is strange, which is why it comes up so frequently in the issue tracker. I'm guessing your devtool / lint package will issue warnings to enforce / warn about the common usage mentioned previously.

StateNotifierProvider is the only provider that has two different "aspects" of the state

As you've said, later, it isn't the only one
StreamProvider/FutureProvider do it too, with:
StreamProvider<int> provider;

AsyncValue<int> asyncValue = ref.watch(provider);
Future<int> lastValue = ref.watch(provider.last);
Stream<int> stream = ref.watch(provider.stream);

Personally, what I'd like is to be able to switch the behaviour of ref.watch(stateNotifierProvider) and ref.watch(stateNotifierProvider.state)
The .state should be the default behaviour IMO. The problem is... We can't with Dart at the moment.
Supporting it would required changing StateNotifierProvider<MyNotifier> into StateNotifierProvider<MyNotifer, MyState> which is quite verbose and redundant. Hence why it is "backward" at the moment.

One solution though could be to:

remove the @protected on StateNotifier.state
have watch(stateNotifierProvider) "watch" the state
make people do final state = watch(provider).state instead of final state = watch(provider.state)
expose a final notifier = watch(provider.notifier) that does not listen to the state
do the same with ChangeNotifierProvider which could benefit from such .notifier too.

Personally I think your last solution is what a lot of people are looking for, and mimics the StateProvider's
watch(provider).state

What I fail to understand is: Why do newcomers write final state = watch(provider).state?
Because the IDE gives a very clear warning: "state is protected"
It seems like newcomers ignore the warning but don't make the connection between the warning and their UI not rebuilding.

Personally I think your last solution is what a lot of people are looking for, and mimics the StateProvider's
watch(provider).state

The biggest issue with that is, the current separation exists to force user to not have any state in their StateNotifier _outside of the state property`.
With this change, people could write:
class Notifier extend StateNotifier<void> {
  int _count = 0;
  int get count => _count;
}

final count = ref.watch(notifier).count;

Because the IDE gives a very clear warning: "state is protected"

Agreed, the IDE is rather clear already as far as it being protected, but they might assume if watch(provider).state is protected, than provider.state might also be protected. Or I think the more common thing is that they don't see / understand the warning since it mentions nothing about the right way to do it, and they don't know about the .state on the provider.

The biggest issue with that is, the current separation exists to force user to not have any state in their StateNotifier outside of the state property.
With this change, people could write:
class Notifier extend StateNotifier<void> {
  int _count = 0;
  int get count => _count;
}

final count = ref.watch(notifier).count;

And that's already a mistake newcomers do. This change would reinforce it

I can see that this would happen. In the case of ChangeNotifier this misunderstanding can also happen, but it is clear that you have to call notifyListeners. But maybe a lint on StateNotifier subclasses to not declare any public accessible fields or getters would be a solution to that problem.

But maybe a lint on StateNotifier subclasses to not declare any public accessible fields or getters would be a solution to that problem.

But then we can have a lint to improve the error message on watch(provider).state
Maybe a first step would be to better document that.
Something like:
My Widget is not rebuilding when my StateNotifier changes, why?

do watch(provider.state) instead of watch(provider).state

But then we can have a lint to improve the error message on watch(provider).state

Understood, but it doesn't change the fact that the most common thing to watch on a StateNotifier is the .state. Which means when you want to watch a provider you have to remember if the provider you are watching is a StateNotifier or not. Of course autocomplete and intellisense helps with that.

Another possibility is to have an assert in debug mode that prevents reading .state inside ConsumerWidget.build
\

^ This might work. Would it also work for useProvider, I assume so.

On the other side of things, depending on debug mode means that the developer has to potentially touch every screen / function of the app to get this error. (i.e. it is not compile safe)?

That shouldn't be necessary, because the @protected still exists.
The main issue is that newcomers ignore it. But for people writing tests and caring about compile-safety, they'd have their CI fail if there's a warning

What I fail to understand is: Why do newcomers write final state = watch(provider).state?
Because the IDE gives a very clear warning: "state is protected"
It seems like newcomers ignore the warning but don't make the connection between the warning and their UI not rebuilding.

I'm a newcomer and I don't know any better and am pretty confused. For example, this snippet here:
https://riverpod.dev/docs/concepts/reading#consumer
uses final state = watch(provider).state. It means that the syntax is different for a Provider and StateNotifierProvider. I did notice the warning, but then I checked what is done in StateController, found this:
// Remove the protected status
  @override
  T get state => super.state;

and did the same for my class. The fact that both watch(aProvider.state) and watch(aProvider).state compile (a warning is just this, a warning) doesn't help at all.

I think that these two snippets will cause a lot of bugs and confusion, the 'incorrect' syntax should ideally be a compilation error, if possible. Lints are just this, lints, warnings - don't forget not everyone is as good a developer as you guys (I work on a project where many teams don't even use the analyzer...), but they still want to get the job done, and you probably also want them to get it done using your package. The idea @rrousselGit had about an assertion for watch(provider).state is kind of contradicting to one of the main tenets of Riverpod - compile time safety. In this case, the code will compile, but I will get a runtime exception at most, not ideal.

The problem is, this @protected warning has more uses than just avoiding this bug. Removing it has undesired side-effects.
This @protected also promotes a centralised business logic. Removing it also allows:
class Counter extends StateNotifier<int> {
  Counter(): super(0);
}

final provider = StateNotifierProvider<Counter>((ref) => Counter());

RaisedButton(
  onPressed: () =>
    // The logic shouldn't be here. Instead this ++ logic should be in a method inside `Counter`
    context.read(provider).state++,
)

Also, the syntax watch(aProvider.state) depends on an extensions in base.dart, so someone who knows StateNotifier doesn't necessarily expect it to suddenly have a state property. When I command-click the state property, I land here:
/// Adds [state] to [StateNotifierProvider.autoDispose].
extension StateNotifierStateProviderX<Value>
    on StateNotifierProvider<StateNotifier<Value>> {
  /// {@macro riverpod.statenotifierprovider.state.provider}
  StateNotifierStateProvider<Value> get state {
    _state ??= StateNotifierStateProvider<Value>._(this);
    return _state as StateNotifierStateProvider<Value>;
  }
}

Why is my provider suddenly an 'autoDispose' one? Or is it just an IDE quirk?

@rrousselGit
There are extremely rare, but valid use cases for ignoring a @protected annotation. Personally I feel like that is why the dart team made it an annotation rather than a language feature. But that is not the point of this issue.
Back to the topic of the issue. I feel like I'm not the only one who would prefer the alternative

have watch(stateNotifierProvider) "watch" the state
make people do final state = watch(provider).state instead of final state = watch(provider.state)
expose a final notifier = watch(provider.notifier) that does not listen to the state
do the same with ChangeNotifierProvider which could benefit from such .notifier too.

Personally I think you could do this without having removing the @protected annotation from StateNotifier.state
by returning a new object.
For example:
class StateNotifierWatchedState<T> {
  const StateNotifierWatchedState(this.state);
  final T state;
}

And have StateNotifierWatchedState<T> be the return type for watch(someStateNotifierProvider<T>);
What do you think?

I don't see the benefit of such object.
If we're really going for that, might as well update StateNotifierProvider to:
class MyNotifier extends StateNotifier<int> {...}

final provider = StateNotifierProvider<MyNotifier, int>(...)

int state = watch(provider);
MyNotifier notifier = watch(provider.notifier);

The inconvenience of having to specify a redundant generic parameter is much lower than that of having to specify .state all over the place

Thinking about it, a case could be made that the inconvenience of having to write watch(provider.state) is higher than having a redundant generic argument...
Maybe I should just add that extra generic parameter and have the expected default behaviour.
Then when Dart evolves, we'll be able to remove that generic parameter.

Can this generic parameter be inferred or will it be necessary to always specify it explicitly?

It cannot be inferred. Which is one of the main reasons why I used this workaround.

I guess this is the issue to watch: https://github.com/dart-lang/language/issues/102

I think this can be closed now that we have #341


DEV: FsR: Construct Graph QL DOT Call

EPIC:
#491 Fluids Report Implementation
PARENT:
#492 Get from Store (R) Fluids Report Object
SIZE:
16 hours = 2 days = 4 points
DEV: Fluids Report: Construct Graph QL DOT Call – Size (4pts)

@sukhendumukherjee  will confirm if this is still valid. ETA for update noon 7/25.

We are not using grapgql and we can close this.


Support bypass for expressurl.net

Can u guys add bypass for https://expressurl.net/ ?

If you provide a link I could try to tackle it


fix(behavior_velocity_planner): support u-turn for stop line (partial-apply #1700)

Description

https://github.com/tier4/autoware.universe/pull/612 にて依存している変数があるため、
#1700 を部分的に（stoplineのみ）適用します。
Related links

https://github.com/tier4/autoware.universe/pull/612
#1700

Tests performed

build testをパスできていることを確認済みです。
リグレッション確認済みです。
planning simulatorにて、障害物に対する停車挙動および、障害物撤去時の再発車挙動が既存と同一であることを確認しました。

Notes for reviewers

Interface changes

Effects on system behavior

Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[x] I've confirmed the contribution guidelines.
[x] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.
[ ] The PR has been properly tested.
[ ] The PR has been reviewed by the code owners.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.
[ ] The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

Codecov Report

:exclamation: No coverage uploaded for pull request base (feat/keep_stopping_dist/v0.3.16@0201fa9). Click here to learn what that means.
Patch has no changes to coverable lines.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@                        Coverage Diff                        @@
##             feat/keep_stopping_dist/v0.3.16    #614   +/-   ##
=================================================================
  Coverage                                   ?   6.38%
=================================================================
  Files                                      ?      70
  Lines                                      ?    5448
  Branches                                   ?     579
=================================================================
  Hits                                       ?     348
  Misses                                     ?    4882
  Partials                                   ?     218

Flag
Coverage Δ

differential
6.38% <0.00%> (?)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

一部適用であることからその他シーンモジュールに対する悪影響の見積もりが困難。
変更内容の妥当性を提示することが困難であるため、本PRはCanceledとします。


Allow using protobuf 4.X versions

Extend the protobuf version range to include 4.X versions. Code generated with at least version 3.19 of the protoc compiler is compatible with the 4.X versions.
This corresponds to the ansys-tools-protoc-helper version 0.4.0 and later.
See also:
https://github.com/ansys/ansys-api-template/issues/10
https://github.com/ansys/ansys-tools-protoc-helper#upgrade-plans

Incorporated this change in #177. Also changed ci/cd workflow to allow setting labels in a PR from a fork per https://github.com/actions/labeler#permissions.


Making complicated rules reduces the performance

Hey can you show me a screenshot of a rule that is running slowly? :)

Well, you can see one in the "Question?" discussion


QTimer warning

QObject::killTimers: timers cannot be stopped from another thread
terminate called after throwing an instance of 'std::bad_alloc'
terminate called recursively
what(): std::bad_alloc
Program "[15092] prefetch.exe" program ended with code 3 (0x3).

#56 QTimer warning is just normal behavior during unclean shutdown.


chore(devimint): ability to force-disable lnv2 module

Successfully created backport PR for releases/v0.5:

#6423


In some specific cases ng-scrollable gets stuck when page loads

I just found another weird case when for some reason mouse wheel on ng-scrollable does not work right after page loads, if spy-y is enabled:
http://plnkr.co/edit/xF30Qp3q3c1DqEVitBcb
This is really strange. For some reason it works just fine if I use ng-repeat instead of some static text. And it suddenly starts working again if I drag the scrollbar down manually (sometimes it does not react immediately, you have to try a few times) or if I change spy-y variable programmatically.

Oh, I found the reason - this happens if I don't have spy variables defined on the scope - then scroll functions receive undefined value and all hell breaks lose with NaNs propagated through  calculation.
The fix is simple:
            $scope.$watch(attrs[attr], function (val) {
              // protect against nulls and undefineds, which are possible
              // on first request if spy variables haven't been set
              // on outer scope
              val = val || 0;

Thanks for finding this. Will include your fix in the next version.


Add string helper methods for inline CHAR arrays

Is your feature request related to a problem? Please describe.
As brought up in https://github.com/microsoft/CsWin32/discussions/635, although we have string comparison and conversion helper methods for inline char arrays, we lack them for inline CHAR arrays. This makes it extremely cumbersome for users to initialize or read them.
Describe the solution you'd like
Generate similar code for CHAR inline arrays that we have for char inline arrays.

Just a small question:
Why we can't generate code like below?
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
internal struct ModuleEntry32
{
    internal int dwSize;
    internal uint th32ModuleID;
    internal uint th32ProcessID;
    internal uint GlblcntUsage;
    internal uint ProccntUsage;
    internal IntPtr modBaseAddr;
    internal uint modBaseSize;
    internal IntPtr hModule;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 256)]
    internal string szModule;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]
    internal string szExePath;
}

It will be more convenient to use.

That may be an option when allowMarshaling: true in the nativemethods.json file. I wonder if we already do something similar for the char inline array case. I doubt it. We generally shy away from non-blittable structs when we can use generated code to get a user friendly API while retaining blittability whereever possible.

Is there a recommended way to perform string to ``? I am currently falling over that exact problem:
internal partial struct BLUETOOTH_LOCAL_SERVICE_INFO
{
	internal winmdroot.Foundation.BOOL Enabled;

	internal winmdroot.Devices.Bluetooth.BLUETOOTH_ADDRESS btAddr;

	internal winmdroot.__char_256 szName;

	internal winmdroot.__char_256 szDeviceString;
}

Want to correctly fill in szName from a managed string.

Pin the sz field and your string, use Buffer.Copy or whatever you like to copy string's internal data to the field.

Pin the sz field and your string, use Buffer.Copy or whatever you like to copy string's internal data to the field.

👍 but what about encoding?

If copy from string(utf16) to CHAR(utf8) inline use Utf8.FromUtf16
If copy from ReadOnlySpan (u8 string) just copy.
If copy from CHAR to string use Utf8.ToUtf16
These API just take Span as input and it's really easy to convert inline array or buffer pointer to Span.


how to disable auto sync

Is this a similar or duplicate feature request?

[ ] I don't know. I will go check it.
[x] No.

I work with a lot of files and every time I start vscode they are synced despite not having modified them. Can't turn off auto sync and leave it to only upload files after autosave?
only sync with autosave
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Does this project help you?

[x] Yes. SFTP IS AWESOME!
[ ] Yes. I want to support you.

I'll add this to the backlog.

@E2estudio, I may not be understanding exactly what you're hoping to do here.
Can you disable by adding the uploadOnSave parameter to your sftp.json like so:
"uploadOnSave": false
Or are you hoping for some other feature here?

@E2estudio, I may not be understanding exactly what you're hoping to do here.
Can you disable by adding the uploadOnSave parameter to your sftp.json like so:
"uploadOnSave": false
Or are you hoping for some other feature here?

I have not explained well.
I have a project with thousands of files, and every time I open Vscode, almost all the files upload to the server without having made any changes to them. This greatly slows down the system. What I say is that you can disable automatic synchronization and that only files that are saved manually are uploaded.

@E2estudio ah ok I understand now, that is unfortunate!


[#7579] Improve the performance of finding active agents

Codecov Report

Merging #7580 (3b7b6f5) into master (5f68e2c) will decrease coverage by 0.01%.
The diff coverage is 63.15%.

@@            Coverage Diff             @@
##           master    #7580      +/-   ##
==========================================
- Coverage   38.66%   38.64%   -0.02%
==========================================
  Files        3506     3506
  Lines      100371   100383      +12
  Branches    12338    12340       +2
==========================================
- Hits        38811    38797      -14
- Misses      58560    58578      +18
- Partials     3000     3008       +8

Impacted Files
Coverage Δ

...orp/pinpoint/web/service/AgentInfoServiceImpl.java
0.00% <0.00%> (ø)

...vercorp/pinpoint/web/service/AdminServiceImpl.java
75.00% <100.00%> (-0.37%)
:arrow_down:

...main/java/com/navercorp/pinpoint/web/vo/Range.java
78.84% <100.00%> (+2.25%)
:arrow_up:

...p/pinpoint/rpc/client/WriteFailFutureListener.java
50.00% <0.00%> (-18.75%)
:arrow_down:

...vercorp/pinpoint/rpc/packet/ServerClosePacket.java
52.00% <0.00%> (-16.00%)
:arrow_down:

...vercorp/pinpoint/rpc/packet/ClientClosePacket.java
55.55% <0.00%> (-11.12%)
:arrow_down:

...nitor/DefaultDataSourceMonitorRegistryService.java
77.77% <0.00%> (-11.12%)
:arrow_down:

...pinpoint/profiler/instrument/AccessorAnalyzer.java
57.14% <0.00%> (-8.58%)
:arrow_down:

...rp/pinpoint/rpc/server/PinpointServerAcceptor.java
84.10% <0.00%> (-4.64%)
:arrow_down:

...cluster/zookeeper/ZookeeperClusterDataManager.java
52.90% <0.00%> (-1.94%)
:arrow_down:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f68e2c...3b7b6f5. Read the comment docs.

Codecov Report

Merging #7580 (3b7b6f5) into master (5f68e2c) will decrease coverage by 0.01%.
The diff coverage is 63.15%.

@@            Coverage Diff             @@
##           master    #7580      +/-   ##
==========================================
- Coverage   38.66%   38.64%   -0.02%
==========================================
  Files        3506     3506
  Lines      100371   100383      +12
  Branches    12338    12340       +2
==========================================
- Hits        38811    38797      -14
- Misses      58560    58578      +18
- Partials     3000     3008       +8

Impacted Files
Coverage Δ

...orp/pinpoint/web/service/AgentInfoServiceImpl.java
0.00% <0.00%> (ø)

...vercorp/pinpoint/web/service/AdminServiceImpl.java
75.00% <100.00%> (-0.37%)
:arrow_down:

...main/java/com/navercorp/pinpoint/web/vo/Range.java
78.84% <100.00%> (+2.25%)
:arrow_up:

...p/pinpoint/rpc/client/WriteFailFutureListener.java
50.00% <0.00%> (-18.75%)
:arrow_down:

...vercorp/pinpoint/rpc/packet/ServerClosePacket.java
52.00% <0.00%> (-16.00%)
:arrow_down:

...vercorp/pinpoint/rpc/packet/ClientClosePacket.java
55.55% <0.00%> (-11.12%)
:arrow_down:

...nitor/DefaultDataSourceMonitorRegistryService.java
77.77% <0.00%> (-11.12%)
:arrow_down:

...pinpoint/profiler/instrument/AccessorAnalyzer.java
57.14% <0.00%> (-8.58%)
:arrow_down:

...rp/pinpoint/rpc/server/PinpointServerAcceptor.java
84.10% <0.00%> (-4.64%)
:arrow_down:

...cluster/zookeeper/ZookeeperClusterDataManager.java
52.90% <0.00%> (-1.94%)
:arrow_down:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f68e2c...3b7b6f5. Read the comment docs.


Added features

Added some more features and methods to Robinhood.py, and adapted example.py to showcase a few of those features.
Also organized the code and added comments to both Robinhood.py and example.py
Updated the README.md to reflect those changes and provide more details

Made the recommended changes

:+1: @dsouzarc looks good, combine changes into one commit and let's merge this!

I have an issue with the use of raw_input which is CLI and Python 2 bias (raw_input removed in Python3). IMO, the API should require a symbol and let a new CLI module handle arguments and/or input.
Also, is there a reason why session, username, password, etc are done as class variables rather than instance variables? The latter seems much more reasonable because if you create more than one instance of this class, things will get weird :)


updated for firmware V10

updated for firmware V10

I didn't realize this pull request was here before i made all the same changes.
Please process pull requests so we can all help contribute to this project.


hack: test athens upgrade handler

Hack to test the v17-athens upgrade handler.
https://github.com/zeta-chain/node/actions/runs/9214733457/job/25351583884

Upgrade tests pass


Support storage.total_limit_size for elasticsearch

Is your feature request related to a problem? Please describe.
Fluent Bit supports limiting filesystem space for chunks(doc here). this is to control the amount of filesystem chunks are using. it started support from Fluent Bit 1.6 and it can be configured in OutPut section.
Describe the solution you'd like
Add support storage.total_limit_size for Elasticsearch plugin ClusterOutput.
Additional context
No response

Fixed in PR https://github.com/fluent/fluent-operator/pull/1196


feat(api,robot-server): Add Dockerfile and docker-compose file

Overview
This adds a Dockerfile to the root of the monorepo. It will build a backend in which to run the robot-server and hardware emulator.
A docker-compose file is added to start the robot-server connected to emulators.
closes #7674
Changelog

added .docker-ignore to root of monorepo
created Dockerfile that builds an image with api, robot-server, notify-server and dependencies.
the docker file copies source files then builds packages. in a separate stage it installs the built packages. Not sure if this is a smart approach vs just using a shared volume
add docker-compose.yml to run the robot-server and hardware emulation together.

Review requests
To run:

docker-compose up --build

I am no Docker wiz. Any advice is welcome.
If we like this approach I will add some notes in the read me for usage.
Risk assessment
None

Love it! Works for me. Left a couple suggestions on how to improve workflow- mostly just prepping for a docker-compose up -d situation~

Thanks! I made a little change to use a shared-volume for persistence of config files. Otherwise, every time one launches the image they'd have to re-calibrate.

This is a general question for everyone -- I've used docker-compose for some personal projects but I can't figure out how to get the CLI to work without having the desktop app open. Is it simply supposed to work like that?

This is a general question for everyone -- I've used docker-compose for some personal projects but I can't figure out how to get the CLI to work without having the desktop app open. Is it simply supposed to work like that?
Maybe cc @X-sam

Do you get the old docker daemon not running?

This is a general question for everyone -- I've used docker-compose for some personal projects but I can't figure out how to get the CLI to work without having the desktop app open. Is it simply supposed to work like that?
Maybe cc @X-sam

You need to be running docker desktop, it's the docker daemon controller that docker-compose works on. In fact, it's even been integrated with the docker CLI now: docker compose up --build

Really cool. I feel like the readme guy, but does it make sense to document the existence of it in the robot-server or tippy top readme?

Dang, READMEGUY, I was just about to do that!!!

One more bit of wondering, it seems like we could have lots of Docker possibilities. Is the idea that we stick whatever we want to be runnable in the container and use compose to run those services?  Seems like a nice  means of having a single container that can do anything in our vast store of monorepo possibilities.

One more bit of wondering, it seems like we could have lots of Docker possibilities. Is the idea that we stick whatever we want to be runnable in the container and use compose to run those services? Seems like a nice means of having a single container that can do anything in our vast store of monorepo possibilities.

Yes, and specify docker-compose to define entry points and env variables. It's not the only pattern, but it works.


Several code blocks in this article that should be normal text

There are several code blocks in this page that should be normal text, e.g.
"""
For more information, see Best Practices for using Windows Azure Cache/Windows Server Appfabric Cache and Application Configuration Settings (Windows Server AppFabric Caching). For additional information about Windows PowerShell scripts and .ps1 files, see Running Windows PowerShell Scripts.
"""

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: aff8221d-8995-2bd1-9ab4-e0043afe1c5f
Version Independent ID: 2b157758-daa4-dad2-5edb-891e421d921f
Content: Manage the Distributed Cache service in SharePoint Server
Content Source: SharePoint/SharePointServer/administration/manage-the-distributed-cache-service.md
Product: sharepoint-server-itpro
GitHub Login: @Techwriter40
Microsoft Alias: kirks

@AndreaBarr Assign this to me please.


Imogen web version

Compiled 0.13 (because 0.14 is not ready yet) with emscripten.
http://skaven.fr/imogen/imogen.html
Many, many, like, many broken stuff BUT it runs. Pretty happy with the result.
Branch is https://github.com/CedricGuillemet/Imogen/tree/dev-0.13-emscripten
run emb.bat to compile it once you have emscripten sdk installed and activated.

Main issue is to disable some features using #ifdef (ffmpeg, python,...) and rework  a lot of OpenGL calls. Some abstraction might help.


where are labels

Whether labels in the dataset have not been uploaded

CQU-BPDD datasets

The dataset is divided into three parts: train, test, and val. Each zip file contains folders that are categorized by labels. Therefore, the name of each folder corresponds to its respective label.


drivers: lsm303dlhc_accel: support for sensor_attr_set

Driver for LSM303DLHC accelerometer now supports API function
sensor_attr_set for attribute SENSOR_ATTR_SAMPLING_FREQUENCY.
Signed-off-by: Dennis Wildmark<EMAIL_ADDRESS>

Codecov Report

Merging #11018 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #11018   +/-   ##
=======================================
  Coverage   53.21%   53.21%
=======================================
  Files         217      217
  Lines       26667    26667
  Branches     5913     5913
=======================================
  Hits        14192    14192
  Misses      10059    10059
  Partials     2416     2416

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 27ec9fa...3f1d8a3. Read the comment docs.


Release with ember-fetch^7.0.0

The latest Ember app blueprint goes with ember-fetch^7.0.0.
The latest Ember-Simple-Auth release doesn't allow ember-fetch^7.0.0.
It causes dependency conflict.
If<EMAIL_ADDRESS>is not ready I suggest releasing a patch version of ESA^2 with support of ember-fetch@^7, so, it is compatible with the latest Ember.

https://github.com/simplabs/ember-simple-auth/releases/tag/3.0.0


Custom LLM from API for QA chain

Issue you'd like to raise.
Hi,
Currently, I want to build RAG chatbot for production.
I already had my LLM API and I want to create a custom LLM and then use this in RetrievalQA.from_chain_type function.
I don't know whether Langchain support this in my case.
I read about this topic on reddit: https://www.reddit.com/r/LangChain/comments/17v1rhv/integrating_llm_rest_api_into_a_langchain/
And in langchain document: https://python.langchain.com/docs/modules/model_io/llms/custom_llm
But this still does not work when I apply the custom LLM to qa_chain.
Below is my code, hope for the support from you, sorry for my language, english is not my mother tongue.
from pydantic import Extra
import requests
from typing import Any, List, Mapping, Optional

from langchain.callbacks.manager import CallbackManagerForLLMRun
from langchain.llms.base import LLM

class LlamaLLM(LLM):
    llm_url = 'https:/myhost/llama/api'

    class Config:
        extra = Extra.forbid

    @property
    def _llm_type(self) -> str:
        return "Llama2 7B"

    def _call(
        self,
        prompt: str,
        stop: Optional[List[str]] = None,
        run_manager: Optional[CallbackManagerForLLMRun] = None,
        **kwargs: Any,
    ) -> str:
        if stop is not None:
            raise ValueError("stop kwargs are not permitted.")

        payload = {
            "inputs": prompt,
            "parameters": {"max_new_tokens": 100},
            "token": "abcdfejkwehr"
        }

        headers = {"Content-Type": "application/json"}

        response = requests.post(self.llm_url, json=payload, headers=headers, verify=False)
        response.raise_for_status()

        # print("API Response:", response.json())

        return response.json()['generated_text']  # get the response from the API

    @property
    def _identifying_params(self) -> Mapping[str, Any]:
        """Get the identifying parameters."""
        return {"llmUrl": self.llm_url}

llm = LlamaLLM()

#Testing
prompt = "[INST] Question: Who is Albert Einstein? \n Answer: [/INST]"
result = llm._call(prompt)
print(result)

Albert Einstein (1879-1955) was a German-born theoretical physicist who is widely regarded as one of the most influential scientists of the 20th century. He is best known for his theory of relativity, which revolutionized our understanding of space and time, and his famous equation E=mc².

# Build prompt
from langchain.prompts import PromptTemplate
template = """[INST] <<SYS>>

Answer the question base on the context below.

<</SYS>>

Context: {context}
Question: {question}
Answer:
[/INST]"""
QA_CHAIN_PROMPT = PromptTemplate(input_variables=["context", "question"],template=template,)

# Run chain
from langchain.chains import RetrievalQA

qa_chain = RetrievalQA.from_chain_type(llm,
                                       verbose=True,
                                       # retriever=vectordb.as_retriever(),
                                       retriever=custom_retriever,
                                       return_source_documents=True,
                                       chain_type_kwargs={"prompt": QA_CHAIN_PROMPT})

question = "Is probability a class topic?"
result = qa_chain({"query": question})
result["result"]

Encountered some errors. Please recheck your request!

Suggestion:
No response

@dosu-bot
The first change is
def _llm_type(self) -> str:
    return "LlamaLLM"

However, for the second and third changes that you mentioned, I don't really understand. Could you write the detailed code for me? And how it use in RetrievalQA.from_chain_type function?

@dosu-bot
Could you define the _import_llama_llm function for me?

@dosu-bot
How can I add some logging in your _call method to print out the request and response details for testing this correct?

@dosu-bot
I still have this error
INFO:__main__:Received response: {'generated_text': 'Encountered some errors. Please recheck your request!'}

> Finished chain.
Encountered some errors. Please recheck your request!

Have any problem here? Btw, the format api of my side is same below:
curl --location 'https:/myhost:10001/llama/api' -k \
--header 'Content-Type: application/json' \
--data-raw '{
    "inputs": "[INST] Question: Who is Albert Einstein? \n Answer: [/INST]",
    "parameters": {"max_new_tokens":100},
    "token": "abcdfejkwehr"
}

Is this have any problem with my setup to wrap the LLM from api
class LlamaLLM(LLM):
    llm_url = 'https:/myhost/llama/api'

    class Config:
        extra = Extra.forbid

    @property
    def _llm_type(self) -> str:
        return "Llama2 7B"

    def _call(
        self,
        prompt: str,
        stop: Optional[List[str]] = None,
        run_manager: Optional[CallbackManagerForLLMRun] = None,
        **kwargs: Any,
    ) -> str:
        if stop is not None:
            raise ValueError("stop kwargs are not permitted.")

        payload = {
            "inputs": prompt,
            "parameters": {"max_new_tokens": 100},
            "token": "abcdfejkwehr"
        }

        headers = {"Content-Type": "application/json"}

        response = requests.post(self.llm_url, json=payload, headers=headers, verify=False)
        response.raise_for_status()

        # print("API Response:", response.json())

        return response.json()['generated_text']  # get the response from the API

    @property
    def _identifying_params(self) -> Mapping[str, Any]:
        """Get the identifying parameters."""
        return {"llmUrl": self.llm_url}

@dosu-bot
API Response: If none of the above suggestions help, you might need to look at the response from your API more closely. If the API provides more detailed error messages or codes, those could help you figure out what's going wrong. You could modify your _call method to log the entire response from the API, not just the 'generated_text' field, to get more information.
please support me modify the _call method to log the entire response from the API, not just the 'generated_text' field, to get more information.

@dosu-bot
What the format of llm in RetrievalQA.from_chain_type function?
I testting the function is OK, but maybe the format in the RetrivalQA is the different, so that cause the failure.
llama_llm = LlamaLLM()
prompt = "[INST] Question: Who is Albert Einstein? \n Answer: [/INST]"
result = llama_llm._call(prompt)
print(result)

Albert Einstein (1879-1955) was a German-born theoretical physicist who is widely regarded as one of the most influential scientists of the 20th century. He is best known for his theory of relativity, which revolutionized our understanding of space and time, and his famous equation E=mc².

@dosu-bot
> Finished chain.
Encountered some errors. Please recheck your request!

How can show exactly what is encountered in the request? because now it's hard to debug

@dosu-bot
Have any easy way to check it?
Again, consider my setup below and give the best as you can for the solving problem and debugging.
from pydantic import Extra
import requests
from typing import Any, List, Mapping, Optional

from langchain.callbacks.manager import CallbackManagerForLLMRun
from langchain.llms.base import LLM

class LlamaLLM(LLM):
    llm_url = 'https:/myhost/llama/api'

    class Config:
        extra = Extra.forbid

    @property
    def _llm_type(self) -> str:
        return "Llama2 7B"

    def _call(
        self,
        prompt: str,
        stop: Optional[List[str]] = None,
        run_manager: Optional[CallbackManagerForLLMRun] = None,
        **kwargs: Any,
    ) -> str:
        if stop is not None:
            raise ValueError("stop kwargs are not permitted.")

        payload = {
            "inputs": prompt,
            "parameters": {"max_new_tokens": 100},
            "token": "abcdfejkwehr"
        }

        headers = {"Content-Type": "application/json"}

        response = requests.post(self.llm_url, json=payload, headers=headers, verify=False)
        response.raise_for_status()

        # print("API Response:", response.json())

        return response.json()['generated_text']  # get the response from the API

    @property
    def _identifying_params(self) -> Mapping[str, Any]:
        """Get the identifying parameters."""
        return {"llmUrl": self.llm_url}

llm = LlamaLLM()

#Testing
prompt = "[INST] Question: Who is Albert Einstein? \n Answer: [/INST]"
result = llm._call(prompt)
print(result)

Albert Einstein (1879-1955) was a German-born theoretical physicist who is widely regarded as one of the most influential scientists of the 20th century. He is best known for his theory of relativity, which revolutionized our understanding of space and time, and his famous equation E=mc².

# Build prompt
from langchain.prompts import PromptTemplate
template = """[INST] <<SYS>>

Answer the question base on the context below.

<</SYS>>

Context: {context}
Question: {question}
Answer:
[/INST]"""
QA_CHAIN_PROMPT = PromptTemplate(input_variables=["context", "question"],template=template,)

# Run chain
from langchain.chains import RetrievalQA

qa_chain = RetrievalQA.from_chain_type(llm,
                                       verbose=True,
                                       # retriever=vectordb.as_retriever(),
                                       retriever=custom_retriever,
                                       return_source_documents=True,
                                       chain_type_kwargs={"prompt": QA_CHAIN_PROMPT})

question = "Is probability a class topic?"
result = qa_chain({"query": question})
result["result"]

> Entering new RetrievalQA chain...

> Finished chain.
Encountered some errors. Please recheck your request!

@dosu-bot
Could I ask for support from some dev of Langchain to solve this case?

This happens because of the context length setting of the API. So I already fixed it and it works fine now. Thanks for your support!

I am trying something similar, with RetrievalQA from chain type, but I get below error for my custom Mistral7B LLM
'Mistral7B_LLM' object has no attribute 'callbacks'
@dosu-bot


Fix broken contributing guide link.

Please describe the changes this PR makes and why it should be merged:
The Discord.js documentation welcome page has a broken link (link is at "See the contribution guide if you'd like to submit a PR."). This PR fixes the broken link.
Semantic versioning classification:

[ ] This PR changes the library's interface (methods or parameters added)
[ ] This PR includes breaking changes (methods removed or renamed, parameters moved or removed)
[x] This PR only includes non-code changes, like changes to documentation, README, etc.

Fixed in 8c6167d3. Thanks for your contribution, though.


Pease allow importing json files

Node allows requiring .json files out of the box, in fact it's common to expose version of the library by importing it from package.json  but that does not work with flow.

This is a dupe. #620