Expanded balance point search range

Default grid search parameters (heating: 55-65 F, cooling: 66-75 F) may be restrictive in more extreme climates.
We propose the following:
Allowing non-overlapping user-defined grid search ranges.
Allowing the balance point search to be conducted in temperature increments up to 2 deg F.

Testing: For a test dataset calculate savings using Caltrack as well as arbitrary expanded search ranges

Explore balance point estimates by climate zone and building type? Document the diversity of results.

To achieve accurate regressions HEA has found it is necessary to adjust balance point temperatures for each home independently across a broad temperature range, even in mild climates like the SF Bay Area.

Due to the non-linearity of degree days, getting the right balance point temperature
for each home is crucial to accurately disaggregate heating and cooling loads.
HEA has observed a 25F degree variation in balance point temperatures from homes in the same area, due largely to high internal gains (from occupants & plug loads) combined with increasingly well-insulated homes.
HEA utilizes a balance point range of 50-75F for heating and 65-90F for cooling.

That's helpful. Are you able to provide test results that show differences in results if you constrain your balance points? For example, see some of the work that was done in CalTRACK 1.0 where we established a methodology for floating balance points for daily methods.

@margaretsheridan I wonder if you could say more about your proposal? Are you suggesting that there might be variable ranges of balance points based on the particular climate zone or building type? So we would advise limiting balance point ranges in temperate climates, but allowing more extreme values in more extreme climates?

Energy Trust has also observed a wide range of balance points creating site-level models with good fits in various billing analyses over the years. We support increasing the range of balance point temperatures tested, but I'm not sure that it should be a user-defined. I think we should use the same ranges of balance points to be tested for every site, to be consistent. I don't have a specific recommendation for ranges of balance point temperatures to test, but we have often tested a 30 degree range of balance point temps for HDD and CDD. For example: HDD 40-70 and CDD 60-90.
The HDD and CDD balance point ranges should be allowed to overlap. You may want to test both HDD60/CDD60 and HDD65/CDD70. So long as you do not allow the CDD balance point to be less than HDD balance point, I think it is fine and maybe even desirable to have some overlap between the ranges of balance point values tested.
We have a slight preference for a one-degree increment in balance point temperatures tested, but it probably doesn't make that big of a difference.

I have been looking at balance points in reference to some work I have been doing here at SMUD and I thought we may see some variation based on a number of characteristics such as climate zone, state, building type, and building age. Looking at heating and cooling slopes and balance points could help to identify outliers in cohort groups. For example, looking at large samples of homes within California building standard vintages can provide quite a bit of insight into the efficiency and use of individual homes.

FWIW, I use a multi-pass grid search so that a wider search increment can be used initially.
A fairly wide search range is needed for commercial buildings.

TEST RESULTS
Background
The CalTRACK model uses variable base degree day regression to model baseline and reporting period energy use. Part of this process involves fitting a number of models across a range of heating and cooling balance points. One decision point that is typically decided by an analyst, is the actual search range of balance points. A larger search range requires more computational time and may yield non-physical results (e.g. if the heating balance point is 100 F), especially when data quality is subpar. On the other hand, a search range that is too narrow may lead to poor model fit.
Dataset
Billing data from 1000 residential buildings in Oregon and daily data from 1000 participants in an HVAC program in California.
Tested parameters
Default Caltrack methods were used with different search ranges for the HDD and CDD balance points.
Results
Caltrack models were fit to the Oregon building usage dataset using 4 balance point search ranges:
• 10 degree spread: 55-65 heating, 65-75 cooling
• 20 degree spread: 45-65 heating, 65-85 cooling
• 30 degree spread: 40-70 heating, 60-90 cooling
• 40 degree spread: 40-80 heating, 50-90 cooling
Figure 1 shows the distribution of best-fit HDD balance points for three of those ranges. It is clear that when the search grid is constrained, many models tend to settle at either end  of the search range (e.g. for the default search range, almost 30% of buildings have an HDD balance point of exactly 65 F).  When the search range is expanded, the distribution of best-fit balance points tends towards a Gaussian distribution (centered at 63 F for this particular dataset). Therefore, we can conclude that a search grid that is too constrained may result in sub-optimal model fits.
Interestingly, Figure 2 demonstrates that even though many models may be sub-optimal, the effect on out-of-sample model performance is marginal. Only a handful of models see a significant change in out-of-sample mean absolute error in predictions and the change in overall performance for the sample is negligible.

Figure 1. Effect of balance point search range on distribution of best-fit HDD balance points.

Figure 2. Effect of balance point search range on out of sample mean absolute errors (MAE)
In a separate study, the effect of the balance point range on estimated savings was also analyzed using a dataset of HVAC program participants. Baseline models were fit to the full set of program participants five times, varying the search range for the HDD balance point and keeping all other parameters constant. Figure 3 shows clearly that the HDD search range has a negligible effect on calculated savings.

Figure 3. Effect of HDD balance point search range on program savings.
We also tested several search increments between 1 and 4 F and in general, the fits do not seem to be much affected when the best-fit balance point is off from the optimal one by up to 2 degrees (Figure 4). This implies that having search increments up to 3 degrees is acceptable as it ensures that the optimal temperature will always be within one degree of a point on the search grid.

Figure 4. Effect of varying heating balance point by 1-2 degrees on model fit.
Recommendations

Expand balance point search range to 40-80 F for heating and 50-90 F for cooling.
Allow variable search increments up to 3 F, including multipass grid search and non-uniform search increments.

Great data!
These results make sense to me, except for Figure 3...
Can anyone explain why all five results in this chart look absolutely identical?
I would have expected different types of energy savings (HVAC vs baseline) to produce variations in this test.

Good question, @steevschmidt . Most models didn't change with an expanded search grid and I think it's just a matter of the dense plot masking the points that did change.  I went back and pulled out a 100 buildings where the optimal balance point and total savings did change and as you can see below the changes are very modest. It appears that when we change the balance point search grid, the savings breakdown (heating vs cooling vs baseload) does change quite a bit, however, the aggregate savings are relatively unchanged (usually only a 2-3% change). I'll dig in deeper when I get an opportunity.

One comment about this:  Since we will eventually, if not now, be concerned with the timing of savings, e.g. the load shape of savings, and since you wrote that the savings breakdown (heating vs cooling vs baseload) does change quite a bit, getting the appropriate balance or change point may be much more important than just for total savings, especially for interval data.
Bill Koran, P.E., CMVP
SBW Consulting, INChttp://www.sbwconsulting.com/
503-974-9741
From: hshaban<EMAIL_ADDRESS>Sent: Thursday, March 1, 2018 11:52 AM
To: CalTRACK-2/caltrack
Cc: Bill Koran; Comment
Subject: Re: [CalTRACK-2/caltrack] Expanded balance point search range (#72)
Good question, @steevschmidthttps://github.com/steevschmidt . Most models didn't change with an expanded search grid and I think it's just a matter of the dense plot masking the points that did change. I went back and pulled out a 100 buildings where the optimal balance point and total savings did change and as you can see below the changes are very modest. It appears that when we change the balance point search grid, the savings breakdown (heating vs cooling vs baseload) does change quite a bit, however, the aggregate savings are relatively unchanged (usually only a 2-3% change). I'll dig in deeper when I get an opportunity.
[image]https://user-images.githubusercontent.com/13769526/36866252-a5dc1be2-1d46-11e8-847e-39576d7a4425.png
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/CalTRACK-2/caltrack/issues/72#issuecomment-369709984, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ai08ipbKJXlYS7LRLuCoPq3CYNRQHoJOks5taFFCgaJpZM4R1exa.

Hassan wrote:

... savings breakdown (heating vs cooling vs baseload) does change quite a bit, however, the aggregate savings are relatively unchanged ...

If you have an inaccurate disaggregation of heating & cooling loads, and significant changes in weather between the base and reporting years, I believe your savings calculations will be inaccurate.

@bkoran @steevschmidt I agree that we'd ideally get an accurate load disaggregation. The results show that for some buildings, the balance points and load disaggregation change when expanding the search range, but we don't have the ground truth. The question raised by this issue is what search range we should explore when selecting models. Does using a wide search range give us a better chance at finding the best balance point or is a constrained one more appropriate? Your thoughts?

Does using a wide search range give us a better chance at finding the best balance point or is a constrained one more appropriate? Your thoughts?

I think wider is better, but for balance points outside of expected ranges more work can (& should?) be done. For example, I mentioned pool heating during today's call. There are homes with two different heating balance points throughout the year: one very high for pool heating during an extended summer swim period, and another during the "home heating only" parts of the year. Depending on whether your goal is "fit" or "accurate load disaggregation", one may chose different approaches.

I also would like to emphasize the quality of the temperature data going into the model - particularly for hourly estimation. I performed correlations of hourly weather station data by proximity to fill in missing data and to help identify outliers. This cleaning activity greatly improved my adjusted R squared values. I am now using a third-party product to ensure the quality of the weather data. This should be foundational and should be part of a standardized process.

I think wider is better, but for balance points outside of expected ranges more work can (& should?) be done.

I agree with you @steevschmidt . A wider range makes sense and we could take anomalous balance points into account with building qualification (those buildings may need custom models separate from the default Caltrack path)

@margaretsheridan Definitely useful input. If you don't mind, could you add some thoughts around weather data cleaning and how to treat missing values in Issue #65? Or any relevant literature/documentation would also help.

Also agree with margaretsheridan.  We've seen cases where nearby NWS sites don't track each other; there are anomalous deviations for a period of time of months or more.  (I've also seen where temperatures deviate dependent upon which way the wind is blowing, which is not anomalous.)  This is in addition to the dropouts and bad data.  Data cleaning, in general, is an important issue.

Several related comments:
These models will almost inevitably be used in ways you don’t expect. For example, I have seen similar models used in industrial applications.  In those cases, the constrained search range could be far from the truth.  More below about how constrained searches can be an issue.
There is little merit, in my view, to using a narrow search range, nor to use a wide tolerance.  A search for a single change point is pretty fast:
Checking using ECAM, with a search resolution of 0.05 ºF:
365 (daily) data points:  less than 0.5 seconds
5110 (hourly) data points:  less than 1.0 seconds
(I’m not using a decent resolution timer to get a better number.)
Of course, finding 2 change points greatly increases the iterations.
846 (hourly) data points:  less than 3 seconds with a resolution of 0.18 ºF; add less than 1 second to get the resolution <0.02 ºF.
(Yes, I know these resolutions are much, much finer than we actually know the temperatures at the site.)
This is in a spreadsheet with interpreted code and worksheet functions!  With compiled code and better programmers than me, on a server, maybe you can probably beat that by two orders of magnitude?
These searches use the full range of the data, merely requiring at least 3 points on each side of a change point.
Also, there is no reason to use degree-days with daily or hourly data.  Just use temperature directly.  There is no loss of information since degree-days are already at the daily level.  Why use an intermediate calculation?  It is extra, less transparent, and doesn’t plot as easily.  (In most climates, we have seen very little if any value to even using degree-days at the monthly level instead of monthly average temperature, but theoretically degree-days should be better, and likely is in CA climates.)
Another point is that you need to use a wide search range because the temperatures found are not necessarily heating and cooling base temperatures:  For example, in a residence with a heat pump with supplemental electric heat, there will be 2 heating slopes and hence 2 heating coefficients:  One coefficient at moderately cool temperatures for heat pump heating, and a different coefficient at colder temperatures when the supplemental electric heat kicks in.  Similar things occur on the cooling side with economizers and mechanical cooling.  Yet another change point can occur due to equipment running out of capacity.  This can occur at quite cold or quite warm temperatures.
A final, less important point:  The best fit balance point temperature or change point is not always the maximum likelihood change point.  This doesn’t seem to be an issue with higher-frequency data, but it can rarely be an issue with monthly billing data, and can result in one of those change points that doesn’t seem to make sense.  I’ve never implemented it, but a maximum likelihood change point might avoid a few change points that fall outside of expectations.
Bill Koran, P.E., CMVP
Sr. Energy analytics engineer
SBW Consulting, Inc.
bill.koran<EMAIL_ADDRESS>503-974-9741
3945 parker rd.
west linn, or 97068
www.SBWConsulting.comhttp://www.sbwconsulting.com/
From: hshaban<EMAIL_ADDRESS>Sent: Thursday, March 1, 2018 1:29 PM
To: CalTRACK-2/caltrack
Cc: Bill Koran; Mention
Subject: Re: [CalTRACK-2/caltrack] Expanded balance point search range (#72)
@bkoranhttps://github.com/bkoran @steevschmidthttps://github.com/steevschmidt I agree that we'd ideally get an accurate load disaggregation. The results show that for some buildings, the balance points and load disaggregation change when expanding the search range, but we don't have the ground truth. The question raised by this issue is what search range we should explore when selecting models. Does using a wide search range give us a better chance at finding the best balance point or is a constrained one more appropriate? Your thoughts?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/CalTRACK-2/caltrack/issues/72#issuecomment-369737008, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Ai08isB3GAn7JL83ugJeCnUUuzVvX9NPks5taGgXgaJpZM4R1exa.

Final comment.  Again, I don't see a strong rationale for constraining the grid search.
Related to this, it is important that the daily methods are not limited to a single slope for heating and a single slope for cooling.  This often doesn't match the physics of the building and can result in poorer fits.  It may not mean that savings estimates will change greatly, but it probably will:  Least squares slopes tend to aim at the points that are at the extreme cold or extreme warm temperatures.  However, the majority of temperatures are in the more moderate range.  So the slopes, and hence predictions will be off.  Attached is a model for gas; similar situations for electricity can also occur.
You may consider that the attached gas use could be modeled with a negative coefficient for the cooling slope.  That is possible, but the point is that a model limited to 2 slopes, or 2 slopes plus a flat range between the balance/change point temperatures, is not always sufficient for daily data.

http://www.calmac.org/publications/AMI_Report_Vol_2_Appendices_FINAL.pdf

@bkoran I've been thinking a lot about the "elbow" issue. There's no doubt that it's observable. There are three questions that seem important to answer about it. First, what does the poor model fit do to savings estimates? If the model errors are evenly distributed over the course of a year, would we be concerned about its effects? Second, are there quantitative criteria for determining when an "elbow" would be valuable to search for? Should it be default? Should it be configurable? Should it be based on probability of occurrence? Finally, how do we test for this? The image you showed above is a kW function, not a kWh function. Is that relevant?
It strikes me that when we get to hourly modeling that this is going to be a much more relevant conversation. Don't give up on this. We are benefitting from your insights on this issue. We just need a way to generalize it and test it so that it can be fit into the methods specification in a clear, broadly applicable fashion.

I can't answer most of your questions very well.  I covered savings up above; I do think it can have an effect.  It could be that it's effects could be minimized by using a different fit than least squares; not sure about that.
There are physical characteristics when an "elbow" should be searched for, but I don't know how to know when analytically except by brute force trying and testing significance.
I don't think it should be default; it is common but not the most common.
Configurable?  Don't know what you mean.
kW vs. kWh is just time integration; don't think it matters

A bit more--this is more specific to hourly models, although it can affect daily models:
One of the most common measures in building commissioning is improving economizer operation, i.e. cooling with outside air. Economizers operate at moderate temperatures, from the cooling change/balance point up to somewhere between 62 and 73 degF, depending on climate and humidity.  In a model they are characterized as having a steeper slope than the rest of the cooling.  Since the savings are at temperatures close to the change/balance point, I believe the effect of this improvement would be all but invisible if only a single slope is used for cooling.  Maybe that is good enough in a whole building project with many measures, but this is a very common measure, and the 2 slopes are often obvious when an economizer is performing well.  (Corollary is that the base case often doesn't show it, but the post case does.)

When I was forecasting hourly system load for systems operations at SMUD years ago, I used multiple techniques including a regression with a third order polynomial for the temperature-load relationship. This would track fairly well with piecewise linear load-temp relationship. The greater simplicity of modeling would need to be examined relative to any loss in accuracy.

This update has been integrated in CalTRACK 2. Closing this issue


Improve framework support

Changelog
Added

The buildpack will add a default process type if a supported framework is detected and the expected build output is found. This mirrors the same feature from the Maven buildpack. (#726)
Support for the Micronaut and Quarkus frameworks. Both previously worked with the buildpack but required some configuration. Unless heavily customized, no build task needs to be specified anymore. (#726)

Also, will these frameworks be supported by the Maven buildpack?

Yes! :)


Make modules reusable

It should be able to dispatch module-self action for one module and another one but the same one will not reflect to that action.

done in v1.0.0


AIV editor - Attkunitally field - AI to help other AIs during siege ?

Hi guys, does anyone know if you can use the AIV editor to set the AI to automatically send troops and help other AI who was under siege ?

no

So far, you best chance for such functionality would be one of the still-unknown AIC variables.


Make iter_files deterministic

Fix #5145.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


Add get_query_executions generating DataFrames from Athena query executions detail

Feature or Bugfix-

Feature

Detail

Add list_query_executions to Athena module for fetching list query execution IDs.
Add get_query_executions to Athena module for returning DataFrames contain query executions details

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 1730739ff7c86dd2b2a9b604e30ad3ab40d1a49a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 5eaa5ac1e46e0fe494a0753f8f27143f93224ef4
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 44baa9d816b5e7df2c943ce9291164c08a154244
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 6a26fe6591cc0c01d46443df12a3ff2caf99206b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


Trouble with traefik, always return 404

I installed and run these two line:
TAG=prod FRONTEND_ENV=production bash ./scripts/build.sh
docker-compose up -d

Every Server is up and work fine, but when I try to go http://localhost or http://localhost/docs, it's always return 404.
How can I fix it, I'm using Mac with Catalina(10.15.4)
Check traefik log about 507 & 509:
<IP_ADDRESS> - - [10/Jul/2020:14:59:08 +0000] "GET /api/overview HTTP/1.1" 200 435 "-" "-" 506 "api@internal" "-" 0ms
<IP_ADDRESS> - - [10/Jul/2020:14:59:11 +0000] "GET / HTTP/1.1" - - "-" "-" 507 "-" "-" 0ms**
<IP_ADDRESS> - - [10/Jul/2020:14:59:13 +0000] "GET /api/overview HTTP/1.1" 200 435 "-" "-" 508 "api@internal" "-" 0ms
<IP_ADDRESS> - - [10/Jul/2020:14:59:15 +0000] "GET /docs HTTP/1.1" - - "-" "-" 509 "-" "-" 0ms**
<IP_ADDRESS> - - [10/Jul/2020:14:59:18 +0000] "GET /api/overview HTTP/1.1" 200 435 "-" "-" 510 "api@internal" "-" 0ms

I assume you solved it, thanks for closing the issue :+1:

I am having the same issue. How did you solve this issue where traefik keeps returning 404 error


UserWarning: Polars found a filename when using EagerPolarsDataset

Description
First reported at https://linen-slack.kedro.org/t/22699778/hi-everyone-i-just-upgrade-to-kedro-1-1-and-i-m-now-getting-#1f0022c7-f023-4f1f-b963-4410aae0e6d2
UserWarning: Polars found a filename. Ensure you pass a path to the file instead of a python file

Comes from:
https://github.com/kedro-org/kedro-plugins/blob/064c5d9bc491a3f96f4341a2c2d1f034b7b354cb/kedro-datasets/kedro_datasets/polars/eager_polars_dataset.py#L158-L159
Context
Nothing breaks, but it puzzles users and adds noise.

Similarly to https://github.com/kedro-org/kedro-plugins/issues/590, this comes because of how we use fsspec. But as explained in https://github.com/kedro-org/kedro-plugins/issues/625, we rely on fsspec because of our versioning capabilities.

Unsure if this is still an issue, see https://github.com/pola-rs/polars/issues/19150


Incompatibility elementary-data with Airflow (MWAA)

Describe the bug
We are currently using Airflow (MWAA) 2.7.2, we would like to use elementary with Airflow and run edr commands in operators.
But there is a constraint for the library pydantic which has to be 2.4.2 but elementary-data is using pydantic <2.0.0.
Please find all airflow constraints here:
https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.11.txt
To Reproduce
Create a file requirements.txt with:
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.11.txt"
elementary-data==0.14.0
Then run the command:
pip download -d tmp/plugins --requirement requirements.txt

You must obtain the following error:

Expected behavior
We should be able to install all requirements without conflicts between airflow and elementary.
Environment (please complete the following information):

edr Version: 0.14.0
dbt package Version: 0.14.0
airflow: 2.7.2

One way to get around conflicting requirements between Airflow and dbt/packages is to run dbt inside a virtualenv within the airflow workers. Can set up a virtualenv using a startup script and activate it before running the dbt commands.

One way to get around conflicting requirements between Airflow and dbt/packages is to run dbt inside a virtualenv within the airflow workers. Can set up a virtualenv using a startup script and activate it before running the dbt commands.

That's exactly what we do.

Hi @mattxxi !
Thanks for opening this issue and apologies for only replying now.
The issue with the pydantic requirement has actually been solved since version 0.14.1 of the package!
(Running in a virtualenv though as suggested above is a good practice in general, if you are already doing so)


WIP: Use core-dns over kube-dns for clusters >= 1.10

What this PR does / why we need it:
Use core-dns instead of kube-dns by default for clusters >= 1.10
fixes #422
Default to core-dns over kube-dns for clusters >= 1.10

/assign

/assign @JoshVanL
/unassign


Replace HTTP request multipart encoder with Netty's implementation

Pros:

heavily tested encoder
support for chunked input without a necessity to read everything into memory right away
easily might be extended to support multipart/mixed, not only multipart/form-data

Cons:

custom transfer encodings are not supported right now, only binary and plain text (to add base64 or qp we need to extend Netty's implementation first)

I'm very happy to limit our multipart implementation to whatever Netty can offer out of the box.  Thanks for this.


SyntaxError.Unexpected identifier

I am getting this following error on linux os
(function (exports, require, module, __filename, __dirname) { import mysqldump from 'mysqldump'
^^^^^^^^^
SyntaxError: Unexpected identifier

That looks like a problem with whatever bundler/compiler you're using.
The end of the line you provided: import mysqldump from 'mysqldump'
We don't import our own package into our package, so the problem is with your code somewhere...


BeatMods rejection

ServerBrowser was rejected from BeatMods for now. This is for the latest release (v0.1.1).

Server Browser was denied on BeatMods- theres absolutely nothing stopping people from joining a custom lobby if they dont have the song or mpex, which soft locks the game if they do try

it makes it really easy for someone to just see something in the server browser, click a map expecting to join, and then the game just crashing

if approved there will be a lot of "i cant join half of the server browser games and my game crashes!" in pc-help

This is a totally valid point.
Issue #7 addressed this to an extent (prevent unmodded players from seeing modded lobbies), but this hasn't been shipped as a release yet.
BeatMods approval will take a while, and with cross-play mods on the horizon I think v0.2.0 should be the next version I try to submit as it will be relatively future-proof.
Checklist

[x] Finish implementing v0.2.0 features (#11, #9, #5)
[x] Testing scenarios for players without MpEx
[x] Testing scenarios for unowned songs (e.g. official DLC song in progress)
[x] Testing with cross-play mods / master server switching
[ ] Ship v0.2.0

Tested & ready, preparing to ship v0.2.0.

Shipped


"Why do these operations take so long?" should be on separate line in "Too many files" section

I believe the bullet points in that section were meant to be spaced like so:

‘git status’ takes up to 10 minutes, even where there are no dirty files in the repo
‘git checkout’ can take 3 hours
‘git commit’ takes 30 minutes

Why do these operations take so long? Two fundamental reasons:

Git maintains an index file that contains a flat list of every file in the repo. This creates overhead for every single command that needs the index, since Git has to read and parse the index every time you run those commands.
When you run an operation like status or checkout, Git essentially has to walk through every single file in your working directory, compare those files to the related entry in the index, and then compare the entries in the index to the corresponding objects in the Git tree. With over 3M files in the repo, the sheer cost of all that file IO means there’s no way for the operation to complete quickly enough to feel responsive.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b891ab97-f15e-79b7-df68-38c29e45efaf
Version Independent ID: 52b38d10-59c6-7c3b-a00c-7ccdab65a5b8
Content: Technical Scale Challenges with Git - Azure DevOps
Content Source: https://mseng.visualstudio.com/DefaultCollection/TechnicalContent/_git/devops-learn-pr?path=%2fdocs%2fgit%2ftechnical-scale-challenges.md&version=GBlive&_a=contents
Product: devops
Git Login: samgu
Microsoft Alias: sanoursa

@judemars Looks like the formatting got messed up when these articles were migrated (OK I just did a quick sweep of the Git at Scale articles and only this one had that issue). I just merged a fix for this into our private repo and it should be live soon. Thanks for letting us know!

No problem! Thanks for the awesome documentation


Sesame Street Counting Cafe (USA).md nothing but a black screen

This depends on the 68000 getting to execute one more instruction before handling VINT if it enables vertical interrupts while a vertical interrupt is pending:
https://gendev.spritesmind.net/forum/viewtopic.php?t=2202


[Feature] Option for Blinking Red When low

at low health (e.g. <30%)
2.1. increase the frequency the lower their health gets

@shemetz request


[simple-hub] change heroku application naming

As soon as this PR is merged in, https://dashboard.heroku.com/apps/simple-hub-staging/settings should be ranamed from simple-hub-staging to simple-hub-production

I regenerated this PR against the latest master over at #1704


Update redirects

Created with https://github.com/dkhamsing/frankenstein

Neat, thanks!

@natecook1000 if you know any other repos that could use this, let me know 😄


Avoid repetitions in transform

There are some repetitions in transform if looped over frequencies or offsets that could be avoided. In the case of the lagged convolution Hankel DLF this can make as much as 10 % of the runtime (but then, in the case of the lagged convolution Hankel DLF, the runtime is VERY fast, so overall gain is probably rather small in absolute numbers and might not be worth the effort).

[x] Move lambd-calculation outside of empymod.transform, to avoid repetition if looped over frequencies.
[x] Pass factAng instead of angle to empymod.transform, to avoid repetition if looped over frequencies or offsets.

The first point, lambd/int_pts, should be in utils. However, this would be heavily backwards incompatible, as off would new have to be provided to utils.check_hankel.
Move there in version 2.0.

This will be undone again. It is a very particular case. Very fast anyway. And not worth introducing inconsistencies for that reason.


No such file or directory

diff: /../Podfile.lock: No such file or directory
diff: /Manifest.lock: No such file or directory
error: The sandbox is not in sync with the Podfile.lock. Run 'pod install' or update your CocoaPods installation.

pod install

sorry


Resolving a consul flap issue introduced by recent branch merge

reverting some fixes to enable stabler master.
Test and renable the fixes while addressing #69
Signed-off-by: Madhu Venugopal<EMAIL_ADDRESS>

Tried testing and got intermittent weird ping time outs..  Here is the sequence...
vagrant up
vagrant ssh socketplane-1
sudo socketplane network create web <IP_ADDRESS>/16
vagrant ssh socketplane-2
sudo scoketplane run -n web -itd ubuntu:14.04
sudo scoketplane run -n web -itd ubuntu:14.04
go back to socketplane-1
sudo scoketplane run -n web -itd ubuntu:14.04
sudo socketplane attach
ping <IP_ADDRESS>
ping <IP_ADDRESS>
With intermittent results as follows:
vagrant@socketplane-1:~$ sudo socketplane run -n web -itd ubuntu:14.04
7d5e9238ef921a053d739998b9ecb9db0d678109ada3106b4e553041f227eea8
vagrant@socketplane-1:~$ sudo socketplane attach 7d5e9238ef9
root@7d5e9238ef92:/#
root@7d5e9238ef92:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet <IP_ADDRESS>/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
17: ovs15bf5e0: <BROADCAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default
    link/ether 02:42:0a:02:00:04 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/16 scope global ovs15bf5e0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:aff:fe02:4/64 scope link
       valid_lft forever preferred_lft forever
root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
From <IP_ADDRESS> icmp_seq=1 Destination Host Unreachable
From <IP_ADDRESS> icmp_seq=2 Destination Host Unreachable
From <IP_ADDRESS> icmp_seq=3 Destination Host Unreachable
^C
--- <IP_ADDRESS> ping statistics ---
6 packets transmitted, 0 received, +3 errors, 100% packet loss, time 5031ms
pipe 3
root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=0.028 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.037 ms
64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=64 time=0.230 ms
64 bytes from <IP_ADDRESS>: icmp_seq=4 ttl=64 time=0.064 ms
^C
--- <IP_ADDRESS> ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.028/0.089/0.230/0.082 ms
root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=1.88 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.471 ms
64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=64 time=0.528 ms
^C
--- <IP_ADDRESS> ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.471/0.960/1.883/0.653 ms
root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=1.71 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.498 ms
64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=64 time=0.665 ms
64 bytes from <IP_ADDRESS>: icmp_seq=4 ttl=64 time=0.477 ms
^C
--- <IP_ADDRESS> ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3006ms
rtt min/avg/max/mdev = 0.477/0.838/1.712/0.509 ms
root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
^C
--- <IP_ADDRESS> ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4012ms

root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
^C
--- <IP_ADDRESS> ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7001ms

root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
^C
--- <IP_ADDRESS> ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1008ms

root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=1.87 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.526 ms
^C
--- <IP_ADDRESS> ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.526/1.199/1.872/0.673 ms
root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=0.883 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.573 ms
^C
--- <IP_ADDRESS> ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 0.573/0.728/0.883/0.155 ms
root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=1.81 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.461 ms
64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=64 time=0.599 ms
64 bytes from <IP_ADDRESS>: icmp_seq=4 ttl=64 time=0.671 ms
^C
--- <IP_ADDRESS> ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 0.461/0.886/1.816/0.543 ms
root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=1.23 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.620 ms
64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=64 time=0.512 ms
^C
--- <IP_ADDRESS> ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.512/0.788/1.233/0.318 ms
root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=0.665 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.509 ms
^C
--- <IP_ADDRESS> ping statistics ---
3 packets transmitted, 2 received, 33% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.509/0.587/0.665/0.078 ms
root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
^C
--- <IP_ADDRESS> ping statistics ---
9 packets transmitted, 0 received, 100% packet loss, time 8041ms

root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
^C
--- <IP_ADDRESS> ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9012ms

root@7d5e9238ef92:/# ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
^C
--- <IP_ADDRESS> ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2008ms

root@7d5e9238ef92:/#

Going to merge this issue because it fixes the the original issue. Will open a new issue for the PING issue.


Issue installing on CentOS

2 syntax errors in install.js

You probably have a very outdated version of Node far past the official current LTS release, which is 8.x.x.
Most (somewhat outdated) distributions won't come with the current version of Node in their repositories, to give an idea, Ubuntu 16.04 currently comes with Nodejs 4.2.6 and support for the ES let and const bindings were added around the start of version 6 of Node.
Same concept probably applies to CentOS.
If you want to get a more up to date version of Node, I suggest you check out https://github.com/nodesource/distributions#rpminstall for an easy installation process.
To give an idea, you might want to do this :
# Remove currently installed node (package can also be called nodejs in case)
sudo rpm -e node
# Get current LTS
curl -sL https://rpm.nodesource.com/setup_8.x | bash -

And what you were trying to do should work.

Hi Kevin,
Thanks for the response. I installed node 10 last week, as node 8 was not available. I then got this error:
Error:
/usr/lib/node_modules/terminalizer/node_modules/electron/dist/electron: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory
Do you have a workaround for this? The only package on CentOS 6.9 with libgtk-3.so in it is Firefox.

The closest pointer I could give you is this http://itvision.altervista.org/compiling-and-installing-gtk3-in-centos6.html
Another option would be to upgrade to Centos7 if viable, it comes with gtk3 support (and systemd).
Also may I ask you what desktop environment you are currently running currently ?

I heard about terminalizer through a newsletter email that Twilio sent out, and thought it was pretty awesome. I'm realizing that it requires or assumes that I have a desktop environment. This is a dev server i'm installing terminalizer on. I wanted to use the gifs as short tutorials for my dev team. I have image libraries on our web servers, so I assumed this would render a gif, and I could sent it off to my team via link or even the slack API. Maybe not though. :)

Best alternative I can give without a DE is asciinema, Centos probably has it available in its repositories.
What you can do is record your cast, upload it to the website, then take the link it gives you and append .gif to it.

The command looks like this (taken from the page above):
docker run --rm -v $PWD:/data asciinema/asciicast2gif https://asciinema.org/a/170671.json demo.gif
It's a nice little trick that works without a DE.

Thanks Kevin, I'll check that out. Appreciate the help!

Yes it doesn't support os without GUI for now. Terminalizer is still under development so there are a lot of enhancements we are doing.
Should I close the issue?

Yes, you can close. Thanks Mohammad. I'll keep an eye out for updates.

@faressoft you might want to add a requirements section to the README to keep people aware.

We are waiting Electron to release the feature that allows us to use it without GUI
https://github.com/electron/electron/issues/228


changed regex to get filename

this is so it works on windows, mac and linux.
The existing regex expression only worked on filenames on mac/linux - eg /home/anthony/git/Source/Repos/comms-api/apiproxy/resources/node/api/swagger/swagger.yaml
This should allow it work on windows as well - eg c:\git\Source\Repos\comms-api\apiproxy\resources\node\api\swagger\swagger.yaml
I have tested this on windows and it works, but dont have a mac or linux machine to test it on so cant confirm it works on there. I have however tested the regex expression and it works with both types of filenames

great!! thanks for correction


Fieldset and question can convert legend to page title

What is the context of this PR?
Added option to make legend a H1 and a question title.
Usage:

When using the question macro include the parameter "legendIsPageTitle": true inside the onsQuestion configuration. Also include "dontWrap": true in the configuration of the input you are using.
When just using an input component without the question component only include the parameter "legendIsPageTitle": true in the configuration of the input you are using.

https://trello.com/c/3VplFHOj
How to review

Ensure all fieldset calling components work as expected.
Check accessibility

 Labels and Legends as headings


[request] returning 500 error should contain an "error" field

the spec says this: https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#service-broker-errors
It would be nice to be able to return/define that error or have the lib default to "InternalServerError" or something like this...

I could see closing this as not important.

No, I think it's valid, just not the biggest priority probably right now. Lets leave it open! :)


Use latest available Apache Struts2 GA 2.5.20 release

Use latest Struts2 version 2.5.20 in dependencies
Use latest support plugin 1.1.0 with additional useful interceptors and predefined stack in samples and archetypes

@sapessi I fixed the dependency conflict and it builds now. I upgraded the fileupload to 1.4 project wide. Hope this is fine for you.

Merged changes in the core branch where I'm working on some other issues. This will go out as a patch release with the bug fixes.


[bug]: ERROR: Could not find a version that satisfies the requirement mediapipe (from invokeai) (from versions: none) ERROR: No matching distribution found for mediapipe

Is there an existing issue for this?

[X] I have searched the existing issues

OS
Linux
GPU
cpu
VRAM
No response
What version did you experience this issue on?
3.0 and 3.0.1
What happened?
I tried installing the program got to the part where it is initializing the invokeai application then got a message saying
ERROR: Could not find a version that satisfies the requirement mediapipe (from invokeai) (from versions: none)
ERROR: No matching distribution found for mediapipe. After that the log goes on and gives me other errors in lines 50, 179, 235, 311, 145, 113, 252, 215, 299, 17, 27 (in that order) I expected installation to complete and to be able to use the program

Screenshots

Additional context
So far it happens EVERY single time I've tried to install it though 2.3.5.post2 worked fine
Contact Details
<EMAIL_ADDRESS>

Can you provide the following information:

Which Linux distribution (e.g. Ubuntu 22.04)
Output of uname -a
Output of python -V

@SamohtSirrom1 is there a reason you are installing invoke as a root user?

The first time I ever got it working it was under this same conditions.


DNS resolution does not work for  "A Record" DNS records.

**Rancher versions: Build from master
Steps to reproduce the problem:
Create a DNS record choosing "One or more external IP addresses" option.
DNS entry name does not resolve from within pods in the same namespace.
Just name or using fully qualified path does not resolve.
root@test1-5fbdf9fd46-t4797:/# dig  test2.default.svc.cluster.local +short
root@test1-5fbdf9fd46-t4797:/# ping test2
ping: unknown host test2
root@test1-5fbdf9fd46-t4797:/#

Tested with latest build from master.
Able to resolve  DNS entry  created with "One or more external IP addresses" option successfully.


feat(widget-meet): Call controls

Added call controls, and some additional components to support the controls.

Tests are passing, but CircleCI is having some github issues


fixit: new sample for response streaming

Description
New sample for streaming responses in v2 functions in Java.

Thank you for the sample, please incorporate the suggestions before merging.

Done, thank you for reviewing!


Add spiritual creatures to task list

Adds spiritual creatures to task list and defines available targets to highlight, should also fix NPE when building target highlight list
fixes #2295

Use the icon from the "Spiritual Fervor" extension entry? http://oldschoolrunescape.wikia.com/wiki/Slayer_reward_point#Extend

for now the item ID is defined by its absolute value since it doesn't have a name associated to it, this doesn't feel like the best practice but it works for now.

Add the ID to ItemID.java in a separate commit? Would be handy to have defined for future reference. :+1:
	public static final int SPIRITUAL_MAGE = 11844;

Do not add items to ItemID.java, as it is auto generated.

We should regenerate ItemID then before doing this


DNS probe error

Describe the bug

I am using last NPM version on Raspberry Pi official OS
I keep getting on every redirection (even on NPM itself)
DNS_PROBE_POSSIBLE
Containers are able to communicate. Everything has access to internet and all nslookup are working.
I believe that there is a bug since I don't see any possible solution (or problem :))
Regards

Does this only happen for npm proxied pages? And does 'on every redirect' mean that the proxied pages are not available at all or only when a http redirect is sent? Or something else entirely?
When searching the web for this error it comes up with lots of results where the issue seems to be the router, especially if some form of wrt was installed. Have you checked whether that could be the issue?

Whatever I'm redirectig to, I'm getting DNS Probe error (even redirecting on google.com). I can still access to container with http:://localhost:port_of_container.
It's an ISP Router (Bouygues). I have found out that it's impossible to modify the DNS on the router but I have used dnsmasq to try to solve this issue without success.

Maybe the solution may come from DNSmask. I have tried different configuration and the result is now a bit different (it's going to the error without waiting), but I'm still not mastering this tool.

After weeks on it, It's still not working. I will try with other OS...

Keep getting same error on all OS....
I till have ;
slookup portainer
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>#53
Non-authoritative answer:
Name:   portainer
Address: <IP_ADDRESS>
So I have no idea what is going wrong....

did you ever resolve this? im going through the same thing, its like the router grinds to a stop and no devices can get internet access at all... i changed the ip on my server because i could figure out what the issue was but now it seems as though it is NginX setup... been on this for a couple days and im fed up!

Hello,
I have abandoned this setup, sorry :)
Regards
Le mar. 20 juin 2023, 20:34, aj1322001 @.***> a écrit :

did you ever resolve this? im going through the same thing, its like the
router grinds to a stop and no devices can get internet access at all... i
changed the ip on my server because i could figure out what the issue was
but now it seems as though it is NginX setup... been on this for a couple
days and im fed up!
—
Reply to this email directly, view it on GitHub
https://github.com/NginxProxyManager/nginx-proxy-manager/issues/1534#issuecomment-1599305886,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARHH2XXBN6OGIQEBDCDA5DDXMHUM5ANCNFSM5G5F675Q
.
You are receiving this because you authored the thread.Message ID:
@.***>


DisplayConfig refactor

Much of the display config parsing in SteamVR-OSVR has since moved into OSVR-Core. Specifically, in ClientKit's display.h (C++) or displayC.h (C). Since the display config parsing code is out of date and causing errors, it's probably a good time to replace it with code that gets the information from ClientKit.
So, osvr_display_configuration.h and projection_matrix.h can be removed I think. In OSVRTrackedDevice, we need to re-implement GetWindowBounds, GetRecommendedRenderTargetSize, GetEyeOutputViewport, GetProjectionRaw, GetHeadFromEyePose, ComputeDistortion, GetIPD.
Also, the display config code in OSVR-Core internally subscribes to /me/head and provides pose states for you, so we can also remove for example the handler in OSVRTrackedDevice::HmdTrackerCallback. The API lets you specify matrix row order and coordinate system handedness, so there shouldn't be a need to use Eigen to do the conversions. Once that is removed, look into whether there is any code that can be removed from matrix_cast.h. I suspect only the final matrix/pose conversion functions will still be needed.
Note: for this iteration, it's ok to only support OSVR display configurations that will match what OpenVR expects. That would be a single display input, a single viewer, with two eyes (one surface each) mapped to the same display input. I don't believe OpenVR supports multi-input HMDs yet, so we can probably defer support for them in the plugin at this time.
A question was brought up about what to do for ClientDriver_OSVR::AttachToWindow. Previously I thought this was asking the driver to actually implement the attachment to the window, but this is what the documentation says:
"[Windows only] Notifies the driver that the VR output will appear in a particular window."
I'm not exactly sure what they are expecting the HMD driver to do in response to this event, but it looks like it's called after the window has been attached elsewhere (perhaps by the game engine), so I don't think we need to do anything for this callback.

Based on a quick peek at Valve's Oculus driver, that function sure isn't Windows only. I'm not sure exactly what they're responding with, but it is something.

Also, with regard to the figuring out the display - that's not in OSVR Core yet due to a variety of reasons as I listed in my email. If SteamVR needs it, for now, either pick the first monitors with the right resolution, or just ask for it to be configured - there does appear to be a config file provided to steamvr drivers.

@rpavlik I'm aware; I'm just wondering if there's any planned solution for this that doesn't involve a full render manager.

I'd like there to be one - would be the "fabled 'display interface'" - you'd basically have plugins corresponding to displays as well as input devices, and those plugins for displays could find them (with some wrapper functions from core to make it cross-platform if possible)

So looking over this:

GetRecommendedRenderTargetSize
SteamVR assumes that an HMD is always a single window or screen. OSVR does not. How do we compute this?
GetEyeOutputViewport
I implemented this using getRelativeViewport().
GetProjectionRaw
I implemented this using getProjectionClippingPlanes().
GetHeadFromEyePose
How best to implement this?
GetIPD
I don't think we have API for this at all.

Any suggestions?

For GetRecommendedRenderTargetSize, is that meant to be the actual viewport size or are they asking for extra pixels for distortion/time-warp? Either way, you can just use the display input API to get this (if they're asking for extra pixels, just scale the display input resolution by some small hard-coded factor). For now, just add support for a single display input. We'll probably need to talk to the Epic folks about supporting HMDs with multiple display inputs, but it's not a priority.
As for IPD, hard code this for now. There should be a hard-coded value in OSVR-Core - please use the same value.
As for GetHeadFromEyePose, hmm. I can't remember offhand but I think you can use osvrClientGetViewerPose to get the head pose in world space, and osvrClientGetViewerEyePose to get the eye poses in world space. If so, then you should be able to use those to get the eye poses in head-space.

Based on https://github.com/ValveSoftware/openvr/wiki/IVRSystem::GetRecommendedRenderTargetSize, it does seem like they want some extra space. I'm uncertain how best to calculate this, as they want it "matched with the projection matrix and distortion function".

When I tried an overfill of 1.5X on the Oculus parameters and a 10x10 distortion mesh, I got quite bad texture clamping effects.  This disappeared with an overfill of 2.0.  This does not account for large time warp needs (which would be less than this).  Given an estimate of maximum latency and motion speed and the distortion correction, we could compute an overfill ratio.  For now, I just made it a parameter -- people may want to trade this off because it could impact render speed.

@JeroMiya returning double the normal dimensions via GetRecommendedRenderTargetSize causes extremel performance degradation on my computer; we need to figure out a better way of computing the actual ideal render target size.

Initial implementation is in https://github.com/OSVR/SteamVR-OSVR/pull/14.
Is there anything that currently uses distortion with OSVR? It's going to be difficult to test as-is...

Overfill scale factor is now part of the rendermanager configuration, which is going to be in core (there's a PR for it, but not merged). But that's an internal API (in osvrClient) without a public API yet, which will take a little time. So for now, using a 1.0 overfill factor is a reasonable default.

Wouldn't the overfill factor depend on how much 'timewarp' you expect to do? Does 1.0 overfill factor mean that instead of "W x H" we are asking the app to do '2W x 2H' pixels?

Currently, the amount of overfill required depends on two things: (1) the distortion parameters, which is fixed; (2) the amount and direction (and rotation) required by timewarp, which depends on user motion, which is statistical.
We could compute for some 90th-percentile estimate of user head motion how much margin to leave and set it to that.
The reason it is a parameter is because I anticipated the performance hit required to have a high confidence that we will have enough coverage would be larger than some users would want to accept.  This makes it sort of like a "high quality, very high, insanely high" knob, which trades off frame rate (number of pixels goes up quadratically with overfill) and artifacts (chance of getting a clamped texture coordinate at the edge of the render texture).  When the clamping happens, we get a constant-color artifact near the edge of the screen.
Unfortunately, the factor of 1.5 I chose was not enough to handle the distortion mesh from the Oculus example config, which is why I changed it to 2.  No impact on my cube-world rendering on my high-end card, but that's not a good test; the above report of slowdown makes me lean towards leaving it a parameter with a default that covers most cases.
A factor of 1.0 leaves the display size the same.  A factor of 2 doubles it in each dimension, quadrupling the number of pixels rendered.

nVidia's response to this has been their non-uniform display buffer in VR mode.  I've not looked into that.

@russell-taylor unfortunately VR APIs that are under NDA may not be all that helpful here, at least for me. Some platform abstraction layer may be necessary. I'm not sure what solutions the other graphics card vendors are planning to implement.

I agree, which is what RenderManager is for.  It hides the DirectMode API in a DLL that lets you use it without having to link to it.  I don't know of plans to implement the non-uniform rendering within RenderManager (we're still pounding on latency reduction).  For now, the adjustable parameter is what we have.
We may be able to handle smaller overfills by increasing the resolution of the distortion mesh (another parameter, but it may not be exposed yet).  This would get vertices closer to the edge whose coordinates are not clipped.  Haven't though much about this.

We're open to wrapping other vendors' APIs inside RenderManager as well.

and the DK2 (and a bunch of other HMDs, look in the Displays dir) use the distortion stuff

In fact, that is the point of OSVR - to create a vendor-independent abstraction layer. So we will wrap AMD, Intel and others in a similar way

Yep, I'm aware. I just dislike portions of the stack being locked up from community contribution due to vendor SDK restrictions — hopefully the portions of the stack locked up like this will be relatively minor.

Amen.

Is there any more work to be done on refactoring the display config in SteamVR-OSVR at this time? Or is it all pending further work in OSVR-Core and RenderManager?


Remove non-libuv code from Node extension

Using the libuv code was already the default, and all of the 1.2.x artifacts use the libuv code. I have seen no bugs against 1.2.x that were specifically caused by the libuv implementation, so I think it is safe to remove the fallback to the other code.
This mainly just removes a few files that will be unused, but it also moves completion_queue_uv.cc to completion_queue.cc and merges the contents of server_uv.cc into server.cc.

Looks like one more ifdef GRPC_UV in the renamed completion_queue.cc file.
otherwise LGTM

test this please

The failures of Asan C and C++, the Bazel Basic build, and the performance tests were all caused by #10850, and actually passed. The Mac failure is #10582, #10368, #9727, and similar. The Windows failure is #9088. The Linux and Windows portability failures are being fixed in #10848. And the interop failure appears to be an infrastructure failure.
Jenkins: test this please. I'd like to see actual results for the interop tests.


Please don't abbreviate to "holli"

Auto-translated czech-language version is below.
Níže je uveden automatický překlad do češtiny.
The "Holešovice" network is named after Nádraží Holešovice in Prague in Czech Republic.
I can understand why the Goerli testnet, named after "Goerlitzer Bahnhof" in Berlin, is abbreviated to "goerli"... everyone in Berlin calls it "goerli", same same like everyone called Kottbusser Tor "kotti".
But nobody (I repeat, nobody) in Prague calls Holešovice "holli". Abbreviating to this would be using a Berlin-centric pattern for abbreviation, and is not appropriate in the context of Prague. This is not to mention the use of 2 "L" letters, which don't appear in the elongated version.
Proposed alternatives:

Don't abbreviate: use the full name of the station: -holesovice, it really isn't that complicated to spell,

Only truncate, don't alter: "hole" or "holes" are both better than "holli"

If none of these are acceptable, then I propose to choose a different name for the testnet, because using "holli" as an abbreviation for Holešovice is incongruous.
Translated to Čeština via deepl:
Síť "Holešovice" je pojmenována podle Nádraží Holešovice v Praze v České republice.
Chápu, proč se testnet Goerli, pojmenovaný podle "Goerlitzer Bahnhof" v Berlíně, zkracuje na "goerli"... všichni v Berlíně mu říkají "goerli", stejně jako všichni říkají Kottbusser Tor "kotti".
Ale nikdo (opakuji, nikdo) v Praze neříká Holešovicím "holli". Zkracování na tuto adresu by bylo používáním berlínského vzoru pro zkracování a v kontextu Prahy není vhodné. A to nemluvím o použití 2 písmen "L", která se v prodloužené verzi nevyskytují.
Navrhované alternativy:

Nezkracovat: používat plný název stanice: Holešovice, opravdu není tak složité ho napsat,

pouze zkracovat, neměnit: "díra" nebo "díry" jsou obě lepší než "holli".

Přeloženo do češtiny přes deepl:

I wholeheartedly agree - using holli to represent holesovice is unfathomable.
Make Holesovice great again and keep the original spelling!

I totally agree. I'm native Czech living in Prague almost all my life and I've never heard "Holli".
Holešovice is not just a train stop, but an entire Prague district (before city) with a rich history dating back to the 12th century. So calling it Holli is nonsense.
Greetings from Prague! 🇨🇿  Díky moc, kamarádi

Agree, I'm also native Czech living in Prague and Holli is pure nonsense.
If we still would like to abbreviate, I will choose --holesky with pronunciation of Holešky (which is imho acceptable and used abbreviation of Holešovice) because it will be funny to see people pronunciating as a testnet that is a hole in the sky.

It can also sound like "whole sky", which could be either good, great or terrible.

Thank you for raising this. Fortunately, the name is not set in stone yet.
Unless anyone vetos, I would love to go with Holešky. 🖤

I also support this act of local patriotism!! 💪
Also for more context that part of Prague 7 is sometimes referred to as "mokřad", meaning a wetland (https://en.m.wikipedia.org/wiki/Wetland)

Question: why does it need to be abbreviated, why not just let it be called Holesovice, and write --holesovice in full?
Olympic, Morden, Rinkeby, Ropsten, Kovan were always defined via full names. Testnet names only started being abbreviated with Goerli (and arguably the short-lived Kotti)... but that precedent doesn't need to be followed...
For example, ./geth --holesovice can work...

Why are test nets and hardforks named after cities? Why cant we name it after something else like people perhaps?

Question: why does it need to be abbreviated, why not just let it be called Holesovice, and write --holesovice in full?
Olympic, Morden, Rinkeby, Ropsten, Kovan were always defined via full names. Testnet names only started being abbreviated with Goerli (and arguably the short-lived Kotti)... but that precedent doesn't need to be followed...
For example, ./geth --holesovice can work...

To make it more accessible for an international audience.

To make it more accessible for an international audience.

I don't think it's worth adding the complexity to truncate "holesovice" to "holesky" and this arguably makes it less accessible for an international audience, having to learn both words.
Learning to spell "holesovice" feels the same rough order of magnitude of complexity as to learn "holesky", but learning just one term is half as complex as learning two.

(I mean, it's not as though there are any accents to type)

On the other hand, when people will learn "holesky", it will give them additional "flex factor" during their visit of Prague

Please keep it simple, "holesovice" can be very uncommon to some languages and lead to mistakes. Paying homage is one thing, but let's not forget that the goal is to find a simple name that can be easily remembered and reused by developers around the world.

imho i prefer the -holesovice flag, it is as straightforward to spell as it is true to the original name of the Prague district


erlang_service_lint is failing when parsing lager_transform

Describe the bug
Since updating to ELP 1.1.0 (2024-07-16) (VS Code extension version 0.27.0), erlang_service_lint is failing when parsing lager_transform.
To Reproduce**

Clone danielfinke/elp-repro-lager-transform
rebar3 compile

Expected behavior
eqWAlizer reports incompatible_types at elp_repro_lager_transform_app.erl#L16
Actual behavior
erlang_service_lint fails with:
Could not parse, error: erlang service failed with: exception error: undefined function lager_transform:parse_transform/2
  in function  lists:foldl/3 (lists.erl, line 1594)
  in call from erlang_service_lint:run/1 (erlang_service_lint.erl, line 38)
  in call from erlang_service_server:'-process_request_async/4-fun-0-'/4 (erlang_service_server.erl, line 161) elp(L0002)

(highlight shown on line 1 col 1)
Downgrade to ELP extension version 0.26.0 and the expected behaviour is observed.
Context

ELP Version (output of elp version): elp 1.1.0+build-2024-06-07
Editor used: VS Code 1.91.1
ELP Extension Version: 0.27.0

Thanks for the accurate bug report! Having a look!

@danielfinke The regression should be solved in the latest version (VS Code 0.28.0) which I just released.
Please try the latest version and let us know if this is still an issue.

@robertoaloi looks fixed to me 🙂


RequestPermissionsAsync just returns the current permission status (iOS - location permission)

On iOS I've noticed that if I deny the location permission that is prompted by RequestPermissionsAsync that subsequent calls of this method with the same permissions no longer "prompt" the user, but instead just return the current status of the permission.  On android the behavior is what I would expect, regardless of the decision the user made regarding the permission before, RequestPermissionsAsync will always prompt the user to allow or deny.
In my case I'm only using the Location permission, so other permissions may not have this problem.
Am I misunderstanding the API's function, or is this the behavior of iOS, or is this a legit bug?
Bug Information
Version Number of Plugin: <IP_ADDRESS>
Device Tested On: iOS Simulator (MacBook Pro)
Simulator Tested On: iPhone 8 iOS 11.4
Version of VS: 2017 (15.8.6)
Version of Xamarin:
Versions of other things you are using:
Steps to reproduce the Behavior

(on iOS) Request for the location permission
Deny the permission via the prompt UI
Request for the permission again

Expected Behavior
Regardless if the user has denied or allowed the permission, I would expect the RequestPermissionAsync method to prompt the user (since we can get the current status of the permission via CheckPermissionStatusAsync.
Actual Behavior
No user notification is presented and the current permission status (denied) is returned.
Code snippet
I believe [this is the problem] in my particular case.  I didn't investigate other permissions for this issue.(https://github.com/jamesmontemagno/PermissionsPlugin/blob/master/src/Plugin.Permissions/Permissions.ios.cs#L210)
Screenshots

That is iOS behavior. It will only present the system dialog to the user one time. If they denied you the permission, you will need to prompt them to change the permission from the Settings app, all subsequent attempts in-app will simply return the current setting.

^ yup!


Chainlink local

https://documentation-git-ccip-local-chainlinklabs.vercel.app/chainlink-local

Some comments above left open intentionally; these items need discussion and a follow-up PR, but aren't blockers to initially publishing these docs.


Interpolator Memory Improvement

Description
Seeks to improve the memory efficiency of the VectorInterpolator class which is presently causing an explosion in memory usage in high dimensional cases
Changes

Detect when the input data is a single unique value and skip most of the processing if so

Tested and works great...should see a speed boost here as well, but I need to run it side-by-side.  Thanks James.


Initialize Operation parameters via macro attributes

Some Operation parameters might be initialized via macro attributes, I believe this was discussed prior.
This change adds parsing of api_v2_operation macro attributes, it will validate if attribute can be initialized via macro parameter it will add that initialization to the operation init code.
Supported set of attributes:

summary
description
operation_id
consumes
produces

If summary attribute is empty it will try to obtain summary and description from a function doc comments.

I can also quickly add move to fix that bug #216 here unless you do it separately?

Feel free to!

Done. I found that this issue may apply only to async case, as in non async case we cannot move ownership into future


Building the SDK fails on Windows fails trying to execute python3

I'm attempting to build the dart-lang SDK under Windows 10. As far as I can tell I have the environment configured as per the documentation at Building. These builds fail trying to execute python3. (Or, at least, I assume it is the python step that causes the overall build failure.)
My command:
tools\build.py --no-goma --mode release --arch x64 create_sdk
The error:
ERROR at //sdk_args.gni:22:24: Could not execute interpreter.
  default_git_folder = exec_script("$_dart_root/tools/get_dot_git_folder.py",
                       ^----------
I was trying to execute "python3".
See //BUILD.gn:6:1: whence it was imported.
import("sdk_args.gni")
^--------------------
Command failed: C:\Users\Chris\Desktop\dart-sdk\sdk\buildtools\gn.exe gen out\ReleaseX64 --ide=vs --args=target_os="win" host_cpu="x64" target_cpu="x64" dart_target_arch="x64" dart_use_compressed_pointers=false dart_use_crashpad=false dart_snapshot_kind="app-jit" dart_use_fallback_root_certificates=true dart_use_tcmalloc=false is_debug=false is_release=true is_product=false dart_debug=false dart_runtime_mode="develop" exclude_kernel_service=false is_clang=true dart_vm_code_coverage=false is_asan=false is_lsan=false is_msan=false is_tsan=false is_ubsan=false is_qemu=false dart_platform_sdk=false use_goma=false goma_dir="None" verify_sdk_hash=true
ninja -C out\ReleaseX64 create_sdk
ninja: error: loading 'build.ninja': The system cannot find the file specified.

ninja: Entering directory `out\ReleaseX64'
BUILD FAILED

I believe the environment is set up correctly:

All steps are run as admin
Visual Studio is installed
The Windows 10 SDK is installed, and the Debugging Tools for Windows were added to it.
The source was retrieved using fetch dart.
The Chromium depot tools were installed.
I ensured that python.bat (as well as python3.bat) were at the front of my path. I double-checked this by adding echo lines to the start of the bat files and testing python and python3 on the command line. The echo lines appeared, so the path seems correct.

Any suggestions as to what might be wrong here? Thanks!

Are you running this in a cmd.exe shell or PowerShell? cmd.exe is the one that should work (preferably as administrator, though it should not be necessary after fetch dart).
Some things to try:
where python3
where ninja
where gclient
gclient sync
python3 tools/build.py --no-goma --mode release --arch x64 create_sdk


train_and_evaluate does not preserve the Dataset iterator state across train/eval

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): N/A
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): v1.8.0-0-g93bc2e2072 1.8.0
Python version: 3.6
Bazel version (if compiling from source): N/A
GCC/Compiler version (if compiling from source): N/A
CUDA/cuDNN version: N/A
GPU model and memory: N/A
Exact command to reproduce: see below

Describe the problem
When the input function is based on the one-shot dataset iterator, the training phase always starts from the beginning of the iterator. That is, the iterator state gets reset in between the train/eval phases. Therefore, if the dataset is big enough, then the training would only see a subset of the data, which can be processed in eval_spec.throttle_secs.
I think the issue is caused by the fact that the graph is persisted before transitioning to the next phase, and restored upon reentering training. However, I find the behaviour a bit counterintuitive, so if it is not a bug, it should be mentioned in the train_and_evaluate docs.
Source code / logs
Here is a small example demonstrating the issue:
import tensorflow as tf

def input_fn(data):
    dataset = tf.data.Dataset.from_tensor_slices(data)
    dataset = dataset.batch(batch_size=1)
    x = dataset.make_one_shot_iterator().get_next()
    return {"x": tf.Print(x, [x])}, x

if __name__ == "__main__":
    model = tf.estimator.LinearRegressor(feature_columns=[
        tf.feature_column.numeric_column("x")
    ])

    train_spec = tf.estimator.TrainSpec(
        input_fn=lambda: input_fn(list(range(2**20))),
        max_steps=2)
    eval_spec = tf.estimator.EvalSpec(
        input_fn=lambda: input_fn([42]),
        steps=1,
        start_delay_secs=1,
        throttle_secs=1)

    tf.logging.set_verbosity("INFO")
    tf.train.create_global_step()
    tf.estimator.train_and_evaluate(model, train_spec, eval_spec)

The code produces the following log output (I've omitted irrelevant lines):
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after 1 secs (eval_spec.throttle_secs) or training is finished.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
[0]
INFO:tensorflow:Saving checkpoints for 1 into /var/folders/wr/s7brqkzj74v4pmdwkwn19321l34xg2/T/tmpgtq1ap9_/model.ckpt.
INFO:tensorflow:loss = 0.0, step = 1
INFO:tensorflow:Loss for final step: 0.0.
...
INFO:tensorflow:Starting evaluation at 2018-05-03-16:21:51
...
INFO:tensorflow:Finished evaluation at 2018-05-03-16:21:51
...
INFO:tensorflow:Restoring parameters from /var/folders/wr/s7brqkzj74v4pmdwkwn19321l34xg2/T/tmpgtq1ap9_/model.ckpt-1
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
[0]
INFO:tensorflow:Saving checkpoints for 2 into /var/folders/wr/s7brqkzj74v4pmdwkwn19321l34xg2/T/tmpgtq1ap9_/model.ckpt.
INFO:tensorflow:loss = 0.0, step = 2
INFO:tensorflow:Loss for final step: 0.0.
...

Just today I came across this problem, and I can't think of a workaround. Also, I think this is also related to #13895.

@JuanjoAlegria the workaround is to run the TF cluster on a single machine with just two tasks: chief doing the training, and evaluator doing solely the evaluation. Each task should be started in a separate process, e.g. via the ProcessPoolExecutor.

Is there any news about this issue? I agree with @superbobry that, if this is not a bug, at least is an issue that should be highlighted in the docs.
Thanks!

This caused also problems to the cache. Check https://github.com/tensorflow/tensorflow/issues/18266

Nagging Assignee @ispirmustafa: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Hi @superbobry
We have updated the behavior of train_and_evaluate. Now train is called only once. So you should not have cache problem in train input-fn.
Having said that eval input-fn will be called for every evaluate.

@ispirmustafa So also  https://github.com/tensorflow/tensorflow/issues/18266 is it solved?

@ipirmustafa is there a plan also for evaluate?

Thanks @ispirmustafa, that is a welcome change. Looks like this documentation should be updated then:
https://github.com/tensorflow/tensorflow/blob/3edb609926f2521c726737fc1efeae1572dc6581/tensorflow/python/estimator/training.py#L280-L284

Thanks for looking into this, @ispirmustafa! Am I right that the training graph is no longer persisted in local mode? If not, how did you solve iterator persistence issue?
Also, for future reference, here's the commit fixing the train part: 3edb609926f2521c726737fc1efeae1572dc6581.

Hi @guillaumekln , Yes that should be updated. we'll do that

Hi @bhack ,
For evaluate we don't have any plan to change graph re-creation.

Hi @superbobry,
I guess you mean whether we keep same graph with multiple train graph by 'persisting'.
No we're not doing that. We call train only once. We do evaluation via a hook (listener)

@ispirmustafa yes, thanks for clarifying this. I think we can close the issue once the documentation clearly reflects the evaluation graph re-creation.
One final question: is there a reason for not caching the evaluation graph in the hook? i.e. do we really need to recreate it?

Keeping evaluation graph needs to hijack existing estimator API.
On Wed, Jun 27, 2018 at 3:02 AM Sergei Lebedev<EMAIL_ADDRESS>wrote:

@ispirmustafa https://github.com/ispirmustafa yes, thanks for
clarifying this. I think we can close the issue once the documentation
clearly reflects the evaluation graph re-creation.
One final question: is there a reason for not caching the evaluation graph
in the hook? i.e. do we really need to recreate it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/19062#issuecomment-400614504,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASZl7ejW6wjWGIaeFKlSN21V-UUL6gGdks5uA1grgaJpZM4TxZax
.

I'm sorry, I think I miss context to understand what needs to be hijacked.

Nagging Assignee @ispirmustafa: It has been 30 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Am I  understanding this right? Suppose my input_fn returns a tf.data.TextLineDataset(filename).map (lamda line: tf.decode_csv(line)) that references a csv with, say, 1,000,000 lines, and it shuffles with dataset.shuffle(buffer_size=256). If the evaluation runs after a few hundred steps of training, when training resumes, will the input_fn used in training start reading thr cvs from where it left off before doing the evaluation, or will it start over from the first line?

@formigone your statement is correct. train input_fn will not be impacted by eval. this is the case for tf.version>=1.9


Referring to other components with specific props

I have a use case where I need to change the background colour of a nested StyledComponent, but some of these are behind a prop. As an example, imagine a grid library exposes the following styled component:
const Row = styled.div`
  background-color: white;

  &:hover {
    background-color: lightgray;
  }

  ${({ selected }) =>
    selected &&
    `
    color: white;
    background-color: black;

    &:hover {
      background-color: gray;
    }
  `}
`;

With this declaration, if I want to change the colour to a green palette I have a problem: From the parent component I can't specify the style when the row is selected.
Would it be posible to provide a way of refering to another component with a specific prop set? Something like:
const Container = styled.div`
  ${Row} {
    background-color: green;

    &:hover {
      background-color: lightgreen;
    }
  }

  ${Row.withProps({ selected: true })}:hover {
    background-color: darkgreen;
  }
`;

One of the patterns that we recommend in general are "reverse-self-selectors" https://styled-components.com/docs/advanced#referring-to-other-components

We could have nested the color-changing rule within our Link component, but then we'd have to consider both sets of rules to understand why Icon behaves as it does.

This already helps in a lot of cases not to run into this problem, since the state of the element that's changing can directly be used when you reverse the relationship here.
Another thing to consider is that the two elements affecting each other shouldn't be widely apart. They could for instance be in the same file, which means passing on props is also an option.
You could also as a last resort use attribute selectors. I've noticed that you have a div that presents itself as selected which isn't quite accessible; so eventually you may add aria-selected to also represent this state in your DOM, which is an attribute you could target using [aria-selected="true"]
As another alternative, if you don't support IE11, then custom properties, aka CSS variables, could be used here.

Closing due to inactivity.
Please don't forget to be a good netizen on GitHub and revisit issues, especially ones that have been resolved and need to be closed, or ones that have received replies ❤


Use non-default account in integration tests

Integration tests use geth in devmode, and rely on the first account for executing. However, using the first account is not recommended, so we should set up a second account, transfer funds to it, and use that one for testing.
Note that part of this process is already in place for the hdwallet suite of tests.

Pushing back to backlog


filter with "greater_than" using dates

Currently, the only way to use dates in a get_filtered() call is by defining a date range.  We also need to use dates with "greater_than", specifically so we can get all BlackoutPeriod objects that occur in the future.
While we're at it, it will probably be trivial to implement this for "less_than" as well.

This needs a test. As much as it pains me to commit without a test and abandon this issue, I must work on UI code as much as possible right now. However, we definitely won't commit a release to the master branch until all issues like this have received satisfactory tests.

Is there any particular reason why this is not applied to greater_than_or_equal and less_than_or_equal as well?

The date created in handle_endpoint also needs to have tzinfo set to properly compare the dates. I'll take care of it.
As a side note: Is there any reason why this isnt done always in pr_time.iso8601_to_datetime?

No reason at all.  Feel free to add them.  I was just adding the ones
I needed for the UI.
On Sat, Jan 28, 2012 at 7:31 PM, John O'Connor
<EMAIL_ADDRESS>wrote:

Is there any particular reason why this is not applied to greater_than_or_equal and less_than_or_equal as well?

Might as well for the sake of completion.

I think that when tzinfo is None, that means UTC.  The final return
value is a product of calling datetime.utcfromtimestamp(), which as
you can imagine, is designed to always return a datetime in UTC.
https://github.com/TireSwingSoftware/openassign-server/blob/0764f7b0197f45e2c86a48777c9c5daab9a4fc35/pr_services/pr_time.py#L34
http://docs.python.org/library/datetime.html#datetime.datetime.utcfromtimestamp
On Sat, Jan 28, 2012 at 7:47 PM, John O'Connor
<EMAIL_ADDRESS>wrote:

The date created in handle_endpoint also needs to have tzinfo set to properly compare the dates. I'll take care of it.
As a side note: Is there any reason why this isnt done always in pr_time.iso8601_to_datetime?

Right but the problem occurs when compared to dates with tzinfo=pr_time.UTC(). If you want to see what I mean checkout issue/77 and run the pr_services.TestGetFiltered with and without commit dc94678681797da5ee9a4001ad8940faec535d94 present.

I created a separate issue #90 for the date consistency issue.
For now the date created in the filter uses the pr_time.UTC class which is sufficient for this issue.
This is now merged into dev.


mainnet: user @eeksskee doesn't get claim button on winning position

User has winning position, owns winning NO shares in 0xd01ef4c1006deff19c20bafbb6cb7657ff160b0f market.
Position says they have NO shares.

The positions back from augur-node look weird.

transaction history for the market looks correct:

user wants to keep public address private. contact me to get it.
...
After talking to Scott, user has already claimed, issues
Etherscan doesn't recognize SHARE token burns so tokens still show up in user wallets.
UI doesn't update correctly sometimes after claiming. Still confused about numSharesAdjustedForUserIntention and how to display them, see above user's position based on data coming back from augur-node.
This might it handled as a side-effect of working PL stuff. Is this true @pgebheim ?
-bthaile
...
This will be fixed with Alex's changes since we will be basing all these token balances off of the actual balance tables which are affected by burns/transfers/etc.
This looks to be an artifact of the bug where the positions aren't tracked properly when you receive a share by creating a sell order for another share which you don't own and escrowing CASH.
-pgebheim
...
@achapman -- Should work with the new code yah?
-pgebheim
...
We could probably use the new tables for this data yea. In general I think we should get rid of the concept of numSharesAdjustedForUserIntention entirely. For now in the current UI for Binary/Scalar we could just check for No/Short shares and display them as negative and in categorical just show what people have.
The net qty for binary/scalar is redundant anyway and for categoricals it breaks (logically) as soon as someone makes a trade on an outcome they received shares of from shorting.
-nuevoalex
...
I agree on that -- and was thinking of the same solution.
From getPL we can return the NO shares and the UI can look them up for the negative share values, or I can add that as a case to the getter for binary markets.
-pgebheim

Blocked by https://github.com/AugurProject/augur/issues/263

@petong can you tell me the reproduction steps ?

User buys 1 no share in Yes/No market. The market ends and the No outcome is reported on then market finalizes.
User doesnt see claim footer on market card. So user can’t claim their winnings.
We have made small changes to hopefully address a few underlining issues. This issue is block waiting for another change.

@bthaile can you tell the impact ?

Depends on if user was ever able to claim winnings. I’ve not heard of this issue being common. Because if this, I’d say medium.


Use of salmon counts for DESeq2

Hi @drpatelh
Question/Suggestion on this - https://github.com/nf-core/rnaseq/blob/master/bin/salmon_tximport.r
I believe that if one has to use Salmon quant.sf counts for downstream DESeq2, the counts have to be "non-normalized".
See below (different links and my thread with Mike Love on tximport with salmon counts for DESeq2)

https://hbctraining.github.io/DGE_workshop_salmon/lessons/01_DGE_setup_and_overview.html

The pseudocounts generated by Salmon are represented as normalized TPM (transcripts per million) counts and map to transcripts. These need to be converted into non-normalized count estimates for performing DESeq2 analysis.

Note that although there is a column in our quant.sf files that corresponds to the estimated count value for each transcript, those values are correlated by effective length. What we want to do is use the countsFromAbundance=“lengthScaledTPM” argument.

https://bioconductor.org/packages/devel/bioc/vignettes/tximport/inst/doc/tximport.html#Downstream_DGE_in_Bioconductor

Do not manually pass the original gene-level counts to downstream methods without an offset. The only case where this would make sense is if there is no length bias to the counts, as happens in 3’ tagged RNA-seq data (see section below). The original gene-level counts are in txi$counts when tximport was run with countsFromAbundance="no". This is simply passing the summed estimated transcript counts, and does not correct for potential differential isoform usage (the offset), which is the point of the tximport methods (Soneson, Love, and Robinson 2015) for gene-level analysis.

My discussions with Mike Love

https://github.com/COMBINE-lab/salmon/issues/581#issuecomment-719838691
https://github.com/COMBINE-lab/salmon/issues/581#issuecomment-719871117

Thank you for reporting this @tamuanand :) Yes, I included the Salmon PCA plots as a bonus really because I got carried away with the modularisation of the pipelines. I am guilty of not using Salmon very much. I think @olgabot @lpantano added this functionality to the pipeline. How would you propose we fix this?

we can use a parameter to the pipeline and this passed to the R script? that would be the countsFromAbundance option?

Hi @drpatelh @lpantano
I would propose that the pipeline can give all possible outputs from Salmon+tximport, but only use the countsFromAbundance=lengthScaledTPM part for DESeq

this way, the end user can use other output files for other needs (say find out Differential Transcript usage, etc)

This is how I have implemented within my code - In my case, I have quant.sf within foldernames salmon_Sample1, salmon_Sample2, and so on.  Feel free to use/repurpose as needed.
#!/usr/bin/env Rscript
args = commandArgs(trailingOnly=TRUE)

# test if there is at least one argument: if not, return an error
if (length(args) < 1) {
      stop("This script requires at least 2 arguments\nArg1=path to directory that has sample info which then have quant.sf file in
their subfolders\nArg2=Folder path to a directory that has a 2 column file that ends in tx2gene.tsv - column1 is transcript, column2 is gene\n",
call.=FALSE)
}

get_quants <- function(path1, ...) {
    additionalPath = list(...)

    suppressMessages(library(tximport))
    suppressMessages(library(readr))

    salmon_filepaths=file.path(path=path1,list.files(path1,recursive=TRUE, pattern="quant.sf"))

    samples = data.frame(samples = gsub(".*?quant/salmon_(.*?)/quant.sf", "\\1", salmon_filepaths) )
    row.names(samples)=samples[,1]
    names(salmon_filepaths)=samples$samples

    # no tx2Gene available
    salmon_tx_data_1 = tximport(salmon_filepaths, type="salmon",
                              txOut = TRUE)

    salmon_tx_data_2 = tximport(salmon_filepaths, type="salmon",
                              txOut = TRUE, countsFromAbundance = "lengthScaledTPM")

    salmon_tx_data_3 = tximport(salmon_filepaths, type="salmon",
                              txOut = TRUE, countsFromAbundance = "scaledTPM")

    ## Get transcript count summarization
    write.csv(as.data.frame(salmon_tx_data_1$counts),
              file = "tx_NumReads.csv", quote=FALSE)
    write.csv(as.data.frame(salmon_tx_data_2$counts),
              file = "lengthScaledTPM_tx_NumReads.csv", quote=FALSE)
    write.csv(as.data.frame(salmon_tx_data_3$counts),
              file = "scaledTPM_tx_NumReads.csv", quote=FALSE)

    ## Get TPM
    write.csv(as.data.frame(salmon_tx_data_1$abundance),
              file  =  "tx_TPM_Abundance.csv", quote=FALSE)
    write.csv(as.data.frame(salmon_tx_data_2$abundance),
              file  =  "lengthScaledTPM_tx_TPM_Abundance.csv", quote=FALSE)
    write.csv(as.data.frame(salmon_tx_data_3$abundance),
              file  =  "scaledTPM_tx_TPM_Abundance.csv", quote=FALSE)

    rm(salmon_tx_data_1)
    rm(salmon_tx_data_2)
    rm(salmon_tx_data_3)

   #tx2gene available

   if(length(additionalPath > 0)) {
        tx2geneFile = additionalPath[[1]]
        my_tx2gene=read.csv(tx2geneFile,sep = "\t",stringsAsFactors = F, header=F)
        salmon_tx2gene_data_1 = tximport(salmon_filepaths, type="salmon",
                                       txOut = FALSE, tx2gene=my_tx2gene)

        salmon_tx2gene_data_2 = tximport(salmon_filepaths, type="salmon",
                                       txOut = FALSE, tx2gene=my_tx2gene,
                                       countsFromAbundance = "lengthScaledTPM")

        salmon_tx2gene_data_3 = tximport(salmon_filepaths, type="salmon",
                                       txOut = FALSE, tx2gene=my_tx2gene,
                                       countsFromAbundance = "scaledTPM")

        salmon_tx2gene_data_4 = tximport(salmon_filepaths, type="salmon",
                                       txOut = TRUE, tx2gene=my_tx2gene,
                                       countsFromAbundance = "dtuScaledTPM")

        ## Get Gene count summarization
        write.csv(as.data.frame(salmon_tx2gene_data_1$counts),
                  file = "tx2gene_NumReads.csv", quote=FALSE)

        write.csv(as.data.frame(salmon_tx2gene_data_2$counts),
                  file = "lengthScaledTPM_tx2gene_NumReads.csv", quote=FALSE)

        write.csv(as.data.frame(salmon_tx2gene_data_3$counts),
                  file = "scaledTPM_tx2gene_NumReads.csv", quote=FALSE)

        write.csv(as.data.frame(salmon_tx2gene_data_4$counts),
                  file = "dtuScaledTPM_NumReads.csv", quote=FALSE)

        ## Get TPM
        write.csv(as.data.frame(salmon_tx2gene_data_1$abundance),
                  file  =  "tx2gene_TPM_Abundance.csv", quote=FALSE)
        write.csv(as.data.frame(salmon_tx2gene_data_2$abundance),
                  file  =  "lengthScaledTPM_tx2gene_TPM_Abundance.csv", quote=FALSE)

        write.csv(as.data.frame(salmon_tx2gene_data_3$abundance),
                  file  =  "scaledTPM_tx2gene_TPM_Abundance.csv", quote=FALSE)
        write.csv(as.data.frame(salmon_tx2gene_data_4$abundance),
                  file  =  "dtuScaledTPM_TPM_Abundance.csv", quote=FALSE)

        rm(salmon_tx2gene_data_1)
        rm(salmon_tx2gene_data_2)
        rm(salmon_tx2gene_data_3)
        rm(salmon_tx2gene_data_4)

    }
}

if(length(args) == 1){
    quant_dirpath =   gsub("/$", '', args[1])

    get_quants(quant_dirpath)
}else if(length(args) == 2){

    quant_dirpath =   gsub("/$", '', args[1])

    tx2gene_File =  gsub("/$", '', args[2])

    suppressWarnings(get_quants(quant_dirpath, tx2gene_File))

}else{
    stop("This script requires at least 2 arguments\nArg1=path to directory that has sample info which then have quant.sf file in
      their subfolders\nArg2=Folder path to a directory that has a 2 column file - column1 is transcript, column2 is gene\n",
call.=FALSE)
}

this is fine with me, if this version works, you could just do the PR with this update? not sure if the output section in the pipeline needs to be updated to store all these new files into the final folder. I think is good to have all and explained in the output documentation.

@lpantano @olgabot @drpatelh
I am not really comfortable with me doing the PR part as there are so many intricacies with this workflow and I am afraid I would inadvertently break some key functionality.
I can exchange messages on this thread about my code above --  I can vouch that I have pressure tested all facets on the code I shared.
Code aside - the key thing to be noted in the nf-core/rnaseq documentation is that if salmon is used, the counts have to be non-normalized before DESeq and the above code does that.
Using the code example above, I would then use the file lengthScaledTPM_tx2gene_NumReads.csv with DESeqDataSetFromMatrix after reading in the file, rounding the counts, etc

I am planning on doing a minor release soon so be good if we can fix this. If someone is able to submit a PR to fix the salmon_tximport.r R script that generates the counts then I am happy to do the rest. Does that sound reasonable?

I can try this week, I am planning on Wednesday, if that is ok?

Absolutely 👌

I believe this has now been addressed @tamuanand. Mostly by the work from @lpantano in https://github.com/nf-core/rnaseq/pull/519. I updated the main script to use these counts for DESeq2 here. This will use the length scaled counts generated by Salmon as initially created here. How does this look now? Are we ok to close this issue?

Thanks @lpantano
@drpatelh - I think it looks good and this issue can be closed.
Thanks a lot everyone.

@drpatelh  I closed this - probably you should remove the 'bug' label on this issue?

Thanks @tamuanand 👍🏽 We can leave the label as it is for posterity.

I got pointed here by Rob, thanks everyone here for the effort to include Salmon and tximport in the nf-core.
Just to say, I typically use:

counts + offset for use with DESeq(). This is what you get with basic calls of tximport, DESeqDataSetFromTximport, DESeq without additional arguments.
counts alone with countsFromAbundance="lengthScaledTPM" if I am prevented from passing an offset matrix for some reason. Maybe if you want a single matrix for use with DESeq2, edgeR, limma then this is a good compromise. You can still use DESeqDataSetFromTximport, it will figure out to not build an offset.
scaledTPM or dtuScaledTPM for DTU analysis.

Happy to answer any remaining questions as they come up.

Thanks @mikelove ! I should be releasing the shiny new version of the pipeline tomorrow! I have pointed users to this issue] in the documentation so hopefully, that will help us refine the docs within the pipeline.

@drpatelh Often people just use the default kmer size of 31 from salmon
Here is a post from @rob-p on k-mer selection - https://github.com/COMBINE-lab/salmon/issues/482#issuecomment-583799668
Suggestion: If ok with you, probably the bolded/italicized note below can be added to the nf-core documentation - where this section is listed

NB: The default Salmon parameters and a k-mer size of 31 are used to create the index. As discussed here), a k-mer size off 31 works well with reads that are 75bp or longer. Please also see this

Done 🙂 https://github.com/nf-core/rnaseq/pull/535/commits/ccd21ea1efd43297cf9c6dc70e738a6dd00d8337