Low performance of TextResponse and another related responses

Feature Request

Q
A

New Feature
no

RFC
no

BC Break
no

Summary
Currently TextResponse always allocating new buffer for string output - https://github.com/laminas/laminas-diactoros/blob/2.6.x/src/Response/TextResponse.php#L75
This approach working really slow, my suggestion is to implement pure RAM string stream.
Here is what I mean - https://github.com/makise-co/framework/blob/master/src/Http/FakeStream.php (this approach is ~30% more performant)

@codercms are you always experiencing that low performance or it depends on the text length? Does anything change if /maxmemory:NN gets added to php://temp? The filesystem should only be used when the text is big (default limit is 2Mb).

my suggestion is to implement pure RAM string stream.

Overall agree, php://memory could suffice here, but  we also are not sure if the provided string|StreamInterface will fit in memory, hence why a more conservative php://temp was used here.
I suggest sending a patch, and then we discuss the implications there: it is also acceptable to tell end-users that TextResponse is not intended for large payloads (which is in fact the original design anyway)

@lcobucci there is two cases:

I'm experiencing low performance because TextResponse doing excess allocation of PHP native stream (allocation time + string copying time)
With large text length performance difference is even more

Patched TextReponse:
$size = 10 * 1024 * 1024;
$body = new Stream("php://temp/maxmemory:{$size}", 'wb+');

It is still slower than approach without PHP native stream allocation.
320 RPS without native PHP stream
260 RPS with native PHP stream

@Ocramius I guess this patch will not affect end users, because php native stream is only allocated when string was passed to the constructor - this means that the memory for response is already allocated.

Looking back at this, I think that for text, we can use php://memory, and let consumers with large text/plain payloads implement their own streamed response perhaps?
Unsure if this should be considered a BC break.

Similar for html, which is the more common response object.

@Ocramius I think there is one more approach (in two different ways):

Create a “fake” stream object that is just simulating stream behavior (under the hood its just a string wrapper which implements StreamInterface) - end user will have to create stream object
Create a true plain string response which will create a “fake” stream object under the hood - no additional actions for end user

I didn't really understand the difference between those suggested approaches :thinking:

@Ocramius actually its only one approach which can be implemented in two manners

But basically, if I understand it correctly, avoiding usage of a stream at all, and having a string variable instead (since html/text responses are generally well within the memory limits)

@Ocramius yes it is. The performance problem occurs when the Response object allocates stream for string (I mean PHP stream), but actually there is no need to allocate memory one more time.


Add DonateBtn component with basic styling and Open Collective link

I would change it to be a white button with black text. Green is too primary. Also the white outline is somewhat odd to me.

Something like this for mobile? https://www.pngitem.com/middle/ibmmxoT_donate-icon-pink-hd-png-download/

I was thinking like this.

@Tijani-zainab im testing the donate button on the dev site. When i hover it turns to white which makes it so you cant read the text. If we do a hover color change can we make it so the color white goes to a gray?

Ohh, yes I am. will make the changes


chore: merge dbt-fal and fal packages into dbt-fal

The idea of this PR is to merge the 2 packages into one, and we would start releasing only the dbt-fal package.
New dbt-fal command line tool
We now install the fal and dbt-fal CLIs, they are the same but the fal one gets a warning when started.
❯ fal flow run
The dbt tool `fal` and `dbt-fal` adapter have been merged into a single tool.
Please use the new `dbt-fal` command line tool instead.
Running `pip install dbt-fal` will install the new tool and the adapter alongside.
Then run your command like

    dbt-fal <command>

00:29:04  Running with dbt=1.5.1
00:29:04  Unable to do partial parsing because config vars, config profile, or config target have changed
...

❯ dbt-fal flow run
00:29:10  Running with dbt=1.5.1
00:29:10  Unable to do partial parsing because config vars, config profile, or config target have changed
...

Warning on fal imports
Another thing to consider is that now imports from fal moved to fal.dbt.  Now we get warnings for fal direct imports.
>>> import fal
>>> fal.FalDbt
The dbt tool `fal` and `dbt-fal` adapter have been merged into a single tool.
Please import from the `fal.dbt` module instead.
Running `pip install dbt-fal` will install the new tool and the adapter alongside.
Then import from the `fal.dbt` module like

    from fal.dbt import FalDbt

<class 'fal.dbt.utils.FalDbt'>
>>> fal.FalDbt # no more warnings
<class 'fal.dbt.utils.FalDbt'>
>>> from fal import NodeStatus
The dbt tool `fal` and `dbt-fal` adapter have been merged into a single tool.
Please import from the `fal.dbt` module instead.
Running `pip install dbt-fal` will install the new tool and the adapter alongside.
Then import from the `fal.dbt` module like

    from fal.dbt import NodeStatus

>>> from fal import NodeStatus # no more warnings
>>> from fal.dbt import Context

To later free the fal namespace.

Run tests for:

postgres
bigquery
snowflake

Closes FEA-1227
Closes FEA-1263
Closes FEA-1267

Will a person be able to use fal-serverless (new fal) and dbt-fal? I see that we're putting the fal cli in fal namespace, so just curious if the CLI and serverless functions can work together in the same environment.

Even though fal cli is in the fal namespace, notice that the whole project is in src/fal/dbt/*. That means that to import anything from it you use import fal.dbt.
This is the same behavior we see for dbt adapters. They install modules in the dbt namespace, but inside the dbt.adapters.<adapter-name> module, so it does not conflict with imports from dbt-core .
Let me know if I did not explain myself well and I can clarify!

Some tests are set to @todo, does this mean that this feature doesn't work for those databases for both CLI and adapter?

The tests right now keep being separate, we test the cli and adapter separately.
So a todo in the cli_tests folder applies to cli and a todo in the integration_tests folder applies to adapter.

🚢 thank you @chamini2


SynapseToSQL_ADF IncrementalExport_SQL Merge Activity not working correctly

In the last step of the Merge Activity in the IncrementalExport pipeline, the input for the @TargetTable variable is on the form [Schema].[TableName]

In the Script task of the Merge Activity the first part of the query which sets the @UpdateColumns and @Columns variables doesn't return a result because in the where clause the TABLE_NAME column form INFORMATION_SCHEMA.COLUMSN is being evaluated to [Schema].[TableName].

This can be fixed by changing the where clause to WHERE CONCAT(TABLE_SCHEMA,'.',TABLE_NAME) = @TargetTable

Thanks for raising the issue..Closing the issue with attached pull request


Android's latest picture won't be found

I solved this issue and can now save photos to the public 'Pictures' directory on my device (android, for now), however, now I have a new issue.
The new captured photos appear properly if viewed via Android's native 'Photos' app (in the appropriate 'Pictures' directory),  however, when I call PhotoManager.getAssetPathList() again from within my app, it won't refresh and show the new image. Are results cached somehow? Strangely, taking a new photo will make the last one appear.
Eg: if I save 1.jpg with the camera, it appears in android's Photos app, but not within my app. Then if I save 2.jpg and refresh by calling PhotoManager.getAssetPathList() again, 1.jpg will appear, but not 2.jpg, and so on.

@ROTGP  I fix it .
try use version 0.1.7

@CaiJingLong thank you! I've tested it and it works. I can also confirm that this bug does not occur on iOS.

@CaiJingLong thank you! I've tested it and it works. I can also confirm that this bug does not occur on iOS.

gook luck


[Bug]: Multiverse-Core v4.3.9 has registered a listener for org.bukkit.event.entity.EntityCreatePortalEvent on method but the event is Deprecated.

/mv version -p output
https://paste.gg/p/anonymous/1726304dbde54e4790ec9c94d38d26f6
Server logs
http://ix.io/4Bv3
Server Version
Bukkit Version: 3836-Spigot-3374045-08ec344 (MC: 1.20.1)
Built with BuildTools #160 https://hub.spigotmc.org/jenkins/job/BuildTools/160/

Bug Description
Error output in server log: http://ix.io/4Bv3
Steps to reproduce
Start a server with multiverse core installed.
Read the log
Agreements

[X] I have searched for and ensured there isn't already an open or resolved issue(s) regarding this.
[X] I was able to reproduce my issue on a freshly setup and up-to-date server with the latest version of Multiverse plugins with no other plugins and with no kinds of other server or client mods.

The last 3 sentences of your error log is your answer.
This will be resolved in a future release.

Closing as a duplicate of #2000.


project_roles not available unless logged in

Looks like this should have been fixed by d2b0300ab6c3c3bb1cfbef9e62f27d1eb3fc1d23, but I still get a 401 with "You must be logged in to access this resource."

The index route is working for me, https://panoptes-staging.zooniverse.org/api/project_roles.
The show resource route still requires a login https://panoptes-staging.zooniverse.org/api/project_roles/1.

Are there any differences between what the show and index routes return? We should just expose both without a login if there isn't.

I don't think so.
The original issue asked for project_roles for a project so i exposed the index route with the project_id filter param /project_roles/?project_id=X.


Grunt ignores bootswatch

My gruntfile ignores the dependency in bower and does not install bootswatch-dist. I am able to install other bower components such as jquery, jquery-ui, and etc. Since the only dependency that will not install into my application wwwroot folder is bootswatch, I feel it merits investigation.
Here are the steps to reproduce my problem:
Basic Steps
0. Obtain CTP6

You can use commandline in windows and not CTP6 if you wish. See how
Make sure basic empty application or you can use my open source project here.

How to reproduce problem. Open the commandline of your choice and go to the project root directory.

bower init
npm install grunt -save
npm install grunt-bower-task -save
grunt

What you should see is something like I have outlined here
Thanks.

I don't know anything about grunt-bower-task. If you use it to wire dependencies in html, I suggest you to use grunt-wiredep. It perfectly wires bootswatch-dist.

Thanks for your reply.
I am new to this myself, but I use grunt-bower-task and invoke it with a node package manager command (npm). So something like npm install grunt-bower-task -save, and then I have it reside in the gruntfile.js. What I mean by all this is to bring the dependency into the wwwroot of the application. Only after it is brought into wwwroot do I make a reference to it in the html (which I can't because it doesn't work).
My temporary workaround is to use thomaspark/bootswatch with the above process I gave (steps 1-4), but I feel your approach is superior as it allows specific dependencies to be brought in. Unfortunately, I doubt we can determine the cause.
Since I found a workaround, I will go ahead and close this issue, but if you start looking into grunt-bower-task in the future for your repo. Please let me know here.

If it is all about installing bower_components in different directory you should be using .bowerrc file. Take a look at docs.


Bugfix/resolve domain

Codecov Report

Merging #52 (b912116) into master (833f63a) will decrease coverage by 2.71%.
The diff coverage is 97.43%.

@@            Coverage Diff             @@
##           master      #52      +/-   ##
==========================================
- Coverage   62.41%   59.69%   -2.72%
==========================================
  Files          30       30
  Lines        3344     3302      -42
==========================================
- Hits         2087     1971     -116
- Misses       1257     1331      +74

Impacted Files
Coverage Δ

turbo_tunnel/tunnel.py
61.37% <87.50%> (+0.09%)
:arrow_up:

tests/test_https.py
100.00% <100.00%> (ø)

turbo_tunnel/__init__.py
85.71% <100.00%> (ø)

turbo_tunnel/__main__.py
0.00% <0.00%> (-56.32%)
:arrow_down:

tests/__init__.py
80.00% <0.00%> (-20.00%)
:arrow_down:

turbo_tunnel/ssh.py
51.08% <0.00%> (-6.46%)
:arrow_down:

turbo_tunnel/socks.py
27.39% <0.00%> (-3.60%)
:arrow_down:

turbo_tunnel/utils.py
71.00% <0.00%> (-2.91%)
:arrow_down:

turbo_tunnel/server.py
73.15% <0.00%> (-2.82%)
:arrow_down:

turbo_tunnel/conf.py
76.11% <0.00%> (-1.66%)
:arrow_down:

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 833f63a...b912116. Read the comment docs.


Posts table isn't included in migrations

I get this error:
"SQLSTATE[42S02]: Base table or view not found: 1146 Table 'database.posts' doesn't exist (SQL: select * from posts limit 1)"
And I looked in the migrations and there aren't any for the posts table.

Hi,
$post is just an example given.
please use your own entity for the review. Thanks


Android production build crashes on startup

Hi i'm having a problem with android production builds building fine, but when running them on a device it crashes after the splash screen. I'm having the same problem when running android development builds with the --no-dev flag.
This only happens when I install this package, even if I don't use it, and I'm not able to recreate it with a clean create-expo-app project.
The initial error I'm shown is a Invariant Violation: "main" has not been registered^mainnotregistered error. The fatal error I'm getting when running with adb logcat is the following, which I believe is a red herring:
--------- beginning of crash
02-13 09:09:15.207  6076  6139 E AndroidRuntime: FATAL EXCEPTION: expo-updates-error-recovery
02-13 09:09:15.207  6076  6139 E AndroidRuntime: Process: no.my.app, PID: 6076
02-13 09:09:15.207  6076  6139 E AndroidRuntime: java.lang.RuntimeException: Could not invoke RNSentry.fetchNativeAppStart

Furthermore I also find the following error in adb logcat
2024-02-19 15:18:23.651 12792-12981 ReactNativeJS           app.app.app                   E  ReferenceError: Property 'viewManagersMetadata' doesn't exist, js engine: hermes
2024-02-19 15:18:23.653 12792-12981 ReactNativeJS           app.app.app                   E  Invariant Violation: "main" has not been registered. This can happen if:
                                                                                * Metro (the local dev server) is run from the wrong folder. Check if Metro is running, stop it and restart it in the current project.
                                                                                * A module failed to load due to an error and `AppRegistry.registerComponent` wasn't called., js engine: hermes

The only references to viewManagersMetadata I can find is in the following packages: expo-modules-core, expo-dev-menu, expo-dev-launcher. Seeing as expo-modules-core is the only dependency this package uses I'm guessing the problem is related to it, but once again, because it works fine on a clean create-expo-app project I don't know.
Also found a NullPointerException in adb logcat, but I'm suspecting it's related to the expo-updates red herring.
02-13 09:09:31.291  6222  6289 E ExpoModulesCore: Invalidated JavaCallback was invoked
02-13 09:09:31.291  6222  6289 E ExpoModulesCore: java.lang.NullPointerException: java.lang.NullPointerException
02-13 09:09:31.291  6222  6289 E ExpoModulesCore:       at expo.modules.kotlin.jni.JavaCallback.invoke(Native Method)
02-13 09:09:31.291  6222  6289 E ExpoModulesCore:       at expo.modules.kotlin.jni.JavaCallback.invoke(JavaCallback.kt:26)
02-13 09:09:31.291  6222  6289 E ExpoModulesCore:       at expo.modules.kotlin.jni.PromiseImpl.reject(PromiseImpl.kt:95)
02-13 09:09:31.291  6222  6289 E ExpoModulesCore:       at expo.modules.kotlin.PromiseKt$toBridgePromise$1.reject(Promise.kt:45)
02-13 09:09:31.291  6222  6289 E ExpoModulesCore:       at expo.modules.kotlin.KPromiseWrapper.reject(KPromiseWrapper.kt:17)
02-13 09:09:31.291  6222  6289 E ExpoModulesCore:       at expo.modules.updates.UpdatesModule$definition$1$4$1.onSuccess(UpdatesModule.kt:121)

I'm at wits end, the only idea I have is to upgrade the expo-modules-core dependency in this project, but I logically don't see how that would help when it runs fine on a clean create-expo-app project.
Any ideas? I've been stuck with this for a week.

It's possible that one or more of the other packages you have installed is conflicting with this one, which could be an explanation for why it works on a clean app and not on yours. You could try a resolution for expo-modules-core and see if that addresses it. If not, then if you can include a minimal reproducible example I can investigate it some more.

Finally made some progress, seems like the RNSentry.fetchNativeAppStart wasn't a red herring after all. It seems to be working after removing @sentry/react-native and sentry-expo.
Seeing as sentry-expo has been merged into @sentry/react-native and been deprecated as of SDK 50, that might be the problem. I'll keep troubleshooting - hopefully I'll be able to pinpoint the issue.


Rasa input file

Rasa NLU version:
Operating system windows:
Content of model configuration file:

Issue:  does rasa work on other then json formate

Yes, per the docs it can support JSON and Markdown.


[BUG] Window resize event triggers editing mode

Description
The Windows resize event triggers a function which put the page in edit mode.
Steps to reproduce

As user with editing permission, open any page.
The toolbar must be active.
Ensure that the page is in published mode, ie. the canonical URL of the page must be visible.
Resize the window.

This immediately puts the page into edit mode, ie. the URL changes to something such as:
/de/admin/cms/placeholder/object/53/edit/72715/
Expected behaviour
Resizing the window should not have an effect in edit mode.
Additional information (CMS/Python/Django versions)
Happens in version 4.1.1 with the latest bundle of *.js files.
I was able to drill down a little bit. If I debug the page using the developer tools I can see that the window resize event is registered twice, in toolbar.js and bundle.toolbar.min.js. If I remove the second event handler, this effect does not happen anymore.
Unfortunately, even rebundling the JS files using gulp bundle --debug doesn't show me the handler, where this happens.
Do you want to help fix this issue?

[ ] Yes, I want to help fix this issue and I will join #workgroup-pr-review on Slack to confirm with the community that a PR is welcome.
[x] No, I only want to report the issue.

Hm. I cannot reproduce this. Do I understand correctly, that "Ensure that the page is in published mode, ie. the canonical URL of the page must be visible" means, there's

the canonical URL in the browser's address bar (no admin endpoint)
Versioning offers a yellow "New Draft" button.

I now found out more about this issue.
I logged into Django from the same browser instance with the same user but in different Chrome profiles. Actually Version 126.0.6461.0 (Official Build) canary (arm64), but that doesn't matter.
In one browser profile this effect occurs, in the other it doesn't. Therefore my assumption is that the local storage must be the culprit, and indeed, this is how I'm able to reproduce it:
In the local storage there is a key named cms_cookie and this key has different values:

The browser profile where this effect happens is light green, and where it does not happen is light red. I then edited the value in the green profile and changed "mode" from "structure" to "edit", since this is the value from the red profile. Et voilà, now that effect does not occur anymore.
Now the question is, why does the key in local storage > cms_cookie > mode has different values, once structure and once edit?

Can you pls. compare the end point urls?

With endpoints you mean /de/admin/cms/placeholder/object/53/edit/73244/ which is referenced in edit, edit_off and structure? This URL has nothing to do with the page currently edited/viewed. It actually points onto a completely different page.
In local storage, this endpoint does not even change, when accessing a different page.

So those might be "leftovers"?

Might be "leftovers". However, consider the second browser profile. Here the local storage also contains 3 endpoints, ie. /de/admin/cms/placeholder/object/53/edit/74716/. This also is a completely unrelated page.

Can you compare the cms_cookie values to CMS.config.mode and CMS.config.settings.mode value?

In Published View:
"CMS.config.mode": "live"
"CMS.config.settings.mode": "edit"
Local storage > cms_cookie > "mode": "structure"
In Edit View:
"CMS.config.mode": "draft"
"CMS.config.settings.mode": "edit"
Local storage > cms_cookie > "mode": "structure"

Apparently #7920 caused a regression and was partially reverted. The ramification of this is that this bug unfortunately reappeared.

@jrief Sorry, I cannot reproduce the error. Can you give more specifics?

Ups, might be my bad. I just rebundled (gulp bundle) and now everything works as expected.


Collapse CNAME chain

This allows more than a single CNAME record to be collapsed.
Also addresses introduces go.mod, and a moved package.

Awesome, thanks!


fixed missing extract_path

Reference Issues/PRs
Fixes #1466

What does this implement/fix? Explain your changes.
Added missing extract_path in line 178 in sktime/datasets/_data_io.py to fix the bug

Does your contribution introduce a new dependency? If yes, which one?

What should a reviewer concentrate their feedback on?

Any other comments?

PR checklist

For all contributions

[x] I've added myself to the list of contributors.
[ ] Optionally, I've updated sktime's CODEOWNERS to receive notifications about future changes to these files.
[x] I've added unit tests and made sure they pass locally.

For new estimators

[ ] I've added the estimator to the online documentation.
[ ] I've updated the existing example notebooks or provided a new one to showcase how my estimator works.

Hi, I've tested the branch locally and it passes. I'm not sure how should I add the test for the custom folder. Should I create a custom folder for extract inside test, load a custom dataset there, and check that it exists?

Should I create a custom folder for extract inside test, load a custom dataset there, and check that it exists?

Yes, that would work, though the problem would be to do this that it works on all the operating systems.
Previously, we had a lot of problems with tests that were downloading files somewhere, setting off virus scanners or accidentally cluttering system folders.
So, in this case, I'd exceptionally say that it's perhaps better to avoid testing this, unless we can come up with a robust way to avoid all the unintended consequences...

070     >>> from sktime.classification.feature_based import TSFreshClassifier
071     >>> from sktime.datasets import load_italy_power_demand
072     >>> X_train, y_train = load_italy_power_demand(split="train", return_X_y=True)
073     >>> X_test, y_test = load_italy_power_demand(split="test", return_X_y=True)
074     >>> clf = TSFreshClassifier()
075     >>> clf.fit(X_train, y_train)
UNEXPECTED EXCEPTION: TypeError("acf() got an unexpected keyword argument 'unbiased'")
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/_internal/cpython-3.7.12/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/opt/_internal/cpython-3.7.12/lib/python3.7/site-packages/tsfresh/utilities/distribution.py", line 43, in _function_with_partly_reduce
results = list(itertools.chain.from_iterable(results))
File "/opt/_internal/cpython-3.7.12/lib/python3.7/site-packages/tsfresh/utilities/distribution.py", line 42, in
results = (map_function(chunk, **kwargs) for chunk in chunk_list)
File "/opt/_internal/cpython-3.7.12/lib/python3.7/site-packages/tsfresh/feature_extraction/extraction.py", line 337, in _do_extraction_on_chunk
return list(_f())
File "/opt/_internal/cpython-3.7.12/lib/python3.7/site-packages/tsfresh/feature_extraction/extraction.py", line 317, in _f
result = func(x, param=parameter_list)
File "/opt/_internal/cpython-3.7.12/lib/python3.7/site-packages/tsfresh/feature_extraction/feature_calculators.py", line 404, in agg_autocorrelation
a = acf(x, unbiased=True, fft=n > THRESHOLD_TO_USE_FFT, nlags=max_maxlag)[1:]
TypeError: acf() got an unexpected keyword argument 'unbiased'
This error failed the build (more than once), any idea why it happened?
Everything passed locally

Hm, looks like this is coming from an interface change in statsmodels, as a tsfresh dependency?
Where tsfresh is calling stattools.acf which no longer has an unbiased argument in 0.13.0, but used to have?

I've opened an issue for the bug  #1478, it seems unrelated to your contribution


Add option to specify solc include path

Not 100% sure how to implement this. Solc has an option to do path remappings and it is also possible to specify a fallback path. @GNSPS, would it be sufficient to add an option for setting the fallback path?

This is really hard... Because sometimes you have to do a combination of fallback remapping + allowed paths (for zeppelin-solidity libraries for example). Maybe have an option that is exactly mapped as a string to solc?

Yeah, that's probably the best way.

I added the --solc-args argument (7c0dedd). You can now do this:
$ myth -x solidity_examples/underflow.sol --solc-args='--arg1 --arg2'


Interactive mode crashes when a \ is typed after a ".

When I run wren in interactive mode and start to type out a string literal with a backslash escape sequence, the program immediately crashes. The crash is not very graceful, printing the error without a newline in front, and leaving the terminal text in the colour of the syntax highlighted ".
The error is:
Range end out of bounds.
[repl line 555] in text
[repl line 344] in refreshLine(_)
[repl line 28] in run()
[repl line 929] in (script)
[<repl> line 1] in (script)

My system is GalliumOS, which is based on Xubuntu. I compiled wren per the instructions on the Getting Started page.
EDIT: I get the same problem on Arch Linux, with wren installed from community repo. The line numbers have changed, but this is probably just because they're different versions.

I can confirm this issue on Fedora 28 as well, with latest wren compiled with gcc 8.1.1


fix: update crypto lib

Motivation and Context

Jira Link:
Description

Fix RN example with updated crypto, tss-lib
How has this been tested?

Screenshots (if appropriate):
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[ ] My code follows the code style of this project. (run lint)
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

Thanks for the changes, I'll follow that while creating a new app. Currently due to older react native versions, this app is not working properly in build. We will have to remake it. I'll do that on a separate PR.


Normalise permission bits to 755 or 644

Apparently, the only permission bit git tracks is executable or not. So this flattens the permission bits to 755 for executable files, or 644 for non-executable, matching the default umask on Debian systems.
I still want to think about this:

Does wheel installation preserve any more permission bits? If not,
this is definitely fine.
Do other version control systems store more permissions?
Is there a reason you might want other permissions on files installed
from wheels, e.g. to make a file not world readable? It seems redundant
when packages come from a public index, but maybe for privately
installed packages it makes sense.

Apparently, the only permission bit git tracks is executable or not

What about other versioning system. I though you didn't want to special case git in flit ?

I don't. That's why it's on my list of things to check. ;-)

I just checked Mercurial and Subversion and both don't track more than execution.

Thanks Brett for investigating that. With that and the existence of distutils644, which I heard about recently, I think this is the right thing to do.


Replace Jade Template with Pug

Removed Jade templating and replaced with Pug jsantell/poet#121

Changed documentation references from Jade to Pug
Changed extensions of jade to pug
Updated code to look for pug instead of had
Updated tests to test pug templates
Updates examples to have modern pug syntax
Update test scripts to exit properly

These updates were tested with the examples locally

Thanks for this! I'll push out a release shortly


fix(to_query_str): properly quote query string parameters

Closes #1871

Indeed, it's awkward in Python. It's one of those few cases where functional languages like Haskell would shine like anything :slightly_smiling_face:


Passing in fully configured matrix file into binary for a non-preconfigured chain fails

Between the chain name, chain spec name, chain config name and "registered chain labels", something is getting tangled.
Example of failing config (chihuahua is fully configured in matrix and not a pre-configured chain):
https://github.com/strangelove-ventures/ibctest/blob/dan/matrix-fix/cmd/ibctest/deleteme/example_matrix_custom2.json
Yielded Error:
--- FAIL: TestConformance (0.02s)
    ibctest_test.go:201: View chain and relayer logs at /Users/dan/.ibctest/logs/ibctest.log
    --- FAIL: TestConformance/chain_pairs (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x5405e77]

goroutine 67 [running]:
testing.tRunner.func1.2({0x5600d00, 0x6fe22b0})
	/usr/local/go/src/testing/testing.go:1389 +0x24e
testing.tRunner.func1()
	/usr/local/go/src/testing/testing.go:1392 +0x39f
panic({0x5600d00, 0x6fe22b0})
	/usr/local/go/src/runtime/panic.go:838 +0x207
github.com/strangelove-ventures/ibctest.(*BuiltinChainFactory).Name(0xc0012e1360)
	/Users/dan/go/src/github.com/strangelove-ventures/ibctest/chainfactory.go:137 +0x97

FIX WIP:
https://github.com/strangelove-ventures/ibctest/blob/e7538a9892b3390de885261c69b2d0a657291305/chainspec.go#L71-L102
NOTE:
https://github.com/strangelove-ventures/ibctest/blob/e7538a9892b3390de885261c69b2d0a657291305/ibc/types.go#L95-L98


Integration with IDEs or FlexUnit

Have you given any thought to integration with IDEs like Flash Builder or IntelliJ IDEA? Is it possible, at the moment?
Failing that, do you think it'd be possible to extend Jasmine-Flex to build FlexUnit tests under the hood? That way it we could leverage the work they've already done with FlexUnit.

Wmadden:  Good question.  I haven't looked at doing either although it seems like both should be possible.  For the second option, are you suggesting that Jasmine tests generate wrappers that call the Jasmine blocks?  Or are you suggesting a translation to FlexUnit tests?  I think I'd rather look into IDE integration... so that you can take advantage of the hierarchical nature of the tests.
Have you thought much about this?  What do you think it would take?  What would be most useful?


sorting functionality for API documentation listing page - Publisher

Description:
API documentation listing page in publisher does not have the sorting functionality.
Suggested Labels:
Suggested Assignees:
Affected Product Version:
APIM 3.0.0
**OS, DB, other environment details and versions: N/A
**Steps to reproduce: N/A
**Related Issues: N/A

Fixed with https://github.com/wso2/carbon-apimgt/pull/5794


Question: Please can someone help with this sample code from the tutorial?

Hi,
Really sorry to bother anyone with this but I am struggling. I have no previous experience of dotspatial.
I am working in VB.Net
I downloaded one of the tutorials from the codeplex website and the specific example shows how to draw points, lines and polygons and save them to a shp file.
The point example works fine but I cant get the line or polygon to work.
The section of the tutorial Select Case statement code is below and I used nugget to install the dotspatial packages in my project. (v2.0.0-rc1)
It seems to be where I try to add coordinates that I have problems. I understand that the methods may be different for the v2.0 but I cant seem to find the correct way of doing it.
Dotspatial looks really interesting so any help would be much appreciated.
Thanks in advance.
Case "line"
                If e.Button = MouseButtons.Left Then
                    'left click - fill array of coordinates
                    'coordinate of clicked point
                    Dim coord As Coordinate = Map1.PixelToProj(e.Location)
                    If (linemouseClick) Then
                        'first time left click - create empty line feature
                        If firstClick Then
                            'Create a new List called lineArray.
                            'In List we need not define the size and also
                            'Here this list will store the Coordinates
                            'We are going to store the mouse click coordinates into this array.
                            Dim lineArray As New List(Of Coordinate)
                            'Create an instance for LineString class.
                            'We need to pass collection of list coordinates
                            Dim lineGeometry As New LineString(lineArray)
                            'Add the linegeometry to line feature
                            Dim lineFeature As IFeature = lineF.AddFeature(lineGeometry)
                            'add first coordinate to the line feature
                            lineFeature.Coordinates.Add(coord)
                            'set the line feature attribute
                            lineID = lineID + 1
                            lineFeature.DataRow("LineID") = lineID
                            firstClick = False
                        Else
                            'second or more clicks - add points to the existing feature
                            Dim existingFeature As IFeature = lineF.Features(lineF.Features.Count - 1)
                            existingFeature.Coordinates.Add(coord)
                            'refresh the map if line has 2 or more points
                            If existingFeature.Coordinates.Count >= 2 Then
                                lineF.InitializeVertices()
                                Map1.ResetBuffer()
                            End If
                        End If
                    End If
                Else
                    'right click - reset first mouse click
                    firstClick = True
                    Map1.ResetBuffer()
                End If

            Case "polygon"
                If e.Button = MouseButtons.Left Then
                    'left click - fill array of coordinates
                    Dim coord As Coordinate = Map1.PixelToProj(e.Location)
                    If (polygonmouseClick) Then
                        'first time left click - create empty line feature
                        If firstClick Then
                            'Create a new List called polygonArray.
                            'Here this list will store the Coordinates
                            'We are going to store the mouse click coordinates into this array.
                            Dim polygonArray As New List(Of Coordinate)
                            'Create an instance for LinearRing class.
                            'We pass the polygon List to the constructor of this class
                            Dim polygonGeometry As New LinearRing(polygonArray)
                            'Add the polygonGeometry instance to PolygonFeature
                            Dim polygonFeature As IFeature = polygonF.AddFeature(polygonGeometry)
                            'add first coordinate to the polygon feature
                            polygonFeature.Coordinates.Add(coord)
                            'set the polygon feature attribute
                            polygonID = polygonID + 1
                            polygonFeature.DataRow("PolygonID") = polygonID
                            firstClick = False
                        Else
                            'second or more clicks - add points to the existing feature
                            Dim existingFeature As IFeature = polygonF.Features(polygonF.Features.Count - 1)
                            existingFeature.Coordinates.Add(coord)
                            'refresh the map if line has 2 or more points
                            If existingFeature.Coordinates.Count >= 3 Then
                                'refresh the map
                                polygonF.InitializeVertices()
                                Map1.ResetBuffer()
                            End If
                        End If
                    End If
                Else
                    'right click - reset first mouse click
                    firstClick = True
                    Map1.Cursor = Cursors.Arrow
                End If

It seems to be where I try to add coordinates that I have problems.

Please explain what problems you mean. Knowing what doesn't work helps fixing the problem. ^^

thanks for the response. I have included the errors that visual studio lists below in the error window.

Value of type 'List(Of Coordinate)' cannot be converted to 'Coordinate()'.
'Coordinates' is not a member of 'IFeature'

These errors are raised from the following lines in the sample code from the tutorial marked below with  '########
Case "line"
                If e.Button = MouseButtons.Left Then
                    'left click - fill array of coordinates
                    'coordinate of clicked point
                    Dim coord As Coordinate = Map1.PixelToProj(e.Location)
                    If (linemouseClick) Then
                        'first time left click - create empty line feature
                        If firstClick Then
                            'Create a new List called lineArray.
                            'In List we need not define the size and also
                            'Here this list will store the Coordinates
                            'We are going to store the mouse click coordinates into this array.
                            Dim lineArray As New List(Of Coordinate)
                            'Create an instance for LineString class.
                            'We need to pass collection of list coordinates

                         '########    Dim lineGeometry As New LineString(lineArray)

                            'Add the linegeometry to line feature
                            Dim lineFeature As IFeature = lineF.AddFeature(lineGeometry)
                            'add first coordinate to the line feature

                            '########  lineFeature.Coordinates.Add(coord)

                            'set the line feature attribute
                            lineID = lineID + 1
                            lineFeature.DataRow("LineID") = lineID
                            firstClick = False
                        Else
                            'second or more clicks - add points to the existing feature
                            Dim existingFeature As IFeature = lineF.Features(lineF.Features.Count - 1)

                            '########  existingFeature.Coordinates.Add(coord)

                            'refresh the map if line has 2 or more points

                       '########      If  existingFeature.Coordinates.Count >= 2 Then

                                lineF.InitializeVertices()
                                Map1.ResetBuffer()
                            End If
                        End If
                    End If
                Else
                    'right click - reset first mouse click
                    firstClick = True
                    Map1.ResetBuffer()
                End If

            Case "polygon"
                If e.Button = MouseButtons.Left Then
                    'left click - fill array of coordinates
                    Dim coord As Coordinate = Map1.PixelToProj(e.Location)
                    If (polygonmouseClick) Then
                        'first time left click - create empty line feature
                        If firstClick Then
                            'Create a new List called polygonArray.
                            'Here this list will store the Coordinates
                            'We are going to store the mouse click coordinates into this array.
                            Dim polygonArray As New List(Of Coordinate)
                            'Create an instance for LinearRing class.
                            'We pass the polygon List to the constructor of this class

                            '########  Dim polygonGeometry As New LinearRing(polygonArray)

                            'Add the polygonGeometry instance to PolygonFeature
                            Dim polygonFeature As IFeature = polygonF.AddFeature(polygonGeometry)
                            'add first coordinate to the polygon feature

                            '######## polygonFeature.Coordinates.Add(coord)

                            'set the polygon feature attribute
                            polygonID = polygonID + 1
                            polygonFeature.DataRow("PolygonID") = polygonID
                            firstClick = False
                        Else
                            'second or more clicks - add points to the existing feature
                            Dim existingFeature As IFeature = polygonF.Features(polygonF.Features.Count - 1)

                         '########  existingFeature.Coordinates.Add(coord)

                            'refresh the map if line has 2 or more points

                          '########   If existingFeature.Coordinates.Count >= 3 Then

                                'refresh the map
                                polygonF.InitializeVertices()
                                Map1.ResetBuffer()
                            End If
                        End If
                    End If
                Else
                    'right click - reset first mouse click
                    firstClick = True
                    Map1.Cursor = Cursors.Arrow
                End If

Dim lineArray As New List(Of Coordinate)
Dim lineGeometry As New LineString(lineArray) '- Value of type 'List(Of Coordinate)' cannot be converted to 'Coordinate()'.

This is self-explanatory. You're trying to use a list even though an array is expected.
Try this instead: Dim lineGeometry As New LineString(lineArray.ToArray())
You could replace

existingFeature.Coordinates.Add(coord)   - 'Coordinates' is not a member of 'IFeature'

by  existingFeature.Geometry.Coordinates.Add(coord) to get rid of the given error message. This is caused by the fact that in DS 2.0 it's no longer possible to use Feature for geometry based operations. You have to use Feature.Geometry instead.
But you'll cause the next error because Coordinates is an array and you can't add things to arrays.
So basically you have to change the way you work.
First get all the points you want to use to create your geometry. Then add them to an array and use this array to create your geometry.
Be aware of issue #801.

Thank you so much for your help.
I really appreciate it and I think that has helped me to get around my issue.


Request for Instructions for Building

Hello, I am interested in your project and trying to install it in my own PC.
It will be very helpful if you can document the necessary requirements for installing and running the project.

Sure, I will document it for you
Before that, I will briefly write instructions for you here
Try following these instructions
(Assuming Ubuntu or wsl)

git clone & git submodule init & git submodule update
cd Sapphire
python3 run_tests.py -d

These commands should work if you satisfied requirements

I documented it on the notion page (link present in README)
Please tell me if you have any issues installing and running tests

Very helpful. Thank you


Support streaming download with Fetch

The first limitation listed is: "Playback can only start when the file is fully loaded (when not streaming through WebSockets). I'm waiting for chunked XHR with ArrayBuffers to arrive in browsers."
Now that the Fetch API is available in Chrome, Firefox, and Opera (i.e. over 50% of browsers worldwide) [1], can progressive download become a reality?

http://caniuse.com/fetch

Older versions of Chrome support fetch without ReadableByteStream, which might be a concern.
Progressive download has always been a reality in those browsers anyway (using the charset=x-user-defined) trick.  Partial requests and responses have been another viable alternative for a while too (see #63).  It might be worth combining all these approaches to get the best possible performance in each browser when streaming the video.

It seems ReadableByteStream is not supported by current Firefox versions either - as far as I can see, that's the only way to have progressive loading via fetch.
In any case, I have implemented a solution that at least works in Chrome. It falls back to non-progressive loading in all other browsers.

So can we stream the file now !!??? How can I test it??

Streaming only in Chrome. Maybe I'm missing something, but it seems Firefox does not yet support this in any way.
In Chrome, this should start playing pretty much immediately, instead of download the whole 13mb file beforehand:
http://phoboslab.org/files/jsmpeg/


HW3: add issues badge to README

Just separated this PR (which used to have parts 1 and 2 together) into two separate PRs!


Handling simultaneous SIP requests

Hi,
One issue I see from one of the clients we are testing is that after REGISTER, a SUBSCRIBE and PUBLISH are sent as separate transactions, but at relatively the same time. It appears as though drachtio-server (latest develop branch) seems to lose state. For example, it may respond properly tot he PUBLISH. In our code using drachtio-srf, we send a NOTIFY in response to the SUBSCRIBE (to reg events in this case). That NOTIFY ends up going out without a "To tag", with the wrong (a new?) "From tag", but with the correct Call-ID (of the SUBSCRIBE dialog).
It looks like in this case, SUBSCRIBE is received by drachtio, then a PUBLISH, PUBLISH is responded with 200 OK, then the SUBSCRIBE gets 200 OK, then the NOTIFY goes out with the incorrect information.
Sometimes it works, perhaps some minor timing changes cause different call flow. If we disable PUBLISH, it works fine as well. We are also looking to make the SUBSCRIBE/PUBLISH flow sequential, but, I am not sure if that is limited by standards for non-INVITE requests.
I am working on some logs and a packet capture to send to you.
Thanks

I will try to recreate.  So the sequence is
client         drachtio
———> REGISTER
<—— 20 OK
————> SUBSCRIBE
—————> PUBLISH
<———— 202 (SUBSCRIBE)
<—————200 (PUBLISH)
<—————NOTIFY
Is that right?
On May 24, 2018, at 8:15 PM, James<EMAIL_ADDRESS>wrote:
Hi,
One issue I see from one of the clients we are testing is that after REGISTER, a SUBSCRIBE and PUBLISH are sent as separate transactions, but at relatively the same time. It appears as though drachtio-server (latest develop branch) seems to lose state. For example, it may respond properly tot he PUBLISH. In our code using drachtio-srf, we send a NOTIFY in response to the SUBSCRIBE (to reg events in this case). That NOTIFY ends up going out without a "To tag", with the wrong (a new?) "From tag", but with the correct Call-ID (of the SUBSCRIBE dialog).
It looks like in this case, SUBSCRIBE is received by drachtio, then a PUBLISH, PUBLISH is responded with 200 OK, then the SUBSCRIBE gets 200 OK, then the NOTIFY goes out with the incorrect information.
Sometimes it works, perhaps some minor timing changes cause different call flow. If we disable PUBLISH, it works fine as well. We are also looking to make the SUBSCRIBE/PUBLISH flow sequential, but, I am not sure if that is limited by standards for non-INVITE requests.
I am working on some logs and a packet capture to send to you.
Thanks
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/davehorton/drachtio-server/issues/31, or mute the thread https://github.com/notifications/unsubscribe-auth/ABzDLQx9FhDtjbPOptHBAwYCTiefHBS3ks5t100igaJpZM4UNL2i.

Just a minor modification, PUBLISH is answered first:
client         drachtio
———> REGISTER
<—— 200 OK
————> SUBSCRIBE (event: reg)
—————> PUBLISH
<————— 200 (PUBLISH)
<———— 200 (SUBSCRIBE)
<———— NOTIFY (event: reg)

Yes, agreed that PUBLISH and SUBSCRIBE are separate, at least PUBLISH is not part of dialog started by SUBSCRIBE. Both have different From tag and Call-ID when sent from our client.
What is mostly interesting is that 4 out 5 tests show the problem, but it sometimes works. That 1 time out of 5 the NOTIFY comes with the correct Call-ID and From tag, with the new To tag from. Possibly a timing or concurrency issue?

Here are packet dumps of a good example and bad showing the SIP headers, in case it helps build a test vector.
sub_pub_good.txt
sub_pub_bad.txt

I'm having trouble recreating, and I think I need more insight into what your code looks like.
I tried recreating by using srf.createUAC when receiving the SUBSCRIBE, then on the created dialog calling dlg.request({method: 'NOTIFY'...  However, I cannot recreate the race condition, and also you must be doing something different since srf.createUAC would generate a 202 response to the SUBSCRIBE whereas you are generating a 200 in your trace.
can you comment on how I can make my simple test case look more like yours?  Here is what I am testing (which seems to not hit this race condition)
    // respond to publish and register immediately
    this.srf.register((req, res) => {
      console.log('sending 200 OK to register');
      res.send(200);
    });
    this.srf.publish((req, res) => {
      console.log('sending 200 OK to publish');
      res.send(200);
    });

    // slight delay in responding to subscribe
    this.srf.subscribe((req, res) => {
      console.log('got SUBSCRIBE');
      setTimeout(() => {
        console.log('sending 200 OK to SUBSCRIBE');
        this.srf.createUAS(req, res)
          .then((dlg) => {
            console.log('sending NOTIFY');
            return dlg.request({
              method: 'NOTIFY',
              headers: {
                'Subscription-State': 'active',
                'Event': req.get('Event')
              }
            });
          })
          .catch((err) => console.log(`Error responding to SUBSCRIBE: ${err}`));
      }, 200);

I use the above test app, and then send the following sequence from a sipp uac:
--> REGISTER
<-- 200 OK
--> SUBSCRIBE
--> PUBLISH
<-- 200 OK (PUBLISH)
<-- 202 Accepted (SUBSCRIBE)
<-- NOTIFY

We aren't using createUAS. Also, I think we are more aligned to RFC 6665 in which 202 response for SUBSCRIBE is deprecated, but nonetheless the 200/202 are supposed to be treated the same. But if you are saying the drachtio-* will behave differently if 202 is sent, then we can try that.
Also, just to clarify, the PUBLISH and SUBSCRIBE are separate. PUBLISH from the client is for 'presence' and the SUBSCRIBE is for 'reg' (events).
Here is the code for pub/sub we are using, with some field processing and user management parts removed.
srf.publish((req, res) => {
    var sipEtag = 'some_generated_value';

    res.send(200, {
        headers: {
            'Expires': req.msg.headers.expires,
            'SIP-ETag': sipEtag
        }
    });
});

srf.subscribe((req, res) => {
    // Send a 200 response to the subscribe.
    res.send(200, {
        headers: {
            expires: 3600
        }
    });

    // Send NOTIFY using srf.request()
});

I think, perhaps we should be using createUAS and dlg.request to send the NOTIFY as per your example? We've just been doing it manually.

BTW, it is fine doing things like that, you do not necessarily need to change to use dlg.request.  Totally up to you.  As long as you include the Call-ID in the headers of the request, the server should set things up properly

In your example, one more curly bracket is needed after the headers block since the callback is the next parameter after the options.
This seems to help resolve the issue, but there are some new problems. If I send 5 users through the reg/pub/sub process:

Some users get registered and the sub/pub/notify works, sometimes only one.
Sometimes "Error: invalid message from server, did not start with length#:"
Sometimes "Error: unexpected message with type: undefined"

It is always thrown from here:
drachtio/lib/wire-protocol.js:280
at WireProtocol._onData (drachtio/lib/wire-protocol.js:244:21)
or
at DrachtioAgent._onMsg (drachtio/lib/drachtio-agent.js:675:15)
I will look over my code to find other causes.

Hmm, 'invalid message from server, did not start with length’ is a pretty low level error, it means that somehow the message between server and client was formatted incorrectly or was corrupted somehow.  I’m not totally sure what it points to.
The best way for me to help troubleshoot this is if you can set the drachtio server loglevel to debug and capture a full log from drachtio server for this sequence
On May 29, 2018, at 5:19 PM, James<EMAIL_ADDRESS>wrote:
In your example, one more curly bracket is needed after the headers block since the callback is the next parameter after the options.
This seems to help resolve the issue, but there are some new problems. If I send 5 users through the reg/pub/sub process:
Some users get registered and the sub/pub/notify works, sometimes only one.
Sometimes "Error: invalid message from server, did not start with length#:"
Sometimes "Error: unexpected message with type: undefined"
It is always thrown from here:
drachtio/lib/wire-protocol.js:280
at WireProtocol._onData (drachtio/lib/wire-protocol.js:244:21)
or
at DrachtioAgent._onMsg (drachtio/lib/drachtio-agent.js:675:15)
I will look over my code to find other causes.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/davehorton/drachtio-server/issues/31#issuecomment-392948911, or mute the thread https://github.com/notifications/unsubscribe-auth/ABzDLbo1EuGnUCJ5fvB2BjBB7MZxFI67ks5t3btxgaJpZM4UNL2i.

Log file is uploaded. Here, user 0<PHONE_NUMBER>0) causes the problem immediately:
--> REGISTER
<-- 401/200 OK
--> SUBSCRIBE
--> PUBLISH
PROBLEM
Then, later the other clients get handled after retries or server restarts.
problem.txt
log.txt

could you re-run with DEBUG=drachtio:* on the client side, and include that log?

also in the logs, pidf xml content in the PUBLISH that is coming through looks like it has some unprintable characters in the 'entity' attribute of the 'presence' tag.  What is that?  I am thinking that it is blowing up on the PUBLISH message for some reason, and if the content is not utf8 encoding that may be the reason.

can you confirm also you are running drachtio-srf v 4.2.17 ?  That is the latest, and includes some fixes for proper handling of utf8 strings....I had actually ran into issues having to do with content that included emojis, which made me realize that I was not handling utf8 correctly in prior releases.

Thank you very much for taking a detailed look!
app_log.txt
log.txt

I'm pretty sure it has something to do with those unprintable characters in the pidf payload, so I would be interested to know what they are, and what the encoding is.  If its utf8 encoding, I should be handling it, but maybe I have a bug.
Perhaps we can try to determine whether that is in fact the problem by having you send a PUBLISH without those special characters (simple ascii encoding) to see if the problem does then not occur?

Yes, that entity field looks corrupt, even some Wireshark traces are broken because of it. It should be a sip uri. I am checking client code and will either send proper information or none for testing. This certainly looks like our problem. I will test as soon as we have a fix and let you know. Really sorry if this is causing some of the issues. But I think the advice for sending the NOTIFY in the callback was right on. Thanks.

OK, let me know.  no need to be sorry -- all good.  I should probably burp out some more useful log/diagnostic to help pinpoint these types of issues.

Using a PUBLISH body with valid characters, operation was as expected without issue.
.


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-source-version-target/pull/101 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-archive-test-source-version-target.git

Docs Build status updates of commit db6dcf6:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

versioningArchiveSourceFolder/TOC.yml
:white_check_mark:Succeeded
View (sql-server-2014 || azuresqldb-current || azuresqldb-mi-current || azure-sqldw-latest || >= aps-pdw-2016 || >= sql-server-2016 || >= sql-server-linux-2017 || sql-server-previousversions || sqlallproducts-allversions)

versioningArchiveSourceFolder/docfx.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Adopt guess_max a la readr

Propose read_excel() follow readr and gain a guess_max parameter with similar default behaviour:

guess_max   Maximum number of records to use for guessing column types.
... n_max = Inf, guess_max = min(1000, n_max), ...

Do you approve in principle @hadley?
Would subsume issues #206 #135 #131 , PRs #138 #143

👍

@jennybc is there a way I could update PR #138 in order to implement this, or are you planning on reworking readxl internals considerably, which would make the PR obsolete?

I haven't gotten that far, I'm just triaging issues. I think there will be several changes to make ingest more consistent with rest of tidyverse, most especially with readr. If there are easy ways to update #138 that seem likely to be what's needed, that would be great and I'll try to use it.

Good example of workbook with 3 columns that are assumed to be numeric but then are, in fact, not.
library(readxl)

theURL <- "https://github.com/tts/villisika/raw/master/SpeciesObservations.xlsx"
tf <- tempfile(fileext = ".xlsx")
download.file(theURL, tf)
data_se <- read_excel(tf, sheet = "SLW Data")
#> Warning in read_xlsx_(path, sheet, col_names = col_names, col_types =
#> col_types, : [123, 70]: expecting numeric: got 'Skyddad vik vid lerig
#> älvstrand'
#> Warning in read_xlsx_(path, sheet, col_names = col_names, col_types =
#> col_types, : [139, 103]: expecting numeric: got 'Tomtskådning Dyudden'
#> Warning in read_xlsx_(path, sheet, col_names = col_names, col_types =
#> col_types, : [139, 104]: expecting numeric: got 'Arter noterade från
#> tomten Gäddeholm 2:59 på Dyuddens sydspets, med adress Gäddeholm 73, 725 97
#> Västerås. Startade 29 oktober 2006, dagen efter inflyttning. '
#> Warning in read_xlsx_(path, sheet, col_names = col_names, col_types =
#> MORE OF THE SAME ... LOTS MORE

## which columns are the problematic ones?
cellranger::num_to_letter(c(70, 103, 104))
#> [1] "BR" "CY" "CZ"

Apologies if this was covered elsewhere already, but perhaps read_excel could also gain the n_max argument used by readr::read_csv?
I find it often useful to read a particularly difficult worksheet in chunks, e.g. by combining readr::read_csv's n_max and skip arguments.

@tomsing1 I was wondering if/when someone would request that! Can you open a new issue for that please?

Done! The new issue is #281


v2.0.1 fails build of @hospitalrun/frontend

When using @hospitalrun/components v2.0.1 in @hospitalrun/frontend, the build fails.
🐛 Bug Report
https://github.com/HospitalRun/hospitalrun-frontend/runs/1088553167
Expected behavior
The build should not fail.
Technical Notes
Caused by #581

@jackcmeyer Perhaps the SCSS should all be moved to the scss folder? Then we wouldn't need to reference the src folder when importing.

Can I pick this with the following questions? Please bear with me if am asking the obvious here. I am just getting to know the @hospitalrun codebase.
Current npm package - @hospitalrun/components - ships raw *.scss files.

Assuming the components only get modified via props, is there a need for .scss files in the front-end other than the _variables.scss? I could not trace any ties in the front-end to these .scss files. ( front-end only uses a single .css file)

Therefore, can we inject all component/*.scss into js as css at the component level (during the build?)

       // ex: in Toaster/index.tsx
       import 'react-toastify/dist/ReactToastify.min.css';
       import './toaster.scss';

This way front-end does not need to do any .scss reference as these will be picked up by imports in the @hospitalrun/components/dist/components.esm.js.

Also, @hospitalrun/components/scss/_variables.scss can be shipped for variable reference in the front-end if needed.

I tested this with both @components and @front-end successfully. And both worked w/o any errors and passed all tests. Would like to know everyone's thoughts on this as well. Thanks.

@CreativeCreate Go for it! I think this is the best possible fix at the moment!
I really like what you've written. Your assumptions are correct!

Please note: now that there's no need for .scss references in the front-end, the main.scss reference in the front-end must be removed as follows.
   ...
   import { Provider } from 'react-redux'
 - import<EMAIL_ADDRESS>  // ***to be removed
 - import './index.css'                               // ***include after App import
    import App from './App'
 // ***must import after App to override component styles
 + import './index.css'
  ...
 ReactDOM.render(

Not sure whether I should make a pull-request for the above front-end changes w/o getting assigned?
Should I?


Instance size

Is there a way to configure the instance size?

what do you mean? you can set the cpu and memory. There are no "instances" in kubernetes, only in the underlying infrastructure
BTW the correct repo location is https://github.com/jenkinsci/kubernetes-plugin


Big git diff can't be interrupted

Type: Bug
If I accidentally trigger git diff for a large text file with a lot of changes, the CPU goes high, and the diffing "task" cannot be stopped - even if the file has been reverted or a new diff started.  In fact, a new diff cannot be started until the previous diff finishes (or times out?) or the VSCode window is reloaded.
 Can you try with empty settings?

I haven't changed that:

It cannot be empty, but I tried 0.
It keeps diffing (CPU high), even if I close the file tab or revert the file.

Can you share the diff that causes this timeout? Thanks!

Hey @hediet, this issue might need further attention.
@dzid26, you can help us out by closing this issue if the problem no longer exists, or adding more information.

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


Pgcopy stream refactor2

Solves: #506
Also, I have removed the old metrics in the headers

It is a first approach to the changes requested in #480
I need to test it a lot before continue. But I suppose that it possible to review if it fits with the changes requested. I am not sure if it is better than the original.
@dgaubert @enekid @rafatower

As per our chat, closing this for the moment (but keeping the branch and all). Another issue/PR will be created instead. Feel free to reopen if needed.

I am going to reopen this PR, because I am going to follow with the work made here, changing the base branch

I have developed some different solutions:

the current one: the event emitter
double callback: it was into the commit https://github.com/CartoDB/CartoDB-SQL-API/pull/503/commits/8a15b2f8d10d096bedd43ee119ce8258dabebae9 The main difference is: Stream Copy, instead of emitting an event at the end with the results, receives a second callback and call it with the results.

Anyway, I prefer the current solution in master branch.

are you sure you wanted to merge this?

It was merged automatically when I merged https://github.com/CartoDB/CartoDB-SQL-API/pull/508 (#508 was created from this)
I came here to remove it and I have found it as "merged"

ha! wonders of github. Makes sense if it was the departing point of the final refactor. It's ok, let's leave it as it is. Thanks!


update golang.org/x/net from v0.20.0 to v0.23.0 (backport #2781)

BLOCKED ON https://github.com/cometbft/cometbft/pull/2784


Setting - Help - Unnecessary Active Button

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Go to Setting> Help
Click on one of the Options below (e.g. send money).
Click on "Expensify help" at the top corner.
Returns Back to welcome to Expensify help home page.
Observe the  "Expensify help" at the top corner.

Expected Result:
It should be Deactivated like "Home" option button at welcome to Expensify help home page.
Actual Result:
It is Active and refresh the Help home page
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

Version Number: 1.3.28-5
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/93399543/84896a06-90c7-4d73-9197-4c13ffd59315
https://github.com/Expensify/App/assets/93399543/4dbbcc4a-e9d0-42a8-8b32-0a04931ca9dc
Expensify/Expensify Issue URL:
Issue reported by: Amanuel Mulugeta Arega
Data Analyst and QA S@Amnex-08
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1687182997820449
View all open jobs on GitHub

I don't think this is a bug since most sites behave the same. But if we have too
Proposal
Please re-state the problem that we are trying to solve in this issue.
Setting - Help - Unnecessary Active Button
What is the root cause of that problem?
It is not a bug but a feature request
What changes do you think we should make in order to solve the problem?
We can conditionally render a tag or span based on route to make it disabled, like this:
{% if page.url == "/" %}

            {% else %}

            {% endif %}

What alternative solutions did you explore? (Optional)
We can also prevent click events by applying this js code:
if (window.location.pathname === '/') {
    // Disable the anchor tag by preventing its default behavior and removing the href attribute
    document.querySelector('a[href="/"]').addEventListener('click', (event) => {
        event.preventDefault();
    });
}

As per the slack discussion, not a bug


Provide loading indicator for tab Chat

Follow-up of issue #827.
It would be nice to have a loading indicator because I changed tabs a few times because I thought the tab Chat displays wrong content. But maybe I got confused by the green tab coloring. Currently I am not so sure anymore.

I dont get what you wanna say :(

When tab Leaderboards is the initial tab on starting up the UI and then changing to tab Chat, there is no Loading indicator (green circle):

So in the end I got very confused by the combination green chat indicator (#827) + no loading indicator when changing to tab chat.

Fixed on 0.9.0-beta


[AIRFLOW-6944] Allow AWS DataSync to "catch up" when Task is already …

…running

Issue link: WILL BE INSERTED BY boring-cyborg
Make sure to mark the boxes below before creating PR: [x]

[ ] Description above provides context of the change
[ ] Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID*
[ ] Unit tests coverage for changes (not needed for documentation changes)
[ ] Commits follow "How to write a good git commit message"
[ ] Relevant documentation is updated including usage instructions.
[ ] I will engage committers as explained in Contribution Workflow Example.

* For document-only changes commit message can start with [AIRFLOW-XXXX].

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

TODO: Still have to provide documentation

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@fa932e8). Click here to learn what that means.
The diff coverage is 25%.

@@            Coverage Diff            @@
##             master      #19   +/-   ##
=========================================
  Coverage          ?   86.35%
=========================================
  Files             ?      896
  Lines             ?    42641
  Branches          ?        0
=========================================
  Hits              ?    36824
  Misses            ?     5817
  Partials          ?        0

Impacted Files
Coverage Δ

airflow/providers/amazon/aws/hooks/datasync.py
16.66% <0%> (ø)

airflow/providers/amazon/aws/hooks/sns.py
95.65% <100%> (ø)

airflow/providers/amazon/aws/operators/datasync.py
32.63% <23.8%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa932e8...9d14df0. Read the comment docs.


Default import is not the default export when using ESM

I tried using this in a NextJS ESM project, and the default import is not the default export, rather the module object.
My workaround probably does a better job at explaining the problem:
import diffViewerModule from "react-diff-viewer-continued"
const DiffViewer = diffViewerModule.default

I know that getting typescript and ESM to play nice can be a real PITA, but would it be possible to provide a build for ESM?
Esbuild can handle this, you could perhaps build lib/index.mjs and then put that in the module field in package.json.

@joostdecock I think the next build now has an ESM version. Can you let me know if that works for your purposes?


Testing : Fix vfs pointer leak and use automatic mocking

@elharo @patflynn
One of these mocks wasn't working with idea14, so updated it in a way we could have common code in both branches.
Also fixed that pesky vfs pointer not disposed thing.

lgtm


[minor request] CTRL+D to duplicate a line into the Script Editor

Hi,
When editing a script into the editor, Ctrl+D is still assigned to the <somewhere else> where it duplicates nodes.
It would be cool if it was assigned to a duplicate-line function when editing a script.

In fact I've just noticed there is a Ctrl+B to duplicate lines, and that it is available since at least 1.0RC1.
It's just a matter of getting used to Ctrl+B instead of Ctrl+D now ...
So, if no one objects, I propose to close this feature request.


sc-5196: implement reset password feature

Type of change

[x] bug fix
[x] new feature
[ ] documentation
[ ] other (describe)

Scope of changes
sc-5196: implement a reset password feature
Acceptance criteria
Go to the account/reset route and fill out your email. Auth0 will send you an email to change your password.
Author checklist

[x] I have manually tested the change and/or added automation in the form of unit tests or integration tests
[ ]  I have updated the dependencies list
[ ]  I have recompiled and included new protocol buffers to reflect changes I made
[ ]  I have added new test fixtures as needed to support added tests
[ ]   Check this box if a reviewer can merge this pull request after approval (leave it unchecked if you want to do it yourself)
[x]  I have moved the associated ShortCut or Jira ticket to "Ready for Review"

Reviewer(s) checklist

[ ] Any new user-facing content that has been added for this PR has been QA'ed to ensure correct grammar, spelling, and understandability.

Codecov Report

Merging #525 (a9e9ee9) into main (10805bb) will decrease coverage by 9.36%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #525      +/-   ##
==========================================
- Coverage   46.84%   37.47%   -9.37%
==========================================
  Files         234      152      -82
  Lines       11737     2391    -9346
  Branches      608      608
==========================================
- Hits         5498      896    -4602
+ Misses       5270     1320    -3950
+ Partials      969      175     -794

Impacted Files
Coverage Δ

...b.com/trisacrypto/directory/pkg/gds/store/store.go

...m/trisacrypto/directory/pkg/gds/admin/v2/decode.go

...github.com/trisacrypto/directory/pkg/bff/status.go

...com/trisacrypto/directory/pkg/sectigo/endpoints.go

...ub.com/trisacrypto/directory/pkg/gds/store/load.go

...com/trisacrypto/directory/pkg/utils/wire/rewire.go

...sacrypto/directory/pkg/gds/models/v1/pagination.go

...github.com/trisacrypto/directory/pkg/trtl/peers.go

...risacrypto/directory/pkg/gds/store/trtl/connect.go

...com/trisacrypto/directory/pkg/trtl/replica/sync.go

... and 72 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 10805bb...a9e9ee9. Read the comment docs.


[ngrx-slice] depend on Angular 14 packages

Hey!
First of all, thank you for this awesome package, it really makes development a lot faster.
I have tried upgrading a project to Angular 14 that uses slice, but I get the following error:
npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR!
npm ERR! While resolving<EMAIL_ADDRESS>npm ERR! Found<EMAIL_ADDRESS>npm ERR! node_modules/@angular/common
npm ERR<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm ERR! node_modules/ngrx-slice
npm ERR<EMAIL_ADDRESS>from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.

Peer dependencies like ngrx-immer seem to be compatible with Angular 14. Could you please update the ngrx-slice package?
The project does work well with the --legacy-peer-deps option though, so I hope upgrading won't mean too much refactoring.

@dJani97 Thank you for the kind words. I've just released 6.3.0 which bumps the version of Angular and NgRx to 14.

Thank you for the quick release!
I suggest adding v14 compatibility to the docs > peer-dependencies section, like so:

NgRx Slice
Angular
NgRX

v6.3
v14
v14

v6
v13
v13

v5
v11, v12
v11, v12

No support for Angular <11 because of TypeScript version

Yep, I can push a commit for docs later. I always forget updating docs xD


Test

Testing to make sure the config is setup properly and we all get emails

Adding a comment as a second test

This was just a test issue, closing.


Schedule PDX Finder SAB 2021 meeting

Description
PDX Finder is due to have its next SAB meeting in 2021
Acceptance criteria

[x] schedule the meeting

Meeting scheduled fro Nov 15 2021


Make Cask tests work without tapping caskroom/cask.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[ ] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you written new tests for your changes? Here's an example.
[x] Have you successfully run brew tests with your changes locally?

Nice work :clap:


HTTPS call is not working

I followed this tutorial, but the last step is not working (serving through HTTPS).
The secret is created, I can access it on HTTP but not on HTTPS.
Can you guide me to the fix?
this is the output of the describe ingress:
Name:             helloweb
Namespace:        default
Address:          <IP_ADDRESS>
Default backend:  helloweb-backend:8080 (<IP_ADDRESS>:8080)
TLS:
dogs-com-tls terminates app-solidair-vlaanderen.com
Rules:
Host                         Path  Backends

app-solidair-vlaanderen.com
/.well-known/acme-challenge/Q8kcFSZ0ZUJO58xZyVbK6s-cJIWu-EgwPcDd8NFyoXQ   cm-acme-http-solver-mhqnf:8089 ()
Annotations:
url-map:          k8s-um-default-helloweb--17a833239f9491d9
backends:         {"k8s-be-30819--17a833239f9491d9":"Unknown","k8s-be-32482--17a833239f9491d9":"HEALTHY"}
forwarding-rule:  k8s-fw-default-helloweb--17a833239f9491d9
target-proxy:     k8s-tp-default-helloweb--17a833239f9491d9
Events:
Type     Reason  Age                From                     Message

Normal   ADD     45m                loadbalancer-controller  default/helloweb
Normal   CREATE  44m                loadbalancer-controller  ip: <IP_ADDRESS>
Warning  Sync    7m (x22 over 28m)  loadbalancer-controller  Error during sync: error while evaluating the ingress spec: could not find service "default/cm-acme-http-solver-mhqnf"

Did you solve it? I'm facing the same problem

It is solved now.
I recreated the ingress and changed 3 things:
-1: I added " certmanager.k8s.io/acme-http01-edit-in-place: "true" " to the annotations
-2: I changed the secret from "www-dogs-com-tls" to "dogs-com-tls" as it was defined without "www"
-3: I added a health path
optional: if you want to force HTTPS, you can add 2 other annotations.
Now it's working like a charm. Many thanks to the creators, maintainers!!!!
my ingress config:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: helloweb
  annotations:
    kubernetes.io/ingress.global-static-ip-name: helloweb-ip
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
    ingress.kubernetes.io/force-ssl-redirect: "true"
    ingress.kubernetes.io/ssl-redirect: "true"
  labels:
    app: hello
spec:
  rules:
    - host: app-solidair-vlaanderen.com
      http:
        paths:
        - path: /*
          backend:
            serviceName: helloweb-backend
            servicePort: 8080
  tls:
  - secretName: dogs-com-tls
    hosts:
    - app-solidair-vlaanderen.com

What do you mean by "changed the image to a keycloak one" ?

I use a keycloak docker image in stead of the hello world image
On Sun, 7 Oct 2018, 21:31 John<EMAIL_ADDRESS>wrote:

What do you mean by "changed the image to a keycloak one" ?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ahmetb/gke-letsencrypt/issues/18#issuecomment-427680046,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQHRWShUhKU0vtFmRNHP1InvwAb0UXtSks5uilZ5gaJpZM4XLVN5
.

Bu I meant, Do I need to have two deployments? or "gcr.io/solidair-vlaanderen-204118/sv-security:0.0.1" should be replaced with my app image?

You should replace it with your image :)
On Sun, 7 Oct 2018, 21:49 John<EMAIL_ADDRESS>wrote:

Bu I meant, Do I need to have two deployments? or "
gcr.io/solidair-vlaanderen-204118/sv-security:0.0.1" should be replaced
with my app image?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ahmetb/gke-letsencrypt/issues/18#issuecomment-427681450,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQHRWRKAnxyDz8EMumPuhlV_wyP-5vDMks5uilrZgaJpZM4XLVN5
.

Folks, if you hit this issue reproducibly, please file a bug at https://github.com/jetstack/cert-manager/issues


add developer guide

Really just a scratchpad really for notes to help get new people
set up quickly based on my experience, for instance to
document the logging setup when running system-tests locally (which is
very smart but just not obvious until you dig into it).

:+1:


Why does it consume so much memory?

I have 1 video (already encoded offline as 16 images), 1 image (encoded in .pth), 1 title, 8 captions, and I use a custom decode function to decode them. But I find that this consumes a lot of memory and crashes my program after a while of training. I have used methods like deepcopy/del/gc.coolect(), but it doesn't seem to do much.
Is there something wrong with my code? Or how does webdataset release memory?

`class Decode:
def init(self, args=None):
self.target_len = args.target_len
self.freq = args.freq
self.aug_roll_mag = args.aug_audio_roll_mag
self.num_frames = args.num_frames
self.tokenizer = CLIPTokenizerFast.from_pretrained(args.tokenizer_name, cache_dir=args.cache_dir)
self.tokenizer.mask_token_id = 0
self.max_seq_length = args.max_seq_length
self.mlm_probability = args.mlm_probability
self.use_cap = args.use_cap
self.pretrain_clip = args.pretrain_clip
def __call__(self, sample):
    # try:
    #     with warnings.catch_warnings(record=True) as w:
    video = self.decode_video([sample[f"frm{i}.jpg"] for i in range(self.num_frames)])

    audio = self.decode_audio(sample['pth'], self.target_len, self.freq)
    captions = [sample[f'cap{i}.txt'] for i in range(8)] if self.use_cap else None
    input_ids, labels, attention_mask = self.decode_text(sample['txt'], captions)

    video_new, audio_new = deepcopy(video), deepcopy(audio)
    input_ids_new, labels_new, attention_mask_new = deepcopy(input_ids), deepcopy(labels), deepcopy(attention_mask)
    del video, audio, input_ids, labels, attention_mask, captions
    gc.collect()
    return video_new, audio_new, input_ids_new, labels_new, attention_mask_new
    # except Exception as e:
    #     logging.info(f"Error occurred: {e}")
    #     return None

def decode_video(self, frames):
    results = []
    for frame in frames:
        with io.BytesIO(frame) as stream:
            img = Image.open(stream)
            img.load()
            img = img.convert('RGB')
        result = np.asarray(img)
        results.append(torch.from_numpy(result))
    results = torch.stack(results, dim=0)
    frames = aug_frame(results, 224)
    frames_new = deepcopy(frames)
    del frames, results
    gc.collect()
    return frames_new

def decode_audio(self, x, target_len=1024, freq=128):
    with io.BytesIO(x) as stream:
        fbank = torch.load(stream)

    n_frames = fbank.shape[0]
    assert freq == fbank.shape[1]
    p = target_len - n_frames

    # cut and pad
    if p > 0:
        m = torch.nn.ZeroPad2d((0, 0, 0, p))
        fbank = m(fbank)
    elif p < 0:
        fbank = fbank[0: target_len, :]
    fbank = (fbank - (-4.2677393)) / (4.5689974 * 2)
    fbank = fbank.unsqueeze(0)
    fbank_new = deepcopy(fbank)
    del fbank
    gc.collect()
    return fbank_new

def decode_text(self, title, captions=None):
    title = title.decode("utf-8")
    if captions is not None:
        cap_pre = np.random.randint(0, 3)
        cpa_mid = np.random.randint(3, 5)
        cap_last = np.random.randint(5, 8)
        text = title + ', ' + ', '.join([captions[cap_pre].decode("utf-8"),
                                         captions[cpa_mid].decode("utf-8"),
                                         captions[cap_last].decode("utf-8")])
        text = whitespace_clean(basic_clean(text)).lower()
        del captions
        gc.collect()
    else:
        text = title

    text = self.tokenizer(text, padding='max_length', truncation=True, max_length=self.max_seq_length,
                          return_attention_mask=True, return_tensors="pt")
    input_ids = text.input_ids.clone()
    labels = input_ids.clone()
    if not self.pretrain_clip:
        input_ids, labels = self.mask_text(input_ids, self.tokenizer.vocab_size, labels)
    attention_mask = text.attention_mask
    input_ids, labels, attention_mask = input_ids.squeeze(), labels.squeeze(), attention_mask.squeeze()

    input_ids_new, labels_new, attention_mask_new = deepcopy(input_ids), deepcopy(labels), deepcopy(attention_mask)
    del input_ids, labels, attention_mask, title
    gc.collect()

    return input_ids_new, labels_new, attention_mask_new

def mask_text(self, input_ids, vocab_size, targets):
    probability_matrix = torch.full(targets.shape, self.mlm_probability)
    masked_indices = torch.bernoulli(probability_matrix).bool()

    masked_indices[input_ids == self.tokenizer.bos_token_id] = False
    masked_indices[input_ids == self.tokenizer.eos_token_id] = False      # cls and pad is eof in clip

    targets[~masked_indices] = -100  # We only compute loss on masked tokens
    # 80% of the time, we replace masked input tokens with tokenizer.mask_token ([MASK])
    indices_replaced = torch.bernoulli(torch.full(input_ids.shape, 0.8)).bool() & masked_indices
    input_ids[indices_replaced] = self.tokenizer.mask_token_id
    # 10% of the time, we replace masked input tokens with random word
    indices_random = torch.bernoulli(torch.full(input_ids.shape, 0.5)).bool() & masked_indices & ~indices_replaced
    random_words = torch.randint(vocab_size, input_ids.shape, dtype=torch.long)
    input_ids[indices_random] = random_words[indices_random]
    # The rest of the time (10% of the time) we keep the masked input tokens unchanged
    return input_ids, targets

def get_wds_dataset(args, is_train=True, epoch=0, floor=False):
input_shards = args.train_data
assert input_shards is not None
resampled = getattr(args, 'dataset_resampled', False) and is_train
num_samples, num_shards = get_dataset_size(input_shards)
if not num_samples:
    num_samples = args.train_num_samples
    if not num_samples:
        raise RuntimeError(
            'Currently, number of dataset samples must be specified for training dataset. '
            'Please specify via `--train-num-samples` if no dataset length info present.')

shared_epoch = SharedEpoch(epoch=epoch)  # create a shared epoch store to sync epoch to dataloader worker proc

def build_pipeline(input_shards):

    pipeline = [wds.SimpleShardList(input_shards)]

    # at this point we have an iterator over all the shards
    pipeline.extend([
        detshuffle2(
            bufsize=_SHARD_SHUFFLE_SIZE,
            initial=_SHARD_SHUFFLE_INITIAL,
            seed=args.seed,
            epoch=shared_epoch,
        ),
        wds.split_by_node,
        wds.split_by_worker,
        # at this point, we have an iterator over the shards assigned to each worker at each node
        tarfile_to_samples_nothrow,  # wds.tarfile_to_samples(handler=log_and_continue),
        wds.shuffle(
            bufsize=_SAMPLE_SHUFFLE_SIZE,
            initial=_SAMPLE_SHUFFLE_INITIAL,
        ),
        wds.map(Decode(args))
    ])
    return pipeline

pipeline = build_pipeline(input_shards)

pipeline.append(
    wds.batched(args.batch_size, partial=not is_train)
    # batched1(args.batch_size, partial=not is_train)
)

dataset = wds.DataPipeline(*pipeline)

if not resampled:
    assert num_shards >= args.workers * args.world_size, 'number of shards must be >= total workers'
# roll over and repeat a few samples to get same number of full batches on each node
round_fn = math.floor if floor else math.ceil
global_batch_size = args.batch_size * args.world_size
num_batches = round_fn(num_samples / global_batch_size)
num_workers = max(1, args.workers)
num_worker_batches = round_fn(num_batches / num_workers)  # per dataloader worker
num_batches = num_worker_batches * num_workers
num_samples = num_batches * global_batch_size
dataset = dataset.with_epoch(num_worker_batches)  # each worker is iterating over this

dataloader = wds.WebLoader(
    dataset,
    batch_size=None,
    shuffle=False,
    num_workers=args.workers,
    persistent_workers=True,
)

dataloader.num_batches = num_batches
dataloader.num_samples = num_samples

return DataInfo(dataloader=dataloader, shared_epoch=shared_epoch)

`

the pipeline is from open_clip (https://github.com/mlfoundations/open_clip/blob/main/src/training/data.py), I just use my custom decode function.
I don't know why it is consuming so much memory, how can I fix this problem? I have 200 .tar files, each .tar file contains 5k sets of data, each set has 16 frames, 1 image (.pth encoding), 1 title, 8 captions. i use 8 A100 GPUs for training, each GPU has a batchsize of 80 and 10 num workers.

btw. I set the batchsize to 40, it's also failed, and the training time is longer than that of the batchsize 80.

WebDataset doesn't store or buffer any data unless you ask it to.
Most likely, your memory usage comes from the shuffle buffer. If your SHUFFLE_SIZE is 5000, that means that WDS keeps 5000 samples in memory. Try choosing a smaller size and see whether that reduces memory usage.

Hey @LinB203 did this resolve your issue? I also ran into the same issue where my CLIP training jobs get killed because of OOM, did you try reducing the buffer sizes and did that do any trick?
This is the exact issue I am facing : https://github.com/webdataset/webdataset/issues/325#issuecomment-2120638533


Problem with failover latency based

Describe the bug
Hi, the failover based on latency are buggy, when latency is high the gateway are removed from group gateways many times, and when the latency is normal again, the gateway returns to group only one time, and the connections don't go through it and I need apply firewall rules again.
Expected behavior
If latency is already high, and the gateway was removed from group gateways, it not should removed again and again
Relevant log files
Jan 8 01:31:08	monit[39059]: 'gateway_alert' status succeeded (0) -- no output
Jan 8 01:29:08	monit[39059]: 'gateway_alert' status failed (1) -- MONITOR: FLINGW has high latency, removing from routing group ESSSFLNGW
Jan 8 01:27:08	monit[39059]: 'gateway_alert' status failed (1) -- MONITOR: FLINGW has high latency, removing from routing group ESSSFLNGW
Jan 8 01:25:08	monit[39059]: 'gateway_alert' status failed (1) -- MONITOR: FLINGW has high latency, removing from routing group ESSSFLNGW
Jan 8 01:23:07	monit[39059]: 'gateway_alert' status failed (1) -- MONITOR: FLINGW has high latency, removing from routing group ESSSFLNGW
Jan 8 01:21:07	monit[39059]: 'gateway_alert' status failed (1) -- MONITOR: FLINGW has high latency, removing from routing group ESSSFLNGW
Jan 8 01:19:07	monit[39059]: 'gateway_alert' status failed (1) -- MONITOR: FLINGW has high latency, removing from routing group ESSSFLNGW
Jan 8 01:17:07	monit[39059]: 'gateway_alert' status failed (1) -- MONITOR: FLINGW has high latency, removing from routing group ESSSFLNGW
Jan 8 01:15:07	monit[39059]: 'gateway_alert' status failed (1) -- MONITOR: FLINGW has high latency, removing from routing group ESSSFLNGW
Jan 8 01:13:07	monit[39059]: 'gateway_alert' status failed (1) -- MONITOR: FLINGW has high latency, removing from routing group ESSSFLNGW
Jan 8 01:11:07	monit[39059]: 'gateway_alert' status failed (1) -- MONITOR: FLINGW has high latency, removing from routing group ESSSFLNGW
Jan 8 01:09:06	monit[39059]: 'gateway_alert' status failed (1) -- MONITOR: FLINGW has high latency, removing from routing group ESSSFLNGW
Jan 8 01:07:06	monit[39059]: 'gateway_alert' status failed (1) -- MONITOR: FLINGW has high latency, removing from routing group ESSSFLNGW
Environment
OPNsense 19.781 (amd64, OpenSSL).

The message isn't totally correct, its Monit reporting the status (not performing actions).
https://github.com/opnsense/core/blob/168be32071ea4739ddab0aef5dd53ade49544c66/src/opnsense/scripts/OPNsense/Monit/gateway_alert#L63-L69
You can safely disable the monit check if your not interested in the messages, dpinger takes care of the actual switches using a monitor event.
https://github.com/opnsense/core/blob/a7b528be2b5a5128612ea95f65beb96862988977/src/etc/inc/plugins.inc.d/dpinger.inc#L236-L237

Thanks @AdSchellevis , I will remove monit checks, but my doubt is why connections still outing by secondary gateway after primary returns to group. Does any command to verify which gateway are currently operating?

The gateway page should show the current statuses. Low level you can always check /tmp/rules.debug for the rules that are generated for policy-based routing.

Very thank you @AdSchellevis, have a nice day

Hi Again, I just confirmed, the failover not return to primary gateway always.
All gateways are with status Online (so outgoing traffic should  going to 187.XXX.XXX.XXX), but in rules.debug is:
pass in quick on em1 route-to {( em2 200.XXX.XXX.XXX )} sticky-address inet from {<IP_ADDRESS>/23} to {any} keep state label "8f91442a5874adfddf4c7d7--------"

there are two things you can look at, first one is what happens if you regenerate the rules from the console (configctl filter reload), next you could check if the monitor event actually fired when you thought it should (I think it should be in the gateway log)

I just did the filter reload, and will wait for the high latency occur

but did the rules.debug change afterwards?

Yes, like I says before:
from
pass in quick on em1 route-to {( em2 200.XXX.XXX.XXX )} sticky-address inet from {<IP_ADDRESS>/23} to {any} keep state label "8f91442a5874adfddf4c7d7--------"
to
pass in quick on em1 route-to {( em0 187.XXX.XXX.XXX )( em0 187.XXX.XXX.XXX )} sticky-address inet from {<IP_ADDRESS>/23} to {any} keep state label "8f91442a5874adfddf4c7d---------"

Good morning (from Brazil)
OK, some firsts interaction works, high latency changed to
pass in quick on em1 route-to {( em2 200.XXX.XXX.XXX )} sticky-address inet from {<IP_ADDRESS>/23} to {any} keep state label "8f91442a5874adfddf4c7d79---------"

when normalize returned to
pass in quick on em1 route-to {( em0 187.XXX.XXX.XXX )( em0 187.XXX.XXX.XXX )} sticky-address inet from {<IP_ADDRESS>/23} to {any} keep state label "8f91442a5874adfddf4c7d----------"

but after some swaps it stops to return to primary gateway, the last swap to secondary gateway it stay on secondary until now... below are the log, this last interaction of dpinger registered Clear latency, but the gateway stay on secondary
pass in quick on em1 route-to {( em2 200.XXX.XXX.XXX )} sticky-address inet from {<IP_ADDRESS>/23} to {any} keep state label "8f91442a5874adfddf4c----------"

The log files Gateways:
Jan 10 09:16:07 | dpinger: GATEWAY ALARM: FLINGW (Addr: <IP_ADDRESS> Alarm: 0 RTT: 25619ms RTTd: 9302ms Loss: 0%)
-- | --
Jan 10 09:16:07 | dpinger: FLINGW <IP_ADDRESS>: Clear latency 25619us stddev 9302us loss 0%
Jan 10 09:15:46 | dpinger: GATEWAY ALARM: FLINGW (Addr: <IP_ADDRESS> Alarm: 1 RTT: 26073ms RTTd: 9403ms Loss: 0%)
Jan 10 09:15:46 | dpinger: FLINGW <IP_ADDRESS>: Alarm latency 26073us stddev 9403us loss 0%
Jan 10 08:44:18 | dpinger: GATEWAY ALARM: FLINGW (Addr: <IP_ADDRESS> Alarm: 0 RTT: 22115ms RTTd: 197ms Loss: 0%)
Jan 10 08:44:18 | dpinger: FLINGW <IP_ADDRESS>: Clear latency 22115us stddev 197us loss 0%
Jan 10 08:43:25 | dpinger: GATEWAY ALARM: FLINGW (Addr: <IP_ADDRESS> Alarm: 1 RTT: 27256ms RTTd: 9155ms Loss: 0%)
Jan 10 08:43:25 | dpinger: FLINGW <IP_ADDRESS>: Alarm latency 27256us stddev 9155us loss 0%
Jan 10 08:02:27 | dpinger: GATEWAY ALARM: FLINGW (Addr: <IP_ADDRESS> Alarm: 0 RTT: 21957ms RTTd: 176ms Loss: 0%)
Jan 10 08:02:27 | dpinger: FLINGW <IP_ADDRESS>: Clear latency 21957us stddev 176us loss 0%
Jan 10 08:01:39 | dpinger: GATEWAY ALARM: FLINGW (Addr: <IP_ADDRESS> Alarm: 1 RTT: 27431ms RTTd: 13045ms Loss: 0%)
Jan 10 08:01:39 | dpinger: FLINGW <IP_ADDRESS>: Alarm latency 27431us stddev 13045us loss 0%

I don't did any changes on opnsense in mean time to avoid external influence.
Does exists a number of swaps that gateways groups can do?
Thanks

you try to inspect the system log, usually it should trigger /usr/local/etc/rc.filter_configure on monitor event, if it did on the time you expect, maybe it has some more context about it.

the system log :
Jan 10 09:16:07 | opnsense: /usr/local/etc/rc.filter_configure: ROUTING: creating /tmp/em0_defaultgw using '187.XXX.XXX.XXX'
-- | --
Jan 10 09:16:07 | opnsense: /usr/local/etc/rc.filter_configure: ROUTING: removing /tmp/em2_defaultgw

Apparently it runs... but the rules not applied

that's the output of the default gateway switching, which means that at that point in time it did trigger and preferred the other gateway as default. The groups are populated with the active gateways (I don't think they are in the log by themselves).

anything else I can do to find where is the issue?

I must disable "Allow default gateway switching" ?

default gateways switching is independent, so you can just leave it on if you would like to use it. My community support time is over, maybe someone else wants to step in or has additional ideas.

Ok, thanks for your time

Hi @AdSchellevis , just to inform you, I found the problem, maybe is not an expected behavior,
the range of latency are 10ms ex: beetwen 30 and 40ms, when the gateway primary latency cross the highest value of range, it is swapped to secondary, IF the latency reduce to 35ms, it became yellow and not return to primary, after if it reduce to 20ms, I guess from RED do yellow and yellow to green the trigger not change the rules, juste RED do Green.
I put the same value to From and To in Latency thresholds and the issue not occurred anymore.
Thanks for you help

thanks for letting us know, I'm not sure what the exact behaviour of the timings should be from the top of my mind (and when it signals an event), it might be good to extend the docs a bit for these settings.

This issue has been automatically timed-out (after 180 days of inactivity).
For more information about the policies for this repository,
please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.
If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.