An example on how to use node-formidable to upload big files with express.js

Currently I am using body parser (in my express app) for parsing the bodies of my requests, etc, but so far I have not had the need to upload "big" files, i.e. songs and videos.
I would like to keep using body parser, because my whole application is using body parser. Now I have a form where I need to upload pictures, songs and videos, and therefore I need some kind of parser that allows me to upload and store these files to the server. It would be nice if I could simply use node-formidable to parse specific post requests, instead of all of them. As far as I have understood, I would need some kind of middleware since I am working with express, but node-formidable is not a middleware...
I know there exists the express-formidable module which should theoretically be what I need, but the documentation is so poor, there are no examples, and honestly it does not seem so reliable.
My current app.js file looks file:
var config = require('./config');
var express = require('express');
var path = require('path');
var logger = require('morgan');
var bodyParser = require('body-parser');

var dustjs = require('adaro');
var app = express();
var mongoose = require('mongoose');
var passport = require('passport');
var expressSession = require('express-session');

// Connect to MongoDB here
mongoose.connect(config.mongoUrl + config.mongoDbName);

// Register model definition here
require('./models');

// dustjs view engine setup
app.engine('dust', dustjs.dust());
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'dust');

app.use(express.static(path.join(__dirname, 'public')));

//configure app
app.use(logger('dev'));

app.use(bodyParser.urlencoded({
  extended: false
})); // parse application/x-www-form-urlencoded

app.use(bodyParser.json()); // parse application/json

// Configuring Passport
app.use(expressSession({
          // some options
}));

app.use(passport.initialize());

// for persistent login sessions
app.use(passport.session());

// Using the flash middleware provided by connect-flash to store messages in session
// and displaying in templates
var flash = require('connect-flash');
app.use(flash());

// Initialize Passport
var initPassport = require('./passport/init');
initPassport(passport);

var routes = require('./routes/root/router')(passport);
app.use('/', routes);

// Initialize routers here
var routers = require('./routes/routers');

// app.use('/', routers.root);
app.use('/library', routers.library);
app.use('/albums', routers.albums);
app.use('/artists', routers.artists);
app.use('/tracks', routers.tracks);
app.use('/users', routers.users);
app.use('/playlist', routers.users);

module.exports = app;

I have then a form (in a Dust.js view) where the action is set to /tracks/upload where I should handle the upload of big files, i.e. tracks and videos. (Or should I handle this process in another way?)
Could you please show me an example of how could I do what I need to do based also on my description of my app? If node-formidable is not a good option for me, what options would you suggest?
I am using express 4.

@nelson-brochado probably not relevant anymore, but this should help: http://stackoverflow.com/a/11296989/6853928

@nbro I don't see need to have middlewares for formidable - both koa and express. It just need to be passed with nodejs's request stream, nothing more. If middleware exists for it, the middleware should be some cool wrapper of common things and providing cool features. Because formidable is kind of low-level lib.
So you can just
var app = express()

app.use(function (req, res) {
  var form = new formidable.IncomingForm({ some: 'options here' })
  form.once('error', console.log)
  form.parse(req, function (err, fields, files) {})
})


Set values for time axis

I'm unclear as to how I might set the values for a time axis. I'm using Axis as follows...
Axis(title=my_title, values=my_values)
...and have tried setting my_values to a list of ISO formatted strings, datetime objects, datetime64 objects (np) and a DatetimeIndex object (pd). In each case, the stacktrace indicates that a float is required.
Thanks for any help in advance. I appreciate that this is less an issue and more a request for assistance, but there isn't an Altair discussion forum.

Hi @iaindillingham – great question, and after poking around a bit I don't have a good answer.
The Vega-Lite schema specifies that Axis.values should be a list of numbers... Altair interprets number as something convertible to floating point. Maybe @kanitw can comment on how this can best be accomplished in Vega-Lite.

@kanitw can speak to the vega-lite schema issues.
In the underlying Vega implementation UNIX time stamps (long integers,
which fall under type "number" in JavaScript) can stand in for JS Date
objects. So one path to try would be outputting Date values as time stamps
when generating the VL JSON from Altair. Of course, the scale transform
driving the axis would need to be a Date-type (temporal) scale to interpret
the values correctly.
On Thursday, September 1, 2016, Jake Vanderplas<EMAIL_ADDRESS>wrote:

Hi @iaindillingham https://github.com/iaindillingham – great question,
and after poking around a bit I don't have a good answer.
The Vega-Lite schema specifies that Axis.values should be a list of
numbers... Altair interprets number as something convertible to floating
point. Maybe @kanitw https://github.com/kanitw can comment on how this
can best be accomplished in Vega-Lite.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ellisonbg/altair/issues/187#issuecomment-244103843,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMYL8GBz3y7oVIBsaqdF0CNoxR3del9ks5qluZ3gaJpZM4JyXl4
.

Thanks @jheer – going off that, I was able to set time axes by converting pandas dates to microseconds-since 1970:
import pandas as pd
import numpy as np

data = pd.DataFrame({'x': pd.date_range('2015-01', periods=10, freq='M'),
                     'y': np.sin(np.arange(10))})

unix_dates = list(data['x'].astype(int) // 10**6)

dates = pd.to_datetime(['2015-03-15', '2015-05-15', '2015-07-15']).astype(int) / 10 ** 6

Chart(data).mark_line().encode(
    x=X('x', axis=Axis(values=list(dates))),
    y='y'
)

If this is the best way to do it, it might be useful to automatically do this kind of conversion of date values on the Python side.

This is probably the best way, seeing as how JSON has no built-in way of
expressing Date value literals.
On Thursday, September 1, 2016, Jake Vanderplas<EMAIL_ADDRESS>wrote:

Thanks @jheer https://github.com/jheer – going off that, I was able to
set time axes by converting pandas dates to microseconds-since 1970:
import pandas as pdimport numpy as np
data = pd.DataFrame({'x': pd.date_range('2015-01', periods=10, freq='M'),
'y': np.sin(np.arange(10))})
unix_dates = list(data['x'].astype(int) // 10**6)
dates = pd.to_datetime(['2015-03-15', '2015-05-15', '2015-07-15']).astype(int) / 10 ** 6
Chart(data).mark_line().encode(
x=X('x', axis=Axis(values=list(dates))),
y='y'
)
[image: vega 15]
https://cloud.githubusercontent.com/assets/781659/18174355/244a8956-7022-11e6-88d7-7ee0e3f8157c.png
If this is the best way to do it, it might be useful to automatically do
this kind of conversion of date values on the Python side.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ellisonbg/altair/issues/187#issuecomment-244126070,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMYLw_HLIw9aS4AaH5mKCNJyDgfb4c6ks5qlvY0gaJpZM4JyXl4
.

This reminds me that we should support our DateTime Definition object (that we introduced for temporal filter) for axis/legend's values as well (https://github.com/vega/vega-lite/issues/1516).  So this will be reasonable to use when people use Vega-Lite from JSON.
That said, I think from Altair's perspective, I think automatically date conversion to timestamp would be better.   :)

probably oweing to differing treatments of midnight between pandas and vega-lite.

Possibly because the timestamp is UTC but our label displays local time.
We'll have to work on support for UTC (https://github.com/vega/vega-lite/issues/446).
cc: @domoritz

Hello everyone,
Thank you all for your time. It's reassuring to find such a committed bunch of core developers!
I followed @jakevdp and passed timestamps to Axis. Rather than hard-coding dates, I used pd.date_range as follows:
date_min = '2016-01-01'
date_max = '2016-02-01'
my_values = list(pd.date_range(date_min, date_max, freq='W-MON').astype(int) / 10 ** 6)

The curious can find the associated notebook here. (It's a quick exploration of bike usage in Bath, UK, which I prepared for an upcoming hack day.)


Incompatibility with qutip 5

The package expects some qutip.qobj module that seems missing. I have only managed to make sqcircuit work by fixing qutip to 4.7.6
conda create -n sqcircuit python matplotlib-base
pip install qutip=4.7.6
pip install sqcircuit

QuTiP developer here. In the short term while SQcircuit does not yet support QuTiP 5, changing its qutip dependency to specify qutip~=4.7 seems like a sensible quick fix.
If there are questions regarding porting to QuTiP 5, feel free to ask here, or on the QuTiP mailing list or GitHub issues depending on which seems easiest / most appropriate. We did a fair amount of work to ease migration before the release, and we're keen to help as much as we can.

Thank you, @juanjosegarciaripoll, for using SQcircuit and bringing this issue to our attention. I also want to extend my thanks to @hodgestar for providing a quick fix. I'm pleased to inform you that we've released Version 1.0.0 of SQcircuit, which addresses the QuTiP dependency issue. Our latest version now fully supports QuTiP 5.0.0 and above.

@taha1373 That's awesome! How did the migration to QuTiP 5 go?

@hodgestar Overall moving to QuTiP 5 was mostly smooth. Before we relied on the QuTiP .data being stored as a CSR matrix, so it took a little bit to get used to the new options for data format and make sure we dealt with that correctly (without calling .full() and making it dense).

@hodgestar Apologies for the delayed response. I was waiting for @sambonkov's input since he handled the migration.

Glad to hear things went mostly smoothly. Woot.


Fixes for X11 build (Linux)

As posix filesystems are usually case-sensitive, I've fixed openhmd to OpenHMD in the SConstruct file.
Fixed a "" by a "/", so all plataforms interpret it as a path separation. (linux doesn't like "")
Added libusb folder in the include search path, since config.h resides there and not in libusb/libusb (it is created after properly making libusb)
Added some INSTALL information to README file to help others build it in X11 systems, specially to properly clone the dependencies and build then. (Without it, build will fail to find libusb config.h in Linux)

Looks good to me:)


caja-with-extensions: init

Motivation for this change
The standalone caja package does not find extensions. These are all configured in the mate desktop environment. This package builds a wrapper round caja which sets the extension directory correctly.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build mate.caja-with-extensions

This pull request has been mentioned on Nix community. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review-may-2019/3032/16


Update Register-SPOHubSite.md

Add -Principals parameter (required).
Change -Site to required parameter.

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sharepoint/sharepoint-ps/sharepoint-online/Register-SPOHubSite.md
:white_check_mark:Succeeded
View (sharepoint-ps)

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

thx Aaron, appreciated.


Feat: adding description of agro products #1209

Related Issue
#1209
Description
I have added all products separate description in a modal description button, when the description button is clicked the description will pop out, and  to go out of the description details we can select the (cross)X button on right most above corner.
Type of PR

[ ] Bug fix
[x] Feature enhancement
[ ] Documentation update
[ ] Other (specify): _______________

Screenshots / videos (if applicable)
https://github.com/user-attachments/assets/882b5611-807d-4cd0-bada-d82f560e6000
Checklist:

[x] I have performed a self-review of my code
[x] I have read and followed the Contribution Guidelines.
[x] I have tested the changes thoroughly before submitting this pull request.
[x] I have provided relevant issue numbers, screenshots, and videos after making the changes.
[x] I have commented my code, particularly in hard-to-understand areas.

Additional context:
[Include any additional information or context that might be helpful for reviewers.]

Navbar is looking like this now. Also when i hover on description the box size keep on increasing and decreasing.

Navbar is looking like this now. Also when i hover on description the box size keep on increasing and decreasing.

Sorry for the inconveniences I will make the desired changes properly


CASMCMS-8163: Make v1 and v2 report db_status instead of _status

Allows both versions of the API to use the same openapi schema for status reporting in a db agnostic way.
Issues and Related PRs

Resolves CASMCMS-8163

Testing
I built out the resultant change via our openapi, both through the pipeline and locally and examined the contents of the generated connexion model for healthz reporting. The updated image contains a model that has a db_status field instead of an etcd_status field.
Code was then deployed onto groot to ensure the healthz endpoints were functioning appropriately.
Tested on:

groot
Local development environment

Risks and Mitigations
Pull Request Checklist

[X] Target branch correct
[X] CHANGELOG.md updated
[X] Testing is appropriate and complete, if applicable

requests.get('http://<IP_ADDRESS>:9000/v1/healthz').text
'{\n  "apiStatus": "ok",\n  "dbStatus": "ok"\n}\n'
requests.get('http://<IP_ADDRESS>:9000/v2/healthz').text
'{\n  "apiStatus": "ok",\n  "dbStatus": "ok"\n}\n'


New version: DIMESampler v0.1.2

Registering package: DIMESampler
Repository: https://github.com/gboehl/DIMESampler.jl
Created by: @gboehl
Version: v0.1.2
Commit: 192d1895161ace817f3f4be09e87fae5dcad9ca3
Reviewed by: @gboehl
Reference: https://github.com/gboehl/DIMESampler.jl/commit/192d1895161ace817f3f4be09e87fae5dcad9ca3#commitcomment-85639971
Description: Differential-Independence Mixture Ensemble MCMC sampling for Julia

Ja, warte doch nochmal kurz, öhm.


Customize CSV file

Currently I have 2 columns "category" and "customers" but I really want to change the "category" from header column, and change it to "date & time".

The CSV options were recently added.
You may find the options to customize the CSV file here - https://apexcharts.com/docs/options/chart/toolbar/#csv


JAMES-3476 improve documentation

Improve readme and add examples for file based configuration
Add tag name "James" for each file (good to distinguish if you have other non-James dashboards running in the same Grafana server)
Fix elasticsearch version (2 -> 6)
Use unified data source name (DS_JAMES_ES) in all scripts (currently some dashboards like Imap_board and JMAP_board use DS_ELASTICSEARCH instead as datasource name)

I just want to be sure you are aware of an alternative way to collect metrics through promotheus via a /metrics webadmin endpoint.

Actually I'm not aware of that. I know that when I take a random Prometheus dashboard from https://grafana.com/grafana/dashboards?dataSource=prometheus then these dashboards all are are capable to use Prometheus as a data source (so all data flows to Prometheus first and Grafana picks up it from there). So it would be nice if James dashboards would be capable ot that as well but I don't know how to achieve that.

Hello @aasaru
Sorry for the delay, I did a quick sweep and noticed your PR got a bit forgotten. I just merged it. Thanks again for your contribution :)
Can you please help me close this PR though? Thanks!

Hello @aasaru
Sorry for the delay, I did a quick sweep and noticed your PR got a bit forgotten. I just merged it. Thanks again for your contribution :)
Can you please help me close this PR though? Thanks!

No problem. Thanks for merging it!

No problem. Thanks for merging it!


AnimationPlayer now resets on every frame it is called

Godot version
4.2 dev 6
System information
Windows 11
Issue description
Prior to Godot 4.2 dev 6, calling play("animation") on an animation repeatedly simply continues playback of the animation.
In Godot 4.2 dev 6, repeatedly calling a looping animation causes it to restart/reset every time it is called.
func _process(delta): $AnimationPlayer.play("animation_name")
For ex. the above example now restarts at 0 on every frame.
Steps to reproduce

Create a new project in Godot 4.2 dev 6
Add an AnimationPlayer node to your scene, and create a looping animation for a parameter of your choice.
Playing the animation in the editor works as expected
Create a script to call the animation at regular intervals (for ex. every frame)
Press F5 to launch run the project -> now the animation player resets to 0 on every call

Minimal reproduction project
N/A

I'm hit by this bug as well. When I tried to open my project with 4.2 dev 6 I noticed that my pawns aren't properly animating. But when I switch on my older compile, it does properly animating and no longer freeze in their frame 0. Could be a regression somewhere.


"SyntaxError: invalid syntax" error on a line with a ternary conditional operator

I have a Python source file (see attachment) that gives the following error when I try to convert it with f2format v0.4.3 on Ubuntu 18.04 64-bit using Python 3.6.5:
Traceback (most recent call last):   File "/opt/design/python/lib/python3.6/multiprocessing/pool.py", line 119, in worker     result = (True, func(*args, **kwds))   File "/opt/design/python/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar     return list(map(*args))   File "/opt/design/python/lib/python3.6/site-packages/f2format/core.py", line 244, in f2format     text = convert(string, lineno)   File "/opt/design/python/lib/python3.6/site-packages/f2format/core.py", line 185, in convert     val = ast.parse(temp_expr).body[0].value                        # parse AST   File "/opt/design/python/lib/python3.6/ast.py", line 35, in parse     return compile(source, filename, mode, PyCF_ONLY_AST)   File "<unknown>", line 1     address:0{self.axi_address_width // 4 + (1 if self.axi_address_width % 4 != 0 else 0)              ^ SyntaxError: invalid syntax
I suspect it because of the ternary conditional operator in that line of code.
xilinx.py.zip

Still… this is the same grammar f2format currently do not support (c.f. issue#2).

Issue resolved, see #1


ConnectedRouter is re-rendering when Redux store updates

I recently migrated from the deprecated react-router-redux package to connected-react-router in order to successfully update react-redux to v6. The migration went well enough thanks to the detailed docs for this project.
However, now that I am on v6.0.0 of both connected-react-router and react-redux I am noticing some strange things occurring with the ConnectedRouter component.
First issue
As others have already stated, the ConnectedRouter seems to be rendering two times on initial page load.
Second issue
This is the larger one for me, and I'm still trying to figure out why it is happening. Basically, on any update to the state of my redux store, the ConnectedRouter component re-renders, causing unnecessary re-renders of any child components below it.
I set up an example repo that you can clone to get more details. I'll give a brief description here:
Description
I've setup my redux store according to the docs in this repository. Assuming this, here is a basic example using ConnectedRouter:
// App.js

const App = () => (
  <Provider store={store}>
    <ConnectedRouter history={history}>

        <Switch>
          <Route exact path="/" component={Home} />
          <Route path="/about" component={About} />
        </Switch>

    </ConnectedRouter>
  </Provider>
)

The Home component renders a child IncrementCount component which dispatches an action to increment the count value stored in redux store:
// IncrementCount.js

import { increment } from './actions'

const IncrementCount = ({ increment }) => (
  <button onClick={increment}>Increment</button>
)

export default connect(null, { increment })(IncrementCount)

The About component renders a child CountValue component which displays the current count:
// CountValue.js

const CountValue = ({ value }) => (

{value}

)

const mapStateToProps = ({ app }) => ({
  value: app.count
})

export default connect(mapStateToProps)(CountValue)

When incrementing the count on the '/' route, the ConnectedRouter re-renders along with the Nav component (doesn't even connect to the redux store), and the Home component.
This is not the expected behavior.
The expected behavior occurs when replacing ConnectedRouter with BrowserRouter from the react-router-dom library:
// with BrowserRouter

const App = () => (
  <Provider store={store}>
    <BrowserRouter>

        <Switch>
          <Route exact path="/" component={Home} />
          <Route path="/about" component={About} />
        </Switch>

    </ConnectedRouter>
  </Provider>
)

In the code above, incrementing the count value on the '/' route does not cause any unnecessary re-renders to the Nav or Home components.

Same bug here, I tried different ways of rendering the component (prop, children prop, render prop) and it does not have any effect. The issue seems to be linked to connected-react-router.

@jakewies @merildev this workaround works for me:
I found that changing any routes from using component prop to render prop did the trick:

Works

        <Switch>
          <Route path={'login'} render={props => <LoginContainer {...props} />} />
        </Switch>
Renders 2x

        <Switch>
          <Route path={'login'} component={props => <LoginContainer {...props} />} />
        </Switch>

https://github.com/supasate/connected-react-router/issues/193#issuecomment-446212265

@radziksh This has no affect on either of the problems this issue is trying to examine.
What versions of react-redux and connected-react-router are you using? The comment you posted is coming from an issue that references a different version of connected-react-router.

@jakewies i cloned your repository to make some tests and try to fix the problem.
Looks like the problem was indeed inside the ConnectedRouter component.
If you look at the file, in the function createConnectedRouter, it creates the ConnectedRouter extending React.Component.
Then, it creates a Container Component called ConnectedRouterWithContext, that connects it's children with the redux Context.
What happens is that ConnectedRouterWithContext is a functional component, and ConnectedRouter is just a React.Component class. They don't have shouldComponentUpdate implementations.
This is probably what's causing the re-renders. I've forked this repo, switched the ConnectedRouter to a PureComponent, and it seemed to work just like expected (like the BrowserRouter example).

FWIW I'm getting an Invariant Violation: Maximum update depth exceeded. This can happen when a component repeatedly calls setState inside componentWillUpdate or componentDidUpdate. React limits the number of nested updates to prevent infinite loops. error when calling setState in componentDidMount due to the multiple renders

It should be fixed in v6.1.0. Please let me know if it helps and feel free to re-open this issue if the problem still exists.
Thank you all for your contributions!

This issue seem to be exists still on 6.4.0. Any help please?

same issue here with 6.4.0.

Same issue here on 6.4.0.

Hello,
I am facing the same issue on 6.5.2. Is there any update on this?
Thank you

Issue still present on 6.5.2

Yep, same here (also on 6.5.2)

Issue still present on 6.5.2

Still seeing this on 6.8.0

Same issue for me too ..

Still seeing rerenders on 6.8.
Any workaround until an official fix comes around?

same issue for me too..

Still seeing this on 6.9.1

the issue is still existing 6.9.2


How to completely remove aws-sdk-cpp libraries and headers from ubuntu

Hi all,
I have installed on a Ubuntu 16.04 Vm the 1.0.24  version c++ aws-cpp-sdk-core , aws-cpp-sdk-config, aws-cpp-sdk-kinesis. I'd like to completely remove them in order to install the latest version 1.0.76. I tried the command make uninstall but I got the following error message :
make: *** No rule to make target 'uninstall'.  Stop.
I there any other safe way to remove the libraries or can I install the new version on top of the old one?

we need to add an uninstall target.  In the meantime rm -r /usr/include/aws
and rm /usr/lib/libaws-cpp-*
should do the trick.
Sent from my iPhone

On Mar 6, 2017, at 6:59 AM, dk1301<EMAIL_ADDRESS>wrote:
Hi all,
I have installed on a Ubuntu 16.04 Vm the 1.0.24 version c++ aws-cpp-sdk-core , aws-cpp-sdk-config, aws-cpp-sdk-kinesis. I'd like to completely remove them in order to install the latest version 1.0.76. I tried the command make uninstall but I got the following error message :
make: *** No rule to make target 'uninstall'. Stop.
I there any other safe way to remove the libraries or can I install the new version on top of the old one?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Thanks a lot.

Also, rm -rf  /usr/lib/cmake/aws-cpp-* (see #455)


E2 function holoSetPose cannot affect multiple pose parameters

Hello,
So I noticed a while back that the holoAnim extension had E2 functions for pose parameters, and I decided to try and make a model that uses them. I got halfway through making it work when I discovered a very strange problem; it seems that the holoSetPose() function can only affect one pose parameter at a time.
A more technical way of describing the problem is that when holoSetPose() is used to set a pose parameter, all other pose parameters appear to reset to their "zero" position.
At first I thought it was a problem with just my model, so I did a test using the HL2 Jeep/buggy model, which uses pose parameters to move the wheels up and down (among other things).
This is the test E2 I made, as you can see it's pretty simple.
@name Pose Parameter Test
@inputs Left Right
@outputs
@persist
@trigger

if(first()){
    holoCreate(0,entity():pos(),vec(1),entity():angles(),vec(255),"models/buggy.mdl")
}

holoSetPose(0,"vehicle_wheel_fl_height",Left)
holoSetPose(0,"vehicle_wheel_fr_height",Right)

Controlling the first pose parameter works fine. This is what happens when I lowered the first wheel:

This is what happens when I try the other one independently:

This is where the bug occurs. Setting the First, then the Second, does this:

When the sequence is reversed, the same thing happens, but it affects the other wheel:

Examining the model in HLMV shows both wheels being controllable at the same time; This only makes sense as both source vehicles and certain addons such as Simfphys use all necessary pose parameters to make their vehicles work.

I have not tested holoGetPose() to see if it correctly retrieves the pose parameter info.
Thanks for your time and attention, and please let me know if you need any more information relating to the issue.

Wiki page doesn't seem to indicate this should happen, though it does say we should call http://wiki.garrysmod.com/page/Entity/InvalidateBoneCache. I'll give that a try once I have a chance.

Thanks, I really appreciate it!

Do you have your buttons set to toggle? I've just pasted this chip but replaced the Left and Right variables with constants (0 or 1), it's working as expected.

You can tell their toggled because those buttons lights are both on at the same time.

I see. After testing again it I can see the issues, though I don't know of a way to fix it, my only guess would be this has to be called client side and http://wiki.garrysmod.com/page/Entity/InvalidateBoneCache has to be called.

Any progress on this?

Any progress on this?


ratio between two related operations

Hey there! I was hoping to use the package to measure the wall clock time of two related operations in a single benchmark. So that I can report on both their execution times and also return the ratio between them as a result of that single benchmark.
What is the best way to go around this? I know I can make custom metrics but I'm not sure if it's possible to do two time measurements within one benchmark?

Hi,
There's nothing built in for that, but there are a couple of things you could try:

Generate two benchmarks runs (just parameterize the benchmark) and use start/stop measuring to get eg. A/B measurements - possibly ok if you runtime isn't too bad - statistically it'll give you something useful. Then export this in one of the supported formats and calculate the ratio

You could also try with custom measurements using the BenchmarkClock.now and register custom metrics for A/B/ratio as you suggest - that's probably cleaner and what I'd try first. Then you measure the time yourself basically.

I've taken the second approach and have the following extension on Benchmark now:
extension Benchmark {
    @discardableResult
    convenience init?(_ name: String, forward: @escaping (Benchmark) -> (), reverse: @escaping (Benchmark) -> ()) {
        self.init(name, configuration: .init(metrics: [CustomMeasurement.forward, CustomMeasurement.reverse, CustomMeasurement.ratio])) { benchmark in
            let startForward = BenchmarkClock.now
            for _ in benchmark.scaledIterations {
                forward(benchmark)
            }
            let endForward = BenchmarkClock.now
            let startReverse = BenchmarkClock.now
            for _ in benchmark.scaledIterations {
                reverse(benchmark)
            }
            let endReverse = BenchmarkClock.now

            let forward = Int((endForward - startForward).nanoseconds())
            let reverse = Int((endReverse - startReverse).nanoseconds())

            benchmark.measurement(CustomMeasurement.forward, forward)
            benchmark.measurement(CustomMeasurement.reverse, reverse)
            benchmark.measurement(CustomMeasurement.ratio, reverse / forward)
        }
    }
}

Only downside here is that we can only measure integer values so can't turn the ratio into a display of factors of say 1.5 etc. Any ideas and/or tips on that? (or the implementation itself)

How about keeping microseconds for forward/reverse, but nanoseconds for the ratio?
Need to tweak the runtime of the measurement such that microseconds (or more) are suitable though.
Something like:
extension Benchmark {
    @discardableResult
    convenience init?(_ name: String, forward: @escaping (Benchmark) -> (), reverse: @escaping (Benchmark) -> ()) {
        self.init(name, configuration: .init(metrics: [CustomMeasurement.forward, CustomMeasurement.reverse, CustomMeasurement.ratio])) { benchmark in
            let startForward = BenchmarkClock.now
            forward(benchmark)
            let endForward = BenchmarkClock.now
            let startReverse = BenchmarkClock.now
            reverse(benchmark)
            let endReverse = BenchmarkClock.now

            // Calculate times in nanoseconds
            let forwardTimeNanos = (endForward - startForward).nanoseconds()
            let reverseTimeNanos = (endReverse - startReverse).nanoseconds()

            // Convert times to microseconds for storage
            let forwardTimeMicros = forwardTimeNanos / 1000
            let reverseTimeMicros = reverseTimeNanos / 1000

            // Ensure forwardTimeNanos is not zero to avoid division by zero
            guard forwardTimeNanos != 0 else {
                print("Forward time in nanoseconds is zero, cannot compute ratio.")
                return
            }

            // Calculate ratio using nanosecond precision
            let ratio = reverseTimeNanos / forwardTimeNanos

            benchmark.measurement(CustomMeasurement.forward, forwardTimeMicros)
            benchmark.measurement(CustomMeasurement.reverse, reverseTimeMicros)
            benchmark.measurement(CustomMeasurement.ratio, ratio)
        }
    }
}

That still doesn't allow to display a ratio of 1.5 right? So in terms of precision if execution time is 2,000,000 ns vs 3,000,000 the ratio will still be 1.5 regardless of precision

Right, only integer samples - but in that case you would store 2,000(us) vs 3,000(us) with 1,500,000ns for your custom measurements, so you'd see the fractional part in that case.
Another approach is to just export the raw data in one of the output formats and do your own post processing for that specific case.


Revert "CHORE: Update dark mode develop theme"

Reverts nationalarchives/ds-wagtail#887

Issue resolved


fix: make it easier to disable scheduler generation

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign jstrachan
You can assign the PR to them by writing /assign @jstrachan in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jstrachan: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

vault-kapp
350c5b0e879485502c36d14bd74a7e219b094de8
link
/test vault-kapp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

/retest

/approve

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: jstrachan
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [jstrachan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jstrachan: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

vault-kapp
350c5b0e879485502c36d14bd74a7e219b094de8
link
/test vault-kapp

kube
350c5b0e879485502c36d14bd74a7e219b094de8
link
/test kube

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

/retest


[Mario Kart 8] Why 5120x1440 (32:9) resolution has been removed ?

I just updated CEMU graphic pack and realized that 5120x1440 (32:9) resolution for MK8 has been removed. (the "src/MarioKart8/Graphics/rules.txt" file has been edited in the last commit).
Can you explain why ?
I have a Samsung C49HG90 and the 5120x1440 (32:9) resolution was absolutely perfect ! MK8 was displayed on the whole screen. 5120x1440 (32:9) resolution is just perfetc for racing games on that type of monitor.
I'm very sad and disappointed.
Can you get this [Preset] back please ?

Forgot to check for resolutions when copy-pasting the Splatoon ones, whoops
Will fix in a moment

Thank you :)


Bulk Similarity Search

I'd like a way to do bulk similarity search so I can more efficiently query to see if a set of embeddings either already exists or already has a similar match greater than some threshold. I think if the search API supported querying for multiple embeddings it would allow me to build any other needed logic on top easily. It could just return multiple results back in the same order as the input query embeddings. Is something like this already doable? Thanks.

Hi, do you mean batch search? It's already built-in: https://qdrant.tech/documentation/search/#batch-search-api

Yup, that looks like exactly what I need. Sorry I missed it in the docs. Thanks for your help.


Documents Widget

build to api/links first
add in 'document upload' native afterwards
value for mines division as well

Sid wants ability to name "common" document items that a public user would see throughout the engagements/common to each type of engagement.
ie. Project Description

Documents widget have been implemented aside from "uploading" a file which is in a different ticket. closing.


cpeng: round up img size to nearest cacheline

Round the image buffer to the nearest cacheline size (64).
Signed-off-by: Rodrigo Rojo<EMAIL_ADDRESS>

Coverage decreased (-0.008%) to 65.77% when pulling 959aa47c670452e3088b6a9182ad80a69e5e662d on rrojo/cpeng-padding into 30b54790ad92152b8af12cb024d4039498b5e5b2 on master.


Add AllPrimitives and AllPrimitivesWithKeyManager

These new methods on the keyset handle complement Primitives and PrimitivesWithKeyManager, but return all primitives, not just enabled primitives. This can be useful for key implementations to display information about key parameters, but should not be used to create sets for encryption duties, since only enabled keys should be used for encryption.
Done by adding a filter function to the private primitives method, and having Primitives and PrimitivesWithKeyManager use it to filter for enabled keys, while AllPrimitives and AllPrimitivesWithKeyManager don't filter. This keeps the code changes to a minimum. Also: add test to ensure that disabled keys are included in the All methods but excluded in the non-All methods.
Requires disabling the check for enabled keys in primitiveset; should probably change it to allow the adding of other keys through a separate method or something, but I wanted to get some feedback and discussion going before I dove into that, so simply commenting out that check (and the corresponding test) for now.
See discussion in #14 for some of the background on this proposal.

I don't think we want to add this. Sorry.
We do have plans to extend the API of keyset.Handle, similar to what is already possible in Java, see:
https://javadoc.io/static/com.google.crypto.tink/tink/1.13.0/com/google/crypto/tink/KeysetHandle.html
But since this is a bigger redesign, I don't think we will accept pull requests for this until the majority of the re-work is done.

I don't think we want to add this. Sorry.

Okay, I'll keep my ugly hack for now, then.

We do have plans to extend the API of keyset.Handle, similar to what is already possible in Java, see: https://javadoc.io/static/com.google.crypto.tink/tink/1.13.0/com/google/crypto/tink/KeysetHandle.html

Do you mean something like getAt, assuming that it returns keys regardless of their status? I think that would be fine for my use case.


How can I contribute?

I was denied permission when trying to a push a local branch here. 😞

Basically you need to fork the repo, create a branch there and create a Pull request here. Just google "Github contribute", you will find more detailed instructions.

Wondering if who all are interested can collaborate to take this forward.
Let me know if anyone is interested.


[Bug]: iOS AppName

Platform
iOS
Plugin
package_info_plus
Version
3.0.1
Flutter SDK
3.3.5
Steps to reproduce
I have this crash on start app on simulator iPhone 14 pro max iOS 16.0

Code Sample
No response
Logs
[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: type '_InternalLinkedHashMap<Object?, Object?>' is not a subtype of type 'String'
#0      MethodChannelPackageInfo.getAll (package:package_info_plus_platform_interface/method_channel_package_info.dart:15:32)
<asynchronous suspension>
#1      PackageInfo.fromPlatform (package:package_info_plus/package_info_plus.dart:41:26)
<asynchronous suspension>
#2      _initPackageInfo (package:facemassage/main.dart:52:28)
<asynchronous suspension>
#3      main (package:facemassage/main.dart:38:16)
<asynchronous suspension>

Flutter Doctor
[✓] Flutter (Channel stable, 3.3.5, on macOS 12.6 21G115 darwin-arm, locale ru-BY)
    • Flutter version 3.3.5 on channel stable at /Users/kirilltupikov/Projects/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision d9111f6402 (3 days ago), 2022-10-19 12:27:13 -0700
    • Engine revision 3ad69d7be3
    • Dart version 2.18.2
    • DevTools version 2.15.0

[!] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
    • Android SDK at /Users/kirilltupikov/Library/Android/sdk
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.

[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14A400
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] Connected device (4 available)
    • sdk gphone64 arm64 (mobile) • emulator-5554                        • android-arm64  • Android 12 (API 32) (emulator)
    • iPhone 14 Pro Max (mobile)  • AE1BA1C2-048A-4879-A15D-ADC3FBFC53D2 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-16-0 (simulator)
    • macOS (desktop)             • macos                                • darwin-arm64   • macOS 12.6 21G115 darwin-arm
    • Chrome (web)                • chrome                               • web-javascript • Google Chrome 106.0.5249.119

[✓] HTTP Host Availability
    • All required HTTP hosts are available

Checklist before submitting a bug

[X] I Google'd a solution and I couldn't find it
[X] I searched on StackOverflow for a solution and I couldn't find it
[X] I read the README.md file of the plugin
[X] I'm using the latest version of the plugin
[X] All dependencies are up to date with flutter pub upgrade
[X] I did a flutter clean
[X] I tried running the example project

What are the contents of your info.plist? Do you have more than one app name through localization or similar?

BundleName doesn't mutate, only bundleIdentifier changes through .xcconfig files


Fixes

Running the examples/test.js fails with the following error:
ERRO[0000] TypeError: Value is not an object: undefined
        at killPod (file:///Users/paul/Develop/Extensions/xk6-chaos/examples/test.js:12:20(3))
        at file:///Users/paul/Develop/Extensions/xk6-chaos/examples/test.js:5:2(2)
        at native  executor=per-vu-iterations scenario=default source=stacktrace

running (00m00.0s), 0/1 VUs, 1 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs  00m00.0s/10m0s  1/1 iters, 1 per VU
ERRO[0000] handleSummary() failed with error "TypeError: Value is not an object: undefined", falling back to the default summary  source=console

For setup, I created multiple pods, then then modified the script to remove a pod with name containing "new" as shown:

I have made a fixup as this was simply me uppercasing a letter :facepalm:

Thank you @javaducky !! I meant to get around to this but didn't get a chance.


How to assign more CPU on this python script

Is there any possible way to assign more CPU on this script. Honestly, is's super fast on my windows machine. However, I discover that it only takes maybe 60%-70% CPU, so if there's any possible way to make fully use of my CPU. Or is there any other way to improve the speed without losing the quality.

By default it uses 4 CPU threads. You can set the constructor argument cpu_threads to use a different value:
model = WhisperModel(model_path, device="cpu", cpu_threads=6)

By default it uses 4 CPU threads. You can set the constructor argument cpu_threads to use a different value:
model = WhisperModel(model_path, device="cpu", cpu_threads=6)

Thank you for your patient explanation! I have another question, can this project support translation for Whisper? It seems that tasks can be specified as transcribe or translate.

Yes, you can set task="translate" to translate any audio to English:
model.transcribe(..., task="translate")

By default it uses 4 CPU threads. You can set the constructor argument cpu_threads to use a different value:
model = WhisperModel(model_path, device="cpu", cpu_threads=6)

I tried to modify the relevant code as per the instructions you provided using the micro-machines.wav file provided on the Whisper website. The audio duration was approximately 29 seconds. I tested the subtitle recognition using the large-v2 model on a Windows computer CPU. Here are the specific results:

Using 4 cores, the processing time was 1 minute and 38 seconds.
Using 12 cores, the processing time was 1 minute and 35 seconds.
Using 16 cores, the processing time was 1 minute and 29 seconds.

It appears that allocating more CPU cores did not significantly improve the processing time. Should I try using a GPU for testing, or should I attempt to further modify the code to allow for concurrent processing of a single file, even though it may affect the order of processing? Additionally, I noticed that there is a num_workers field in your code, but it seems to refer to the number of tasks being run simultaneously. Therefore, I would like to know what other ways there are to further optimize the processing time. Would specifying the language directly before running the model reduce the time cost of language auto-detection?

How many physical CPU cores does your system have? Note that the processing time will barely reduce when using more threads than the number of physical CPU cores.

Should I try using a GPU for testing

Of course if you have access to a GPU it would speed things quite a lot, especially for the large models.

should I attempt to further modify the code to allow for concurrent processing of a single file, even though it may affect the order of processing? Additionally, I noticed that there is a num_workers field in your code, but it seems to refer to the number of tasks being run simultaneously.

num_workers allows multiple concurrent call to transcribe to run in parallel. You could split the audio into 30s chunks and call transcribe from multiple Python threads but it will not be that much faster than setting cpu_threads (also it will not use the previous context so the transcription will be different).
num_workers is mostly useful when this project is used in a multithreaded webserver and you want multiple transcriptions to run in parallel.

Would specifying the language directly before running the model reduce the time cost of language auto-detection?

Sure, that would help.

Therefore, I would like to know what other ways there are to further optimize the processing time.

There a multiple options that can optimize the processing time at the cost of different, usually worse, transcriptions:

Load the model in INT8 with compute_type="int8"
Use beam_size=1
Disable the temperature fallback temperature=0
Disable the context condition_on_previous_text=False (only useful for audio longer than 30s)

I tried to build this program on a GPU server using T4 graphics card. The result was very shocking. A 25-second video takes over a minute to recognize on my computer, but it only took 9 seconds on the GPU server. It's really powerful. Thanks to the author's code!

Cool, that's a nice speedup!
Let me close this issue as all questions were answered. Feel free to reopen other issues if you have other specific questions.


Unable to derive deserializer for a list type alias

I am unable to creating a deriving list type when deserializing data
The code:

type user = {
  id: int;
  name: string;
  some_bool: bool;
  pet_name: string option;
  some_int64: int64;
  some_int32: int32;
  some_float: float;
  pets: string list;
  pets_array: string array;
}
[@@deriving deserialize]

type users = user list [@@deriving deserialize]

The error message.
File "serde-postgres/test/test_serde.ml", line 33, characters 0-47:
33 | type users = user list [@@deriving deserialize]
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: This expression has type string
       This is not a function; it cannot be applied.

This is the same when I am using an array
File "serde-postgres/test/test_serde.ml", line 33, characters 0-48:
33 | type users = user array [@@deriving deserialize]
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: This expression has type string
       This is not a function; it cannot be applied.

Source code: https://github.com/dbcaml/dbcaml/blob/2e2fab3ea4b8d9b4bd6d7108ea6548f1fce51d4f/serde-postgres/test/test_serde.ml#L20-L33
Reproducing
The simplest way to reproduce is to clone gbcaml:
git clone github.com/dbcaml/dbcaml

and checkout to either branch v2 or sha 2e2fab3ea4b8d9b4bd6d7108ea6548f1fce51d4f.
You should be able to do
opam install . --deps-only

and get all the packages needed.
run the test with:  dune runtest

I added reproducing step, let me know if it don't work.
I am going on vacation tomorrow until Sunday so I am not able to look into this myself until next week


No visible @interface for 'ARTRealtime' declares the selector 'subscribeToStateChanges:'

I'm using the connection code from README.md, but I keep getting the error above.
 ARTRealtime * client = [[ARTRealtime alloc] initWithKey:@"xxxxx"];
 [client subscribeToStateChanges:^(ARTRealtimeConnectionState state) {
     if (state == ARTRealtimeConnected) {
         // you are connected
     }
 }];

What's the recommended way to subscribe to connection changes? Is using EventEmitter ok?
ARTRealtime * client = [[ARTRealtime alloc] initWithKey:@"xxxx"];
[client.eventEmitter on:^(ARTRealtimeConnectionState state) {
    NSLog(@"OK");
}];

Library is installed via podfile:
use_frameworks!

target 'demo-ios' do
        pod "ably", :git => 'https://github.com/ably/ably-ios.git', :commit => '006d61ba4c8c2572e5e1e91d786bee6c6800ad29'
end

@fealebenpae are you able to help with this?

-[ARTRealTime subscribeToStateChanges:] is a private method that the event emitter calls internally. You can use the event emitter to monitor state changes. The "state" attribute on ARTRealtime is Key Value Observing-compatible, so you can use Cocoa's KVO methods to observe state changes as well.

@fealebenpae thanks for the info.
@mattheworiordan It would be nice to have some up-to-date documentation for the iOS lib. There is nothing on docs.ably.io at this point and IMO the README is not sufficient.

Currently it uses the ARTEventEmitter: https://github.com/ably/ably-ios/blob/2398b9580d6aaef45ca91c01a99160cf09bbd4d9/Source/ARTEventEmitter.h#L22


Fix internal-testutils-kmp project sync

It fixes this error during IDE sync:
Failed to resolve platform dependency on jsTest
org.gradle.internal.resolve.ModuleVersionResolveException: Could not resolve project :internal-testutils-kmp.
Required by:
    project :compose:foundation:foundation
Caused by: org.gradle.internal.component.NoMatchingConfigurationSelectionException: No matching variant of project :internal-testutils-kmp was found. The consumer was configured to find a library for use during 'kotlin-api', preferably optimized for non-jvm, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'js', attribute 'org.jetbrains.kotlin.js.compiler' with value 'ir' but:

@eymar, could you look at this PR?


filenode not work properly

Have you read a contributing guide?

[X] I have read CONTRIBUTING.md
[X] I have searched the existing issues and didn't find any that were similar
[x] I have considered creating a pull request with fixes instead of a bug report and want to proceed

Current Behavior
After deploying on Linux public server and filling the generated configuration to the client, the files can't be synchronized properly, the articles and emoji can be synchronized, but after inserting the image files, it will be stuck in sync, I suspect it has something to do with s3 emulator, because I observe the logs of the S3 emulator, and I found that it only receives the records of my two requests with python, and no anytype filenode write log (see picture), hopefully can come out an official more complete tutorial, such as heart. yml fill rule.In addition to this, I also observed that my self-hosted backup node is zero kb on the mobile side and infinity on the computer

Expected Behavior
I want container generated network. filenode in yml is valid and S3 emulator can receive requests correctly or filenode can receive files correctly and request S3 emulator, if it is because of emulator, I would like to provide the option to configure my own S3 bucket

Steps To Reproduce

Clone the official any-sync-dockercompose library to your Debian hard drive
cd to this directory
make start Wait for deployment to complete
Open etc/network.
Use any-sync-tools_0.0. 4_amd64. deb generate profile heart. yml
Use etc/network. Replace heart with the corresponding content in yml. The content in yml (for example, according to the networkId, peerId, port, etc. corresponding to types, and the IP address is also changed to the public IP of the Linux server)
Heart. yml upload to the client, and create a new user, start trying to synchronize operations, such as changing documents, uploading files
And then there's these problems that I'm talking about.

Environment
- OS:Debian
- Version:12

Anything else?
No response

Is this a user error? It's still a bug. The official doesn't seem to have a mature deployment document.

I'm sure it's the filenode problem now, because I stop or even delete the any-filenode container, and the result is the same as when I deploy all the containers normally (make start), and the result is that once the file is uploaded, it gets stuck. This means that filenode doesn't seem to be working at all.Thank you for providing this Docker image so that anytype users have more choices. This is very much in line with the concept of anytype. I am now a loyal fan of anytype. I hope the team can solve this problem when they have time.

hi!
I think the problem is in the internal "docker network".
Servers should be available for the clients on same ip addresses, which were recorded in "network config" (mongodb).
It means, that clients should be deployed in the same docker-compose.

All server are exposed with their respective IP adresses which are set inside the clients config. Or do you mean that the ips inside client and server confs need to be the same. If that is the case I can try that. My load balancer can handle that.

I had the same problem with a MinIO instance, but managed to solve it by adding a file storage limit to the coordinator config, for instance (here 1GB) :
fileLimit:
  limitDefault:<PHONE_NUMBER>
  limitAlphaUsers:<PHONE_NUMBER>0
  limitNightlyUsers:<PHONE_NUMBER>0

I'm added setting "EXTERNAL_LISTEN_HOST", by default it's <IP_ADDRESS>.
you can set LAN ip or PUBLIC internet ip (if you have).


Add backend functions in causal_manager.py to handle dashboard requests

Description
This PR adds request_global_cohort_effects, request_local_instance_effects and request_global_cohort_policy to causal_manager.py. These functions are added so that the python backend (responsibleai) can request global cohort level stats for causal effects and causal policy.
The related PR is https://github.com/microsoft/responsible-ai-toolbox/pull/1552, which requires additional work to add the cohort level filtering to RAIdashboard endpoints.
Checklist

[ ] I have added screenshots above for all UI changes.
[ ] I have added e2e tests for all UI changes.
[ ] Documentation was updated if it was needed.

Codecov Report

Merging #1579 (55232f1) into main (e3d0f24) will decrease coverage by 12.46%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             main    #1579       +/-   ##
===========================================
- Coverage   88.04%   75.58%   -12.47%
===========================================
  Files         109       10       -99
  Lines        5295      258     -5037
===========================================
- Hits         4662      195     -4467
+ Misses        633       63      -570

Flag
Coverage Δ

unittests
75.58% <ø> (-12.47%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...bleai/responsibleai/_tools/causal/causal_result.py

...onsibleai/responsibleai/managers/causal_manager.py

raiwidgets/raiwidgets/fairness_dashboard.py

.../responsibleai/modelanalysis/constants/__init__.py

...ponsibleai/responsibleai/_tools/causal/__init__.py

responsibleai/responsibleai/__init__.py

responsibleai/responsibleai/__version__.py

...eai/responsibleai/managers/data_balance_manager.py

raiwidgets/raiwidgets/explanation_dashboard.py

...nsibleai/responsibleai/_managers/causal_manager.py

... and 89 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e3d0f24...55232f1. Read the comment docs.


Elasticsearch.Net.UnexpectedElasticsearchClientException: Exception of type 'System.OutOfMemoryException' was thrown.

NEST/Elasticsearch.Net version: 6.1
Elasticsearch version: 6.1
Steps to reproduce:
I can't reproduce it manually!
From time to time we get this exception:
Elasticsearch.Net.UnexpectedElasticsearchClientException: Exception of type 'System.OutOfMemoryException' was thrown.
---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at Nest.Json.Linq.JContainer.ReadContentFrom(JsonReader r, JsonLoadSettings settings)
at Nest.Json.Linq.JContainer.ReadTokenFrom(JsonReader reader, JsonLoadSettings options)
at Nest.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings)
at Nest.DictionaryResponseJsonConverterHelpers.ReadServerErrorFirst(JsonReader reader, Error& error, Nullable1& statusCode)     at Nest.ResolvableDictionaryResponseJsonConverter3.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
at Nest.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
at Nest.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Nest.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Nest.Json.JsonSerializer.Deserialize[T](JsonReader reader)
at Nest.InternalSerializer.d__201.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at Elasticsearch.Net.ResponseBuilder.<SetBodyAsync>d__51.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Elasticsearch.Net.ResponseBuilder.d__21.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)     at Elasticsearch.Net.HttpConnection.<RequestAsync>d__141.MoveNext() in C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Connection\HttpConnection.cs:line 259
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Elasticsearch.Net.RequestPipeline.d__741.MoveNext() in C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 478  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)     at Elasticsearch.Net.Transport1.d__151.MoveNext() in C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 120     --- End of inner exception stack trace ---     at Elasticsearch.Net.Transport1.d__151.MoveNext() in C:\Users\russ\source\elasticsearch-net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 138  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at Nest.ElasticClient.<Nest-IHighLevelToLowLevelDispatcher-DispatchAsync>d__2734.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Nest.IndicesPointingToAliasExtensions.d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Rm.Backend.Rpm.JobPerformanceRepository.ElasticHelper.d__7.MoveNext() in C:\Development
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

First thing I suggest is updating NEST and Elasticsearch.Net 6.1 to the latest 6.x package. Currently, this is 6.8.6. Please see if you continue to get OutOfMemoryException.
If you continue to see the exception, please can you provide code for how the client is configured and typical usage pattern of the client. If you are also able to capture the size of the response, that would also help.

@AviramYo33 can you provide additional information about how the client is configured and the request/response made when an OutOfMemoryException is seen?

We’re closing this issue due to inactivity; if you’re still in need of resolution, feel free to re-open with the additional details and we'd be glad to help you out 👍

@russcam this happens to me with ElasticSearch.Net 7.6.1 and NEST 7.6.1. I've checked the Release history over at Github and did not find any fix or enhancement that could help with our issue.
Our client is configured in a class that is singleton. This is how we create our client:

	   if (_client == null)
            {
                IEnumerable<Uri> uris = _dbSettings.ElasticSearch.Urls.Select(x => new Uri(x));

                //----------------------------
                //  Mapping for 7.x
                //----------------------------
                var settings = new ConnectionSettings(uris.First())
                  .DefaultMappingFor<TheDoc>(m => m.IndexName("theIndexName").RelationName("parent"))
              ;

                if (!_dbSettings.ElasticSearch.Urls.Contains("localhost"))
                {
                    settings.BasicAuthentication(_dbSettings.ElasticSearch.Username, _dbSettings.ElasticSearch.Password);
                }

                _client = new ElasticClient(settings);

                var indexResponse = _client.Indices.Create("theIndexName", c => c
                    .Map<TheDoc>(m => m.AutoMap())
                    .Settings(s => s
                        // ToDo: settings for dev only.
                        .NumberOfReplicas(_dbSettings.ElasticSearch.DoNotCreateReplicas ? 0 : 1)
                        )
                    );

            }

            return _client;

This is how we build our query, it might help you troubleshoot and see where the issue is. Perhaps we're doing something wrong in the query?

			if (pageSize > 1000) pageSize = 1000;

            var response = await _elasticManager.GetClient().SearchAsync<TheDoc>(s =>
            {
                var sd = s.From(pageNumber)
                .Size(pageSize)
                .Query(q =>
                {
                    QueryContainer qc = null;

                    if (ids != null && ids.Any())
                    {
                        // Serch by ids
                        qc = q.Terms(t => t.Field(f => f.Id).Terms(ids));
                    }
                    else if (string.IsNullOrWhiteSpace(keyword))
                    {
                        // Search by the (default) keyword.
                        qc = q.MatchPhrasePrefix(m => m.Field(f => f.Name).Query(keyword).Boost(1.5));
                    }
                    else {
                        // Search by the keyword.
                        qc = q.QueryString(qs => qs.Name("TheDoc_query")
                            .Boost(1.1)
                            .Fields(fs => fs.Field(f => f.fldX).Field(f => f.fldY).Field(f => f.fldZ).Field(f => f.fldA))
                            .Query($"{keyword} {keyword.Replace(",", " ")}")
                            .DefaultOperator(Operator.Or)
                            .MaximumDeterminizedStates(3)
                            .Escape()
                            .Fuzziness(Fuzziness.Auto)
                            .FuzzyPrefixLength(2)
                            .FuzzyMaxExpansions(3)
                            .MinimumShouldMatch(2)
                            .Lenient()
                        //.AutoGenerateSynonymsPhraseQuery(false)
                        );
                    }

                    qc = qc && q.Match(m => m.Field(f => f.fld_1).Query(filterA);

                    if (!string.IsNullOrWhiteSpace(filter_1))
                    {
                        qc = qc && q.MatchPhrasePrefix(m => m.Field(f => f.fldX).Query(filter_1).Boost(0.9));
                    }
                    if (!string.IsNullOrWhiteSpace(filter_2))
                    {
                        qc = qc && q.MatchPhrasePrefix(m => m.Field(f => f.fldY).Query(filter_2).Boost(0.9));
                    }

                    return qc;
                }); // end Query

                return sd;
            });

@russcam this happens to me with ElasticSearch.Net 7.6.1 and NEST 7.6.1. I've checked the Release history over at Github and did not find any fix or enhancement that could help with our issue.
Our client is configured in a class that is singleton. This is how we create our client:

	   if (_client == null)
            {
                IEnumerable<Uri> uris = _dbSettings.ElasticSearch.Urls.Select(x => new Uri(x));

                //----------------------------
                //  Mapping for 7.x
                //----------------------------
                var settings = new ConnectionSettings(uris.First())
                  .DefaultMappingFor<TheDoc>(m => m.IndexName("theIndexName").RelationName("parent"))
              ;

                if (!_dbSettings.ElasticSearch.Urls.Contains("localhost"))
                {
                    settings.BasicAuthentication(_dbSettings.ElasticSearch.Username, _dbSettings.ElasticSearch.Password);
                }

                _client = new ElasticClient(settings);

                var indexResponse = _client.Indices.Create("theIndexName", c => c
                    .Map<TheDoc>(m => m.AutoMap())
                    .Settings(s => s
                        // ToDo: settings for dev only.
                        .NumberOfReplicas(_dbSettings.ElasticSearch.DoNotCreateReplicas ? 0 : 1)
                        )
                    );

            }

            return _client;

This is how we build our query, it might help you troubleshoot and see where the issue is. Perhaps we're doing something wrong in the query?

			if (pageSize > 1000) pageSize = 1000;

            var response = await _elasticManager.GetClient().SearchAsync<TheDoc>(s =>
            {
                var sd = s.From(pageNumber)
                .Size(pageSize)
                .Query(q =>
                {
                    QueryContainer qc = null;

                    if (ids != null && ids.Any())
                    {
                        // Serch by ids
                        qc = q.Terms(t => t.Field(f => f.Id).Terms(ids));
                    }
                    else if (string.IsNullOrWhiteSpace(keyword))
                    {
                        // Search by the (default) keyword.
                        qc = q.MatchPhrasePrefix(m => m.Field(f => f.Name).Query(keyword).Boost(1.5));
                    }
                    else {
                        // Search by the keyword.
                        qc = q.QueryString(qs => qs.Name("TheDoc_query")
                            .Boost(1.1)
                            .Fields(fs => fs.Field(f => f.fldX).Field(f => f.fldY).Field(f => f.fldZ).Field(f => f.fldA))
                            .Query($"{keyword} {keyword.Replace(",", " ")}")
                            .DefaultOperator(Operator.Or)
                            .MaximumDeterminizedStates(3)
                            .Escape()
                            .Fuzziness(Fuzziness.Auto)
                            .FuzzyPrefixLength(2)
                            .FuzzyMaxExpansions(3)
                            .MinimumShouldMatch(2)
                            .Lenient()
                        //.AutoGenerateSynonymsPhraseQuery(false)
                        );
                    }

                    qc = qc && q.Match(m => m.Field(f => f.fld_1).Query(filterA);

                    if (!string.IsNullOrWhiteSpace(filter_1))
                    {
                        qc = qc && q.MatchPhrasePrefix(m => m.Field(f => f.fldX).Query(filter_1).Boost(0.9));
                    }
                    if (!string.IsNullOrWhiteSpace(filter_2))
                    {
                        qc = qc && q.MatchPhrasePrefix(m => m.Field(f => f.fldY).Query(filter_2).Boost(0.9));
                    }

                    return qc;
                }); // end Query

                return sd;
            });

@vonbv We identified an issue with EventCounters which in certain circumstances seemed to be causing an OOM exception. This was addressed in #5088 and released in 7.10.0 of NEST, As a starting point, could you try your code using the latest release of the client and see if that helps?

@vonbv We identified an issue with EventCounters which in certain circumstances seemed to be causing an OOM exception. This was addressed in #5088 and released in 7.10.0 of NEST, As a starting point, could you try your code using the latest release of the client and see if that helps?

@stevejgordon thanks for the quick response! Good job on finding the issue and fixing it. And thank you for pointing out which issue was that.

@stevejgordon thanks for the quick response! Good job on finding the issue and fixing it. And thank you for pointing out which issue was that.

It seems to work. I stress-tested our searches and the server has not thrown an OOM exception yet. Thank you Elasticsearch.Net team!

It seems to work. I stress-tested our searches and the server has not thrown an OOM exception yet. Thank you Elasticsearch.Net team!

Glad to hear that, thanks for confirming @vonbv

Glad to hear that, thanks for confirming @vonbv

Hello ElasticSearch.Net Team, we are experiencing the OOM issue again. We are still on 7.10.1 and we have the same configuration and implementation as I shared earlier in this thread. Below is the stack trace we got.
An error has occured: Elasticsearch.Net.UnexpectedElasticsearchClientException: Exception of type 'System.OutOfMemoryException' was thrown. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1.Rent(Int32 minimumLength)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.DeserializeAsync[T](Stream stream, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.ResponseBuilder.SetBodyAsync[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType, CancellationToken cancellationToken)
   at Elasticsearch.Net.ResponseBuilder.ToResponseAsync[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType, CancellationToken cancellationToken)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   at Nest.ElasticClient.SearchAsync[TDocument](ISearchRequest request, CancellationToken ct)
   at Nest.ElasticClient.SearchAsync[TDocument](Func`2 selector, CancellationToken ct)
   at OUR_SEARCH_METHOD

Hello ElasticSearch.Net Team, we are experiencing the OOM issue again. We are still on 7.10.1 and we have the same configuration and implementation as I shared earlier in this thread. Below is the stack trace we got.
An error has occured: Elasticsearch.Net.UnexpectedElasticsearchClientException: Exception of type 'System.OutOfMemoryException' was thrown. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1.Rent(Int32 minimumLength)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.DeserializeAsync[T](Stream stream, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.ResponseBuilder.SetBodyAsync[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType, CancellationToken cancellationToken)
   at Elasticsearch.Net.ResponseBuilder.ToResponseAsync[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType, CancellationToken cancellationToken)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   at Nest.ElasticClient.SearchAsync[TDocument](ISearchRequest request, CancellationToken ct)
   at Nest.ElasticClient.SearchAsync[TDocument](Func`2 selector, CancellationToken ct)
   at OUR_SEARCH_METHOD

Hi @stevejgordon just wondering if you guys are able to replicate my issue.

Hi @stevejgordon just wondering if you guys are able to replicate my issue.

@vonbv Apologies but the notification got lost as this issue is marked closed. I'll reopen so we can investigate this with you. Is the OOM exception happening often and repeatable in a small repro? Are you running on a memory constrained environment / container perhaps? Are you able to grab a profile or any memory dumps we could potentially investigate?

@vonbv Apologies but the notification got lost as this issue is marked closed. I'll reopen so we can investigate this with you. Is the OOM exception happening often and repeatable in a small repro? Are you running on a memory constrained environment / container perhaps? Are you able to grab a profile or any memory dumps we could potentially investigate?

repeatable in a small repro

You mean if I can take out the part of our code that throws the exception and put it in small dedicated project?

Are you running on a memory constrained environment / container perhaps?

1 GB memory and 30 GB storage

Are you able to grab a profile or any memory dumps we could potentially investigate?
I'll collect the information needed.

repeatable in a small repro

You mean if I can take out the part of our code that throws the exception and put it in small dedicated project?

Are you running on a memory constrained environment / container perhaps?

1 GB memory and 30 GB storage

Are you able to grab a profile or any memory dumps we could potentially investigate?
I'll collect the information needed.

You mean if I can take out the part of our code that throws the exception and put it in small dedicated project?

I'm interested to know if that query specifically seems to trigger the OOM. If so, does it occur if the same query is used in a simpler app? Is the exception always thrown in the same place? How long is the app typically running before you experience the exception?

1 GB memory and 30 GB storage

How intensive is the application? Currently it sounds like there's a memory leak somewhere in the app. We'd need to narrow down if it's specifically related to NEST.

I'll collect the information needed.

Thanks! I think this will give us a starting point. Also, what environment is this running on specifically?

You mean if I can take out the part of our code that throws the exception and put it in small dedicated project?

I'm interested to know if that query specifically seems to trigger the OOM. If so, does it occur if the same query is used in a simpler app? Is the exception always thrown in the same place? How long is the app typically running before you experience the exception?

1 GB memory and 30 GB storage

How intensive is the application? Currently it sounds like there's a memory leak somewhere in the app. We'd need to narrow down if it's specifically related to NEST.

I'll collect the information needed.

Thanks! I think this will give us a starting point. Also, what environment is this running on specifically?

@vonbv
In addition to @stevejgordon's query the following seems to be worth a try.
By default we use a recyclable memory stream implementation that tries to reuse buffers reasonably aggressively.
Since you are on a memory constraint environment perhaps the following works better more predictable by configuring the client with the following.
settings
    .MemoryStreamFactory(Elasticsearch.Net.MemoryStreamFactory.Default)

This implementation of IMemoryStreamFactory creates new MemoryStream() all the time. The overhead in terms of performance should be limited.

@vonbv
In addition to @stevejgordon's query the following seems to be worth a try.
By default we use a recyclable memory stream implementation that tries to reuse buffers reasonably aggressively.
Since you are on a memory constraint environment perhaps the following works better more predictable by configuring the client with the following.
settings
    .MemoryStreamFactory(Elasticsearch.Net.MemoryStreamFactory.Default)

This implementation of IMemoryStreamFactory creates new MemoryStream() all the time. The overhead in terms of performance should be limited.

@stevejgordon

does it occur if the same query is used in a simpler app?

The query is in a service used by our API. It will take time to reference that service in a new simple API and deploy it and generate the data. I will do this over the weekend though as it'll be interesting to know how it performs in that environment.

Is the exception always thrown in the same place?

Yes. It's always in the same API that implements the service that runs the specific query, which I shared in this thread. No other API runs the query. However, our other queries are not used as much as the one with issue.

How long is the app typically running before you experience the exception?

I will have to monitor this but based on previous instances, after a few hours of restarting the server, the OOM happens.

How intensive is the application? Currently it sounds like there's a memory leak somewhere in the app. We'd need to narrow down if it's specifically related to NEST

We are still in the development phase so it's not intensive. If there's a memory leak somewhere then I'm thinking why are the other searches works. So what I do is when this OOM occurs on the specified query, I call our other queries and those work just fine. The other queries that are running fine are using other type of documents.

Also, what environment is this running on specifically?

Running on Azure S2

200 total ACU
3.5 GB memory
Running on Elastic.co
1 GB memory
30 GB storage
version 7.6.2
Built with:
.Net Core 2.1
ElasticSearch.Net 7.10.1
NEST 7.10.1

@Mpdreamz I will try your suggestion

@stevejgordon

does it occur if the same query is used in a simpler app?

The query is in a service used by our API. It will take time to reference that service in a new simple API and deploy it and generate the data. I will do this over the weekend though as it'll be interesting to know how it performs in that environment.

Is the exception always thrown in the same place?

Yes. It's always in the same API that implements the service that runs the specific query, which I shared in this thread. No other API runs the query. However, our other queries are not used as much as the one with issue.

How long is the app typically running before you experience the exception?

I will have to monitor this but based on previous instances, after a few hours of restarting the server, the OOM happens.

How intensive is the application? Currently it sounds like there's a memory leak somewhere in the app. We'd need to narrow down if it's specifically related to NEST

We are still in the development phase so it's not intensive. If there's a memory leak somewhere then I'm thinking why are the other searches works. So what I do is when this OOM occurs on the specified query, I call our other queries and those work just fine. The other queries that are running fine are using other type of documents.

Also, what environment is this running on specifically?

Running on Azure S2

200 total ACU
3.5 GB memory
Running on Elastic.co
1 GB memory
30 GB storage
version 7.6.2
Built with:
.Net Core 2.1
ElasticSearch.Net 7.10.1
NEST 7.10.1

@Mpdreamz I will try your suggestion

Thanks @vonbv. It may also relate to this issue in .NET runtime. I know Microsoft did a lot of work on the GC to optimise it for container and constrained memory environments in 3.0 and 3.1. If possible, it may be worth migrating this to the 3.1 runtime to see if that helps?
Do you have App Insights enabled to get a view on the memory usage over time?

Thanks @vonbv. It may also relate to this issue in .NET runtime. I know Microsoft did a lot of work on the GC to optimise it for container and constrained memory environments in 3.0 and 3.1. If possible, it may be worth migrating this to the 3.1 runtime to see if that helps?
Do you have App Insights enabled to get a view on the memory usage over time?

We're at the point that upgrading will take a considerable amount of time especially that several components will be possibly hit. But with that issue you linked to, it seems that's our best option to fix our issue.
Here's a screenshot of the memory usage covering the times when the OOM exception from ES.NET occurs. By digging through all the OOM exceptions, it shows that it resolves after sometime and then come back again.

We're at the point that upgrading will take a considerable amount of time especially that several components will be possibly hit. But with that issue you linked to, it seems that's our best option to fix our issue.
Here's a screenshot of the memory usage covering the times when the OOM exception from ES.NET occurs. By digging through all the OOM exceptions, it shows that it resolves after sometime and then come back again.

Hi @vonbv. I wanted to check in and see if you managed to try upgrading and whether that helped?

I thought I replied to you already, sorry about that. It did!  Thanks @stevejgordon

@vonbv can you explain what you did to fix this issue? I have the same with NEST 7.10.0 on .NET6

@A-Ryouta I believe we determined it was linked to a .NET Core 3.0 bug and resolved when moving to .NET Core 3.1. Therefore, on .NET 6, your issue is likely caused by something different. I'd suggest profiling your application or collecting a dump when the OOM exception occurs to determine which types are consuming the heap. You may have a memory leak in your application code.


Sorting breaks whenever I place an RTE anywhere within same DocType

Hi,
I understand that there are several issues mentioned already that are similar to this one (e.g. #282, #322), however I have it happen whenever I have a Rich Text Editor datatype as part of the same Document Type. So, the RTE is not part of my Archetype datatype, it just happens to be in the same Document Type (not even on the same tab).
To put it another way, to get sorting fieldsets working again I don't need to change my Archetype datatype at all, all I have to do is remove any RTE datatypes from my doctype.

I am using Umbraco 7.4.3 and Archetype 1.13
I have tried renaming my archetype alias (as per #322) however that did not fix the problem.
I have also updated underscore.js to the latest version (as per #282) to see whether that would work, but alas it did not. (Although the error message I get isn't the same as #282 either).
Just to clarify even further, I am using several (different and the same) archetype datatypes throughout the project. All Archetype sorting works fine on pages that use doctypes containing no RTE datatypes.
Any Ideas?

Here's a workaround: https://github.com/imulus/Archetype/pull/364

Thanks Nicholas. Got it working.

Thanks for the work-around!


Be able to generate build info content from nodes that are not marked as build in the graph

PR on top of: https://github.com/conan-io/conan-extensions/pull/75
Add --add-cached-deps as argument to be able to generate BuildInfo information for nodes that are not marked as Build
This covers the case, for example, that we want to do a conan install on an application that uses dependencies already cached by conan and  we want to use the BuildInfo to make a security scan of the used libraries, for example

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 2 committers have signed the CLA.:x: memsharded:x: czoidoYou have signed the CLA already but the status is still pending? Let us recheck it.


[docs-infra] Fix search ranking when no productId

I broke this when I refactored the productId logic. The problem is that we were setting productId:null as an optional filter in the homepage. Now, Algolia is configured with
      ranking: [
        "words",
        "filters",
        "typo",
        "attribute",
        "proximity",
        "exact",
        "custom",
      ],

https://www.algolia.com/doc/guides/managing-results/relevance-overview/in-depth/ranking-criteria/#filters
so it would rank first all the pages without a productId, which is right now the MUI X migration page as we have to refactor a bit these pages to get the productId from the markdown.

This solves the migration pages appearing first 👍 But still, do we really want the Toolpad "text field" to appear first on the home page search? I feel like we should rank the results on the home page based on the popularity of the product - meaning Core, X, Toolpad. We can address this separately.

But still, do we really want the Toolpad "text field" to appear first on the home page search? I feel like we should rank the results on the home page based on the popularity of the product - meaning Core, X, Toolpad. We can address this separately.

@mnajdova Yeah, but this would be another problem, different from the one this PR fixes. In this case, Toolpad ranks first because it has an exact word match. Then if you search for "text field", Toolpad still ranks first because it comes first when you sort the URLs in an alphabetical descending order.

@siriwatknp I think we could document this in https://www.notion.so/mui-org/Docs-of-the-docs-infra-7ee89042350745f29803413b28d5b60f?pvs=4#15e70fafa7754b2aa6be5bfca82978ef. I have added a bit of content.
I have added a pageRank.
        // % from https://docs.google.com/spreadsheets/d/1x9uTSMlu4xug2SfHrvr5amZ2HokYBoQaSozHmRRdVqw/edit#gid=907743010
        const baselinePopulartyProductId = {
          "material-ui": 83,
          "mui-x": 12,
          "system": 4,
          "base-ui": 2,
          "joy-ui": 2,
          toolpad: 1,
        };

        const pageRank =
          // @ts-ignore
          baselinePopulartyProductId[productId] ||
          // @ts-ignore
          baselinePopulartyProductId[productCategoryId] ||
          0;

To solve the ranking between products on the homepage.

I have also used the popularity for the ranking between component page
        } else if (url.pathname.startsWith("/material-ui/api/")) {
          popularity = -1;

to solve the ranking between API pages and demo pages:

However, there are still a lot of broken things.  It relies on the lvl0 field: https://github.com/algolia/docsearch/blob/2aef60fcfdecdae0cce2897b6213efa62f7dee80/packages/docsearch-react/src/utils/removeHighlightTags.ts#L19 but this is the same between Material UI, Joy UI, Base UI, so groups are nor correctly formed 🙈

Anyway, this is getting way outside my initial objective, this PR is a quick win, fixing an obvious mistake 😁


Filtering in Relink OrgIdenity redirects to index (CO-1779)

Added in the form the link id and the coperson id
In the controller i check if there is a link id available. If this is the case, i assemble the correct url and redirect

Fixed in 681dea2 (see CO-1779).


added iconId to forecasts with code to identify weather

apologies for reformatting the file. I've only added iconId field.

@tptodorov if you can fix the formatting I will be happy to accept 😄

@tptodorov I missed this PR until recently. Could you redo it without the formatting changes against the latest code?

@tptodorov feel free to reopen this when you're able to get merge issues and formatting resolved


nodejs: Shell initialization blocks user abort

During shell initialization using the nodejs subsystem, the process cannot immediately be aborted by the user using e.g. Ctrl + C. It would be beneficial if this was possible to provide a faster exit hatch in case of a broken configuration, or in the case where a directory is entered using direnv and the user does not wish to wait for the shell initialization to complete.
Tagged as nodejs as this was the subsystem tested on, but the request may be general.

There is a new build system i created for node. Unfortunately it's PR is not merged yet.
you can test if your project runs better with it.

update your flake dream2nix input url:

inputs.dream2nix.url = "github:hsjobeki/dream2nix?ref=nodejs/strict-builder-draft";

Add the strict-builder to your projects.toml

# file: projects.toml
# ...
builder = "strict-builder"
# ...

It should produce a more reliable devShell and can be aborted and resumed at any point.


Change name export

How to change export name ? the default is exports.xls

You can read in the documentation about $fileName

string|array $fileName is a name for file name to export or import. Multiple file name only use for import mode, not work if you use the export mode

So, you can set the name excel you want to export.
I hope this help your problem.
I will close this issue.

thankss!!


No Content visible, error messages

Hello
I'm new to mailslurper and downloaded it for the first time yesterday. You have created a great solution to my problem, but unfortunately it does not work. I just unpacked the ZIP, added the config-file (was missing yesterday -> i have seen you have fixed this already) and startet the EXE. I have to say, i'm german speaking and my system is german (i say this because of datetime formatting).
First i had problems with storing E-Mails (it could not convert the dateSent value from nvarchar to datetime). I tried with sqlite and then with my local MS SQL Database. Here is my config file:
{
"wwwAddress": "<IP_ADDRESS>",
"wwwPort": 8088,
"serviceAddress": "<IP_ADDRESS>",
"servicePort": 8888,
"smtpAddress": "<IP_ADDRESS>",
"smtpPort": 25,
"dbEngine": "MSSQL",
"dbHost": "localhost",
"dbPort": 1433,
"dbDatabase": "Mailslurper",
"dbUserName": "mailslurperuser",
"dbPassword": "password",
"maxWorkers": 1000
}
The MailStore problem i could solve by changing the db-users default language to english. So, this is no more a problem.
But i still have the problem that no content is shown when i start the web frontend. The problem also occours with empty, complete new Database. Both, SQLite and MSSQL, it's always the same behavior.
I have attached a file with the response from the service:
JSON_Answer_from_Server.txt
The JSON respons well, but there are errors on page:

I can not even go to the admin section:

It would be great if you could help me, because i would love this software, if it works.
Thanks a lot and have a great day!
Best wishes, Rolf

Hi Rolf,
I'm sorry to hear the troubles you are having with MailSlurper. When I get a few minutes I'll take a closer look at the data you sent me to see if we can't figure out what may be happening. Thanks.

Can you do me a favor? Can you capture two things and paste them here for me? I'd like to see your terminal log (the MailSlurper log) and you're browser's console log (if there is one). This might help me have a more complete picture of what may be happening.

hi!
I have the same problem!! (version 1.12.0)
my browser log:

Ah, this appears to be a cross-origin complaint. Interesting, as I have code to allow requests to the service tier from the address of the admin tier. What browser is this?

Firefox

Try changing the address for both wwwAddress and serviceAddress to localhost and let me know if that makes a difference please.

mmm no.
But in the first access to mailslurper, shows this error:

That error hints to something wrong with the response. Can you paste a copy of the response Firefox got to generate that error?

So let's try another thing. I've published a minor build release 1.12.1 which allow the service tier to accept connections from any source, instead of restricting it. Please update and give it a try. Thank you for your patience. https://github.com/mailslurper/mailslurper/releases/tag/1.12.1

With the new release the CORS problem was solved. But i still have the problem "There was a problem performing your search"..
This is in /admin:

and home:

yeap, it's works!
But in the first access to mailslurper, still have the problem with JSON, that is solved refreshing web page.
Thanks for help me!

Absolutely! @Nyffeneggerr I haven't heard from you, but I suspect this too might solve your issue. I'm going to close this, but if you continue to experience issues, please open another ticket.

Hey, thanks a lot for your responses. I'm very sorry for not writting something, but i was in holidays. I will try what your wrote and give a short feedback. Thanks a lot and have a good day!

Wow, it looks great. Everything seems to work with latest version 01.12.1. Even with Firefox. I really appreciate your effort and work! Thanks a lot!

@Nyffeneggerr No problem. Glad it's working for you now.