Firestore cleanup signals

This PR will change the following

Break out the errors in Firestore tasks to a per task error
Change the firestore.gd file to accept the different signals from the task errors
Remove null from the delete document function since it was throwing a random error

The idea is that breaking out the signals and functions will allow us to have better control over the data that is sent and not try to force it into a single function. This PR has been built so that the end user should not have to update their code at all, but the behind the scenes functions work better.
This PR will close the following:
close #166

Firestore tester results:
Login with email and password has worked
STARTING FIRESTORE TESTS
Connecting to collection 'Firebasetester'
Trying to get 'Document1
Document got successfully
Trying to print contents of Document1
doc_name: Document1,
doc_fields: {name:Document1},
create_time: 2021-02-27T18:55:18.523129Z
Trying to add a document
Document added successfully
Trying to get 'Document2
Document got successfully
Trying to print contents of Document2
doc_name: Document2,
doc_fields: {active:true, name:Document2},
create_time: 2021-03-29T15:28:21.812096Z
Trying to update Document2
Document Updated successfully
Trying to get 'Document2
Document got successfully
Trying to print contents of Document2
doc_name: Document2,
doc_fields: {active:true, name:Document2, updated:true},
create_time: 2021-03-29T15:28:21.812096Z
Trying to delete Doucment2
Document deleted successfully
Running Firestore Query
[{document:{createTime:2021-02-27T19:52:08.718724Z, fields:{points:{integerValue:10}}, name:projects/roundtable-5c241/databases/(default)/documents/Firebasetester/QueryDoc2, updateTime:2021-02-27T19:59:12.261640Z}, readTime:2021-03-29T15:28:23.583433Z}, {document:{createTime:2021-02-27T19:52:17.602644Z, fields:{points:{integerValue:6}}, name:projects/roundtable-5c241/databases/(default)/documents/Firebasetester/QueryDoc3, updateTime:2021-02-27T19:59:17.958287Z}, readTime:2021-03-29T15:28:23.583433Z}]
FINISHED FIRESTORE TESTS


chore(repo): Update lockfile

Forgot to reset the package-lock after adding in stuff for vue, etc

This will be handled via #149


Merge fix

Description of changes: Removing extra content accidentally added in previous PR.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Recreating this PR with a fresh branch.


May 2024 Release

Issues
Fixes #
Reviewer Suggestions
Where should the reviewer start?
Additional Comments
Anything else?

:tada: This PR is included in version 1.2.35 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Add support for multi-frame messages

hi,
how is it going with this? any active developement?

Haven't had a chance to get started yet. Still planning the implementation.
Do you have any input and/or suggestions? Perhaps some use cases?

Hi @zacnelson,
I'm about to use your iso-tp implementation and I need multi frame support.
I have to play around with the single frame implementation you did so far to  get familiar with your code but it looks really good!
use case:
One of our use cases is the transfer of firmware data to a CAN device.
My assumption is that we do not have to comply with the whole iso-tp standard. From my perspective it looks OK to hardcode values:

Block Size (BS)
Seperation Time (ST)

do you think the TP-layer should re-assemble the whole 4095byte payload? In case of an firmware update, this would not be nearly enough to handle the whole process anyway.
input/suggestions:
I implemented a very primitive sort of single frame handling in my code just to demonstrate the basic functionality of operation and used unions with structs and bitfields to describe the 8 data bytes in a CAN frame:

I really liked the code resulting from it.

what do you think? might it worth to think about using this representation?

maybe we can get some inspiration for multiframe support from this repo:
https://github.com/altelch/iso-tp

Nice find with the Arduino iso-tp repo. The only issue I see with that implementation is the blocking while loop while they wait for the subsequent multi-frames to come in. See here: https://github.com/altelch/iso-tp/blob/master/iso-tp.cpp#L407
We'll need to have an implementation that doesn't block the rest of the CAN parsing going on - as used in https://github.com/openxc/vi-firmware

We've implemented it with a fairly simple state machine in C# and C++... I've been looking for free time to write up a C version for a while but haven't had time yet.  This week might be good though. ;)

@hackrid @peplin I've implemented basic multi-frame capability in the multi-frame branch. Have a look and let me know what you think.

Nice! I think it could use a few unit tests to verify the behavior. If you open a pull request from that branch we can iterate on it there.

Absolutely. I started messing with the tests yesterday. I'll push the latest and create a pull request.

Awesome!  Looks pretty good.  Why the 256byte max frame size?

@notsig11 I don't think we have enough memory to allocate the full 4k for every single diagnostic response for the OpenXC VI. Especially if you queue up a bunch at once. Definitely something to address next.

@zacnelson Right but you are only allocating payload_length bytes not the full 4k unless I missed something.  I don't think it's necessary.
If that malloc fails the FC frame will never get sent and the sender can try again after the FC timer expires and we can try again to allocate payload_length bytes for the receive_buffer.

For combining all the received frames, you're correct - it only allocates the payload length. But then when we return the completed IsoTpMessage, we have a struct defined with a payload allocation of 256 bytes. See here: https://github.com/openxc/isotp-c/blob/multi-frame/src/isotp/isotp_types.h#L46
Then in the library above this, uds-c, we also have a 256 byte payload struct for the DiagnosticResponse.

Ah!  I figured I was missing something there.  :)

Pull request has been merged for allow multi-frame responses. Right now limited to 256 byte response. Closing this issue for now. @hackrid take a look when you get a chance.

The current version supports only single frame ISO-TP messages. This is fine for OBD-II diagnostic messages, for example, but this library needs some additional work before it can support sending larger messages.

Is your README out of date?

Partially. multi-frame read is in there. We ran into a memory issue on the VIs with the larger messages so multi-frame write is not fully implemented yet.


problem on faster / slower options for tempo

Observed that after " marimbabot play faster" and "marimbabot play slower" commands, tempo was not updated properly. Robot speed has not changed and same tempo value is printed from behavior node instead of updated value.

One guess is that the command is not detected, in the current version, the command is just a string match, if the template is "play faster", then if you say "plays faster", it will not be detected.

The issue was that the vision model detects tempo as "\tempo 4=120" but the behavior expects "\tempo 4 = 120". The parser works for both versions, but we should of course go with the syntax of our model.
I fixed the issue in the behavior in my most recent PR (241) which also adds volume adjustments to the behavior 👍

Is this related to https://github.com/UHHRobotics22-23/marimbabot/issues/258 ?

It worked the last time I tested it.


[Bug]:  Cannot find module when running using ESM

Bug expectation
I am trying to run a very basic script to load a test page using Puppeteer.
However, my script is crashing on load and i'm getting the error: MODULE_NOT_FOUND
This error only occurs when I kick off my script using "-r esm" in my script.
If I run node using the "--experimental-modules" flag instead, then it runs fine.
It runs fine if I kick it off using babel-node as well.
Why would "-r esm" and the "--experimental-modules" flag be giving different results?
This is my package.json:
{
  "name": "PuppeteerTest",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "scripts": {
    "clean": "rm -rf build && mkdir build",
    "build": "babel -d ./build ./src -s --copy-files",
    "startesm": "yarn clean && yarn run build && NODE_ENV=production node -r esm ./build/server.js",
    "starttest": "babel-node src/server.js",
    "startexperimental": "yarn clean && yarn run build && NODE_ENV=production node --experimental-modules  ./build/server.js"
  },
  "dependencies": {
    "esm": "^3.2.25",
    "express": "^4.18.1",
    "handlebars": "^4.7.7",
    "puppeteer": "^19.5.2"
  },
  "devDependencies": {
    "@babel/cli": "^7.15.4",
    "@babel/core": "^7.15.5",
    "@babel/node": "^7.15.4",
    "@babel/plugin-transform-runtime": "^7.15.0",
    "@babel/preset-env": "^7.15.4"
  },
  "babel": {
    "presets": [
      "@babel/preset-env"
    ]
  }
}

Bug behavior

[ ] Flaky
[ ] PDF

Minimal, reproducible example
import puppeteer from 'puppeteer';
console.log("puppeteer",puppeteer)

async function functionTest(){
const browser = await puppeteer.launch();

// Create a new page
const page = await browser.newPage();

// Website URL to export as pdf
const website_url = 'https://www.google.com';
}
functionTest()

Error string
Error: Cannot find module 'puppeteer-core/internal/common/DeviceDescriptors.js'Error: Cannot find module 'puppeteer-core/internal/common/DeviceDescriptors.js'
Puppeteer configuration
No response
Puppeteer version
19.5.2
Node version
16.18.1
Package manager
yarn
Package manager version
1.22.19
Operating system
Linux

@Codypinto23 could you share the complete test repo including the package-lock/yarn-lock files? In general, it looks like babel is not correctly building things. Using your code and running it directly in nodejs with modules works without issues.

Hi @OrKoN ,
Here's the complete test repo

Thanks, I have looked into it and I think the problem is that babel compiles the code to commonjs and then you try to run it with esm. I am not sure why node has different behaviour here but I guess that would be an issue for node?


Support for image choice (a la Mirador 1)

Important for parity with Mirador 1 and Yale use cases.  Design needed.

related to #339


Turn on logger

We turned off logger to fix CI, when fixes are done, we should turn it back on.

Or maybe not, it produces a lot of noise on success too

@Yury-Fridlyand
@Muhammad-awawdi-amazon got the task of fixing the noise while leaving the logger on. In the main, we will leave it off, but to let him fix the noise, we should turn it on.


Python.gitignore: editor .swp files

Editor .swp files is annoying  ;)

@zzhou1 these are from Vim, right? We have a separate template for Vim-specific rules...

@zzhou1 these are from Vim, right? We have a separate template for Vim-specific rules...

Aha, nice, perfect, thanks the great work !


Using numbers throw errors (v2.0.0-rc5)

Using numbers in element() like so:

{5}

Throws:
vendor.js:699 Uncaught TypeError: Cannot read property 'forEach' of undefined

The workaround is to turn it into a string:

{ '' + 5 }

Added a fix for this in rc6 too :)


Re-adds Reducer.forEach method removed in 1.4

TCA 1.4 perhaps unintentionally removed the Reducer.forEach code which allowed creating a ForEachReducer with a AnyCasePath<Action, (ID, ElementAction)> as the action parameter.
Unfortunately, it seems that:
AnyCasePath<Action, (ID, ElementAction)>, and
AnyCasePath<Action, (id: ID, action: ElementAction)>
are not (always) considered equivalent by the Swift compiler.
The helper library, TCACoordinators, relies on the previous behaviour by passing an AnyCasePath<Action, (ID, ElementAction)>. I have tried updating TCACoordinators to use AnyCasePath<Action, (id: ID, action: ElementAction)>, but this has the effect of breaking TCACoordinators when used with TCA 1.3. Without re-adding this particular method, TCACoordinators cannot compile in a way that works with both TCA 1.3 and TCA 1.4. With this small change, TCACoordinators compiles and works just fine!

@rhysm94 Thanks for pointing out the problem! I think we introduced more churn here than we should have. I've updated your PR to fix the old AnyCasePath API and require the new CaseKeyPath API go through IdentifiedAction. While it's technically a small breaking change, the API was introduced already deprecated and we don't think folks should have adopted it.

@stephencelis Thanks for that! That's great, I'm glad it was possible to make it simpler!
Tested the change on my own mirror branch, and it worked perfectly. Happily, TCACoordinators works without issue with these changes in!


--version

Would enjoy a padpo --version for our CI.

That's fun, I wrote this on my TODO list one hour ago! I totally agree.


Support authentication with AWS IAM Roles Anywhere | AWS_SESSION_TOKEN support

Description
AWS session token support
Additional context
Referring to the documentation - nothing about AWS_SESSION_TOKEN support
Example usage: on-premise cluster without static AWS credentials, temporary credentials (AWS access and secret keys + session token) are being generated by AWS roles anywhere feature
https://docs.aws.amazon.com/rolesanywhere/latest/userguide/authentication-create-session.html
Does trigger mesh support session token, if so - could you please add this to documentation?

@RealArtemiy we've moved this to the correct repo, we'll start having a look at it.
I'll ping you on LinkedIn, it would be great if we could jump on a quick call to get to know you a bit!

Ok I couldn't add you on linkedIn, here is me: https://www.linkedin.com/in/jmcx/ or<EMAIL_ADDRESS>would love to jump on a quick intro call ! Thanks

@RealArtemiy could you check https://github.com/triggermesh/triggermesh/pull/1446 and tell us if that would work for you ?

@RealArtemiy to make sure that you have the right versions deployed, could you use our dev manifests from here and repeat the test?

Hello,
$ kubectl apply -f https://github.com/triggermesh/triggermesh/releases/download/v1.25.1/triggermesh-crds.yaml
$ kubectl apply -f https://github.com/triggermesh/triggermesh/releases/download/v1.25.1/triggermesh.yaml
$ aws sts get-session-token

copied AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN
Created a kubernetes manifest
apiVersion: v1
kind: Secret
metadata:
  name: aws
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: ""
  AWS_SECRET_ACCESS_KEY: ""
  AWS_SESSION_TOKEN: ""

and applied
kubectl apply -f secret.yml
tm.yml is here https://github.com/triggermesh/triggermesh/issues/1440#issuecomment-1588919638
creating an object in s3 bucket and checking events:
kubectl get events --sort-by='.lastTimestamp'
42s         Warning   FailedSubscribe                awss3source/triggermesh-s3-source                                            (combined from similar events): Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

am I missing something ?

$ kubectl apply -f https://github.com/triggermesh/triggermesh/releases/download/v1.25.1/triggermesh-crds.yaml
$ kubectl apply -f https://github.com/triggermesh/triggermesh/releases/download/v1.25.1/triggermesh.yaml
$ aws sts get-session-token

Patch version v1.25.1 does not include session token support, please deploy the dev manifest and repeat the test:
https://github.com/triggermesh/triggermesh/suites/13595480280/artifacts/749297615

 kubectl get ev

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Normal    LatestReadyUpdate   configuration/awssqstarget-triggermesh-aws-sqs                            LatestReadyRevisionName updated to "awssqstarget-triggermesh-aws-sqs-00003"

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

@tzununbekov
Is there anything else I can check?

kubectl get ev

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Normal    LatestReadyUpdate   configuration/awssqstarget-triggermesh-aws-sqs                            LatestReadyRevisionName updated to "awssqstarget-triggermesh-aws-sqs-00003"

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

12m         Warning   FailedSubscribe     awss3source/triggermesh-s3-source                                         Error creating AWS API clients: determining bucket's owner: getting ID of caller: InvalidClientTokenId: The security token included in the request is invalid....

@tzununbekov Is there anything else I can check?

These events are reporting the issues with the awss3source/triggermesh-s3-source object, but I don't see it in your manifests. There is also one line that marks your awssqstarget as "ready" and this one I see in the manifest you shared:
12m         Normal    LatestReadyUpdate   configuration/awssqstarget-triggermesh-aws-sqs                            LatestReadyRevisionName updated to "awssqstarget-triggermesh-aws-sqs-00003"

Could you clarify which component are you testing - S3 source or SQS target? kubectl describe output for this object also would be useful.

For the record: after debugging session, this PR was created. Further discussion moved to Slack.

Docs added here: https://github.com/triggermesh/docs/pull/394


Support iOS builds for arm64e architecture.

Fixes issue #431.

@billhollings Hmm... I still see the same link errors. It builds fine if I only have arm64 in Architectures. However when I replace it with arm64e, it still fails to link...

@DiligentGraphics
Okay. Thanks for the feedback.

I also checked the binary size: it changed from 10.3 Mb to 10.4 Mb (not sure if it is related to this change). Does that sound reasonable?

Yeah...I noticed almost no size change myself. I was expecting a more typical fat binary...something along the lines of twice the original size. I'm not sure if that is because both arm64 architectures can share a lot of compiled code...or because it's actually not including arm64e for some reason.
Unfortunately I am not able to test arm64e on a device directly myself yet. As it sits now...the MoltenVK projects should be building for both architectures. Let's see what additional results arise from further testing as these devices and apps built for them become more common.

I tried to remove arm64 architecture completely and build arm64e only by editing xcode project files (I still do not know where these settings are hidden in the IDE). So when I removed arm64, the project failed to build.
There was also this warning:
None of the architectures in ARCHS (arm64) are valid for this target. Consider setting ARCHS to $(ARCHS_STANDARD) or updating it to include at least one value from VALID_ARCHS (arm64e). (in target 'MoltenVK-iOS')

Shouldn't you also update Architectures? As far as I understand, the intersection of Architectures and Valid Architectures get built.

I also checked the binary size: it changed from 10.3 Mb to 10.4 Mb (not sure if it is related to this change). Does that sound reasonable?

Yeah...I noticed almost no size change myself. I was expecting a more typical fat binary...something along the lines of twice the original size. I'm not sure if that is because both arm64 architectures can share a lot of compiled code...or because it's actually not including arm64e for some reason.

This is an old thread (and it's basically dead given the merge), but I felt inspired to share my theory for the suspiciously small change on the size of the binary. I believe this observation is due to the fact that the arm64e slice refuses to support bitcode. If you lipo -detailed_info a universal executable you will note that the mac slices (i.e. simulator) are similarly small, as they also eschew bitcode.
I found this thread on the Apple developer forums, and added my two cents: How to create arm64e slice with bitcode enabled?. The answer, as far as I know, you cannot.


Add options to specify GPU memory size for resource cache in Flutter engine

Steps to Reproduce

Any apps that loads a lot of images
Run the app in memory constrained device like 1 GB ram
When GPU memory exhausted, the app is either rendering extremely slow or crashed

In Flutter engine the GPU memory for resource cache is fixed at 512MB https://github.com/flutter/engine/blob/master/shell/gpu/gpu_surface_gl.cc
If we build an engined with lower values, the rendering and crash issue is gone. But it should make much more sense to add an option and pass the value to the engine on  startup.

I modified the kGrCacheMaxByteSize from 512MB to 256MB, and the out-of-memory issue is gone on iPhone6 device, but this value is not modified recently, so the OOM issue is not caused by this issue ,but we can temp fix it by changing this value.
The last modification of this value is committed by @chinmaygarde at Feb8 2017 (commit id 0904cf3)
"Bump up limits on the GrResourceCache used for the main GrContext. (#3394)"

This cache is now unused with Impeller. We don't plan to make these values configurable.


Update create-a-shared-mailbox.md

made changes to the navigation as it has changed with the new EAC

@JohanFreelancer9 Please copy review

Issue#https://github.com/MicrosoftDocs/microsoft-365-docs/issues/8276


Co-Design UI Wireframes

Checklist:

[x] Decide on sync/async/hybrid approach to co-design of wireframes with CIF staff
[ ] Identify specific areas that make sense to do a co-design approach
[ ] Identify CIF team members who  we'd like to participate
[ ] Plan co-design session(s)
[ ] Hold session(s)
[ ] Review results of co-design session(s) and draft 'final' mockup/wireframe
[ ] Pass wireframe back to CIF team for usability testing/comments

New ticket?
Are the following bullets for the co-design or for the overall approach to UI development? Does this relate to the existing UX/UI ticket for Raj to identify approach?

[ ] Plan facilitation of wireframes
[ ] Conduct plan for facilitation of wireframes
[ ] Iterate on wireframes
[ ] Pass wireframes back to CIF team (usability testing, etc.)

Notes:

Depending on how we plan the co-design session(s) - the Epic could be updated to  link to the corresponding EPIC for CIF (i.e., this may not be part of discovery)

(This is a basic ticket - add to me!)

Hi @jaimiebutton - we forgot to review and pull into this sprint. Is this something we should refine and pull in together?

Yes, good call! Something we can discuss in our meeting today.

Co-Design Miro board located https://miro.com/app/board/o9J_lkk44vo=/. Further sessions on those user stories can happen async

Fri, Nov 12: Presented ideas of wireframes to CIF and GGIRCS team, narrowing down the most important items needed for MVP. Wireframe final captured in ticket 63.

Thank you @rajpersram - I really enjoyed the co-design sessions we had both as an internal team and with our CIF colleagues. I look forward to more designs.


Doesn't out of order responses break HTTP?

Why does the README talk about supporting out of order responses and avoiding the head-of-line problem? Won't that break existing HTTP clients? I'm probably just misunderstanding :)

httpteleport client and server use custom protocol which is incompatible with HTTP, so existing HTTP clients cannot talk to httpteleport servers.
The httpteleport client generates an unique id for each request and prepends the request by this id before sending to the remote server. The remote server prepends each response with the request id, so the client could properly associate received responses with sent requests.

HTTP client usually talks to HTTP proxy:
HTTP client <-> HTTP proxy, that uses httpteleport client for proxying reuqests to the server <-> httpteleport server


comment on type checks

you can put the types of the arguments in the function as well. In that case, IDE like PyCharm can auto-detect if certain input arguments are of incorrect type while you are writing the codes. You can also do that for output

For instance,
def euclidean_distance(
ts1: TimeSeries,
ts2: TimeSeries
) -> float:

Also, for type checking, instead of adding an extra class attribute for type check, you can just simply use
isinstance(ts1, TimeSeries) instead.

Also, you can use decorators from existing libraries like typeguard for that (as long as you put the input and output types when defining the function as above), see https://typeguard.readthedocs.io/en/latest/userguide.html
(However, beware that type-checking can significantly slow down efficiency of your codes for large projects. So in some cases, might be worth just use try and except to catch and raise any incorrect type issues)

For Dynamic Time Warping (DTW) distance function, why set the window size as the length of the second time series if it is not given?  Shouldn't it be the min of the 2 input series?

Thanks Godfrey and sorry for the late in my reply.

Typing is really interesting. It is good practice and I am going to introduce it gradually into the development.
Thanks for the tips, indeed testing TimeSeries is very useful. As for typeguard, this deserves more investigation.
I think you're right, it should probably be the minimum. Changes will be done soon after checking.
Thanks again!

Regarding point 3. I think the window's minimal size is abs(N2-N1) (with N1, N2 the size of the two series). Hence we should take w = max(window, abs(N2-N1)).


analyzer: parse emerald genesis

We currently only parse and index the consensus genesis document.
Background on how runtime genesis is stored: At the SDK level, each paratime's genesis_state() is called only once; then each module is responsible for writing its own data in init_or_migrate(). For example, the accounts module delegates to its init() method, which just writes to MKVS. There are no events emitted, no associated txs.
So parsing the emerald genesis is trickier because there is no neat API to retrieve "the genesis document". Instead, we have to:

find the consensus height at which Emerald was registered as a paratime (we could even hardcode this for now because it happens only once for each runtime; otherwise it's available from the node's APIs, e.g. the one that generates a genesis-like doc from current height)
fetch the MKVS at that height
for the accounts module: read the MKVS path(s) under which the initial balances are stored
similarly for any other module we care to index

Alternatively, since genesis only happens once, and since the relevant part of genesis state applied by Emerald is tiny or maybe even empty (!) (no initial balances are created), we can simply manually transcode the genesis into a SQL file, and execute it from the analyzer before processing blocks.
We should almost certainly do the latter. Marking as P2 because although genesis parsing is a blocker to launching emerald Explorer, the genesis sql will likely be empty.

The above is actually incorrect. Genesis will usually be empty (e.g. state root will be equal to the empty root) and will only be initialized in a later block when the runtime is activated as the block needs to be produced by the compute nodes.
We can add an event to identify such a block.

Interesting, thank you. That "activate runtime" block (is that where init() of each SDK module runs?) can do pretty much anything with the state, right? It's just Rust code that directly modifies the state.
If so, having an event that identifies this block won't be all that helpful because the state changes will still be mostly opaque to the indexer. We could strongly encourage runtime developers (and initially, that's only us anyway) to emit events from the init() that correspond to changes performed in init(), if any. Is that feasible?

https://app.clickup.com/t/3u9yp4x

Relabeling p3 because of how far/hypothetical this is for now (and the fact that we can deal with one-offs with manual hacks, if needed)

Copying discussion from elsewhere:
@mitjat:

The existing runtimes had an "empty" genesis in terms of balances (see linked github issue). As for geneses between versions, we effectively cannot parse them automatically as they're just opaque changes in the MKVS. So if there ever is a relevant change, we'll have to encode it manually in the indexer.


GH-17 Add received header and block validation

This does not include "state digest" listed out in https://github.com/aeternity/epoch/issues/17, as state is transformed only whenever a block is appended to the chain, hence we cannot do that when receiving a block (e.g. block may land in a pool to wait for its predecessors in the chain)
aec_headers:validate(Header) and aec_blocks:validate(Block) are hooked up into aehttp_dispatch_ext.erl, but are planned to be moved from it whenever logic is moved out of the http dispatcher
Closes https://github.com/aeternity/epoch/issues/60

I have rebased on master. When tests pass, it's good to merge.

Approved.


Change to LRO pattern in .NET clients

This issue tracks the work to move .NET clients to the new LRO pattern described here: https://github.com/Azure/azure-sdk/issues/3662#issuecomment-985112894
Changes include:

Update .NET libraries to the new pattern
Updates to the .NET Guidelines
Updates to the Azure.Core LRO Samples

@annelo-msft is this still valid? If so, could you share which SDKs need to be changed?


Make creation scripts idempotent (repeatable)

The individual proc/table scripts are all CREATEs, which fail on repeated execution (during development or deployment.) Switch these to if-not-exists-create-a-stub, then alter.

(Sorry about that, was trying to close the issues in my personal repo)

I actually surfed over here to suggest this and found Brent beat me to it. Having the scripts be idempotent would make my life a lot easier in making sure that I have the latest version on all my servers.

Brent, what would you think of adding a extended property for the "Ola" version date on each procedure?  The extended property can be tested so that the alter (or a drop-create) is used only if the existing version is older.
I plan to use a master job to deploy the maintenance scripts, local jobs, and local schedules.  I'm considering extended properties as a means to avoid making changes unless there is an update.

@drstonephd I'd prefer that Ola chimed in on it - since they're his, it'd make the most sense if he picked how he wants it to work.

Thank you for the request. This is a good idea.
Let's look at the options to implement this:

CREATE OR ALTER - Only works on SQL Server 2016 SP1 and later, so that is not an option.

Create a stub and then alter - The advantage is that permissions are preserved. The downside is that if for some reason the alter fails, then you are left with the stub.

Drop and create - This is what I am using in MaintenanceSolution.sql. That downside is that permissions are not preserved. Not sure how big this issue is.

Drop and create would be a bit easier to implement in my release procedure, as SMO that I am using to script out the objects generates create scripts.

There is a SMO scripting option that uses a stub.  It seems like it used to use sp_sqlexecute which would have made it possible to use without separate batches.  With separate batches, it's not as easy to use an extended property to skip the code if it's already current.  We could manually wrap it in an execute.
$scrp = New-Object Microsoft.SqlServer.Management.Smo.Scripter($srv)
$scrp.Options.IncludeIfNotExists = $TRUE
$scrp.Options.ScriptDrops = $FALSE
$scrp.Options.SchemaQualify = $TRUE
The output:
/*    ==Scripting Parameters==
Source Server Version : SQL Server 2016 (13.0.4457)
Source Database Engine Edition : Microsoft SQL Server Standard Edition
Source Database Engine Type : Standalone SQL Server

Target Server Version : SQL Server 2016
Target Database Engine Edition : Microsoft SQL Server Standard Edition
Target Database Engine Type : Standalone SQL Server

*/
SET ANSI_NULLS ON
SET QUOTED_IDENTIFIER ON
IF NOT EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[CommandExecute]') AND type in (N'P', N'PC'))
BEGIN
EXEC dbo.sp_executesql @statement = N'CREATE PROCEDURE [dbo].[CommandExecute] AS'
END
ALTER PROCEDURE [dbo].[CommandExecute]
@Command nvarchar(max),
@CommandType nvarchar(max),
...

IncludeIfNotExists looks like a good option. I will do some testing with this. Thank you.

I have now changed the scripts, so that they first create a stub, if the stored procedure do not exist, and then alter the stored procedure.
I am using the SMO scripting option IncludeIfNotExists.

I changed to use IncludeIfNotExists when creating the table. It will then only be created if it does not exist.


CF cli not displaying the documentation url when running cf service <service_name>

Command
cf service <service_name>
What occurred
cli not displaying the documentation url

What you expected to occur
I expect to see the documentation url

CLI Version
6.36.1+e3799ad7e.2018-04-04
CC API Endpoint Version
api version:    2.103.0
Platform & Shell Details
Mac OS X 10.11 iTerm
Any other relevant information
this appears to be working on 6.29.0+ff886fa93.2017-07-24

Hey @Jammy-Louie
Apologies for the very delayed response! Is/was your service broker exposing the documentation URL in the service[n].metadata.documentationUrl field as described in the spec?

I've just verified that this is a real bug. We will look to resolve this. Thanks @Jammy-Louie and @gberche-orange, and sorry again for the slow reply!

Pull request to follow: https://github.com/cloudfoundry/cli/pull/1480

This should be fixed by #1480 (thanks Services API team!). If this continues to be a problem, please let us know.


Update dependencies

Description
This is a routine update of ensembl-client's dependencies
Notable updates

Typescript 4.5 -> 4.6

Also
Changed the log level in .npmrc from silent to warn.
Deployment URL
http://update-packages.review.ensembl.org
Checks

Unit tests — ✅
Type check — ✅
Development build — ✅
Production build — ✅

This is what i get when i run npm ci on my machine (not sure if there is anything that needs addressing):
[ridwan@W10-3FCZ173-L /mnt/c/Users/ridwan/ensembl-clients/ensembl-client-1] (update-packages)$ npm ci
npm WARN deprecated<EMAIL_ADDRESS>This is a stub types definition. webpack-dev-middleware provides its own type definitions, so you do not need this installed.
npm WARN deprecated<EMAIL_ADDRESS>This is a stub types definition. terser-webpack-plugin provides its own type definitions, so you do not need this installed.
npm WARN deprecated<EMAIL_ADDRESS>This is a stub types definition. copy-webpack-plugin provides its own type definitions, so you do not need this installed.
npm WARN deprecated<EMAIL_ADDRESS>This is a stub types definition. css-minimizer-webpack-plugin provides its own type definitions, so you do not need this installed.
npm WARN deprecated<EMAIL_ADDRESS>This is a stub types definition. compression-webpack-plugin provides its own type definitions, so you do not need this installed.
npm WARN deprecated<EMAIL_ADDRESS>See https://github.com/lydell/source-map-url#deprecated
npm WARN deprecated<EMAIL_ADDRESS>Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated<EMAIL_ADDRESS>See https://github.com/lydell/source-map-resolve#deprecated
npm WARN deprecated<EMAIL_ADDRESS>https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated<EMAIL_ADDRESS>The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated<EMAIL_ADDRESS>See https://github.com/lydell/source-map-resolve#deprecated
npm WARN deprecated<EMAIL_ADDRESS>some dependency vulnerabilities fixed, support for node < 10 dropped, and newer ECMAScript syntax/features added
npm WARN deprecated<EMAIL_ADDRESS>Chokidar 2 does not receive security updates since 2019. Upgrade to chokidar 3 with 15x fewer dependencies
npm WARN deprecated<EMAIL_ADDRESS>The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated<EMAIL_ADDRESS>This is a stub types definition. mini-css-extract-plugin provides its own type definitions, so you do not need this installed.
npm WARN deprecated<EMAIL_ADDRESS>Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.

><EMAIL_ADDRESS>prepare
> husky install

husky - Git hooks installed

added 3067 packages, and audited 3068 packages in 2m

351 packages are looking for funding
  run `npm fund` for details

35 vulnerabilities (16 moderate, 19 high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.

Apart from the warnings above, i also ran npm start and all seems to be working fine.


500 error when indexing on  Directory creation

I was getting a 500 error from the indexer on directory creation, seems like it doesn't know what to do with a directory path.
I have a rough fix in my own main.js, basically just check to see if the file object passed to the createEvent hook has a children key, if it does it's a directory... don't index it, if it doesn't it's a file and it gets indexed.
async createEvent(file) {
    if ('children' in file) {
      return;
    }
    this.requestIndexing("PATCH", file.path);
  }

I'm not sure if you want that fix here, or if you want me to take a look at the indexer. I'm not sure if there's any benefit to indexing directories.
Let me know how/if you want me to proceed. I'm not great with javascript/typescript, so it might take me a bit to get a PR in, but I'm happy to do it.

As far as I see and understand this happens only on new directory creation. I do not see any value trying to index an empty directory, and if you create files in it, it should be indexing them rather.
So if you do not think otherwise I will just exclude create dirs from generating a call to the indexer. WDYT?

I will fix folders here: https://github.com/brumik/ollama-obsidian-indexer/issues/16

The exact issue (on creation) is now fixed in 0.1.2 (released).
https://github.com/brumik/obsidian-ollama-chat/commit/6e74af9b2c67bfc00113eb433b6981f818e213d5


archivesspace response for image record in repository

@jackflaps
can we get a parsed response of the metadata that you intend to put into the repository
include information about how you want titles and values to be mapped and indexed if different from the archivesspace response
lets start with an image, ideally the response will be appropriate for the images you select in https://github.com/dulibrarytech/digitaldu-backend/issues/66

@jackflaps - never mind, we dont need a parsed response. i forgot we are just going to use the ArchiveSpace API directly

It occurs to me that if you're going to query the ASpace API, we don't technically need MODS at all. If it's easier for you to read JSON into the index, I can look into how we can have the API return that instead

@jackflaps sounds good! we won't need MODS then. We'll just need a document of some sort to map the response we get from the API to fields you want to see in the repository

New plugin for this: https://github.com/duspeccoll/repository_sync
Dropped a sample JSON object in slack; will push the plugin out for testing by Monday


Update movr app and examples in docs to use new multi-region features

We should update movr to use the new multi-region features coming in 21.1
The exact scope of this work is TBD

FWIW:
https://github.com/cockroachlabs/movr-flask/pull/16

FWIW:
https://github.com/cockroachlabs/movr-flask/pull/16

We should also coordinate with engineering to update the built-in movr schema so that it uses the new multi-region syntax.

We should also coordinate with engineering to update the built-in movr schema so that it uses the new multi-region syntax.


US-12-Layout-Range-Properties

Range Properties
Value description
As a user
I want to adjust the range of the x-and y-axis
So that I can have a detailed look of my data.
Description

Acceptance Criteria

[ ]  Paragraph with text "Range of values for axes:".
[ ]  User can set the range of values separately for the x- and y-axis from a list with radio-buttons ("yes, normal range"/"yes, reversed range"/"no, set minimum and maximum").
[ ] Only when the user has chosen the "no, set minimum and maximum" option in at least one of the menus above:

[ ] Paragraph with text "Minimum and maximum values for ax(es)".
[ ] User can set the minimum and maximum values separately for the x- and y-axis: one input of type "number" for the minimum value, one input of type "number" for the maximum value.

Tasks

[ ] Create a feature branch "Range-properties".

[ ] Create a  InputTypeNumber component:

[ ] Function with props idString, placeholderString,  onChange.
[ ] Returns label and input elements.

[ ] Create a RangeProperties component:

[ ] Declare the useState and handle functions for:

rangeXAxis , rangeYAxis.
handleRangeXAxis , handleRangeYAxis.
minXAxis, minYAxis,maxXAxis, maxYAxis.
handleMinXAxis, handleMaxXAxis, handleMinYAxis, handleMaxYAxis.

[ ] Return: Paragraph, InputTypeNumber and InputTypeRadio components in the appropriate order and text described in the acceptance criteria.

[ ]  In the index.js, add the RangeProperties component to the return (before Plotting).

Nice 👍


When grading cells with math nbgrader seems to screw up all the mathjax

The math that rendered in the normal jupyter interface is correctly rendered mathjax, but when you open that up in the grading interface lots of things seem removed and fonts are wonky.

Something more clear should be posted to the relevant jupyter repo.


Reduce database egress

Is your feature request related to a problem? Please describe.
Spending too much money on database egress
Describe the solution you'd like
only sync with a filter regularly (no force sync) and pull notes that haven't been seen on last sync

For some reason, the database egress seems to be lower now, so im going to bump this down in priority


0.2.7: Cannot subscribe to channel with numbers in the name

This server code is able to send a message to the client:
channel = 'channel'
@session.channel.subscribe(channel)
SS.publish.channel([channel], 'newMessage', {message: 'this is a message'}

But weirdly, this does not:
channel = 'channel3'
@session.channel.subscribe(channel)
SS.publish.channel([channel], 'newMessage', {message: 'this is a message'}

I believe the problem lies in the subscribe step, since the console shows that it is attempting to send a message to the correct channel, but the client never receives it.

Hi Jamie
Wow this is very surprising! I can't think of any reason off the top of my head why this would be the case and I've had no other reports of this.
Publishing to channels is already implemented in 0.3. If you get chance and are able to please use that instead. I'm almost certain it will not suffer from the same problem, but let me know if it does and I will reopen this issue and fix it immediately.
As I'm sure you can imagine, with so much to do to get 0.3 finished, I'm trying to avoid touching 0.2 unless absolutely necessary.
Owen

Very understandable; I'll give 0.3 a whirl.


where to get client JAR for CLASSPATH

Until we fix https://trello.com/c/RbxNUNHD/17-omeroclientsjar we should warn people how to find omero_client.jar as it was far from clear to me. Staged at https://ci.openmicroscopy.org/job/OMERO-DEV-merge-docs/ws/src/omero/_build/html/developers/Java.html#extended-classpath.

To what extent the linked page is usefully a "download page" is another question but that's an issue outwith this PR that is separately tracked in Trello; this PR just tries to stay consistent with the other developer documentation's approach.

Certainly seems fine as a quick fix. Obviously, the why is confusing but that's also outwith this PR. :smile:


Android64 build fixes

Patch for OpenSSL Configure file for android64 architectures
Fix for broken pthread function detection

Stuff to consider:

Currently does not work with cmake-2.8 (Thread detection in cmake is broken)
Currently does not work with android NDK 11 and above (OpenH264 1.5.0 build system does not support that)

@realjiangms android 64 requires that, and the config is only meant to be to allow quick starting development. if you want to support older api for 32bit builds just use a config appropriate for the arch

That makes sense.

@freerdp-bot test

@freerdp-bot test

@freerdp-bot test

@freerdp-bot test

@akallabeth I'll prepare a new machine with recent ndk/sdk for android that this can be tested.

@freerdp-bot test

@freerdp-bot test

@freerdp-bot test

@freerdp-bot test

@freerdp-bot test

lgtm


Test artifacts is locked on filesystem

After upgrading to 15.3 the test artifacts from working dir on agent is locker after stage is finish. The next stage will then fail as it is not able to clean working directory.
Sometime the files are unlocked after a long time or the Go Agent needs to be re-started to free the filelock.
Is this a known issue?
Stage that complete but seems to lock file

The stage right after that cannot do a clean

Can you open this issue against the gocd/gocd repository? This repository
is for the main go.cd website
On Wed, Dec 23, 2015, 4:21 PM ThomasSkelgaard<EMAIL_ADDRESS>wrote:

After upgrading to 153 the test artifacts from working dir on agent is
locker after stage is finish The next stage will then fail as it is not
able to clean working directory
Sometime the files are unlocked after a long time or the Go Agent needs to
be re-started to free the filelock
Is this a known issue?
Stage that complete but seems to lock file
[image: gostagethatlock]
https://camo.githubusercontent.com/6093ecc52448a6a945bd125c7a885ab6b6606e05/68747470733a2f2f636c6f756467697468756275736572636f6e74656e74636f6d2f6173736574732f363833383030352f31313937353138332f36343436333863362d613936622d313165352d383733622d3964323837663462633931366a7067
The stage right after that cannot do a clean
[image: go]
https://camo.githubusercontent.com/94fa123e1f25ab3cde04acf9f48efe0a371b98ca/68747470733a2f2f636c6f756467697468756275736572636f6e74656e74636f6d2f6173736574732f363833383030352f31313937353133372f30643934326637342d613936622d313165352d386463392d3338653430376433386634376a7067
—
Reply to this email directly or view it on GitHub
https://github.com/gocd/gocd.github.io/issues/236.

Sorry. I have created a new one.


Debug improvements

Debug helpers:

Add -logthreadnames cmd-line option to add thread names to debug messages
Add ability to change debug category on the fly from console
NOTE: Before switching to another debug category you'd need to turn debugging off via debug 0
and wait a bit (each thread (de)activates debug mode on its own)

Few small changes:

Better log output (a bit more granular for mnb, fixing \ns and few other small issues)
Unify variable/class members naming a bit

PS. pls do not squash these on merge, I already did it for the later two (7c0a858) :wink:

Merged, thanks!


Add support for Apollo Federation

This PR makes saleor compatible with Apollo Federation which is an extremely cool way of distributing consistent GraphQL schema among separate services, sharing common types and handling automated data fetching for single query without any effort from the client.
Currently graphene does not support federation out of the box, there is a separate package that, we would prefer to not depend on anything for that and hack schema but it seems the only way possible for now, until we or other contributors add federation to core graphene (or we move away from graphene entirely).
For more information, please see apollo docs and graphene-federation.
Testing if federation works should be handled by integration tests and graphene-federation library, I think no additional unit testing is required in saleor.
Closes #4803
Pull Request Checklist

[x] Privileged views and APIs are guarded by proper permission checks.
[x] All visible strings are translated with proper context.
[x] All data-formatting is locale-aware (dates, numbers, and so on).
[x] Database queries are optimized and the number of queries is constant.
[x] Database migration files are up to date.
[x] The changes are tested.
[x] GraphQL schema and type definitions are up to date.
[x] Changes are mentioned in the changelog.

Here is the report for 861057e5aacf23d9ff089b06a0ee58a279a1b850 (mirumee/saleor @ support-federation)
Base comparison is 7ab9e38e7785f1281eabfd47b970e9469ecf3195.
No differences were found. (click me)

# api.benchmark checkout
  test name                                  	left count 	right count	duplicate count
  -------------------------------------------	-----------	-----------	---------------
  add billing address to checkout            	         34	         34	             20
  add shipping to checkout                   	          7	          7	              0
  checkout payment charge                    	         14	         14	              0
  complete checkout                          	          6	          6	              0
  create checkout                            	         48	         48	             24

# api.benchmark homepage
  test name                                  	left count 	right count	duplicate count
  -------------------------------------------	-----------	-----------	---------------
  retrieve main menu                         	          5	          5	              0
  retrieve product list                      	          4	          4	              0
  retrieve secondary menu                    	          5	          5	              0
  retrieve shop                              	          2	          2	              0

# api.benchmark product
  test name                                  	left count 	right count	duplicate count
  -------------------------------------------	-----------	-----------	---------------
  product details                            	         15	         15	              3
  retrieve product attributes                	         13	         13	              2

# api.benchmark variant
  test name                                  	left count 	right count	duplicate count
  -------------------------------------------	-----------	-----------	---------------
  product variant bulk create                	         51	         51	              3
  retrieve variant list                      	         15	         15	              6

# api product sorting attributes
  test name                                  	left count 	right count	duplicate count
  -------------------------------------------	-----------	-----------	---------------
  sort product not having attribute data     	         21	         21	              0


[Event Request] Table 7002 "Sales Price" - OnValidateItemNoOnBefore

We are migrating our code from NAV 2018 to BC16.4. In the Table 7002 "Sales Price", in the field 1 "Item No.", in the trigger OnValidate, we need to add our code at numerous places. To do so, the easier should be to add a IsHandled flag and an event publisher, like this :

With this integration event :

Do you think it could be possible ?
Thank you for your work.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


Inconsistency between "crv", "curve", and "curveName" in code vs. swagger

Similar to #7683 the swagger states to use "crv" while the x-ms-client-name is "curve", but in code we write "curveName".

The service is returning "crv":
      "ResponseBody": {
        "key": {
          "kid": "https:\u002f\u002fheathskv.vault.azure.net\u002fkeys\u002f1930028727\u002fb968cab3e06344088a86c23e01f34bb8",
          "kty": "EC",
          "key_ops": [
            "sign",
            "verify"
          ],
          "crv": "P-256",
          "x": "NVlwYx3dC5c6OiOnj2_FA4P8OdMeyLZlIUMLfeBC9ig",
          "y": "Gg1QouOH6HslHTQ66ho9syd49V_VbTGjv-1a-R_1WI0"
        },
        "attributes": {
          "enabled": true,
          "created":<PHONE_NUMBER>,
          "updated":<PHONE_NUMBER>,
          "recoveryLevel": "Recoverable\u002bPurgeable"
        }
      }

What are other languages doing? Might be good to look at a couple others and we can have an even broader conversation.

@JonathanGiles @johanste @bterlson

For Python:
The parameter is curve (docs)
The low-level JsonWebKey uses the attribute name crv (docs)
The "helper"/enum type for curve names is KeyCurveName (docs)

With regards to swagger and x-ms-client-name, given that the purpose of x-ms-client-name is to give a (strong) hint to code generators (modulo whatever naming conventions apply to the target language - e.g. pascal case, camel case, snake case etc.) as to what the canonical name is supposed to be for the property/type, it would probably be a reasonable thing for us to match it in our libraries. If we feel like the annotation is wrong in swagger, we could change it there (as long as we are mindful of any breaking changes for autogenerated libraries - e.g. Go)

I already fixed it in our code, but had opened this to make sure we validate across the languages (was focused on fixing other old-test issues). There may be a problem in the auto-gen code, since at least in .NET it seems the x-ms-client-name was used not only for the member name, but the JSON key as well. A particular method didn't have coverage which is probably why this was missed, and I'll be adding coverage.

Yeah, using x-ms-client-name for the json key name is not correct...

Yeah, using x-ms-client-name for the json key name is not correct...

There's a few discrepencies bewteen languages on this we'll have to iron out, e.g. .NET uses CurveName for the property (now) but other languages that do expose it (not all do) use curve (correct) or crv (incorrect).


App changes file data before upload, changing its EXIF data and thus changing SHA1

Summary:
The app makes a cache copy of a file and changes the EXIF data of the copy for upload. This modifies the file and the file is no longer the same file that was supposed to be uploaded.
Steps to reproduce:
Upload an image file from the app (quite evident when u use camera pictures.)
Download the image file from the server using the download button in the image detail.
now check the SHA1 hashes of both images, one from the camera and the other downloaded one.
The SHA1 of both images is different.
Link for SHA1 checksum https://emn178.github.io/online-tools/sha1_checksum.html
The app modified the image which it should not.
Device and Android version:
BetaDebug on Nokia 6.1+ android 10.
Commons app version:
Current master.
Would you like to work on the issue?
Yes.

The exact line which modifies the code:
line 60:
https://github.com/commons-app/apps-android-commons/blob/f8a8f9207028a6fb5a35483c040821f22daf755e/app/src/main/java/fr/free/nrw/commons/upload/FileProcessor.kt#L51-L62
line 101:
https://github.com/commons-app/apps-android-commons/blob/f8a8f9207028a6fb5a35483c040821f22daf755e/app/src/main/java/fr/free/nrw/commons/upload/FileProcessor.kt#L94-L104
Trackback
line 103:
https://github.com/commons-app/apps-android-commons/blob/f8a8f9207028a6fb5a35483c040821f22daf755e/app/src/main/java/fr/free/nrw/commons/upload/UploadModel.java#L91-L108
line 84:
https://github.com/commons-app/apps-android-commons/blob/f8a8f9207028a6fb5a35483c040821f22daf755e/app/src/main/java/fr/free/nrw/commons/upload/UploadModel.java#L80-L85

My fault, I had forgotten about these EXIF modifications for privacy.
We will have to work around that.
And sorry for making you create an issue about it!

Sorry for chiming in. To sere as a reference for someone who doesn't have the context, could someone mention what the problem is here and how we're thinking of to address it? 🙂

@sivaraam We are looking for a way to check whether an image has been uploaded already or not (using any device, potentially via the website/etc).
The best strategy so far seems to be to check the SHA1 of the picture after applying the user's current privacy setting. Not perfect but best we can do, I guess. We could also check the SHA1 of the original image, and various privacy settings variations, but that would increase server load.

Thanks for the context @nicolas-raoul !

The best strategy so far seems to be to check the SHA1 of the picture after applying the user's current privacy setting.

IIUC, that would detect images uploaded via the app with those settings. But wouldn't work for the ones that users uploaded via other ways like web etc. right?

Not perfect but best we can do, I guess.

Yeah. This looks like a tricky problem indeed.

We could also check the SHA1 of the original image, and various privacy settings variations, but that would increase server load.

Quite possible. But I think we could at least try 2 combinations: original image and original image with user's current privacy setting which should help cover the case where the image has been uploaded as it is. What do you think?
Speaking of SHA-1 based file detection, I happened to stumble upon "File Analyzer" which looks like a neat tool that does exactly what we're looking at. I thought of sharing it just in case it might be helpful 🙂

Thanks @sivaraam for the inputs,
@nicolas-raoul  if we look at the "reference link" we can see there is a tool to check for the overwritten and deleted files, but I think the web interface is not working or the tool is not available anymore.
I tested the upload wizard and it does not modify the image, it's the same as checking the original image sha1 against the API.
Not sure about the deleted ones, do we have some deleted images uploaded without any EXIF changes?

@4D17Y4 Yes, I uploaded this image and it got deleted :'-( https://commons.wikimedia.org/w/index.php?title=File:Royal_Norwegian_Airforce_Lockheed_Orion_with_air-droppable_lifeboat_for_search-and-rescue_missions.jpg&action=edit&redlink=1

we could at least try 2 combinations: original image and original image with user's current privacy setting

Sounds like a good idea! The Upload Wizard does not modify files in any way, right?

That's my belief too. Looks like @4D17Y4 has confirmed it :)

Thanks for the tip! :-) I wonder whether this tool can recognize deleted files.

I'm afraid not. Finding deleted files using SHA-1 sounds even trickier 🔍

@nicolas-raoul if we look at the "reference link" we can see there is a tool to check for the overwritten and deleted files, but I think the web interface is not working or the tool is not available anymore.

Yeah. The tool seems to be down and unmaintained for a while. If its really necessary, we could try pulling strings to explore the feasibility of bringing it back up. For now, the source seems to be available, at least.

@4D17Y4 Yes, I uploaded this image and it got deleted :'-( https://commons.wikimedia.org/w/index.php?title=File:Royal_Norwegian_Airforce_Lockheed_Orion_with_air-droppable_lifeboat_for_search-and-rescue_missions.jpg&action=edit&redlink=1

Thanks for the image, no trace from the tool, it cannot look for deleted files.

Yeah. The tool seems to be down and unmaintained for a while. If it's really necessary, we could try pulling strings to explore the feasibility of bringing it back up. For now, the source seems to be available, at least.

It is not really important right now but can be included as a post-GSoC improvement. We can show the user that this file was uploaded on commons and was deleted; therefore, you are not allowed to upload this image even though it's not currently on commons.