Fix json exception encoders

This commit fixes the bug, when exceptions with null messages are failed to be encoded into json by several json libraries supported by finch.

Current coverage is 77.65% (diff: 100%)

Merging #699 into master will not change coverage

@@             master       #699   diff @@
==========================================
  Files            32         32
  Lines           640        640
  Methods         613        617     +4
  Messages          0          0
  Branches         27         23     -4
==========================================
  Hits            497        497
  Misses          143        143
  Partials          0          0

Powered by Codecov. Last update c2311df...098a89e

@vkostyukov Done.

Looks great! Thanks @akozhemiakin!


Improve docs around automatic retries

... explaining that not all implicit transactions are automatically
retried.
Fixes #3748

This change is

If this looks good, I'll add similar changes for 2.0 docs to the PR.
Separately, I'm making several code improvements that should result in more doc updates.

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/1686e4c4a81827dd8efa8827ddb44662cfe04504/

v2.1/transactions.md, line 82 at r1 (raw file):
Previously, jseldess (Jesse Seldess) wrote…
I'd like to keep the terminology simple and consistent here, so let's change a query (or a batch's) results the results of a single statement or batch of statements

I meant: Let's change TO the results of a single statement or batch of statements

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/acdef48ed7855c77c44451a10e4c230d9c72e168/

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/b5e090e0aea1c0f7c12fea4b517e12244a0c5ca9/

Sorry about the, Andrei.
Create the v2.2 today for the first 2.2.0-alpha release. It'll be in place as soon as https://github.com/cockroachdb/docs/pull/4051 lands.

@andreimatei, I'll merge this and apply your changes to the 2.2 version of the page in a follow-up PR. Thanks for your help!


Code Clarification

I'm a little bit confused about the codes, why does it show [0;38;48m, [3;37;40m, [1;37;40m?. I feel it's sort of distracting, or is it a must it's printed out?

It is used for change to change the colour of the text in terminal. It should have changed the colour. Try running in Cmd/powershell directly in the terminal.

It is used for change to change the colour of the text in terminal. It should have changed the colour. Try running in Cmd/powershell directly in the terminal.

Okay, I'll try that now. Thanks :smile:

It is used for change to change the colour of the text in terminal. It should have changed the colour. Try running in Cmd/powershell directly in the terminal.

Okay, I'll try that now. Thanks :smile:

Do update me. I'll still fix it soon

Still the same thing

Sed🥲 will fix it maybe tomorrow.

Sedsmiling_face_with_tear will fix it maybe tomorrow.

Okay, no issues.
I'd like to add a few things to the repo, like Contribution guidelines, Code Of Conduct, and make a few changes to the README. Kindly assign this to me

Sedsmiling_face_with_tear will fix it maybe tomorrow.

Okay, no issues.
I'd like to add a few things to the repo, like Contribution guidelines, Code Of Conduct, and make a few changes to the README. Kindly assign this to me

For sure☺

Still the same thing. Does it work well on your PC?

Added a fix for this in latest commit.
And sorry i didn't read the question. Yeah i works for me.

Still the same thing. Does it work well on your PC?

Added a fix for this in latest commit.
And sorry i didn't read the question. Yeah i works for me.

Okay, I'll give it another shot on my PC.

Still the same thing. Does it work well on your PC?

Added a fix for this in latest commit.
And sorry i didn't read the question. Yeah i works for me.

Okay, I'll give it another shot on my PC.

Still the same thing

Still the same thing. Does it work well on your PC?

Added a fix for this in latest commit.
And sorry i didn't read the question. Yeah i works for me.

Okay, I'll give it another shot on my PC.

Still the same thing

Did you try the option 3 in starting

Still the same thing. Does it work well on your PC?

Added a fix for this in latest commit.
And sorry i didn't read the question. Yeah i works for me.

Okay, I'll give it another shot on my PC.

Still the same thing

Did you try entering the option 3 in starting?

Yes, I did. I'd the same results.

Still the same thing. Does it work well on your PC?

Added a fix for this in latest commit.
And sorry i didn't read the question. Yeah i works for me.

Okay, I'll give it another shot on my PC.

Still the same thing

Did you try entering the option 3 in starting?

Yes, I did. I'd the same results.

😳How come it's working for me

Wow, I'll try running it on Pydroid. Let's see the outcome

Wow, I'll try running it on Pydroid. Let's see the outcome

I tried it, didn't get the [0:38.. , worked fine. But it says 0 total folders scanned, 0 total file and folder

Wow, I'll try running it on Pydroid. Let's see the outcome

I tried it, didn't get the [0:38.. , worked fine. But it says 0 total folders scanned, 0 total file and folder

Which folder did you scan?

Wow, I'll try running it on Pydroid. Let's see the outcome

I tried it, didn't get the [0:38.. , worked fine. But it says 0 total folders scanned, 0 total file and folder

Which folder did you scan?

Oh, I picked 1 I guess..trying again

Wow, I'll try running it on Pydroid. Let's see the outcome

I tried it, didn't get the [0:38.. , worked fine. But it says 0 total folders scanned, 0 total file and folder

Which folder did you scan?

Oh, I picked 1 I guess..trying again

Just did that, worked perfectly!. It's cool  :smile:

But doesn't it display well on my PC? :rofl:

But why doesn't it display well on my PC? :rofl:

😂 It should work after entering 3.

But why doesn't it display well on my PC? rofl

joy It should work after entering 3.

Yes it did!

This is a good tool :clap:

This is a good tool :clap:

Thnx😀

I forgot to mention that the colors didn't change, it worked well. But the color was white all through asides when it displayed '...Removed..' this changed to red

I got this output

Also, what does option 3 do? Does it make the colors active or what?

I forgot to mention that the color did not change and that it worked perfectly..the color was white throughout. But when it displayed '...Removed..' it changed to red.

Yeah this is the only color i had set (atleast for now) the nos. And some sentences Are set to bright for those whose terminal default isn't white would only be able to notice

Also, what does option 3 do? Does it make the colors active or what?

It changes all color variable strings to emptry string so it would be like printing normally. In other words it removed the modifying of colours.

Oh, got that. Thanks for the clarifications!


”Sutești”, ”Boroșești”, ”Măzili”, ”Verdea” are listed in the drop-down menu

Description:
When user writes ”Sutești”, ”Boroșești”, ”Măzili”, ”Verdea” the locations appear in the drop-down menu next to the comune and the county which is a part according to the law.
Precondition:
The website is up an running.

Step 1
Write in the search bar "Sutești"

Expected results
The location appears in the drop-down menu as "Sat Sutești, VÂLCEA (Sutești)".

Step 2
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 3
Write in the search bar "Boroșești"

Expected results
The location appears in the drop-down menu as "Sat Boroșești, VÂLCEA (Sutești)".

Step 4
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 5
Write in the search bar "Măzili"

Expected results
The location appears in the drop-down menu as "Sat Măzili, VÂLCEA (Sutești)".

Step 6
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 7
Write in the search bar "Verdea"

Expected results
The location appears in the drop-down menu as "Sat Verdea, VÂLCEA (Sutești)".


[SUPPORT] Slow Upsert When Reloading Data into Hudi Table

At the moment we have a setup using Spark structured streaming to read Avro files from S3, do some basic transformation and schema cleaning, then persist into Hudi which also resides on S3. This setup is using Spark 3.1.1 and Hudi 0.10.0 and is running on AWS EMR using emr-6.3.1. Currently this is working fine for new incoming data (Spark app running a batch interval query to listen for new data and process it accordingly), however recently we began facing a performance bottleneck when we attempted to stop the real-time flow and reload some old data as fewer but considerably larger batch. The main problem here seems to be that the executors are running out of memory and thus causing numerous failures in the application until the entire application shuts down.
Having said all this, can you kindly assist us with better understanding what might be causing the bottleneck and how we would be able to overcome it?
Data Details

Transactional datain Avro format, all residing on S3
Existing table size (total): 75GB | 2,300 Objects
Existing table size (one partition): 4.5GB | 45 Objects
Data to upsert (total): 50GB | 2,600 objects
Data to upsert (one batch):  7GB | 370 objects

EMR Config
[   {     "classification": "yarn-site",     "properties": {       "yarn.nodemanager.resource.memory-mb": "27852",       "yarn.nodemanager.pmem-check-enabled": "false",       "yarn.scheduler.maximum-allocation-mb": "27852",       "yarn.nodemanager.vmem-check-enabled": "false"     },     "configurations": []   },   {     "classification": "spark-defaults",     "properties": {       "spark.history.retainedApplications": "100",       "spark.rdd.compress": "true",       "spark.network.timeout": "600s",       "spark.history.fs.cleaner.interval": "2d",       "spark.history.fs.cleaner.maxNum": "40",       "spark.sql.shuffle.partitions": "100",       "spark.history.fs.cleaner.enabled": "true",       "spark.history.fs.eventLog": "true",       "spark.eventLog.rolling.maxFileSize": "1g",       "spark.shuffle.spill.compress": "true",       "spark.history.fs.cleaner.maxAge": "3d",       "spark.shuffle.compress": "true",       "spark.default.parallelism": "100",       "spark.history.fs.eventLog.rolling.maxFilesToRetain": "5",       "spark.serializer": "org.apache.spark.serializer.KryoSerializer",       "spark.executor.memoryOverhead": "2680m",       "spark.executor.instances": "10",       "spark.task.maxFailures": "11",       "spark.executor.memory": "24100m",       "spark.driver.memory": "24100m",       "spark.kryoserializer.buffer.max": "128m",       "spark.yarn.scheduler.reporterThread.maxFailures": "5",       "spark.eventLog.rolling.enabled": "true",       "spark.driver.cores": "5",       "spark.executor.heartbeatInterval": "30s",       "spark.executor.cores": "5",       "spark.memory.storageFraction": "0.6",       "spark.sql.session.timeZone": "UTC",       "spark.driver.memoryOverhead": "2680m",       "spark.yarn.max.executor.failures": "12",       "spark.memory.fraction": "0.7",       "spark.executor.extraJavaOptions": "-Xloggc:/var/log/spark-GClog.log -XX:+PrintGC -XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p'",       "spark.dynamicAllocation.enabled": "false",       "spark.driver.extraJavaOptions": "-Xloggc:/var/log/spark-GClog.log -XX:+PrintGC -XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p'"     },     "configurations": []   },   {     "classification": "spark-log4j",     "properties": {       "log4j.logger.org.apache.parquet": "WARN",       "log4j.logger.parquet": "WARN",       "log4j.logger.org.apache.hudi": "WARN"     },     "configurations": []   } ]
Hudi Parameters

hoodie.index.type: BLOOM
hoodie.datasource.write.operation: UPSERT
hoodie.upsert.shuffle.parallelism: 100
hoodie.payload.ordering.field: hoodie.datasource.write.precombine.field (uses same field as the precombine)
hoodie.datasource.write.payload.class: org.apache.hudi.common.model.DefaultHoodieRecordPayload (used so that latest does not automatically override existing)

In addition to this, the current real-time version of this app uses the following parameters:

driver-cores: 3
driver-memory: 5400m
conf spark.driver.memoryOverhead:1024m
num-executors: 3
executor-cores: 3
executor-memory: 5400m
conf spark.driver.memoryOverhead:1024m
conf spark.sql.shuffle.partitions: 18
conf spark.default.parallelism: 18
spark.memory.fraction:  0.7
spark.memory.storageFraction: 0.6
hoodie.index.type: BLOOM
hoodie.datasource.write.operation: UPSERT
hoodie.upsert.shuffle.parallelism: 18

Last but not least, attached please find some screenshots from the Spark history server.
Executors

Jobs

Failed Job 30 (SparkUpsertCommitActionExecutor)

Job Details 30 (SparkUpsertCommitActionExecutor)

Failed Stage 79

@MikeBuh Thanks for providing the detailed configs and Spark UI screenshots.  Based on the information you provided, it looks like each Spark executor takes more input data than it can handle well in memory.  You may try to use the following Spark and Hudi configs and retry the real-time version of the job (assuming 7GB per upsert batch):
hoodie.upsert.shuffle.parallelism=70
conf spark.sql.shuffle.partitions: 70
conf spark.default.parallelism: 70

If you still hit the memory bottleneck, you may increase both the driver and executor memory:
driver-memory: 12g
executor-memory: 8g

@yihua Thank-you for the reply on this however please note the following:

real-time is working fine, I just included those to showcase what it is currently using for new data
batch reload is having the issues and failing with out of memory errors (error code 137)
the Spark UI screenshots and detailed configs are related to this reload job
perhaps I was not clear, the resources used by the reload job are the default EMR ones specified in that config (re-adding below to have full clarity):

Spark Parameters

spark.driver.cores: 5
spark.driver.memory: 24100m
spark.driver.memoryOverhead: 2680m
spark.executor.instances: 10
spark.executor.cores: 5
spark.executor.memory: 24100m
spark.executor.memoryOverhead: 2680m
spark.memory.storageFraction: 0.6
spark.memory.fraction: 0.7
spark.default.parallelism: 100
spark.sql.shuffle.partitions: 100
spark.kryoserializer.buffer.max: 128m
spark.driver.extraJavaOptions: -Xloggc:/var/log/spark-GClog.log -XX:+PrintGC -XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p'
spark.executor.extraJavaOptions: -Xloggc:/var/log/spark-GClog.log -XX:+PrintGC -XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p'

Hudi Parameters

hoodie.index.type: BLOOM
hoodie.datasource.write.operation: UPSERT
hoodie.upsert.shuffle.parallelism: 100
hoodie.payload.ordering.field: hoodie.datasource.write.precombine.field
hoodie.datasource.write.payload.class: org.apache.hudi.common.model.DefaultHoodieRecordPayload

Thanks once again for your prompt reply and I hope you can assist me with this.

@MikeBuh Thanks for the clarification.  What is the input size of your batch reload?  The similar principle can be applied here for calculating the parallelism.  To be conservative at first, you can calculate the parallelism to use by deriving [input size / 100MB].  Say if the input size is 50GB, you can try parallelism as 500 as below:
spark.default.parallelism: 500
spark.sql.shuffle.partitions: 500
hoodie.upsert.shuffle.parallelism: 500

You should also increase the Kryo serializer buffer:
spark.kryoserializer.buffer.max: 1024m

The executor and driver memory should be good as they are already large enough.
Once you get a successful run, you may further reduce the parallelism to find the sweet spot so that more memory can be leveraged without OOM.

Hi @yihua the batches I am trying to load are around 9GB each. For the latest test I have tried to load only 2 of these batches but not even one of them managed to be processed successfully. I tried 2 runs and both failed with the executor running out of memory.
For both runs I was using the same Spark resources, with the only difference being the parallelism (both for Spark and Hudi):
Common Spark Parameters

spark.driver.cores: 5
spark.driver.memory: 24100m
spark.driver.memoryOverhead: 2680m
spark.executor.instances: 10
spark.executor.cores: 5
spark.executor.memory: 24100m
spark.executor.memoryOverhead: 2680m
spark.memory.storageFraction: 0.6
spark.memory.fraction: 0.7
spark.kryoserializer.buffer.max: 1024m
spark.driver.extraJavaOptions: -Xloggc:/var/log/spark-GClog.log -XX:+PrintGC -XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p'
spark.executor.extraJavaOptions: -Xloggc:/var/log/spark-GClog.log -XX:+PrintGC -XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p'

Run 1 Parallelism

spark.default.parallelism: 100
spark.sql.shuffle.partitions: 100
hoodie.upsert.shuffle.parallelism: 100

Run 2 Parallelism

spark.default.parallelism: 250
spark.sql.shuffle.partitions: 250
hoodie.upsert.shuffle.parallelism: 250

Following the above and unfortunate persisting failures, might any of the following effect the performance and/or have anything to do with the required resources?

size of target table: I noticed that reloading the same batches to a near empty table is more successful
file sizes: maybe having less but larger files in the target table can help when comparing and updating
compaction and cleanup: if these are heavy operations that need lots of memory then perhaps they can be tweaked

Thanks once again for your reply

@MikeBuh sorry for getting back late.  If you still haven't figured out the right configs, here is a more detailed tuning guide for upserts.  At this point, tuning other configs, like making spark.memory.fraction and spark.memory.storageFraction small (e.g., 0.2 as mentioned in the guide) to make the job reliably slow instead of crashing intermittently, are also helpful to try out.

size of target table: I noticed that reloading the same batches to a near empty table is more successful

Given that the upsert is going to first index the input based on existing records in the data files, the target table size also matters for choosing the parallelism.  You may try 1000 for the parallelism.

file sizes: maybe having less but larger files in the target table can help when comparing and updating

Larger files with a fewer number of files in the target table definitely help the indexing phase during upsert as fewer bloom index from footers are read.  Yet tunning the memory and fraction should still make it work with small files.

compaction and cleanup: if these are heavy operations that need lots of memory then perhaps they can be tweaked

To start with, you may disable async compaction and clean services so that they don't interfere with the ingestion.

@MikeBuh : did you get a chance to try out the suggestions from Ethan above. let us know of any updates you have. would love to learn how the tuning went.


Create a Storage::ephemeral_test_instance() method

I think the test in question is using tmpdirs to make a testing Storage instance. This is a side note, but along the way to fixing it, I think it'd be useful to move that code into the storage crate as a Storage::ephemeral_test_instance() method that returns (Self, Handle), where Handle is the RAII guard you get from the crate that makes temporary directories and cleans up the directory on drop.
Originally posted by @hdevalence in https://github.com/penumbra-zone/penumbra/issues/797#issuecomment-1124001693

xref #1664


DB Full Text Search: Search button always greyed out due to inability to select entities in databases window.

Ubuntu 18.04
DBeaver 5.1.2
PostgreSQL 9.6.6
Linux kernel: 4.15.0-24-generic

Hello,
in recent versions of DBeaver (5.1.0 - 5.1.2, i did not check earlier versions) the "Search" Button in DB Full Text always stays inactivated since you are not able to select a database, database schema  or table in the selection window.
If you select an entity, the menu collapses and resets to a state where no selection was made.
Please have a look at this issue since the Full text search is very important for us.
Thanks a lot!

Thank you for the bug report

This happens with PostgreSQL because of bugs in active database switcher (full-text searcher tries to search in all databases and fails during connection switch).
But we also need to improve full-text search UI in general.

Data/metadata searchers will be significantly improved in 5.1.3.


External Authentication Configuration

How to config External Authentication for rancher 2.4.5?

Useful
Info

Versions
Rancher v2.4.5 UI: v2.4.28

Route
undefined

All authentication options are described on https://rancher.com/docs/rancher/v2.x/en/admin-settings/authentication/. Please file an issue with steps followed and the error message when you encounter issues.

All authentication options are described on https://rancher.com/docs/rancher/v2.x/en/admin-settings/authentication/. Please file an issue with steps followed and the error message when you encounter issues.
Thank you for your time.
What should I do if I want to use certifications developed by other third parties, such as those developed by myself?


[BUG] nullable enums do not add null to the enum list

Describe the bug
When adding a nullable enum Schema, the operation will fail unless null is directly provided in the enum list. This is because nullable is not processed before the schema is validated and not having null in the values causes an error.
The error is similar to:
openapi_parser.errors.ParserError: OpenAPI validation error:
("None is not one of ['asc', 'desc']", 'enum', deque([]), None, [], ['asc', 'desc'], None, {'type': 'string', 'nullable': True, 'default': None, 'enum': ['asc', 'desc']}, deque(['enum']), None)

To Reproduce
Steps to reproduce the behavior:

Create an endpoint with a nullable string parameter.
Do not include null in the enum values
Attempt to parse the spec

Expected behavior
When adding a nullable field - the property should have null automatically added to the enum list - with the default being null if not supplied while allowing a set value.
System details (please complete the following information):

OS: [Ubuntu]
OpenAPI / Swagger version [OpenAPI 3.0.1]
Python version [3.11]

Hey @Pizzaface, this error is coming from prance package since I do not have any validation for enums. Could you please check if some newer version works with that? If yes, I could upgrade the dependency, but if not, please open the issue in the related repo - Prance

Whoops! Sorry about that - appreciate the quick response. Will open an issue there. Cheers!

Bug opened on the prance repo for those who find this issue:
Bug

I'm closing this issue for now, feel free to raise a new one once new version of dependency should be used


[Anilist] free variable 'default' referenced before assignment in enclosing scope

Hello! When i set a collection with AniList Seasonal Anime (https://github.com/meisnate12/Plex-Meta-Manager/wiki/AniList-Builders#anilist-seasonal-anime)  like this:
collections:
  Fall 2020:
    anilist_season:
      season: fall
      year: 2020
    sync_mode: sync

Logs:
| Scanning Fall 2020 Collection                                                                   |
| Collection Warning: anilist_season sort_by attribute not found using score as default              |
| free variable 'default' referenced before assignment in enclosing scope                            |

But it works very well with studios for example.

ok there should be a fix in develop. can you test?

| Method: anilist_season                                                                             |
| Value: ordereddict([('season', 'fall'), ('year', 2020)])                                           |
| Collection Warning: anilist_season sort_by attribute not found using score as default              |
| Collection Warning: anilist_season limit attribute not found using 0 as default                    |
|                                                                                |
| Method: sync_mode                                                                                |
| Value: sync                                                                                |
|                                                                                |
| Sync Mode: sync                                                                                |
|                                                                                |
| Method: anilist_season                                                                             |
| Values: [{'sort_by': 'score', 'season': 'fall', 'year': 2020, 'limit': 0}]                         |
|                                                                                |
| Value: {'sort_by': 'score', 'season': 'fall', 'year': 2020, 'limit': 0}                            |
| Data: {'sort_by': 'score', 'season': 'fall', 'year': 2020, 'limit': 0}                             |
| Traceback (most recent call last):                                                                 |
|   File "/root/Plex-Meta-Manager/modules/config.py", line 462, in update_libraries                  |
|     builder.run_methods(collection_obj, collection_name, rating_key_map, movie_map, show_map)      |
|   File "/root/Plex-Meta-Manager/modules/builder.py", line 748, in run_methods                      |
|     elif "anilist" in method:                           items_found += check_map(self.config.AniList.get_items(method, value)) |
|   File "/root/Plex-Meta-Manager/modules/anilist.py", line 185, in get_items                        |
|     mal_ids = self.season(data["season"], data["year"], data["sort_by"], data["limit"])            |
|   File "/root/Plex-Meta-Manager/modules/anilist.py", line 95, in season                            |
|     return self.get_pagenation(query, limit=limit, variables=variables)                            |
|   File "/root/Plex-Meta-Manager/modules/anilist.py", line 50, in get_pagenation                    |
|     json_obj = self.send_request(query, variables)                                                 |
|   File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 49, in wrapped_f                 |
|     return Retrying(*dargs, **dkw).call(f, *args, **kw)                                            |
|   File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 206, in call                     |
|     return attempt.get(self._wrap_exception)                                                       |
|   File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 247, in get                      |
|     six.reraise(self.value[0], self.value[1], self.value[2])                                       |
|   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise                       |
|     raise value                                                                                |
|   File "/usr/local/lib/python3.6/dist-packages/retrying.py", line 200, in call                     |
|     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)                                  |
|   File "/root/Plex-Meta-Manager/modules/anilist.py", line 27, in send_request                      |
|     raise Failed(f"AniList Error: {json_obj['errors'][0]['message']}")                             |
| modules.util.Failed: AniList Error: Variable "$season" of type "String" used in position expecting type "MediaSeason". |
|                                                                                |
| Unknown Error: AniList Error: Variable "$season" of type "String" used in position expecting type "MediaSeason". |

and boom can you update and try again i think i got it this time

and boom can you update and try again i think i got it this time

Fixed, thanks! 👍

awesome I'm leaving this open until the fix is in master


Photos aren't passed through the resizer from the new recipe form before upload

The promiseWellSizedFile function will ensure that a photo selected for a recipe isn't "too big" to upload. It was on the pathway to the REST api, but doesn't appear to be for the GraphQL API. It (or something equivalent) needs to intercept the selected photo so users don't have to manually resize w/in our arbitrary byte size.

This is done and merged into production.


Enhancement: Bottom bar with scrollable

Hi
Excellent bottom bar menu;  :)
I was wondering if you can add a scrollable for TabItem on the bottom.
I am referring to iOS tab-based application; where when you add more TabItem; it will have more option to go to other TabItem, instead of TabItem squeezed within the bottom

Hi, sorry for the delay,
Okay, I will investigate and check it,
Thank you for the issue, and also pull requests are welcome ;)
If you have enough time and interest in it, try to fork and working on it.

Hi, I investigated and I realized that it is not common to have a scrollable BottomBar.
Then if you are not convinced, let me know.
For now, I will close this Issue.
Thanks


POSIX permissions on DrvFs as per Services for UNIX

A brief description
Currently POSIX permissions do not seem to be supported on DrvFs. For example, the chmod command does not seem to work. POSIX permissions should be supported on DrvFs and they should be compatible with Microsoft Services for UNIX (SFU) and Cygwin.
https://technet.microsoft.com/en-us/library/bb463216.aspx
Expected results
From Cygwin (which supports the SFU standard for POSIX permissions):
billziss@windows:~$ touch foobar
billziss@windows:~$ ls -l foobar
-rw-r--r--+ 1 billziss None 0 Mar 22 12:25 foobar
billziss@windows:~$ cacls foobar /S
C:\Users\billziss\foobar "D:P(D;;WPDTRC;;;S-1-0-0)(A;;0x1f019f;;;S-1-5-21-XXXX-1001)(D;;WP;;;SY)(D;;WP;;;BA)(A;;FR;;;S-1-5-21-XXXX-513)(A;;0x1201bf;;;SY)(A;;0x1201bf;;;BA)(A;;FR;;;WD)"

billziss@windows:~$ chmod 770 foobar
billziss@windows:~$ ls -l foobar
-rwxrwx---+ 1 billziss None 0 Mar 22 12:25 foobar
billziss@windows:~$ cacls foobar /S
C:\Users\billziss\foobar "D:P(D;;SWRPWPDTRC;;;S-1-0-0)(A;;FA;;;S-1-5-21-XXXX-1001)(A;;0x1201bf;;;S-1-5-21-XXXX-513)(A;;0x1201bf;;;SY)(A;;0x1201bf;;;BA)(A;;0x120080;;;WD)"

billziss@windows:~$ rm foobar
billziss@windows:~$ pwd -P
/cygdrive/c/Users/billziss

Actual results (with terminal output if applicable)
From WSL bash:
billziss@WINDOWS:/mnt/c/Users/billziss$ touch foobar
billziss@WINDOWS:/mnt/c/Users/billziss$ ls -l foobar
-rwxrwxrwx 1 root root 0 Mar 22 12:27 foobar
billziss@WINDOWS:/mnt/c/Users/billziss$ chmod 770 foobar
billziss@WINDOWS:/mnt/c/Users/billziss$ ls -l foobar
-rwxrwxrwx 1 root root 0 Mar 22 12:27 foobar
billziss@WINDOWS:/mnt/c/Users/billziss$ rm foobar
billziss@WINDOWS:/mnt/c/Users/billziss$ pwd -P
/mnt/c/Users/billziss

Running cacls on this file from a different prompt:
billziss@windows:~$ cacls foobar /S
C:\Users\billziss\foobar "D:(A;;FA;;;SY)(A;;FA;;;BA)(A;;FA;;;S-1-5-21-XXXX-1001)"

Your Windows build number
14393

Ping @github-cygwin who may have an interest in this.

Neither do UNIX permissions work when you mount an NTFS partition inside a real Linux.
Please please don't re-implement the Cygwin ACL rewriting approach, it screws up everything! It's horrible.

Please please don't re-implement the Cygwin ACL rewriting approach, it screws up everything! It's horrible.

@rustyx can you please explain how it is horrible? Can you provide a better alternative?

The Cygwin ACLs is a hack, there's no other way to call it. From the official cygwin docs:

Unfortunately the security tab in the file properties dialog of the Windows Explorer insists to rearrange the order of the ACEs to canonical order before you can read them. Thank God, the sort order remains unchanged if one presses the Cancel button. But don't even think of pressing OK...

Or this:

They are not supposed to be modifiable in Explorer.  If you want to change permissions on a Cygwin ACL, use chmod or setfacl.

Is this a joke?

No.

So it's not really interoperating with Windows ACL, it's taking it over.
Also don't forget the poor souls who end up with unchangeable cygwin 000 permissions as soon as the ACL becomes just slightly complicated, and any attempt to chmod it making the ACL unchangeable in Windows as well.
I'm not suggesting to not do anything, I'm just saying don't do it this way.

[Disclaimer: I am not affiliated with Cygwin other than as a user, despite the Cygwin badge on my GitHub profile.]
I think there are a few issues here:

Services for UNIX (SFU) uses Windows ACL's to simulate POSIX permissions.
Cygwin "extends" the SFU mechanism to store bits like setuid/setgid in ACL's.
Cygwin also uses Windows ACL's to simulate POSIX ACL's.

To clarify I am only asking for (1).

The Cygwin ACLs is a hack, there's no other way to call it.

I think the issue is subtle. I do not agree that taken as a whole Cygwin's use of ACL's is a hack. However I do agree that there are some aspects of Cygwin's use of ACL's that could be considered a hack (point (2) above).
Regarding point (2): you can read my sentiments in the post below where I effectively say that Cygwin subverts the Windows ACL mechanism to store permissions bits that it should not:
http://cygwin.1069669.n5.nabble.com/POSIX-permission-mapping-and-NULL-SIDs-td127833.html

From the official cygwin docs:
Unfortunately the security tab in the file properties dialog of the Windows Explorer insists to rearrange the order of the ACEs to canonical order before you can read them. Thank God, the sort order remains unchanged if one presses the Cancel button. But don't even think of pressing OK...

Microsoft has effectively defined a standard for ACL's within their OS. That standard allows for ACL's like the ones that SFU and Cygwin create. [I note here that SFU was a Microsoft product.] While it is a fact that Windows Explorer does not always work well with those ACL's, this is a failing of Explorer and not SFU or Cygwin.
It could be argued that if Explorer does not support those ACL's well, then they should be considered "discouraged". I am sympathetic to that view. Implementations often create de-facto standards and Windows Explorer is one of the most widely used implementations in existence.
OTOH it could also be argued that Windows Explorer can be fixed to support all legal ACL's and that Microsoft may have an incentive to do so if they decide to implement POSIX permissions as per SFU (and this issue).

Please see the changes that were made in Insiders Build 17063 in DrvFS to support additional Linux metadata

@tara-raj  thanks for you response. Your link takes me to an installation guide, but I was able to find the release notes here:
https://docs.microsoft.com/en-us/windows/wsl/release-notes#build-17063
From the link above:

DrvFs supports additional Linux metadata. This allows setting the owner and mode of files using chmod/chown, and also the creation of special files such as fifos, unix sockets and device files. This is disabled by default for now since it's still experimental. Note: We fixed a bug in the metadata format used by DrvFs. While metadata works on this build for experimentation, future builds will not correctly read metadata created by this build. You might need to manually update owner for modified files and devices with a custom device ID will have to be recreated.

Unfortunately it is not explained how this metadata is being stored. Furthermore the term "metadata" leads me to believe that it may not be using ACL's like Cygwin and Microsoft' own past products (SFU).
Microsoft may of course implement file metadata any way it sees fit. However I am unclear on what the resolution of this issue is.


Update entry point

Utilities like nodemon throw an error because index.js is actually not the entry for app. server.js is.

Thanks 😄


Fix bug in right_canonise

Bug where right_canonise() forced chi=0 fixed.

I also fixed up truncated_svd() function a bit.


how to properly skip samples that cause inf/nan gradients/loss

tl;dr
does the approach in the code snippet below look ok, or is there a better alternative for automatically skipping few "bad" samples in the data that cause inf/nan gradients/loss? (is it a good practice altogether?)
details
sometimes, there is a small percentage (but annoyingly large in absolute value) of "dirty" samples in the data that cause the loss to be nan, although the neural-network architecture itself is fine and stable in terms of numerical stability.
one approach is to automatically stop training (use terminate_on_nan) and then somehow isolate all these samples and remove them from the data permanently. but..
sometimes we simply want to automatically skip these samples as if they never existed (perhaps with a warning), and continue training.
I couldn't find any documentation about how to do that, nor anyone who asked this question. so i decided to ask and offer a solution I found, for others that might need it as well.
in the end, i came up with the following approach - override on_after_backwards method in my lightning-module with the following code:
code
    def on_after_backward(self) -> None:
        valid_gradients = True
        for name, param in self.named_parameters():
            if param.grad is not None:
                valid_gradients = not (torch.isnan(param.grad).any() or torch.isinf(param.grad).any())
                if not valid_gradients:
                    break

        if not valid_gradients:
            log.warning(f'detected inf or nan values in gradients. not updating model parameters')
            self.zero_grad()

pros

this code successfully identifies nan/inf gradients, and skips parameter update by zeroing gradients for the specific batch
support multi-gpu (at least ddp which I tested). when done this way, detecting inf/nan gradients (instead of inf/nan loss), we avoid a potential cases of losing synchronization between different processes, because typically one of the processes would generate an inf loss, while the others won't. if we stop only one process from doing a backwards pass, we lose synchronization, and would stumble into a never-ending processes that wait for nothing. training stalls. when checking gradients, it is after all gradients in all processes have been affected by the bad inf loss. so we have synchronization.

cons

can't catch bad samples that way.. need to work harder..
might not be future proof
clutters lightning module code (it is essentially architecture agnostic, boiler-plate code)
perhaps there is a better way..

final question
is it worth having such functionality integrated into lightning as a simple command-line-switch/parameter?

@levhaikin Thanks for the proposal. I think it should be fine to use. However, I am not sure we want to have this as option within lightning.
Thoughts @tchaton  @Borda ?

In the case of invalid losses, you can return None in your training_step to skip it.

thanks @carmocca, this is definitely a much simpler way!

is it expected to work with ddp?
is it documented somewhere? if not, I guess it would be nice to have that documented explicitly, to save some effort for people like me :)

is it expected to work with ddp?

I'm not sure. We don't have a test for it using DDP. I'll try it and report back.
If it doesn't, this could be fixed with https://github.com/PyTorchLightning/pytorch-lightning/issues/3325 cc @rohan-varma

* is it documented somewhere? if not, I guess it would be nice to have that documented explicitly, to save some effort for people like me :)

See the returns of training_step https://pytorch-lightning.readthedocs.io/en/stable/lightning_module.html#training-step
Do you still want to support skipping invalid gradients or is skipping losses enough?

if it works with ddp then I guess it should be enough.
thanks for pointing to the docs. I didn't notice that on my own..
i'll probably test it as well, once my current training finishes (don't want to interrupt it)

In the case of invalid losses, you can return None in your training_step to skip it.

Is there a way to change the result of training_step from Nan to None in the Callbacks?

No. The optimization procedure is completely managed by the loops calling the LightningModule hooks and Callbacks have no access to it.

It seems returning None in training_step is a bad idea when using AMP. I am using the native backend and for me it causes gradient scaler issues.
Interestingly it fails only on GPU, on CPU everything works just fine.
Should I submit a bug report?

No. Returning None from training_step is not supported with AMP

Hi @carmocca , what happens to the batch size when __getitem__ returns None? Will the batch be smaller?

@mfoglio Depends on your collate function, but assuming the batch_size is 1, then the loop will skip that batch:
https://github.com/PyTorchLightning/pytorch-lightning/blob/3f78c4ca7a89fed8f79d9015e9510f0a0dcc46b5/pytorch_lightning/loops/epoch/training_epoch_loop.py#L191-L192
Note that this is a different topic than returning None from training_step as this issue discusses.

The code presented in the first comment does not work for me. I'm using mixed precision with pytorch_lightning 1.5.5.  self.zero_grad() present in the function written above  on_after_backward is somehow causing the loss to explode.  When I comment self.zero_grad() out, even though I get inf gradients, the training converges. I know the gradients are inf since I've added the print statement there.
    def on_after_backward(self) -> None:
        """
        Skipping updates in case of unstable gradients
        https://github.com/Lightning-AI/lightning/issues/4956
        """
        valid_gradients = True
        for name, param in self.named_parameters():
            if param.grad is not None:
                valid_gradients = not (torch.isnan(param.grad).any() or torch.isinf(param.grad).any())
                if not valid_gradients:
                    break
        if not valid_gradients:
            print(f'detected inf or nan values in gradients. not updating model parameters')
            # self.zero_grad()

However, with self.zero_grad() present, the loss diverges to larger values.
My guess is that with float16 datatype, lower values for the gradient would also count as the inf. However, while updating the weights, the value of the gradient is used. inf therefore looks symbolic in that sense. Please correct me if I'm wrong anywhere.
I'm using gradient_clip_val in pl.Trainer to stablize the training.

FWIW, I encountered a similar problem and it seems to have been resolved by switching from Trainer(precision=16) to Trainer(precision="bf16"), if you have a suitable device for that floating-point type.

@ashesh-0 I use like this
def optimizer_step(
    self,
    epoch,
    batch_idx,
    optimizer,
    optimizer_idx,
    optimizer_closure,
    on_tpu=False,
    using_lbfgs=False,
):
    """
    Skipping updates in case of unstable gradients
    https://github.com/Lightning-AI/lightning/issues/4956
    """
    valid_gradients = True
    for name, param in self.named_parameters():
        if param.grad is not None:
            # valid_gradients = not (torch.isnan(param.grad).any() or torch.isinf(param.grad).any())
            valid_gradients = not (torch.isnan(param.grad).any())
            if not valid_gradients:
                break
    if not valid_gradients:
        print("detected inf or nan values in gradients. not updating model parameters")
        self.zero_grad()
    optimizer.step(closure=optimizer_closure)

and precision 16
i just have gradient nan problem
how about this?

@YooSungHyun can I put this code in self.training_step or do I need to create self.optimizer step after the training_step happens?

@unlugi i just override optimizer_step and, it is called on global step in training loop
plz chk this
https://github.com/YooSungHyun/lightning-U2/blob/main/models/u2/lightningmodule.py#L135

Hi, @YooSungHyun! When you mentioned -

i just can have gradient nan problem
because, gradient clipping is done before optimizer_step (so, i think can not happen inf problem)
how about this?

do you mean that it is still possible that nan gradients are not handled by the modified optimizer_step that you shared?
Sorry, I'm unable to get the case that you are trying to describe, could you please explain?

def optimizer_step(
    self,
    epoch,
    batch_idx,
    optimizer,
    optimizer_idx,
    optimizer_closure,
    on_tpu=False,
    using_lbfgs=False,
):
    """
    Skipping updates in case of unstable gradients
    https://github.com/Lightning-AI/lightning/issues/4956
    """
    valid_gradients = True
    for name, param in self.named_parameters():
        if param.grad is not None:
            # valid_gradients = not (torch.isnan(param.grad).any() or torch.isinf(param.grad).any())
            valid_gradients = not (torch.isnan(param.grad).any())
            if not valid_gradients:
                break
    if not valid_gradients:
        print("detected inf or nan values in gradients. not updating model parameters")
        self.zero_grad()
    optimizer.step(closure=optimizer_closure)

and precision 16, gradient_clip_val 1.0
i just can have gradient nan problem because, gradient clipping is done before optimizer_step (so, i think can not happen inf problem) how about this?

FYI, the suggested optimizer_step override throws
TypeError: optimizer_step() missing 1 required positional argument: 'optimizer_closure'

for me in lightning 2.0. Removing optimizer_idx works for me. Ref - #16539

@ashesh-0 I use like this
def optimizer_step(
    self,
    epoch,
    batch_idx,
    optimizer,
    optimizer_idx,
    optimizer_closure,
    on_tpu=False,
    using_lbfgs=False,
):
    """
    Skipping updates in case of unstable gradients
    https://github.com/Lightning-AI/lightning/issues/4956
    """
    valid_gradients = True
    for name, param in self.named_parameters():
        if param.grad is not None:
            # valid_gradients = not (torch.isnan(param.grad).any() or torch.isinf(param.grad).any())
            valid_gradients = not (torch.isnan(param.grad).any())
            if not valid_gradients:
                break
    if not valid_gradients:
        print("detected inf or nan values in gradients. not updating model parameters")
        self.zero_grad()
    optimizer.step(closure=optimizer_closure)

and precision 16, gradient_clip_val 1.0
i just can have gradient nan problem because, gradient clipping is done before optimizer_step (so, i think can not happen inf problem) how about this?

I had "MisconfigurationException: When optimizer.step(closure) is called, the closure should be callable"
To avoid any problems with not "reimplementing" the mothod the correct way, I did:
def optimizer_step(
        self,
        *args, **kwargs
    ):
        """
        Skipping updates in case of unstable gradients
        https://github.com/Lightning-AI/lightning/issues/4956
        """
        valid_gradients = True
        for name, param in self.named_parameters():
            if param.grad is not None:
                # valid_gradients = not (torch.isnan(param.grad).any() or torch.isinf(param.grad).any())
                valid_gradients = not (torch.isnan(param.grad).any())
                if not valid_gradients:
                    break
        if not valid_gradients:
            print("detected inf or nan values in gradients. not updating model parameters")
            self.zero_grad()

        pl.LightningModule.optimizer_step(self, *args, **kwargs)

I'm not sure if this is enough when using gradient accumulation + ddp + amp.


python312Packages.pynordpool: 0.2.3 -> 0.2.4

Diff: https://github.com/gjohansson-ST/pynordpool/compare/refs/tags/v0.2.3...v0.2.4
Changelog: https://github.com/gjohansson-ST/pynordpool/releases/tag/v0.2.4

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 368529

x86_64-linux

:white_check_mark: 5 packages built:

home-assistant-component-tests.nordpool
python312Packages.pynordpool
python312Packages.pynordpool.dist
python313Packages.pynordpool
python313Packages.pynordpool.dist


ACCUMULO-4755 Custom serialization for AbstractId types

Since the monitor is using the annotation @Produces(MediaType.APPLICATION_JSON) it looks like Jersey is taking the JAXB route by default. This was causing my custom Jackson serializer to not work on the front-end. Switched over to customizing the marshaling function with a custom XmlAdapter JaxbAbstractIdSerializer

Can one of the admins verify this patch?

Squashed and merged. I confirmed that jaxb-api is brought in from Hadoop's class path.


atmosexposed should conduct with adjacent tiles for air blockers

Description
like walls and doors
i want to make walls meltable but its basically impossible to heat them up since it only conducts with the air on the tile, which is nothing since walls block air

haha sexposed


Add examples

Works as intented.. but..a lot of the golang community are newbies (i am in the bandwagon too)
I understood how to use the channel given back by the Read thanks to #2
adding a couple lines to the readme can't be so bad, altough i may complile a PR as soon as i've really grasped it

@colthreepv Could you check the example folder and indicate whether it contains sufficient examples?

https://github.com/030/informado


cargo doc --open doesn't work on WSL if any of the folders in the path have a space in the name

Problem
Running cargo doc --open in a rust project if the path has any spaces causes it to not open the rust docs in the browser. Instead, we get an error that says command 'wslview' did not execute successfully; exit status: 1 and the stderr is, assuming your path of is "your\ path/your_proj/target/doc/your_proj/index.html",
Start-Process : A positional parameter cannot be found that accepts argument 'path\your_proj\target\doc\your_proj\index.html'.
At line:1 char:1
+ Start your path/your_proj/target/doc/your_proj/index.html
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: :)) [Start-Process], ParameterBindingException
    + FullyQualifiedErrorId : PositionalParameterNotFound,Microsoft.PowerShell.Commands.StartProcessCommand

I expected cargo doc --open to have already replaced all of the spaces with "\ " so that it wouldn't think each space causes the path to be split up into different arguments.
Steps

Use WSL for the following steps.
Create a directory with a space in the name.
Go in that directory from the terminal and make a new Rust project, using cargo add
Go into that directory and run cargo doc --open.

Possible Solution(s)
No response
Notes
No response
Version
cargo 1.65.0
release: 1.65.0
host: x86_64-unknown-linux-gnu
libgit2: 1.5.0 (sys:0.16.0 vendored)
libcurl: 7.58.0 (sys:0.4.59+curl-7.86.0 system ssl:GnuTLS/3.5.18)
os: Ubuntu 18.04 (bionic) [64-bit]

Which version of wslview do you have? You can view it with: wslview --version.
I can't seem to reproduce it myself, so I suspect that it may be due to using a relatively old distribution (Ubuntu 18). You may want to try updating the version of wslview or try a newer distribution.

It is the version wslu v2.0.0; wslview v05. lsb_release -r  seems to show that is version 18.04. I will update my ubuntu version to the 22 version now.

@ChrisChoi314. Does it still happen with Ubuntu 22?


[11.x] Allows to persist database connection between tests and only reset PDO in afterClassTearDown()

Background
PDO connection doesn't automatically get closed when the application is unset but instead requires explicitly set to null to trigger connection close:

https://www.php.net/manual/en/pdo.connections.php
Since PHPUnit is especially a long-running process when the application gets booted and terminated on each test the PDO may persist and eventually can cause max database connections. To solve we have two options:
Option A: We can explicitly disconnect the database between tests
Disadvantages:

Caused breaking change to the existing application since it is still possible to use PDO and loaded instances (without IoC) after Iluminate\Foundation\Testing\TestCase::tearDown() has been called.
Any cached/static instances holding PDO will still prevent the database from being disconnected.

Option B: Reuse PDO instances between tests (provided in this PR)
Disadvantage:

The only issue I can think of if tests within a TestCase change database configuration between tests, since the PDO instance is loaded using cache it not going to fetch the specific configuration defined for the test. However, I don't see any reason why a developer would have this for the application tests, even for package development this would be a rare requirement.

To cover Option A, you can configure a base TestCase and set $disconnectDatabaseConnections property to true:
<?php

namespace Tests;

abstract class TestCase extends \Illuminate\Foundation\Testing\TestCase
{
    use CreatesApplication;

+   protected $disconnectDatabaseConnections = true;

}

Related to #49373? Please comment there

Since this applies to code that falls outside the scope of Suggested change , I would like to comment on it separately. (I'm not familiar with MySQL, so I'd like to comment only on PostgreSQL. But I think MySQL probably has similar features as well.)
https://github.com/laravel/framework/pull/49385#discussion_r1427684362

There is a risk that the transaction state and session settings will be carried over

There is some changeable state that persists while a session is open, whether inside or outside a transaction, and whether commit or rollback.
However, certain states also allow us to explicitly discard their state.
(Some implementations may already exist. Sorry if there are.)

Session Authorization

Return to initial state with RESET SESSION AUTHORIZATION.

Connection status settings.

RESET ALL resets everything. (Also valid for many settings other than SET statements)
PHP's ext/pgsql extension uses this method to reset the state of persistent connections. I think it's a good idea to implement the same in our framework.

Advisory lock, Named Prepared Statement

All will be discarded with DISCARD ALL .

In order to preserve existing behavior, we should execute these (or other?) statements to return the session state to its initial state as much as possible.

Since PHPUnit is especially a long-running process when the application gets booted and terminated on each test the PDO may persist and eventually can cause max database connections.

What causes it to persist? How do you recreate this on a fresh Laravel application to hit the max connections error?

What causes it to persist?

Mentioned in https://github.com/laravel/framework/discussions/49373#discussioncomment-7854320. It appears that a reference to the connection resource is being held in some code somewhere, but I don't know where. It may be Laravel code, but it may also be processed by PHPUnit.

How do you recreate this on a fresh Laravel application to hit the max connections error?

Mentioned in #49311. It happens by simply repeating the test case.

What causes it to persist? How do you recreate this on a fresh Laravel application to hit the max connections error?

Before

We assume using flush() and unset($app) would be enough to clear PDO reference but it doesn't see so (which let me to believe that the documentation in PHP meant that we do need to explicitly set PDO instances to null.
After

We tried disconnecting the database connection in tearDown via #49327 but this caused a regression (reported via #49359) and proven that you can still interacts with PDO after unset($app)

Sorry, but if the current behavior is a significant problem, why are we not seeing many more reports of this from basically every Laravel application in the world that has tests?

@taylorotwell Possibly because everyone uses RefreshDatabase. But in some reason I prefer DatabaseTruncation, so I critically encountered the problem.

@taylorotwell
Here are simple steps to reproduce this problem.
The following test code:
<?php

namespace Tests\Feature;

use Tests\TestCase;
use Illuminate\Foundation\Testing\DatabaseMigrations;
use Illuminate\Foundation\Testing\DatabaseTruncation;

class ConnTest extends TestCase
{
    use DatabaseMigrations;
    // or use DatabaseTruncation;
    public static function repeat(): array
    {
        return array_fill(0, 10, []);
    }

    /** @dataProvider repeat */
    public function test(): void
    {
        echo "waiting input...";
        ob_flush();
        fgets(STDIN);
        echo "input received\n";
        $this->assertTrue(true);
    }
}

run test:
php artisan test tests/Feature/ConnTest.php

The test will stop until the Enter key is pressed:
waiting input...
input received
waiting input...
input received
waiting input...

...(omit)

For MySQL, you can check the number of connections with the following query:
show status like 'Threads_connected';

If you check the number of connections while stopping the test, you will see that the number of connections increases as the test case progresses:
mysql> show status like 'Threads_connected';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| Threads_connected | 2     |
+-------------------+-------+
1 row in set (0.00 sec)

mysql> show status like 'Threads_connected';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| Threads_connected | 3     |
+-------------------+-------+
1 row in set (0.00 sec)

...(omit)

I found the reason for @taylorotwell's question.
In short, with the current implementation, whether the problem occurs or not depends on when PHP's circular reference collector is triggered.
It varies depending on the PHP settings and the memory capacity of the execution environment, so

The code shown by @rikvdh may not always work.
The problem that @mpyw actually encountered may not be reproduced depending on the environment.

In any case, the current implementation's behavior changes depending on the environment (or luck), and this pull request will resolve that.
I have prepared CI results in my repository that can confirm the above operation. Complete results can be found here.

This image visualizes the increase and decrease in the number of simultaneous connections by repeating the same test. $all->flush() is called every time, but the actual timing of disconnection is different each time.
Complete CI results also include cases where we explicitly call gc_collect_cycles(). Since the number of connections in this case is constant, we can see that the trigger for disconnection is GC.

I'd like to chime in on this since it seems like it may heavily affect me.

The only issue I can think of if tests within a TestCase change database configuration between tests, since the PDO instance is loaded using cache it not going to fetch the specific configuration defined for the test. However, I don't see any reason why a developer would have this for the application tests, even for package development this would be a rare requirement.

The reason I have is because I run a multi-database multi-tenant application and sometimes I have some critical features that need the extra level of security to ensure that:

Given 2 tenants (2 databases) and 1 user in each tenant
When User A performs an action
It reflects on Tenant's database without crossing/impacting the tenant boundary.

From reading the PR, I see Taylor's point. I've been using Laravel since 5.0 and I have never had any max connection issues. I believe many Laravel users don't face this issue at all. Making a significantly drastic change to how the framework handles database connections affecting every Laravel/Phpunit Users to fix an edge case without a way to keep the existing behavior seems like an opportunity to invite many more issues/bug reports than the current situation.
I'm all for reusing the same PDO connection across many tests which could improve performance and speed up tests, but I feel like users need to be able to better control this.
While we're on this subject, a paper cut issue I've had with Laravel for many years is precisely how DatabaseManager is tied with Config Repository, making it unable to work without global state changes that causes confusion.

@taylorotwell Possibly because everyone uses RefreshDatabase. But in some reason I prefer DatabaseTruncation, so I critically encountered the problem.

@mpyw If this problem mainly affects DatabaseTruncation, can't it be fixed inside DatabaseTruncation trait?

I believe many Laravel users don't face this issue at all.

It all depends on the max connection configured locally and on CI, we did face this issue on Framework own tests a few years back: https://github.com/laravel/framework/pull/39877

I'm all for reusing the same PDO connection across many tests which could improve performance and speed up tests, but I feel like users need to be able to better control this.

Welcome any suggestion on this, we did add ability to disconnect the database connection via a property flag in this PR.

If this problem mainly affects DatabaseTruncation, can't it be fixed inside DatabaseTruncation trait?

If the approach is to disconnect all database during tearDown, this have been proposed and rejected, also potentially impacts performance (since we only disconnect database connectio via RefreshDatabase and DatabaseMigrations trait at the moment) with less control.

Would it be simpler and less breaking (particularly for the multi-tenant scenario) to just call gc_collect_cycles periodically while running tests?

If the approach is to disconnect all database during tearDown, this have been proposed and rejected, also potentially impacts performance (since we only disconnect database connectio via RefreshDatabase and DatabaseMigrations trait at the moment) with less control.

I guess my primary point is: I'm a user of the DatabaseTransactions trait and as such it seems I'm more unlikely to face this problem than users of the DatabaseTruncation as it was mentioned.
Perhaps the implemented solution to the problem can be more targeted at where the problem is and leave users that are outside the scope of it unaffected. The problem with the current proposal is that it completely hijacks the DatabaseConnectionFactory at a global level (OverrideProvidersForTesting), causing 100% of Laravel installation to be affected by this change regardless of whether they are subject to the reported bug or not.

@crynobone

Would it be simpler and less breaking (particularly for the multi-tenant scenario) to just call gc_collect_cycles periodically while running tests?

Is this the answer?


Test single transaction save for questionnaires

IMPORTANT: Where possible all PRs must be linked to a Github issue
Fixes [link to issue]
Engineer Checklist

[ ] I have written Unit tests for any new feature(s) and edge cases for bug fixes
[ ] I have added any strings visible on UI components to the strings.xml file
[ ] I have updated the  CHANGELOG.md file for any notable changes to the codebase
[ ] I have run ./gradlew spotlessApply and ./gradlew spotlessCheck to check my code follows the project's style guide
[ ] I have built and run the FHIRCore app to verify my change fixes the issue and/or does not break the app
[ ] I have checked that this PR does NOT introduce breaking changes that require an update to Content and/or Configs? If it does add a sample here or a link to exactly what changes need to be made to the content.

Code Reviewer Checklist

[ ] I have verified Unit tests have been written for any new feature(s) and edge cases
[ ] I have verified any strings visible on UI components are in the strings.xml file
[ ] I have verifed the CHANGELOG.md file has any notable changes to the codebase
[ ] I have verified the solution has been implemented in a configurable and generic way for reuseable components
[ ] I have built and run the FHIRCore app to verify the change fixes the issue and/or does not break the app

Closing this, as most of the changes here were merged through https://github.com/opensrp/fhircore/pull/3586


docs: Added host directory r/w example to guide and cookbook

This commit adds an example of reading and writing a host directory from a container to the host filesystems guide and to the cookbook.

Can you rebase @vikram-dagger so that we can see what changes in this PR?

Done, but the diff still seems to be the same. I ran these commands:
git checkout main
git pull
git checkout docs-35-host-dir-rw
git rebase main
git push -f


Okx - Failed to place orders due to LIMIT_MAKER not supported

Describe the bug
After connecting okx exchange, pmm strategy fails to place orders automatically. The log says

2024-05-08 11:44:35,006 - 15 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - NETWORK - Error submitting buy LIMIT_MAKER order to Okx for 10.00 DOGE-USDC 0.14463.
Traceback (most recent call last):
File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 452, in _create_order
await self._place_order_and_process_update(order=order, **kwargs,)
File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 469, in _place_order_and_process_update
exchange_order_id, update_timestamp = await self._place_order(
File "/home/hummingbot/hummingbot/connector/exchange/okx/okx_exchange.py", line 191, in _place_order
"ordType": CONSTANTS.ORDER_TYPE_MAP[order_type],
KeyError: <OrderType.LIMIT_MAKER: 3>

I tried using different APIs and different IPs, still don't work. Apart from the OrderType.LIMIT_MAKER KeyError, it also reports another error, namely Private websocket connection authentication failed.

2024-05-08 11:08:33,580 - 15 - hummingbot.connector.exchange.okx.okx_api_user_stream_data_source.OkxAPIUserStreamDataSource - ERROR - Unexpected error while listening to user stream. Retrying after 5 seconds...
Traceback (most recent call last):
File "/home/hummingbot/hummingbot/core/data_type/user_stream_tracker_data_source.py", line 45, in listen_for_user_stream
self._ws_assistant = await self._connected_websocket_assistant()
File "/home/hummingbot/hummingbot/connector/exchange/okx/okx_api_user_stream_data_source.py", line 55, in _connected_websocket_assistant
raise IOError("Private websocket connection authentication failed")
OSError: Private websocket connection authentication failed

Steps to reproduce

connect okx, entering api key, secret and passphrase
create pmm strategy
start

Release version
dev-1.28.0
Type of installation
Docker
Attach required files

services:
bot_1:
container_name: bot_1
image: hummingbot/hummingbot:development
volumes:
- ./credentials/master_account:/home/hummingbot/conf
- ./credentials/master_account/connectors:/home/hummingbot/conf/connectors
- ./instances/bot_1/logs:/home/hummingbot/logs
- ./instances/bot_1/data:/home/hummingbot/data
- ./conf/strategies:/home/hummingbot/conf/strategies
- ./conf/scripts:/home/hummingbot/conf/scripts
- ./conf/controllers:/home/hummingbot/conf/controllers
environment:
- CONFIG_PASSWORD=a
- CONFIG_FILE_NAME=v2_generic_with_controllers.py
- SCRIPT_CONFIG=conf_v2_generic_with_controllers_1.yml
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: 5
tty: true
stdin_open: true

2024-05-08 11:44:31,320 - 15 - hummingbot.client.hummingbot_application - INFO - Creating the clock with tick size: 1.0
2024-05-08 11:44:31,323 - 15 - hummingbot.client.hummingbot_application - INFO - start command initiated.
2024-05-08 11:44:32,050 - 15 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - INFO - Network status has changed to NetworkStatus.CONNECTED. Starting networking...
2024-05-08 11:44:32,060 - 15 - hummingbot.core.api_throttler.async_request_context_base - INFO - API rate limit on WSConnection (1 calls per 1s) has almost reached. Limits used is 1 in the last 1 seconds
2024-05-08 11:44:32,428 - 15 - hummingbot.core.data_type.order_book_tracker - INFO - Initialized order book for DOGE-USDC. 1/1 completed.
2024-05-08 11:44:33,723 - 15 - hummingbot.connector.exchange.okx.okx_api_order_book_data_source.OkxAPIOrderBookDataSource - INFO - Subscribed to public order book and trade channels...
2024-05-08 11:44:34,638 - 15 - hummingbot.connector.exchange.okx.okx_api_user_stream_data_source.OkxAPIUserStreamDataSource - INFO - Subscribed to private account and orders channels...
2024-05-08 11:44:35,004 - 15 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (DOGE-USDC) Creating 1 bid orders at (Size, Price): ['1E+1 DOGE, 0.14463 USDC']
2024-05-08 11:44:35,006 - 15 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - NETWORK - Error submitting buy LIMIT_MAKER order to Okx for 10.00 DOGE-USDC 0.14463.
Traceback (most recent call last):
File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 452, in _create_order
await self._place_order_and_process_update(order=order, **kwargs,)
File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 469, in _place_order_and_process_update
exchange_order_id, update_timestamp = await self._place_order(
File "/home/hummingbot/hummingbot/connector/exchange/okx/okx_exchange.py", line 191, in _place_order
"ordType": CONSTANTS.ORDER_TYPE_MAP[order_type],
KeyError: <OrderType.LIMIT_MAKER: 3>
2024-05-08 11:44:35,010 - 15 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - WARNING - Failed to submit BUY order to Okx. Check API key and network connection.
2024-05-08 11:44:35,022 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp":<PHONE_NUMBER>.0, "order_id": "93027a12dac34fBCBDEUC34e53e56208", "order_type": "OrderType.LIMIT_MAKER", "event_name": "MarketOrderFailureEvent", "event_source": "okx"}
2024-05-08 11:44:35,023 - 15 - hummingbot.connector.client_order_tracker - INFO - Order 93027a12dac34fBCBDEUC34e53e56208 has failed. Order Update: OrderUpdate(trading_pair='DOGE-USDC', update_timestamp=1715168675.0, new_state=<OrderState.FAILED: 6>, client_order_id='93027a12dac34fBCBDEUC34e53e56208', exchange_order_id=None, misc_updates=None)
2024-05-08 11:44:36,042 - 15 - hummingbot.core.rate_oracle.rate_oracle - INFO - Network status has changed to NetworkStatus.CONNECTED. Starting networking...
2024-05-08 11:45:36,003 - 15 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (DOGE-USDC) Creating 1 bid orders at (Size, Price): ['1E+1 DOGE, 0.14418 USDC']
2024-05-08 11:45:36,006 - 15 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - NETWORK - Error submitting buy LIMIT_MAKER order to Okx for 10.00 DOGE-USDC 0.14418.
Traceback (most recent call last):
File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 452, in _create_order
await self._place_order_and_process_update(order=order, **kwargs,)
File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 469, in _place_order_and_process_update
exchange_order_id, update_timestamp = await self._place_order(
File "/home/hummingbot/hummingbot/connector/exchange/okx/okx_exchange.py", line 191, in _place_order
"ordType": CONSTANTS.ORDER_TYPE_MAP[order_type],
KeyError: <OrderType.LIMIT_MAKER: 3>
2024-05-08 11:45:36,009 - 15 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - WARNING - Failed to submit BUY order to Okx. Check API key and network connection.
2024-05-08 11:45:36,017 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp":<PHONE_NUMBER>.0, "order_id": "93027a12dac34fBCBDEUC7c2dfe57b56", "order_type": "OrderType.LIMIT_MAKER", "event_name": "MarketOrderFailureEvent", "event_source": "okx"}
2024-05-08 11:45:36,018 - 15 - hummingbot.connector.client_order_tracker - INFO - Order 93027a12dac34fBCBDEUC7c2dfe57b56 has failed. Order Update: OrderUpdate(trading_pair='DOGE-USDC', update_timestamp=1715168736.0, new_state=<OrderState.FAILED: 6>, client_order_id='93027a12dac34fBCBDEUC7c2dfe57b56', exchange_order_id=None, misc_updates=None)
2024-05-08 11:45:39,006 - 15 - hummingbot.client.hummingbot_application - INFO - stop command initiated.
2024-05-08 11:46:30,647 - 15 - hummingbot.client.hummingbot_application - INFO - Creating the clock with tick size: 1.0
2024-05-08 11:46:30,651 - 15 - hummingbot.client.hummingbot_application - INFO - start command initiated.
2024-05-08 11:46:31,384 - 15 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - INFO - Network status has changed to NetworkStatus.CONNECTED. Starting networking...
2024-05-08 11:46:31,396 - 15 - hummingbot.core.api_throttler.async_request_context_base - INFO - API rate limit on WSConnection (1 calls per 1s) has almost reached. Limits used is 1 in the last 1 seconds

2024-05-08 11:08:33,580 - 15 - hummingbot.connector.exchange.okx.okx_api_user_stream_data_source.OkxAPIUserStreamDataSource - ERROR - Unexpected error while listening to user stream. Retrying after 5 seconds...
Traceback (most recent call last):
File "/home/hummingbot/hummingbot/core/data_type/user_stream_tracker_data_source.py", line 45, in listen_for_user_stream
self._ws_assistant = await self._connected_websocket_assistant()
File "/home/hummingbot/hummingbot/connector/exchange/okx/okx_api_user_stream_data_source.py", line 55, in _connected_websocket_assistant
raise IOError("Private websocket connection authentication failed")
OSError: Private websocket connection authentication failed
2024-05-08 11:08:34,002 - 15 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (PEPE-USDT) Creating 1 bid orders at (Size, Price): ['1E+6 PEPE, 0.000007667 USDT']
2024-05-08 11:08:34,006 - 15 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (PEPE-USDT) Creating 1 ask orders at (Size, Price): ['1E+6 PEPE, 0.000007983 USDT']
2024-05-08 11:08:34,007 - 15 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - NETWORK - Error submitting buy LIMIT_MAKER order to Okx for 1000000 PEPE-USDT 0.000007667.
Traceback (most recent call last):
File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 452, in _create_order
await self._place_order_and_process_update(order=order, **kwargs,)
File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 469, in _place_order_and_process_update
exchange_order_id, update_timestamp = await self._place_order(
File "/home/hummingbot/hummingbot/connector/exchange/okx/okx_exchange.py", line 191, in _place_order
"ordType": CONSTANTS.ORDER_TYPE_MAP[order_type],
KeyError: <OrderType.LIMIT_MAKER: 3>
2024-05-08 11:08:34,008 - 15 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - WARNING - Failed to submit BUY order to Okx. Check API key and network connection.
2024-05-08 11:08:34,009 - 15 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - NETWORK - Error submitting sell LIMIT_MAKER order to Okx for 1000000 PEPE-USDT 0.000007983.
Traceback (most recent call last):
File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 452, in _create_order
await self._place_order_and_process_update(order=order, **kwargs,)
File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 469, in _place_order_and_process_update
exchange_order_id, update_timestamp = await self._place_order(
File "/home/hummingbot/hummingbot/connector/exchange/okx/okx_exchange.py", line 191, in _place_order
"ordType": CONSTANTS.ORDER_TYPE_MAP[order_type],
KeyError: <OrderType.LIMIT_MAKER: 3>
2024-05-08 11:08:34,010 - 15 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - WARNING - Failed to submit SELL order to Okx. Check API key and network connection.
2024-05-08 11:08:34,019 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp":<PHONE_NUMBER>.0, "order_id": "93027a12dac34fBCBPEUT9d956b907ee", "order_type": "OrderType.LIMIT_MAKER", "event_name": "MarketOrderFailureEvent", "event_source": "okx"}
2024-05-08 11:08:34,020 - 15 - hummingbot.connector.client_order_tracker - INFO - Order 93027a12dac34fBCBPEUT9d956b907ee has failed. Order Update: OrderUpdate(trading_pair='PEPE-USDT', update_timestamp=1715166514.0, new_state=<OrderState.FAILED: 6>, client_order_id='93027a12dac34fBCBPEUT9d956b907ee', exchange_order_id=None, misc_updates=None)
2024-05-08 11:08:34,024 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp":<PHONE_NUMBER>.0, "order_id": "93027a12dac34fBCSPEUTf2a4b5cdae2", "order_type": "OrderType.LIMIT_MAKER", "event_name": "MarketOrderFailureEvent", "event_source": "okx"}
2024-05-08 11:08:34,024 - 15 - hummingbot.connector.client_order_tracker - INFO - Order 93027a12dac34fBCSPEUTf2a4b5cdae2 has failed. Order Update: OrderUpdate(trading_pair='PEPE-USDT', update_timestamp=1715166514.0, new_state=<OrderState.FAILED: 6>, client_order_id='93027a12dac34fBCSPEUTf2a4b5cdae2', exchange_order_id=None, misc_updates=None)

reproduced on latest dev-1.28.0
logs_conf_pure_mm_1.log
2024-05-09 12:48:05,893 - 1703175 - hummingbot.client.hummingbot_application - INFO - start command initiated.
2024-05-09 12:48:06,164 - 1703175 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - INFO - Network status has changed to NetworkStatus.CONNECTED. Starting networking...
2024-05-09 12:48:06,170 - 1703175 - hummingbot.core.api_throttler.async_request_context_base - INFO - API rate limit on WSConnection (1 calls per 1s) has almost reached. Limits used is 1 in the last 1 seconds
2024-05-09 12:48:06,426 - 1703175 - hummingbot.connector.exchange.okx.okx_api_order_book_data_source.OkxAPIOrderBookDataSource - INFO - Subscribed to public order book and trade channels...
2024-05-09 12:48:07,525 - 1703175 - hummingbot.connector.exchange.okx.okx_api_user_stream_data_source.OkxAPIUserStreamDataSource - INFO - Subscribed to private account and orders channels...
2024-05-09 12:48:08,002 - 1703175 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (DOGE-USDT) Creating 1 bid orders at (Size, Price): ['1E+1 DOGE, 0.14555 USDT']
2024-05-09 12:48:08,004 - 1703175 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - NETWORK - Error submitting buy LIMIT_MAKER order to Okx for 10.000000 DOGE-USDT 0.14555.
Traceback (most recent call last):
  File "/home/nikita/backt7004/hummingbot/connector/exchange_py_base.py", line 452, in _create_order
    await self._place_order_and_process_update(order=order, **kwargs,)
  File "/home/nikita/backt7004/hummingbot/connector/exchange_py_base.py", line 469, in _place_order_and_process_update
    exchange_order_id, update_timestamp = await self._place_order(
  File "/home/nikita/backt7004/hummingbot/connector/exchange/okx/okx_exchange.py", line 191, in _place_order
    "ordType": CONSTANTS.ORDER_TYPE_MAP[order_type],
KeyError: <OrderType.LIMIT_MAKER: 3>
2024-05-09 12:48:08,008 - 1703175 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - WARNING - Failed to submit BUY order to Okx. Check API key and network connection.
2024-05-09 12:48:08,019 - 1703175 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp":<PHONE_NUMBER>.0, "order_id": "93027a12dac34fBCBDEUTeaddfdef64c", "order_type": "OrderType.LIMIT_MAKER", "event_name": "MarketOrderFailureEvent", "event_source": "okx"}
2024-05-09 12:48:08,020 - 1703175 - hummingbot.connector.client_order_tracker - INFO - Order 93027a12dac34fBCBDEUTeaddfdef64c has failed. Order Update: OrderUpdate(trading_pair='DOGE-USDT', update_timestamp=1715258888.0, new_state=<OrderState.FAILED: 6>, client_order_id='93027a12dac34fBCBDEUTeaddfdef64c', exchange_order_id=None, misc_updates=None)

@cardosofede

Fix merged to latest development (dev-1.28.0)


Kanban Board

We are only 2 in the group!
link:
Kanban Board

Hi Team,
Good job so far!
There are some issues that you still need to work on to go to the next project but you are almost there!
Highlights

Create Kanban board 👍

Required Changes ♻

Hi, I noticed that you are 2 in the group. I would advise you to focus on tasks provided for a team of 2 Members. you will be free to add what you want when your project will be approved. Therefore remove part 3 from your Kanban to avoid confusion.

Optional suggestions
Every comment with the [OPTIONAL] prefix is not crucial enough to stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better.
Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear.
Please, remember to tag me in your question so I can receive the notification.
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

As described in the Code reviews limits policy you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.

Approved ✔
Hi @,
Great job, Your project is complete!  There is nothing else to say other than... it's time close this issue and start working on the project :shipit:
GOOD LUCK 🤞

(Changes Required) Aspects to improve: ♻

N/A

[Optional] suggestions:

N/A

Every comment with the [OPTIONAL] prefix is not crucial enough to stop the approval of this PR. However, I strongly recommend you to take them into account as they can make your code better.
**Cheers and Happy coding!👏👏👏
Feel free to leave any questions or comments in the PR thread if something is not 100% clear, and please remember to tag me in your question so I can receive the notification**
Please, do not open a new Pull Request for re-reviews. You should use the same Pull Request submitted for the first review, either valid or invalid unless it is requested otherwise.

As described in the Code reviews limits policy you have a limited number of reviews per project (check the exact number in your Dashboard). If you think that the code review was not fair, you can request a second opinion using this form.

Hello team 👋,
This is an additional review as a result of the second opinion request.
Your CRR appears to have already received approval. Wishing you success on your capstone project  💯 .
Keep rocking 🚀!

Thanks for answering to our request @medaminedev66
happy coding 🥳


Not part SQL Server Option

why is it not part of the SQL server options in portal or db engine; and should go specifically to Power shell to do it?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e1a62365-4312-8aad-d2f1-bb5c4184adf1
Version Independent ID: 2ba5b809-8025-d7ee-c756-d6d1bab033fb
Content: Directing Azure traffic to Azure SQL Database and SQL Data Warehouse
Content Source: articles/sql-database/sql-database-connectivity-architecture.md
Service: sql-database
GitHub Login: @srdan-bozovic-msft
Microsoft Alias: srbozovi

@ndondapati Thank you for the detailed inquiry. These changes are being made directly outside your deployed database + sql server instance in how remote resources connect, and is not something that is currently exposed via the Azure Portal. PowerShell and Azure CLI are two options, and both can be launched from the Cloud Shell icon found in the Azure Portal.

@ndondapati We will now proceed to close this thread. If there are further questions regarding this matter, please comment and we will gladly continue the discussion.


ENH: Load metadatas as pandas.DataFrame

Use pandas 1.3.0 to load metadata as pandas dataframe instead of etree._Elements.
See this

It may not be possible as is, the pandas.read_xml seems very restrictive.


Upgrade qualysapi dependency

We are using an old qualysapi dependency, which is working with the VulnWhisperer project unlike the latest version of qualysapi, but it creates an extra dependency on the project, needing to add the code of the library to the project in order to not depend on another external dependency.
The latest version of qualysapi when downloading directly with pip is qualysapi-5.0.3, it would be good to update both Qualys Web Scanner and Qualys Vulnerability Management modules to work with that dependency and avoid maintaining our own version of the dependency.
Related to #51. Mention to @kraigu :)

Hah, thanks. :) Parag's been active lately so pull requests to him may help. I can't promise to be of much help myself, it's busy here nowadays.

This is already fixed in the testing environment, will merge with master once we have the new version of qualysapi in pip updated.


Display option names in lowercase

Caps... erm, probably aren't the best design choice for case-sensitive material:

"Wait, which one is -r and which is -R?"

my bad.


Prevent Too Long Values in Shipping

It's possible to enter too long/large values in the shipping section, the inputs should have max attributes to prevent the user from entering an invalid value and not knowing until the error message is shown.

Closing this as too broad of an issue.
There are still some gaps in the validation of the listing between what the server validates and what the client validates, but the gaps are quite extreme cases and they are something I plan to address with any random odd hours I find at the end of days / weeks.
Re: validating large numbers - as was discussed, the html max attribute doesn't work as we;d like for numeric input. It would take some time to have to retrofit all models to have this validation and I question it's need, since the max number allowed is  9,223,372,036,854,775,807. It's almost certain that a user will not accidentally enter such a number. If they do, the back end will fail validation and an error message will be displayed.


Fixed Index issue with Custom shuffle function

The random function was causing the issue, So I created a custom shuffle function of my own

The issue was actually fixed by rendering the random array on the server side, but thank you for the PR and for joining in the stream!

Yes, I did see that. But I think the issue with that is, Its not a generic solution, because if you want to use ServerProps, you'll need to open up port on the server, if you use something like AWS. But yeah no worries  :)

Yeah it's all good. I'm going to be hosting this site on Vercel and have experience with getServerSideProps etc 🙃


[AIRFLOW-3978] Add missing types in MySqlToGoogleCloudStorageOperator

Hello,
Here is an ticket related to GCP on Jira:
https://issues.apache.org/jira/browse/AIRFLOW-3978
Love

Add BINARY , VARBINARY

@mik-laj could you take look at the JIRA?

Let's wait for other people's opinions.


Fix issues with BlmValuesCalculator

Substitute this line for a meaningful title for your changes
Overview
Please write a description. If the PR is hard to understand, provide a quick explanation of the code.
Designs
Link to the related design prototypes (if applicable)
Testing instructions
Please explain how to test the PR: ID of a dataset, steps to reach the feature, etc.
Feature relevant tickets
Link to the related task manager tickets

Checklist before submitting

[ ] Meaningful commits and code rebased on develop.
[ ] If this PR adds feature that should be tested for regressions when
deploying to staging/production, please add brief testing instructions
to the deploy checklist (docs/deployment-checklist.md)
[ ] Update CHANGELOG file

thanks @aciddaute - all looks good.
Did you manage to reproduce the issue reported by Ana with negative values appearing in the set of BLM values? I would not block this PR if not, in any case: as is it should be already addressing at least the most obvious issues with the calculation of the BLM values set and merging this will help FE colleagues proceed with their implementation - we could take care of any leftover glitches, if any, in a separate PR.

for reference, this was happening when the upper bound of the range was close enough to the min (e.g [0,1]) for the bug in the calculation formula to cause some values to be negative.


Use velox memory pool based allocator in Parquet reader.

Currently, Parquet reader uses the default DuckDB allocator. This patch introduces an allocator which uses Velox memory pools.

LGTM
Hoping to merge this by tomorrow.


changes between teamlink versions

Hi Stuart,
I hope you are well. I wanted to ask you something, in our company we have your solution installed, we currently have problems with some users in viewing some lists, we currently have version 1.0.0. What changes has the latest version, are there changes to highlight?
thanks and I look forward to your comments

any answer?

Apologies for the delay.  The latest release is <IP_ADDRESS>
All changes are detailed in the releases.


@conda-forge-admin,please add bot automerge

[ ] I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

Environment (conda list):

$ conda list

Details about  conda and system ( conda info ):

$ conda info

Hi! This is the friendly automated conda-forge-webservice.
I just wanted to let you know that I added bot automerge in conda-forge/opencensus-ext-azure-feedstock#6.


Need to add tags to taggable resources

Similar to my recent terraform-aws-vault PR, I have a need to apply tags to all taggable resources (corporate policy). I have a commit on our fork and will be submitting a PR as soon as I test it in our environment.

Sounds good!

https://github.com/hashicorp/terraform-aws-consul/pull/74


Get profile picture of current user

We can only get the profile pictures of our friends but not our own.

500 stars, bravo !

https://github.com/Schmavery/facebook-chat-api/issues/248

I'm going to close this because I think there's a good workaround.  Please don't hesitate to reopen if you run into problems.


Fabs/1.11/fix/dcos 21361 nodes resource failure backports

Backports https://github.com/dcos/dcos-ui/pull/2781

Thanks for the comments @DanielMSchmidt, definitely don't think addressing then here would be a good idea because of it being a backports, but I am glad that you challenged those points anyway, and made my brain work :D.

alfred run tests

The integration failure looks related (as in the visual component that uses the class changed on this pr), but the other build (against master) just passed so it could be a flake. I will rebuild.

run integration tests

run integration tests


Upload image to ACR

The correct command is docker run --name base-image ubuntu.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 67d7bc0a-3fd6-6737-72b7-52cf371cf11c
Version Independent ID: 975ffb45-a7c5-cd28-d166-57372f881e57
Content: Using ACR with an Azure DC/OS cluster
Content Source: articles/container-service/dcos-swarm/container-service-dcos-acr.md
Service: container-service
GitHub Login: @julienstroheker
Microsoft Alias: juliens

Thanks for the feedback! We are currently investigating and will update you shortly.

@dmakaroff I am not having any issues running the command as it is written in the doc. I am using the Azure Cloud Shell.
What bash shell are you using and what version?

I've used local PowerShell. The command in question results in C:\Program Files\Docker\Docker\Resources\bin\docker.exe: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"--name\": executable file not found in $PATH": unknown.

@dmakaroff please try using Azure CLI. This is how the doc is designed to be used. Not with PowreShell.

@dmakaroff I will close this for now. Please let me know if running in CMD on your local machine or in Azure Cloud shell provides you the same error. If so we can reopen and continue.


NameError: name 'channels' is not defined

/anaconda3/envs/inpaint-coreml/lib/python3.6/site-packages/onnx_coreml/_operators_nd.py in _convert_instancenorm(builder, node, graph, err)
791     elif rank == 4:
792         # 2D Batch Norm
--> 793         add_bn_with_expansion(builder, node, err, node.name, channels[0], scale, bias, mean, var,
794                               node.inputs[0], node.outputs[0], epsilon, compute_mean_var=True,
795                               instance_normalization=True, axes_for_expansion=[])
It seems the "channels", "mean", "var" are not defined for this method

I haven't tested if this generalizes, but the following works for my use case:
def _convert_instancenorm(builder, node, graph, err):  # type: (NeuralNetworkBuilder, Node, Graph, ErrorHandling) -> None
    '''
    convert to CoreML BatchNorm Layer:
    https://github.com/apple/coremltools/blob/655b3be5cc0d42c3c4fa49f0f0e4a93a26b3e492/mlmodel/format/NeuralNetwork.proto#L1633
    '''
    epsilon = node.attrs.get("epsilon", 1e-5)
    if node.inputs[1] not in node.input_tensors or node.inputs[2] not in node.input_tensors:
        return err.unsupported_op_configuration(builder, node, graph, "CoreML InstanceNorm requires Scale and Bias to be known")

    scale = node.input_tensors[node.inputs[1]]
    bias = node.input_tensors[node.inputs[2]]

    #### BEGIN EDIT ######
    channels = graph.shape_dict[node.inputs[1]][0]

    # mean, var not used if `compute_mean_var=True`
    mean = None
    var = None
    #### END EDIT ######

    if node.inputs[0] not in graph.shape_dict:
        return err.unsupported_op_configuration(builder, node, graph, "Shape of input unknown")

    rank = len(graph.shape_dict[node.inputs[0]])
    # ONNX converts B x C tensor into B x C x 1 hence
    # Rank 2 BN is mapped to Rank 3 BN
    if rank == 3:
        # 1D Batch Norm
        add_bn_with_expansion(builder, node, err, node.name, channels, scale, bias, mean, var,
                              node.inputs[0], node.outputs[0], epsilon, compute_mean_var=True,
                              instance_normalization=True, axes_for_expansion=[0, 3])
    elif rank == 4:
        # 2D Batch Norm
        add_bn_with_expansion(builder, node, err, node.name, channels, scale, bias, mean, var,
                              node.inputs[0], node.outputs[0], epsilon, compute_mean_var=True,
                              instance_normalization=True, axes_for_expansion=[])
    else:
        # Unsupported 1D, 3D and above
        err.unsupported_op_configuration(builder, node, graph, "provided number axes {} not supported".format(rank))

@edis219 thanks for reporting
This is known issue and will be updating soon
Thanks @gilr00y for providing work-around

@edis219 can you give a try? this was fixed with https://github.com/onnx/onnx-coreml/pull/452

@bhushan23 it works now. Thanks a lot!


Add perfectly-scrollable to the list

Hi, I just released a SolidJS wrapper for Perfect Scrollbar, thought I'd offer it as an addition to the list.
Link to the repo: https://github.com/dolsem/perfectly-scrollable

Thank you @dolsem!


✏ Add markdown lint

Description
Use a tool like https://github.com/markdownlint/markdownlint to keep markdown files consistent.
Acceptance

[x] Markdown lint library added
[x] Tool run on pre-commit and PR checks

Closed by #790


refector: Go version updated to  1.18.4

/cc @ankitm123

Remember to run go mod tidy and the steps in this guide: https://jenkins-x.io/community/maintainer_guide/go-upgrades/


CB-13384: (browser) Added deprecation of video.src compatibility

Platforms affected
Browser
What does this PR do?
Updates CameraProxy.js to support video.srcObject and preserves backwards compatibility with URL.createObjectURL
What testing has been done on this change?
Browser
Checklist

[X] Reported an issue in the JIRA database
[X] Commit message follows the format: "CB-3232: (android) Fix bug with resolving file paths", where CB-xxxx is the JIRA ID & "android" is the platform affected.
[ ] Added automated test coverage as appropriate for this change.

Is something holding up the merging of this PR?

@mvandervliet The solution I used is
            localMediaStream = stream;
            if (typeof video.srcObject !== 'undefined') {
                video.srcObject = stream;
            } else {
                video.src = window.URL.createObjectURL(localMediaStream);
            }


Show inconsistent AzCopy version in 'Releases Notes' and 'About' dialog

Storage Explorer Version: 1.16.0
Build: 20201030.5
Branch: rel/1.16.0
Platform/OS: Windows 10/ Linux Ubuntu 16.04
Architecture: ia32/x64
Regression From: Not a regression
Steps to reproduce:

Launch Storage Explorer.
Click 'Help -> View Releases Notes' -> Check the AzCopy version.
Click 'Help -> About' -> Check the AzCopy version.

Expect Experience:
Keep a consistent current AzCopy version.
Actual Experience:
Show inconsistent AzCopy version.

In Release Notes:

In 'About' dialog.

This is the private build we use for testing. We will be replacing it with a public release when we ship.

Verified on build 20201107.1. This issue has been fixed. So close it.