GJK endless loops

With sphere-box collision with the precision of 1e-4 and box-box collision with the precision of 1e-10 GJK sometimes gets itself into the endless loop.
Currently, there is an endless loop detection workaround. However, an investigation is needed, in order to remove the source of the problem, or otherwise find and describe the precise reason of it.

This issue was partially addressed in https://github.com/Godlike/Pegasus/issues/80

https://github.com/Godlike/Arion/issues/10 Migrated.


HTTPRoute: allow more match clauses

Today, a route is limited to 16 routes with 8 matches each. This is problematic in real world environments.
While its easy to split a route up (you can easily get the same behavior, and similar cognitive complexity by having 1 route per HTTPRoute), its quite hard to split up matches. For instance, https://github.com/istio/istio/blob/d7a9700d5eaa4e9728274b408623670f48deadb5/samples/bookinfo/gateway-api/bookinfo-gateway.yaml#L23-L38 is an example of a route matching the exposed APIs for a frontend. This is only a small tivial example, and it already uses 5. In our user base, we often see far beyond 8.
Splitting these up is both complex for the user, and may actually lead to different behaviors if implementations treat route groups differently (for instance, if we add retry budget -- that could now be split; other core or implementation specific policies may behave similarly, this is just an example).
The limit on 8 here seems quite small given the context of real usage, the other limits in the API (16 routes, 64 listeners), and the cost of manually working around it. Based on this, I propose we raise the limit to match Gateway listeners.
What type of PR is this?
/kind feature
Does this PR introduce a user-facing change?:

Increased the limit on HTTPRoute matches from 8 to 64.

/retest

@howardjohn Historically we've been against this because of the worst case scenarios this could lead to if you max out the length of every list within a list (16 rules * 8 matchers * 16 filters == 2048 possible filters in a Route). Increasing the matchers from 8 to 64 would turn that into 16384, which would obviously be problematic.
With that said, we've got CEL now, so we could likely make a case for the following:

Allow up to 64 matchers in a Rule (that seems rather extreme, but something >8 does seem sensible)
Write CEL validation that ensures that the maximum number of matchers allowed in a Route does not go up from it's current theoretical max (128)

That would mean we're allowing significantly more flexibility here without actually increasing the max size of an individual route. If this ends up working reasonably well, we could apply the same principle elsewhere in the API.

For my understanding, what specifically are we trying to constrain with the overall size limit? Is it...

Not requiring implementations to support large routes
Staying below CEL limits
Making sure we fail hard before etcd size limits
Something else?

If its the CEL limits, is the concern that its within the complexity limits now (which already take into account the 16*64), or that it could in the future if matches get more complex?
I ask since it may impact the solution here

For my understanding, what specifically are we trying to constrain with the overall size limit? Is it...

Not requiring implementations to support large routes
Staying below CEL limits
Making sure we fail hard before etcd size limits

All of the above. I'd also mention that it's possible that we'll add more types of matchers inside this list which will only increase the overall size and complexity, I don't want to get so close to a max size that we end up having no additional room to grow. I'm also strongly biased towards more smaller routes than a few very large routes. In general though, when we're talking about API compatibility, this is already a GA API, and significantly loosening the validation would be problematic. I think we can make a reasonably strong case in favor of a change that allows for more flexibility but still fits within the original constraints though.

+1 to this change, this will reduce the friction users are facing when migrating from exiting APIs to Gateway API, who are used to authoring rules a specific way, and this is currently slowing down Gateway API adoption.
@robscott's suggestion of adding a CEL validation to measure and limit total filters for a Route, is a great idea to account for the concerns of the size of the final object persisting in the API server
crosss linking previous issues raised by users in the past

https://github.com/kubernetes-sigs/gateway-api/issues/1485
https://github.com/kubernetes-sigs/gateway-api/issues/2501
https://github.com/envoyproxy/gateway/issues/396

I've pushed up a change to limit the aggregate size. LMK what you think

Sadly looks like we're running into https://github.com/kubernetes/kubernetes/issues/120973 - CRD is invalid because rule is too complex

That's only from the extra restriction fwiw. It works fine when we just allow it to be unbounded, even with the theoretical 16k entries.

Honestly don't get how that rule can possibly be 100x over the limit..

Actually, would really like @youngnick specifically to sign off on this, will add a hold until he's able to take a look.
/hold

LGTM
Deferring review to Nick as per https://github.com/kubernetes-sigs/gateway-api/pull/3205#issuecomment-2243557250

should we do the same for GRPCRoute as well while we're making this change?

I think updating GRPCRoute to match would be great. Would approve in this PR or a follow up.

Added gRPC route as well

Thanks @howardjohn!
/lgtm
/approve

I think we've got enough other LGTMs on this one to remove the hold.
/hold cancel

The grpc implementation is incorrect here - the limit was never raised.


Automated cherry pick of #1796: 提高openstack创建windows机器成功率

Cherry pick of #1796 on release/2.10.0.
#1796: 提高openstack创建windows机器成功率

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/2.10.0@1582760). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             release/2.10.0   #1798   +/-   ##
================================================
  Coverage                  ?   7.68%
================================================
  Files                     ?     410
  Lines                     ?   62619
  Branches                  ?       0
================================================
  Hits                      ?    4810
  Misses                    ?   57400
  Partials                  ?     409

Flag
Coverage Δ

#aFlag
7.68% <ø> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1582760...931192c. Read the comment docs.

/approve
/lgtm


Xamarin.iOS Validation

[x] Check discovering chromecast
[x] Check connecting to chromecast
[x] Check starting application
[x] Check loading media

.

This works


Bug - no data in widget API for news flash

https://anyway-infographics-staging.web.app/newsflash/102730

Solved with @ziv17 's work on temp cache for road segments


container/lxc: Removes volatile device keys when device is removed

Removes volatile device keys when device is removed.
Or when a device is updated such that it's device type changes.
It will also ensure that if the device type remains the same but the new device's config replaces an old volatile key that the old volatile key is removed.
Fixes issue where hwaddr property was left even if device type changed from profile's bridged type to manually added infiniband type.
Also ensures existing behaviour of a profile bridged device's name property replacing an old name volatile key when a manually added device is removed and the profile device is re-added.
This re-works the old code that was device and key name specific with a device type agnostic version.
Signed-off-by: Thomas Parrott<EMAIL_ADDRESS>

@stgraber ready for review

Testsuite passed


pythonPackages.wrapt: 1.13.3 -> 1.14.1

Description of changes

Changelog: https://wrapt.readthedocs.io/en/latest/changes.html
Mainly, wrapt is now compatible with Python 3.11.
Things done

Built on platform(s)

[ ] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

This is currently on staging too in https://github.com/NixOS/nixpkgs/pull/198741 :)

TIL about staging 👍


Fix links in kubeadm.md

With hugo transition, the links have been to "X.md", correct the
relative links to the new location.
Closes: #8440

/assign @mikedanese


Browser Library & assertion-engine 3.0.

Describe the bug
Browser Library does not have a capped version of assertion-engine at 2.0.
Since robotframework-assertion-engine==3.0.0 was recently released, it will be downloaded and running the browser with the parameter will cause an error:
Initializing library 'Browser' with arguments [ auto_closing_level=AutoClosingLevel.MANUAL | enable_playwright_debug=True | timeout=0:00:20 | run_on_failure=Take Screenshot ../../fail-screenshot--{index} ] failed: TypeError: Formatter() takes no arguments
The solution is to downgrade assertion-engine to 2.0
To Reproduce
Steps to reproduce the behavior:

Fresh Installation of RF & Browser Library (ver. 16.1.0 or 17.3.0)
You receive robotframework-assertion-engine ver. 3.0.0
Utilize Library  Browser  with some settings
Library  Browser  auto_closing_level=MANUAL  enable_playwright_debug=True  timeout=20 s ...  run_on_failure=Take Screenshot \ ..${/}..${/}
See error

Expected behavior
receive robotframework-assertion-engine ver. 2.0.0
Desktop (please complete the following information):

OS: Linux, Windows
Browser: Edge, Chrome
Version 17.3.0

Duplicate of #3181
Only the latest version is supported by us, please upgrade to the latest version. Workaround is explained in here: https://github.com/MarketSquare/robotframework-browser/issues/3181#issuecomment-1773894715


Docker Deployment

Write details about how to deploy librairy by using docker-compose v2

Use of gitbook to write the document


Add Credential Revocation Contracts

Summary of Changes:
Addition of Contracts for Credential Revocation:

Added a directory named revoke to the contracts directory, this directory contains the smart contracts related to credential revocation.
Included the revocation contract in genesis.json.

Modifications to Other Contracts:

Added errors related to revocation in AnouncredsErrors.sol.

Modified the ecrecover method to use the standardized ECDSA method from the OpenZeppelin library in the following contracts:

CredentialDefinitionRegistry
CredentialDefinitionRegistryInterface
SchemaRegistry
SchemaRegistryInterface
IndyDidRegistry
IndyDidRegistryInterface

I'm closing this PR to refactor the code, including the tests, wrappers and so on.


sros readme update (cf disks; variants table)

Added vsr-i to the variants table
Added information for additional CFs

thanks @jcpvdm
I did a minor addition/correction to this doc piece.


fine tuning questions using tiny model

First of all, really appreciate you sharing this. This is my first python project ever but was able to get a model out thanks to your clear instructions and commenting.
Questions:

Is there a reason why the prepared data is prepared in 30 seconds chunk rather than splitting by each line of the provided SRT file? Reason asking is splitting at 30 seconds often cuts off in the middle of a sentence and could cause a bit of in accuracy in the beginning and end whereas each line of SRT usually ends with a short pause.

The fine-tuned model transcribes significantly worse then the original model (my base was tiny model for Mandarin using ~10k files to fine tune). I see in your other comments that adjusting the hyper parameters would be the way to go. I did some research but wasn't able to grasp the exact meaning of this, could you give a quick example of "adjusting the hyper parameters" using this project?

Thank you for giving this project a try, and I apologize that it didn't work well for you. Here are my responses to your questions:

The reason for this is that Whisper is trained in this manner, as far as I understand. When audio is divided in the middle of a sentence, Whisper only generates output before the sentence, shifts to the beginning of the sentence, and starts predicting from there. For more details, please refer to the original Whisper paper. Here's a quote from the paper:

When a final transcript segment is only partially included in the current 30 second audio chunk, we predict only its start time token for the segment when in timestamp mode, to indicate that he subsequent decoding should be performed on an audio window aligned with that time, otherwise we truncate the audio to not include the segment.

In such cases, what I usually do is adjust the learning rate and batch size, along with the gradient accumulation step. Additionally, if you haven't already, I recommend trying the --train-only-decoder parameter, as it often stabilizes the training. However, it is also highly possible that there are issues in my code. This project is still a work in progress, and I currently lack the time to further develop it.


run_evaluation.sh  some questions

The runtime shows that scene 4 was skipped by mistake, is this normal? What can I do if it's not normal? Also how can I check if the loss converges at the end of training? Please let me know when you see it, thanks.
(TCP) (base) rossuihan@rossuihan:~/TCP$ sh leaderboard/scripts/run_evaluation.sh
init statistics_manager
begin
========= Preparing RouteScenario_0 (repetition 2) =========

Setting up the agent
routes_lav_valid_10_13_20_29_08
Loading the world
Town02
Base transform is blocking objects  Transform(Location(x=185.695465, y=257.345886, z=1.210000), Rotation(pitch=0.000000, yaw=360.039185, roll=0.000000))
Base transform is blocking objects  Transform(Location(x=185.695114, y=257.845886, z=1.210000), Rotation(pitch=0.000000, yaw=360.039185, roll=0.000000))
Base transform is blocking objects  Transform(Location(x=185.694778, y=258.345886, z=1.210000), Rotation(pitch=0.000000, yaw=360.039185, roll=0.000000))
Base transform is blocking objects  Transform(Location(x=185.694443, y=258.845886, z=1.210000), Rotation(pitch=0.000000, yaw=360.039185, roll=0.000000))
Base transform is blocking objects  Transform(Location(x=185.694092, y=259.345886, z=1.210000), Rotation(pitch=0.000000, yaw=360.039185, roll=0.000000))
Base transform is blocking objects  Transform(Location(x=185.693756, y=259.845886, z=1.210000), Rotation(pitch=0.000000, yaw=360.039185, roll=0.000000))
Skipping scenario 'Scenario4' due to setup error: Error: Unable to spawn vehicle vehicle.diamondback.century at Transform(Location(x=185.693756, y=259.845886, z=1.210000), Rotation(pitch=0.000000, yaw=360.039185, roll=0.000000))
Running the route

I meet the same question when i evaluation the model, I don't know weather it's normal


Add no-array-index-key rule

This rule will warn when using the array index as the key. As a first
pass, I decided to implement this for Array.prototype.map, since that
is likely the most common case, but it theoretically could be expanded
to find other cases so I kept the naming more generic.
Like many rules, this one is imperfect and is prone to some false
positives and negatives. For instance, if someone defines a .map()
function on an object that doesn't have the same signature as
Array.prototype.map, this will likely end up warning in those cases.
However, I think the value of this rule outweighs its hypothetical
drawbacks.

Nice. Would be really great if this could also support React.cloneElement, ala:
[].map((item, i) => {
    return React.cloneElement(someChild, {
        key: i
    })
})

All feedback has been addressed! PTAL

Also, what about the .forEach method of a Map or Set?

@ljharb What do you mean by this question? Are you referring to the name of the rule and documentation mentioning arrays?

@lencioni i mean, aMapOrSet.forEach(x => { somethingElse.push(<div key={x} />); }) or similar. Will these be caught as well, since there's no way to know if aMapOrSet is an array or not?

@ljharb Yes, these will be caught as well for precisely the reason you mention. Likewise, so will any instance of any class that has one of these methods on it. This is what I was referring to about potential false positives in my commit message.

@ljharb I believe I have addressed your feedback.


replace cat with pv

pv its a tool to show the progress bar of a cat big files..

ping ...

The changes here wouldn't really work if "pv" isn't available on the system. I prepared this function which we will call on every assignment to COMPRESSION/DECOMPRESSION:
cat_or_pv()
{
  type pv 1>/dev/null 2>&1
  if [ $? -ne 0 ];
  then
    echo 'cat'
  else
    echo 'pv'
  fi;
}

I will close this and update the master. Let me know if that works for you.


Time threshold for the violation detection

Currently, i had tried to modify the algorithm so that have the feature for the time threshold to recoginse the violation as red color rectangle box after 4seconds.
The concept is like this, once the algorithm detect  the 2 people are very close to each other, it will show a white color box as the buffer zone of the time threshold for around 4 second. After 4 second, if that 2 person are still close to each other, the box will turn into red color. i had tried many solution for this modification but i still unable to achieve that feature.

So basically you want to add a time limit of 4 seconds before flagging the violation as serious?

So basically you want to add a time limit of 4 seconds before flagging the violation as serious?

yes correct.

Sorry I was busy. We can calculate dwell time for every serious violation detected and if the time is greater than 4 sec, we can simply print it, e.g.,
from mylib import config, thread
from mylib.mailer import Mailer
from mylib.detection import detect_people
from imutils.video import VideoStream, FPS
from scipy.spatial import distance as dist
import numpy as np
import argparse, imutils, cv2, os, time, schedule, datetime

#----------------------------Parse req. arguments------------------------------#
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--input", type=str, default="",
	help="path to (optional) input video file")
ap.add_argument("-o", "--output", type=str, default="",
	help="path to (optional) output video file")
ap.add_argument("-d", "--display", type=int, default=1,
	help="whether or not output frame should be displayed")
args = vars(ap.parse_args())
#------------------------------------------------------------------------------#

# load the COCO class labels our YOLO model was trained on
labelsPath = os.path.sep.join([config.MODEL_PATH, "coco.names"])
LABELS = open(labelsPath).read().strip().split("\n")

# derive the paths to the YOLO weights and model configuration
weightsPath = os.path.sep.join([config.MODEL_PATH, "yolov3.weights"])
configPath = os.path.sep.join([config.MODEL_PATH, "yolov3.cfg"])

# load our YOLO object detector trained on COCO dataset (80 classes)
net = cv2.dnn.readNetFromDarknet(configPath, weightsPath)

# check if we are going to use GPU
if config.USE_GPU:
	# set CUDA as the preferable backend and target
	print("")
	print("[INFO] Looking for GPU")
	net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
	net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)

# determine only the *output* layer names that we need from YOLO
ln = net.getLayerNames()
ln = [ln[i[0] - 1] for i in net.getUnconnectedOutLayers()]

# if a video path was not supplied, grab a reference to the camera
if not args.get("input", False):
	print("[INFO] Starting the live stream..")
	vs = cv2.VideoCapture(config.url)
	if config.Thread:
			cap = thread.ThreadingClass(config.url)
	time.sleep(2.0)

# otherwise, grab a reference to the video file
else:
	print("[INFO] Starting the video..")
	vs = cv2.VideoCapture(args["input"])
	if config.Thread:
			cap = thread.ThreadingClass(args["input"])

writer = None
# start the FPS counter
fps = FPS().start()

# initialize structures for dwell time
object_id_list = []
dtime = dict()
dwell_time = dict()

# loop over the frames from the video stream
while True:
	# read the next frame from the file
	if config.Thread:
		frame = cap.read()

	else:
		(grabbed, frame) = vs.read()
		# if the frame was not grabbed, then we have reached the end of the stream
		if not grabbed:
			break

	# resize the frame and then detect people (and only people) in it
	frame = imutils.resize(frame, width=700)
	results = detect_people(frame, net, ln,
		personIdx=LABELS.index("person"))

	# initialize the set of indexes that violate the max/min social distance limits
	serious = set()
	abnormal = set()

	# ensure there are *at least* two people detections (required in
	# order to compute our pairwise distance maps)

	if len(results) >= 2:
		# extract all centroids from the results and compute the
		# Euclidean distances between all pairs of the centroids
		centroids = np.array([r[2] for r in results])
		D = dist.cdist(centroids, centroids, metric="euclidean")

		# loop over the upper triangular of the distance matrix
		for i in range(0, D.shape[0]):
			for j in range(i + 1, D.shape[1]):
				# check to see if the distance between any two
				# centroid pairs is less than the configured number of pixels
				if D[i, j] < config.MIN_DISTANCE:
					# update our violation set with the indexes of the centroid pairs
					serious.add(i)
					serious.add(j)
                # update our abnormal set if the centroid distance is below max distance limit
				if (D[i, j] < config.MAX_DISTANCE) and not serious:
					abnormal.add(i)
					abnormal.add(j)

	# loop over the results
	for (i, (prob, bbox, centroid)) in enumerate(results):
		# extract the bounding box and centroid coordinates, then
		# initialize the color of the annotation
		(startX, startY, endX, endY) = bbox
		(cX, cY) = centroid
		color = (0, 255, 0)

		# if the index pair exists within the violation/abnormal sets, then update the color
		if i in serious:
			# initiate and calculate dwell time
			if i not in object_id_list:
				object_id_list.append(i)
				dtime[i] = datetime.datetime.now()
				dwell_time[i] = 0
			else:
				curr_time = datetime.datetime.now()
				old_time = dtime[i]
				time_diff = curr_time - old_time
				dtime[i] = datetime.datetime.now()
				sec = time_diff.total_seconds()
				dwell_time[i] += sec
				print(dwell_time) # verify dwell time in cmd

				# print dwell time and flag violation as serios only after passing 'x' seconds
				dwell = "{}s".format(int(dwell_time[i]))
				seconds = 4 # change seconds here
				if dwell_time[i] > seconds:
					color = (0, 0, 255)
					cv2.rectangle(frame, (startX, startY), (endX, endY), color, 2)
					cv2.putText(frame, dwell, (startX-5, startY-5), cv2.FONT_HERSHEY_SIMPLEX, 0.70, color, 2)
					# draw the total number of serios social distancing violations on the output frame
					text = "Total serious violations: {}".format(len(serious))
					cv2.putText(frame, text, (10, frame.shape[0] - 55),
					cv2.FONT_HERSHEY_SIMPLEX, 0.70, (0, 0, 255), 2)

		elif i in abnormal:
			color = (0, 255, 255) #orange = (0, 165, 255)

		# draw (1) a bounding box around the person and (2) the
		# centroid coordinates of the person,
		cv2.rectangle(frame, (startX, startY), (endX, endY), color, 2)
		cv2.circle(frame, (cX, cY), 5, color, 2)

	# draw some of the parameters
	Safe_Distance = "Safe distance: >{} px".format(config.MAX_DISTANCE)
	cv2.putText(frame, Safe_Distance, (470, frame.shape[0] - 25),
		cv2.FONT_HERSHEY_SIMPLEX, 0.60, (255, 0, 0), 2)
	Threshold = "Threshold limit: {}".format(config.Threshold)
	cv2.putText(frame, Threshold, (470, frame.shape[0] - 50),
		cv2.FONT_HERSHEY_SIMPLEX, 0.60, (255, 0, 0), 2)

	text1 = "Total abnormal violations: {}".format(len(abnormal))
	cv2.putText(frame, text1, (10, frame.shape[0] - 25),
		cv2.FONT_HERSHEY_SIMPLEX, 0.70, (0, 255, 255), 2)

#------------------------------Alert function----------------------------------#
	if len(serious) >= config.Threshold:
		cv2.putText(frame, "-ALERT: Violations over limit-", (10, frame.shape[0] - 80),
			cv2.FONT_HERSHEY_COMPLEX, 0.60, (0, 0, 255), 2)
		if config.ALERT:
			print("")
			print('[INFO] Sending mail...')
			Mailer().send(config.MAIL)
			print('[INFO] Mail sent')
		#config.ALERT = False
#------------------------------------------------------------------------------#
	# check to see if the output frame should be displayed to our screen
	if args["display"] > 0:
		# show the output frame
		cv2.imshow("Real-Time Monitoring/Analysis Window", frame)
		key = cv2.waitKey(1) & 0xFF

		# if the `q` key was pressed, break from the loop
		if key == ord("q"):
			break
    # update the FPS counter
	fps.update()

	# if an output video file path has been supplied and the video
	# writer has not been initialized, do so now
	if args["output"] != "" and writer is None:
		# initialize our video writer
		fourcc = cv2.VideoWriter_fourcc(*"MJPG")
		writer = cv2.VideoWriter(args["output"], fourcc, 25,
			(frame.shape[1], frame.shape[0]), True)

	# if the video writer is not None, write the frame to the output video file
	if writer is not None:
		writer.write(frame)

# stop the timer and display FPS information
fps.stop()
print("===========================")
print("[INFO] Elasped time: {:.2f}".format(fps.elapsed()))
print("[INFO] Approx. FPS: {:.2f}".format(fps.fps()))

# close any open windows
cv2.destroyAllWindows()

Hi,
Thank  you so much for your reply. It is very helpful.
On Thu, 3 Jun 2021 at 21:24, Sai Subhakar T @.***>
wrote:

Sorry I was busy. We can calculate dwell time for every serious violation
detected and if the time is greater than 4 sec, we can simply print it,
e.g.,
from mylib import config, threadfrom mylib.mailer import Mailerfrom mylib.detection import detect_peoplefrom imutils.video import VideoStream, FPSfrom scipy.spatial import distance as distimport numpy as npimport argparse, imutils, cv2, os, time, schedule, datetime
#----------------------------Parse req. arguments------------------------------#ap = argparse.ArgumentParser()ap.add_argument("-i", "--input", type=str, default="",
help="path to (optional) input video file")ap.add_argument("-o", "--output", type=str, default="",
help="path to (optional) output video file")ap.add_argument("-d", "--display", type=int, default=1,
help="whether or not output frame should be displayed")args = vars(ap.parse_args())
#------------------------------------------------------------------------------#
load the COCO class labels our YOLO model was trained onlabelsPath = os.path.sep.join([config.MODEL_PATH, "coco.names"])LABELS = open(labelsPath).read().strip().split("\n")
derive the paths to the YOLO weights and model configurationweightsPath = os.path.sep.join([config.MODEL_PATH, "yolov3.weights"])configPath = os.path.sep.join([config.MODEL_PATH, "yolov3.cfg"])
load our YOLO object detector trained on COCO dataset (80 classes)net = cv2.dnn.readNetFromDarknet(configPath, weightsPath)
check if we are going to use GPUif config.USE_GPU:
set CUDA as the preferable backend and target
print("")
print("[INFO] Looking for GPU")
net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)
determine only the output layer names that we need from YOLOln = net.getLayerNames()ln = [ln[i[0] - 1] for i in net.getUnconnectedOutLayers()]
if a video path was not supplied, grab a reference to the cameraif not args.get("input", False):
print("[INFO] Starting the live stream..")
vs = cv2.VideoCapture(config.url)
if config.Thread:
cap = thread.ThreadingClass(config.url)
time.sleep(2.0)
otherwise, grab a reference to the video file
else:
print("[INFO] Starting the video..")
vs = cv2.VideoCapture(args["input"])
if config.Thread:
cap = thread.ThreadingClass(args["input"])
writer = None
start the FPS counterfps = FPS().start()
initialize structures for dwell timeobject_id_list = []dtime = dict()dwell_time = dict()
loop over the frames from the video streamwhile True:
read the next frame from the file
if config.Thread:
frame = cap.read()
else:
(grabbed, frame) = vs.read()
# if the frame was not grabbed, then we have reached the end of the stream
if not grabbed:
break
resize the frame and then detect people (and only people) in it
frame = imutils.resize(frame, width=700)
results = detect_people(frame, net, ln,
personIdx=LABELS.index("person"))
initialize the set of indexes that violate the max/min social distance limits
serious = set()
abnormal = set()
ensure there are at least two people detections (required in
order to compute our pairwise distance maps)
if len(results) >= 2:
# extract all centroids from the results and compute the
# Euclidean distances between all pairs of the centroids
centroids = np.array([r[2] for r in results])
D = dist.cdist(centroids, centroids, metric="euclidean")
  # loop over the upper triangular of the distance matrix
  for i in range(0, D.shape[0]):
  	for j in range(i + 1, D.shape[1]):
  		# check to see if the distance between any two
  		# centroid pairs is less than the configured number of pixels
  		if D[i, j] < config.MIN_DISTANCE:
  			# update our violation set with the indexes of the centroid pairs
  			serious.add(i)
  			serious.add(j)
            # update our abnormal set if the centroid distance is below max distance limit
  		if (D[i, j] < config.MAX_DISTANCE) and not serious:
  			abnormal.add(i)
  			abnormal.add(j)

loop over the results
for (i, (prob, bbox, centroid)) in enumerate(results):
# extract the bounding box and centroid coordinates, then
# initialize the color of the annotation
(startX, startY, endX, endY) = bbox
(cX, cY) = centroid
color = (0, 255, 0)
  # if the index pair exists within the violation/abnormal sets, then update the color
  if i in serious:
  	# initiate and calculate dwell time
  	if i not in object_id_list:
  		object_id_list.append(i)
  		dtime[i] = datetime.datetime.now()
  		dwell_time[i] = 0
  	else:
  		curr_time = datetime.datetime.now()
  		old_time = dtime[i]
  		time_diff = curr_time - old_time
  		dtime[i] = datetime.datetime.now()
  		sec = time_diff.total_seconds()
  		dwell_time[i] += sec
  		print(dwell_time) # verify dwell time in cmd

  		# print dwell time and flag violation as serios only after passing 'x' seconds
  		dwell = "{}s".format(int(dwell_time[i]))
  		seconds = 4 # change seconds here
  		if dwell_time[i] > seconds:
  			color = (0, 0, 255)
  			cv2.rectangle(frame, (startX, startY), (endX, endY), color, 2)
  			cv2.putText(frame, dwell, (startX-5, startY-5), cv2.FONT_HERSHEY_SIMPLEX, 0.70, color, 2)
  			# draw the total number of serios social distancing violations on the output frame
  			text = "Total serious violations: {}".format(len(serious))
  			cv2.putText(frame, text, (10, frame.shape[0] - 55),
  			cv2.FONT_HERSHEY_SIMPLEX, 0.70, (0, 0, 255), 2)

  elif i in abnormal:
  	color = (0, 255, 255) #orange = (0, 165, 255)

  # draw (1) a bounding box around the person and (2) the
  # centroid coordinates of the person,
  cv2.rectangle(frame, (startX, startY), (endX, endY), color, 2)
  cv2.circle(frame, (cX, cY), 5, color, 2)

draw some of the parameters
Safe_Distance = "Safe distance: >{} px".format(config.MAX_DISTANCE)
cv2.putText(frame, Safe_Distance, (470, frame.shape[0] - 25),
cv2.FONT_HERSHEY_SIMPLEX, 0.60, (255, 0, 0), 2)
Threshold = "Threshold limit: {}".format(config.Threshold)
cv2.putText(frame, Threshold, (470, frame.shape[0] - 50),
cv2.FONT_HERSHEY_SIMPLEX, 0.60, (255, 0, 0), 2)
text1 = "Total abnormal violations: {}".format(len(abnormal))
cv2.putText(frame, text1, (10, frame.shape[0] - 25),
cv2.FONT_HERSHEY_SIMPLEX, 0.70, (0, 255, 255), 2)
#------------------------------Alert function----------------------------------#
if len(serious) >= config.Threshold:
cv2.putText(frame, "-ALERT: Violations over limit-", (10, frame.shape[0] - 80),
cv2.FONT_HERSHEY_COMPLEX, 0.60, (0, 0, 255), 2)
if config.ALERT:
print("")
print('[INFO] Sending mail...')
Mailer().send(config.MAIL)
print('[INFO] Mail sent')
#config.ALERT = False
#------------------------------------------------------------------------------#
check to see if the output frame should be displayed to our screen
if args["display"] > 0:
# show the output frame
cv2.imshow("Real-Time Monitoring/Analysis Window", frame)
key = cv2.waitKey(1) & 0xFF
  # if the `q` key was pressed, break from the loop
  if key == ord("q"):
  	break
# update the FPS counter

fps.update()
if an output video file path has been supplied and the video
writer has not been initialized, do so now
if args["output"] != "" and writer is None:
# initialize our video writer
fourcc = cv2.VideoWriter_fourcc(*"MJPG")
writer = cv2.VideoWriter(args["output"], fourcc, 25,
(frame.shape[1], frame.shape[0]), True)
if the video writer is not None, write the frame to the output video file
if writer is not None:
writer.write(frame)
stop the timer and display FPS informationfps.stop()print("===========================")print("[INFO] Elasped time: {:.2f}".format(fps.elapsed()))print("[INFO] Approx. FPS: {:.2f}".format(fps.fps()))
close any open windowscv2.destroyAllWindows()
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/saimj7/Social-Distancing-Detection-in-Real-Time/issues/6#issuecomment-853866409,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AR5BRR44BWVBM3IBT7FGY6TTQ57CJANCNFSM45KGM3WQ
.


Get rid of Result?

Scala 2.12 has a properly biased Either type.
If and when we drop support for scala 2.10 and 2.11, maybe we can get rid of Result.

I think we should. I understand 2.11.9 will have a right-biased either, and I can provide a simple implicit conversion for 2.10 if necessary.


Upgrade slate and gems to a more recent version

there is an issue with saasc that prevents us to upgrade
middleman-sprockets to 4.1.x so we stick with 4.0.x for now until it's
fixed.

If some of you want to check if that's ok:
git switch upgrade-slate
bundle install
bundle exec middleman server

Then open http://localhost:4567/

Yep, still have the same issue with middleman-sprockets 4.1.x, but anyway, it looks like whitesource is happy :)


fix(ulp): Write pin's output mode to the correct register (IDFGH-13065)

Fixes register mixup.
According to the ESP32-S3 TRM (pages 515 - 516), the output pin's mode is set in the RTC_GPIO_PINn_REG,
bit RTC_GPIO_PINn_PAD_DRIVER, not the RTC_IO_TOUCH_PADn_REG field RTC_IO_TOUCH_PADn_DRV,
which instead controls the drive output strength.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hi @LonerDan, thanks for the PR. We will be looking into it as soon as possible.

e5d9d1dd607c4eb26deb0652d1b7d1f291c29bd9


Key to request for administrator privileges

Description of the new feature/enhancement
It would be good having a feature which requests for administrator privileges without opening a new window of the terminal.

Proposed technical implementation details (optional)
With a button and also a keyboard shortcut.

@albertopasqualetto I'm not sure how this would work. There is no "native" Windows "sudo" equivalent. What you are asking for might be better requested from https://github.com/Microsoft/Terminal.

As a quick fix, scoop has a sudo you can download that works reasonably well, but yeah I agree Terminal is probably the better idea, if anything because it would be shell-wide rather than a specific program

Sorry, completely mistaken the git project! Thanks


issue fetching orders using get_history_orders()

I had previously been able to retrieve orders using the get_history_orders() function but as of this morning seems to be raising errors I'm unable to troubleshoot or seen solutions for.
My code is pretty simple,
from webull import webull # for paper trading, import 'paper_webull'

webull_email = **email**
webull_pass = **password**

wb = webull()

did = **did from response headers**
wb._did = did

at_ = **at from response headers**
wb.get_trade_token(at_)

wb.login(webull_email, webull_pass)

I previously used this line to get order history no problem but today am recieving the following response
orders = wb.get_history_orders(status = 'All', count = 20)
{'msg': "org.springframework.validation.BeanPropertyBindingResult: 1 errors\nField error in object 'orderPageQueryParam' on field 'startTime': rejected value [1970-0-1]; codes [typeMismatch.orderPageQueryParam.startTime,typeMismatch.startTime,typeMismatch.java.util.Date,typeMismatch]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [orderPageQueryParam.startTime,startTime]; arguments []; default message [startTime]]; default message [Failed to convert property value of type 'java.lang.String' to required type 'java.util.Date' for property 'startTime'; nested exception is org.springframework.core.convert.ConversionFailedException: Failed to convert from type [java.lang.String] to type [java.util.Date] for value '1970-0-1'; nested exception is java.lang.IllegalArgumentException]",
 'code': '400',
 'success': False}

I've tried uninstalling webull, reinstalling, and upgrading package thinking there may have been an update I missed but no luck. Given the error, I also tried including startTime parameters with no luck. Adding these return a different error which is somewhat expected of "unexpected keyword argument 'startTime'.
I've tested other functions to confirm I'm logged in properly testing wb.is_logged_in() and wb.get_watchlists() which both work as expected.
Any help would be greatly appreciated.
*** also tried following solutions here https://github.com/tedchou12/webull/issues/394 with no luck.

I fix it. just change '1970-0-1' to '1970-1-1' in entpoints.py file

hmm. Doesn't seem to work for me @BobLiu20 Did you happen to change anything else? I replaced in these lines
    def orders(self, account_id, page_size):
        return f'{self.base_ustradebroker_url}/trade/v2/option/list?secAccountId={account_id}&startTime=1970-1-1&dateType=ORDER&pageSize={page_size}&status='

    def history(self, account_id):
        return f'{self.base_ustrade_url}/trading/v1/webull/order/list?secAccountId={account_id}'

    def paper_orders(self, paper_account_id, page_size):
        return f'{self.base_paper_url}/paper/1/acc/{paper_account_id}/order?&startTime=1970-1-1&dateType=ORDER&pageSize={page_size}&status='

@BobLiu20 wanted to bump this. Did you happen to change anything else in the endpoints.py file? I provided the code where I changed the 1970-0-1 to 1970-1-1 like you suggested with no luck. I had also tried variations like 1970-01-01 and others.

Just removed the startTime parameter in request URL and does is the trick. I know it's not the ideal way but does the job for now. Closing and will continue to debug for a more correct solution.
    return f'{self.base_ustradebroker_url}/trade/v2/option/list?secAccountId={account_id}&dateType=ORDER&pageSize={page_size}&status=' ##work around for now, removed the date parameter and works correctly.


feat(enforceStep): Add an enforceStep option (defaults to true)

Add an enforceStep option (defaults to true) to force the value to be rounded to the step, even when modified from the outside.
As requested in #210
Demo: http://jsfiddle.net/ValentinH/ooakat25/

Current coverage is 77.84%

Merging #211 into master will decrease coverage by -0.34% as of 3ffadcf

@@            master    #211   diff @@
======================================
  Files            1       1
  Stmts          628     632     +4
  Branches         0       0
  Methods          0       0
======================================
+ Hit            491     492     +1
  Partial          0       0
- Missed         137     140     +3

Review entire Coverage Diff as of 3ffadcf

Powered by Codecov. Updated on successful CI builds.

Don't want to merge for now. Can reopen if someone really (really) needs it. ;)


Incomptability between Sleuth and Eureka

Hi,
In the LoadBalancerAutoConfiguration, a loadBalancedRestTemplate bean is created and later is Autowired into the class TraceWebClientAutoConfiguration.
The TraceInterceptorConfiguration does the following:
@Autowired(required = false)
private Collection<RestTemplate> restTemplates>

and
@Autowired
private TraceRestTemplateInterceptor traceRestTemplateInterceptor;

Then in a loop we add the interceptor to the array of restTemplates.
The following line will return a ClassCastException:
for (RestTemplate restTemplate : this.restTemplates) {

More specifically:
09:49:00.692 [main] ERROR o.s.boot.SpringApplication - Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.cloud.sleuth.instrument.web.client.TraceWebClientAutoConfiguration$Trace
InterceptorConfiguration': Invocation of init method failed; nested exception is java.lang.ClassCastException: com.sun.proxy.$Proxy96 cannot be cast to org.springframework.web.client.RestTemplate

Any idea why the TraceInterceptorConfiguration  cannot assign the RestTemplate that was created by LoadBalancerAutoConfiguration ?
Thanks a lot in advance,
Rudy

It seems you're missing @EnableAspectJAutoProxy(proxyTargetClass = true)

Thanks a lot for the answer. It solved the problem!


Problem belongsToMany query on search

Bug report
this is the query where the problem occurs:
SQL: select * from type where exists (select * from categories inner join type_has_categories on categories.id = type_has_categories.categories_id where type.id = type_has_categories.type_id and type_has_categories.categories_id = ) and exists (select * from categories inner join type_has_categories on categories.id = type_has_categories.categories_id where type.id = type_has_categories.type_id and categories_id = 1) limit 25
What I did:
I'm adding a search filter:
// Controller

$this->crud->query->whereHas('categories', function ($query) use ($value) {
$query->where('categories_id', '1');
});
or
$this->crud->addClause('whereHas', 'categories', function($query) {
$query->where('categories_id', '=', '1');
});

// model type

public function categories()
{
return $this->belongsToMany('App\Models\Categorie','type_has_categories','types_id','categories_id');
}

What I expected to happen:
I wanted the list of types that belonged to the categorie 1
What happened:
the query does not return anything because it has an error:

type_has_categories.categories_id = ) (is missing)

looking at the query I think it can be simplified

PHP Version 7.2.1
Laravel Framework 5.5.39
backpack/base 0.8.9
backpack/crud 3.3.12
backpack/generators 1.1.13
any suggestions?

I think I found the problem
will update

I reviewed the code, I recalled a superclass, but the id did not pass
now solved

Great to hear that @xdasty . Hope it helps someone. Ok to close this, right? Doesn't sound like a bug in the framework.
Cheers!


jps: Watch logic-square.com

jps requests the watch of the watch_keyword logic-square\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

Approved by tripleee in Charcoal HQ

logic-square\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.


Update Composer dependencies (2022-03-04-00-05)

Visual regression test passed!
View the visual regression test report


pm2 ERR_UNKNOWN_FILE_EXTENSION

Creating new project or jeasx-quickstart from the guide went fine
unfortunatelly on Mac OS with Node.js v20.9.0 I can see this error on launch
$ jeasx dev
node:internal/errors:497
    ErrorCaptureStackTrace(err);
    ^
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension  for /jeasx/jeasx-quickstart/node_modules/pm2/bin/pm2-runtime
    at new NodeError (node:internal/errors:406:5)
    at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:99:9)
    at defaultGetFormat (node:internal/modules/esm/get_format:142:36)
    at defaultLoad (node:internal/modules/esm/load:120:20)
    at ModuleLoader.load (node:internal/modules/esm/loader:396:13)
    at ModuleLoader.moduleProvider (node:internal/modules/esm/loader:278:56)
    at new ModuleJob (node:internal/modules/esm/module_job:65:26)
    at #createModuleJob (node:internal/modules/esm/loader:290:17)
    at ModuleLoader.getJobFromResolveResult (node:internal/modules/esm/loader:248:34)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:229:17) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

Thanks for the report.
Can you upgrade to latest Node 20 (currently v20.11.1)? I've seen this issue before with Node 20.9 at a team member, it seems to be a bug related to Node.
Does it work?

Yes this is correct!
This was related to node version as you mentioned. Upgraded and checked both
node v20.11.1
node v21.6.2
All good now 👍🏻


Adding pagination limits.

Why make this change?
This change allows users to use the runtime config to provide limits on the number of records that can be retrieved through paginated calls. It also gives them the ability to define a default size which will be the size returned when no pagination input is given (no first in case of graphql and no limit in case of rest calls).
What is this change?
It adds the nullable paginationoptions property to runtimeoptions of runtimeConfig.
Default page size is set to 100.
Max page size is set to 100,000 ( can be altered by the user in their runtimeconfig).
A call with -1 pagination input will result in max page size records being returned. Any call with a pagination number higher than max page size will be rejected with bad request.
In config:
default-page-size is an integer
default-page-size default is 100
default-page-size value -1 means "same as max-page-size"
default-page-size value 0 is an error
default-page-size value less than -1 is an error
default-page-size value more than than max-page-size is an error
max-page-size is an integer
max-page-size default is 100,000
max-page-size value -1 means "same as int.MaxValue"
max-page-size value 0 is an error
max-page-size value less than -1 is an error
max-page-size value more than int.MaxValue is an error
In a query (REST or GQL):
$first=-1 means "whatever the max-page-size value is"
$first=less than -1 is an error
$first=0 is an error
$first=(any value more than max-page-size) is an error
Sample configuration file:
{
  "runtime": {
    "pagination": {
      "default-page-size": -1,
      "max-page-size": 1233
    }
  }
}

How was this tested?

For default page value of 100, invalid page value of 0 or <-1 the existing test cases should cover.
Added tests for both GQL and REST for above conditions.

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

pr description says this works for both rest/graphql but only states config property is added for runtime.graphql. Is there also a config for runtime.rest?
Would also be good to know how in our pagination logic you convert $first=-1 into proper sql query. Just  value detection->replacement?

please add example configuration json  to include new property in runtime section.
like this:
{
  "runtime": {
    "pagination": {
      "default-page-size": -1,
      "max-page-size": 1233
    }
  }
}

Documentation https://github.com/MicrosoftDocs/data-api-builder-docs-pr/pull/149

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run


Using the svelte dev tools extension.

Hi,
Thanks for your hard work this is a great tool!
Summary
I want to be able to use the svelte dev tools extension.
https://github.com/sveltejs/svelte-devtools
Currently it seems that any extensions installed for my chrome binary are not available when running my project in dev mode.
Any help would be appreciated.

You can install the dev tools in your regular chrome profile, then tell the plugin to use that profile instead:
# <project-root>/.webextrc.yml
chromiumProfile: "<absolute-path-to-chrome-profile>"

You can find your profile's path at chrome://profile-internals
To learn more, check out the docs.


[FEATURE] Improve the content of About Us Page

Describe the solution you'd like
The content available on about us page is not looking professional. Make some changes to the content and also fix the alignment of the content.
Currently the content is badly formatted and also not aligned properly.

please assign this to me

please assign this to me

ok @amitsethii, I am assigning this issue to you.

The issue is resolved so I am closing it


Availability of sc-rnaseq data

Hi,
I would like to know if the sc-rnaseq (already clustered and labeled) given to CibersortX for example on 6k PBMC and 8k PBMC are available somewhere ?
Thanks for your article and your great work.

Sorry for the late reply!
I'm afraid I don't have that data anymore. If you want to generate data for CIBERSORTx, it should be relatively straight forward to process the 10X genomics PBMC datasets for that.


Add job titles for fr_FR locale

French set of job titles from http://www.emploi.com/emploi/index-metiers.html

The source is © Keljob, I can't put that in a MIT project.


vMBMasterRunResRelease(231): Resource release failure (IDFGH-7906)

Hi, I'm constantly getting a vMBMasterRunResRelease(231): Resource release failure error in the log.
In my application I use polling for about 30 modbus registers every 5 seconds through modbus RTU.
The longer the app runs the more resource release failures I get through time.
So I did some digging where the error originates and got to vMBMasterRunResRelease function inside portevent_m.c file.
As I understand, xMBMasterRunResTake is called before any modbus operation, and vMBMasterRunResRelease is called after.
In vMBMasterRunResRelease function xEventGroupSetBits is called to set event bit, and then immediately check if the bit is set after xEventGroupSetBits returns.
If the bit is not set, the function reports a resource release failure.
The problem here is that xEventGroupSetBits may not always return set bits even if the function has successfully set the desired bits.
If another task is already waiting for these event bits, xEventGroupSetBits will set those bits, then check for any waiting tasks, unblock these tasks and clear the bits. xEventGroupSetBits will in this case return cleared bits, so the error check in vMBMasterRunResRelease will fail even though the bits were successfully set.
I think this line does not work as intended and is pollutting my logs.
MB_PORT_CHECK((uxBits == MB_EVENT_RESOURCE), ; , "Resource release failure.");

@ph2355 ,
This is known thing that has been  originally commented and fixed in preliminary WIP version of modbus_support repo. It does not cause  any critical issue but agree should not be an error.

Thanks for the acknowledgement. Can I expect this to be fixed in a future version?

@ph2355 ,
Yes, sure this will be fixed and I will let you know once it merged. Thank you for the contribution.

I second @ph2355 it's also cluttering my logs as I'm trying to debug fix issue #3 ... can you merge it in @alisitsyn?

@brainstorm, @ph2355 ,
The fix is merged in commit

Close the issue as fixed.


Support an alternative naming scheme for load balancers

Currently we make use of the service's UID. The problem with that is we can't use products like heptio ark to do cluster restores since clients cannot set the UID of a resource (restricted by the api server). The problem is you can associate the load balancer back to the service in the new cluster w/o manual intervention to rename the load balancers.  Alternatively, if we allow the load balancer to be named something like (<optional prefix>-)<namespace>-<service> we can associate an existing load balancer with a given service after restore.
Proposal:
Add an optional configuration argument to use an alternate naming schema. The default will remain UID. If the flag is enabled then we'll first attempt to fetch the load balancer by name then if that fails attempt to fetch the load balancer by uid for backwards compatibility. That gives an upgrade path (albeit manual) to migrate existing LB's to the new naming schema if they want.
cc @prydie

For a default name, instead of uid, you could use a scoped name like
--
i.e., something derived from the names in the existing  configuration values in the tenancy.  The order used here is so that the most meaningful names are first, but what's more meaningful here can certainly be debated.
For customization, I was thinking of a custom kubernetes annotation defined by this project.  With a good default naming scheme people may not need to override it often.  You might also be able to get away with just making the override be totally customer-driven,  i.e., accept the default, or own the naming yourself.  The display name only has to be unique within a tenancy or compartment right?


bankmarketing_train.csv: missing column description

Azure documentation issue guidance
Thanks for opening an issue in the Azure technical documentation repository.
We use GitHub issues as the primary channel for customer and community feedback about the Azure documentation.
Creating an issue
To do a model explanation, it's important to have a describtion of the columns.
"emp.var.rate" what is the meaning of this column or "euribor3m"?
We prefer that you create documentation feedback issues using the Feedback link on the published article - the feedback control on the doc page creates an issue that contains all the article details so you can focus on the feedback part.
You can also create a feedback issue here in the repo. If you do this, please make sure your issue lists:

[ ] The relevant Azure service or technology.
[ ] A link to the published documentation article that you have feedback about.
[ ] Clear, specific feedback that the author can act on.

Pull requests and article contributions
If you know the change that is needed in an article, we encourage you to submit the changes directly using a pull request. If the change is large, or if you want to contribute an entire article, follow these guidelines:

[ ] Don't surprise us with a big pull request or a pull request with a new article! Submit an issue that describes the details of the proposed large change or new article.
[ ] Include the service or technology area.

We'll route the issue to the appropriate content team for review and discussion.
Tech support and product feedback
If you would like to contact Microsoft about other things, such as product feedback or tech support, please review these guidelines:

If you need technical support using Azure, the paid and free support options are described here: https://azure.microsoft.com/support/options/.

Each article in the Azure technical documentation contains a product feedback button - it's best to submit product feedback directly from a relevant article. Otherwise, you can submit product feedback for most Azure products in the following product feedback forum: https://feedback.azure.com/forums/34192--general-feedback.

@fhe-fr thanks for the feedback! Please include the link to the article that you are referencing so that we can better assist.

@fhe-fr please create a new issue linking the document that you want fixed, and we will gladly make this change.


Helm getting into year 2038 issue

lbs.set("modifiedAt", strconv.Itoa(int(time.Now().Unix()))) this line is getting to the unix 2038 issue resulting failure of secret creation with below error.
Error: INSTALLATION FAILED: create: failed to create: Secret "sh.helm.release.v1.operator.v1" is invalid: metadata.labels: Invalid value: "-2147255893": a valid label must be an empty string or consist of alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9.])?[A-Za-z0-9])?')
helm.go:84: [debug] Secret "sh.helm.release.v1.operator.v1" is invalid: metadata.labels: Invalid value: "-2147255893": a valid label must be an empty string or consist of alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9.])?[A-Za-z0-9])?')
https://github.com/helm/helm/blob/e81f6140ddb22bc99a08f7409522a8dbe5338ee3/pkg/storage/driver/secrets.go#L180C2-L180C61

Another option https://play.golang.com/p/rniO4YNgkxg
// Format it as a string
	str := fmt.Sprintf("%v", t.Unix())

Is Kubernetes or Helm in this case running on a 32bit system?

I'm not sure directly.  But I think it is expected that 32-bit systems are affected by the 2028 problem. So even if Helm, or Kubernetes could be compiled for a 32-architecture, it would be expected they would not be able to handle epoch offsets for 2038 and beyond.
ie. a solution should not try and work on 32-bit architectures.

Note, the current PR deals with the configmap and secret storage but not the SQL storage. We need a second PR to deal with SQL storage.


Changed mesh orientations, added normals to triangle mesh, clarified …

…camera.
Cleaned up some Houdini/Karma ApiSchemas or properties.
To make it easier for tools which don't yet read both types of USD meshes, this commit changes the geometry to be rightHanded. There should be no visual differences for renderers/software that recognize meshes could have either winding order. (Issue #70)
The triangulated variant now has normals authored as well now (Issue #69)
Also re-authored the Camera transform using XformCommonAPI, rather than a 4x4 Matrix, so the translate/rotate values are clear.

Not sure why usdchecker is failing to recognize the .mtlx files; probably just doesn't have UsdMaterialX.
Logged as #82

yeah, sadness, I'm guessing there are reasons why mtlx isn't included in usd-core (which I believe is what we're using here)

Two reasons, boiling down to distro from pypi problems. The first is having a zoo of dylibs; that'll be resolvable more easily when USD is able to use the static/monolithic builds of MaterialX; not all the CMake changes have landed in Mx or USD for that though. The other tricky bit is the MaterialX node library itself. I don't know that anyone's investigated how to get that pip installable yet.


Add validation middlewares for challenges api endpoints

First, update the API Contracts for the following changes. Once the API Contracts are approved start working on the requirements.

Requirements

There are no request payload validations for challenges POST APIs, Add the validations.
Change /challenges/subscribe to PATCH as the data is being updated and not created

@whyDontI I am working on refactor PR in which I have added validator for challenges POST api, so we only need to update /challenges/subscribe to PATCH.


Convert to puppet 4+ data types

Also update some Gemfile versions now that puppet 3 (and older rubies)
aren't supported.

I've been working on postgresql support. Those changes were getting a bit big for a single PR, so I'm pulling it apart to create smaller chunks of changes to review.

@alexjfisher sounds like a plan. So far nothing out of the ordinary but the Gemfile needs a little adjustment so everything's in alphabetical order.

Thanks!

@alexjfisher sorry, I merged this to master by accident. Can you please open a new pull request to develop?

My bad.  I'm pretty sure I branched off develop, but at the time the HEAD of develop and master were the same commit.  Then didn't pay attention when opening the PR.
Try https://github.com/sensson/puppet-powerdns/pull/26 and feel free to force push to master to permanently delete the stray commits if that works for you.  (I won't tell anyone! Promise! ;) )

No problem. And I'm not too worried about those commits. It can stay as it is now. Thanks for all your work once again!


Ephemeral branching

relevant for #1844

relevant for #1844

The linked issue tracks multiple issues and "relevant for" is a vague relation, so this does not explain what this PR achieves, how and why. While the changelog entry or the PR title can sometimes be enough for explaining a PR, it's not the case here. Would appreciate a short explanation next time of the what, how and why, as I think it could help make the review faster. I'm open for discussion if you disagree.


The navigator uses 100% of cpu in stdout mode

ISSUE TYPE

Bug Report

SUMMARY

When running any playbook with a navigator in stdout mode it uses 100% of CPU core.
My playbook
- hosts: localhost
  tasks:
    - command: sleep 40

command:
ansible-navigator run ./playbook.yml -m stdout
And when I check the htop it uses 100%.
ANSIBLE-NAVIGATOR VERSION

ansible-navigator 1.1.0

Run on x86_64 centos 8.4.2105 with podman version 3.2.3.

Confirmed issue. I get the same problem on :
ProductName:	macOS
ProductVersion:	11.6
BuildVersion:	20G165
ansible-navigator 1.0.0
Python 3.9.7


number array with empty values

numbers[]

1;2

3;5

Shold be:
[{
  "numbers": [
    1,
    2
  ]
},
{
  "numbers": []
},
{
  "numbers": [
  3,
  5
  ]
}]

and not:
[{
  "numbers": [
    1,
    2
  ]
},
{
  "numbers": [
  ""
  ]
},
{
  "numbers": [
  3,
  5
  ]
}]

screenworker: There is an option in the current version to omitEmptyFields; omit empty values and name-value pairs where the value is empty.
The data in this test is what would be returned in your example - the excel package I use returns '' for empty cells. Note the use of default options
  it 'should omit empty values when directed', ->
    data = [[ 'numbers[]' ],
            [ '1;2'],
            [ ''   ],
            [ '3;5']]
    result = convert data, DEFAULT_OPTIONS
    console.dir result

produces
[ { numbers: [ 1, 2 ] },
  { numbers: [ '' ] },
  { numbers: [ 3, 5 ] } ]

setting omitEmptyFields = true has this effect
  it 'should omit empty values when directed', ->
    o =
      omitEmptyFields: true
    data = [[  'numbers[]' ],
            [ '1;2'],
            [ ''   ],
            [ '3;5']]
    result = convert data, o
    console.dir result

produces
[ { numbers: [ 1, 2 ] },
  {},
  { numbers: [ 3, 5 ] } ]

Does this help?

Hey Stevetarver,
i also tried with and without "omitEmptyFields" option in version 2.0.0 and 2.0.1 but always the same result. Maybe my option combination ignores that option feature:
var src = 'test.xlsx';
var destDir = './output';
var options =  {
            sheet: 1,
            isColOriented: false,
            omitEmtpyFields: true
        };

        var destPath = destDir + '/' + sheet + '.json';
        convert(src, destPath, options, function( err ) {
            if (err) {
                console.log(err);
            }
        });

Your code looks right to me; a project would simplify debugging. Attached is a project that shows the code is working as expected - what is shown above. npm install it, then coffee issue17.coffee to produce the above output.
excel-as-json.17.zip

Thank you for your example project :)
You are right with coffee!
But have you ever tried it with node?
Please try it with my example below (npm install and node issue17.js) and you will be able to see the same result as in my first post.
excel-as-json_node.17.zip

It´s an spelling issue of the option parameter :(
omitEmtpyFields: true has to be omitEmptyFields: false
My mistake. I am sorry.


Add docs for moniroting dashboard for MI

Purpose

Add docs for monitoring dashboard

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.SachithK seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


help~

How to make two rigid bodies not push each other?thanks~

Pinning down an answer would require more information.
Here's one random option: if you made the bodies kinematic, they would no longer respond to any forces (including collisions), so they'd no longer push each other. Instead, if they had velocities implying penetration, they'd go straight through each other.


Cannot use with Create React App

Create a new React App:
npx create-react-app debug-libauth
cd debug-libauth
npm i @bitauth/libauth --save

Add to App.js
import * as bitcoin from '@bitauth/libauth'
console.log(bitcoin)

Run the app
npm run start

Observe the error
<EMAIL_ADDRESS>112:37
Module parse failed: Unexpected token (112:37)
File was processed with these loaders:
 * ./node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
|   }), {});
|   const entityOwnership = Object.entries(template.entities).reduce((all, [entityId, entity]) => ({ ...all,
>     ...Object.keys(entity.variables ?? {}).reduce((entityVariables, variableId) => ({ ...entityVariables,
|       [variableId]: entityId
|     }), {})

Version info:
 "@bitauth/libauth": "^1.15.3",
    "@testing-library/jest-dom": "^4.2.4",
    "@testing-library/react": "^9.5.0",
    "@testing-library/user-event": "^7.2.1",
    "react": "^16.13.1",
    "react-dom": "^16.13.1",
    "react-scripts": "3.4.1"

Pretty sure is this CRA / Babels fault and not this library... this worked a few weeks ago.

As expected... this error goes away when using create react app with typescript...
create-react-app debug-bitcoin-ts-typescript --template typescript

Sigh...

hacking on the files... it looks like ?? and optional chaining are in javascript source files... https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-7.html#optional-chaining

It would appear this is an issue with this module after all...
I made a test module
const derp  = null;
module.exports = {
    data: derp ?? {'foo': derp?.bar}
}

and I can import it just fine without the loader exploding....
import * as foo from './test-module'
console.log(foo);
console.log(require('./test-module'))

Hi @OR13, sorry I missed responding to this earlier! Did you get this issue figured out? If not, could you push an example of a breaking repo to GitHub? (I'm afraid I can't reproduce locally – I suspect that might mean you're getting a different version of some dependencies from your local npm or yarn cache. You could try to clear that too.)
For a working example of a Create React App which uses Libauth, check out Bitauth IDE on GitHub.

Thanks! I ended up switching to another implementation, if you can't repro with the latest versions, its probably been fixed and you should feel free to close this issue.

I just retested this, and it's still happening.
https://github.com/OR13/bitauth-cra
npm i
npm run start

@OR13 thanks for bringing this back up! Right now it seems to be a bug in Create React App – I added a comment to an existing issue here: https://github.com/facebook/create-react-app/issues/9468
I think it may be a regression in CRA, since bitauth-ide is working as expected (react-scripts at v3.4.1).

This has been fixed by https://github.com/bitauth/libauth/pull/62. 🎉
Thanks @rkalis!


No padding around chapter sub titles in docs

There is not padding around the subtitle of a chapter when the browser width is shrunk to about < 800px.

cc @naomiblack @wardbell

Closed by #687


MultipleStreamViewModel with parameters

How would you initialize/use MultipleStreamViewModel when a Stream which needs to take in parameters? And those parameters change over the lifetime of your model; and you have to cancel that stream and listen to a new stream with new values for your parameters.
e.g.:
Stream<int> getIdsByDate(DateTime date) async* {
 ...
}

oooooooo, good question. This was probably something I was going to run into in one of the new apps we're building that will be using firestore and requires streams to change based on filters. I guess there's two routes I can take here.

Add functionality to allow you to mark a model as dirty so it gets rebuilt. Mostly likely by calling like markNeedsRebuild()

Add an interface similar to ChangeNotifier with a function notifyDataSourceChanged() which will rebuild the viewmodel, unsubscribe current source and re-subscribe.

Reading those two things again it'll basically be 1 implementation but I'll leave it as two points so I can process better. I think I'll actually implement this now. I want to show it in the next video as well so the timing is great.

@RollyPeres just pushed the functionality. Basically once you've changed your source for the stream you can call notifySourceChanged and it will cancel the current streams then rerun all the initialisation logic.
In the example project you can see how it's called.


[OPPRO-105] Fix the early release of memory pool

What changes were proposed in this pull request?
Found segfault in Velox's pool free. Fixed this issue by expanding the life time of Velox's memory pool.
Register min and max.
Depends on: https://github.com/oap-project/velox/pull/14.
How was this patch tested?
Verified TPC-H on Jenkins.

cc @JKSelf

Merging this PR. If any issue found, please still comment here.


Different behaviour depending on service account settings even when kr8s should ignore these

Which project are you reporting a bug for?
kr8s
What happened?
kr8s is giving me errors in one of my container environments, where it behaves differently to kubectl run directly in the same container. Because kubectl behaves differently, I'm reporting this as a bug, although I'm not actually certain that it is a bug in kr8s; it could be a configuration error in my environment or just a misunderstanding about authentication. In a different environment, where the code was developed, my code that uses kr8s runs fine, the difference being that no service account is configured there.
The start of the traceback in the problematic environment is as follows - the code in pods.py below is within a FastAPI endpoint, whereas the kr8s api is configured in the FastAPI startup method, described below
2023-12-07 00:33:50,340 - ERROR - pods.py - fastapi - Failed to list the pods with exception pods is forbidden: User "system:serviceaccount:space-mct:mct-service-status-api-dev" cannot list resource "pods" in API group "" in the namespace "space-mct"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/kr8s/_api.py", line 136, in call_api
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://<IP_ADDRESS>/api/v1/namespaces/space-mct/pods'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/src/app/api/routes/pods.py", line 130, in get_pods_in_namespace
    for pod in await kr8s.asyncio.get("pods", namespace=namespace):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kr8s/asyncio/_helpers.py", line 65, in get
    return await api._get(
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kr8s/_api.py", line 341, in _get
    async with self._get_kind(
  File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kr8s/_api.py", line 270, in _get_kind
    async with self.call_api(
  File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kr8s/_api.py", line 149, in call_api
    raise ServerError(
kr8s._exceptions.ServerError: pods is forbidden: User "system:serviceaccount:space-mct:mct-service-status-api-dev" cannot list resource "pods" in API group "" in the namespace "space-mct"
2023-12-07 00:33:50,340 - ERROR - pods.py - fastapi - Failed to list the pods with exception pods is forbidden: User "system:serviceaccount:space-mct:mct-service-status-api-dev" cannot list resource "pods" in API group "" in the namespace "space-mct"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/kr8s/_api.py", line 136, in call_api
    response.raise_for_status()
  File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://<IP_ADDRESS>/api/v1/namespaces/space-mct/pods'

The kr8s auth.token and auth.server settings are defined in a .kubeconfig file, which are the ones I want to use. In the environment that exhibits the problem, there is a service account configured via KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables, but I'm trying to avoid using that. Code using the "old" vanilla kubernetes api and kubectl both behave fine in this environment, but my kr8s functions break, possibly because kr8s attempts to authenticate against the service account. I'm not sure exactly what's going on to trigger the failure.
As an attempted workaround for this, I tried directly calling the _load_kubeconfig() method in my startup code as follows:
client = kr8s.api(kubeconfig=KUBECONFIG, serviceaccount=None)
print(client.auth.server)
await client.auth._load_kubeconfig()
print(client.auth.server)

I added this because the api() method call doesn't run _load_kubeconfig() in the environment with the service account. Looking at the kr8s _auth.py reathenticate() method logic, this is because _load_service_account() populates self.server and thus skips the subsequent _load_kubeconfig(). The kr8s docs say the lookup order is:

~/.kube/config (or the path set by the KUBECONFIG environment variable)
/var/run/secrets/kubernetes.io/serviceaccount
but this seems not to match what actually happens, and having a service account configured seems to preclude specifying anything else in the .kubeconfig, which is not what I expected. Calling _load_kubeconfig() directly updates auth.token and auth.server from the .kubeconfig, but this doesn't actually fix things; it seems that kr8s continues to attempt to authenticate against the service account. The two print statements in the startup code snippet output

[https://<IP_ADDRESS>:443](https://<IP_ADDRESS>/)
https://api.imgnetes.beamline.synchrotron.org.au:6443/

and I would expect kr8s to use the updated server details on subsequent calls until something causes a reauthentication, but as you can see from the traceback, it reverts to using the service host address.
I couldn't find anything in the kubernetes docs that define the expected lookup order, so I'm not sure if the kr8s authentication is implemented correctly or not, or what else I can look at to understand the problem.
Anything else?
No response

Thanks for raising this @gazzar. I agree that the code doesn't match the documentation.
I've opened #237 with a potential fix. Would you mind installing it and checking that it resolves your problem?
pip install git+https://github.com/jacobtomlinson/kr8s.git@fix-auth-order

Thank you @jacobtomlinson for looking into this so quickly. I'll have to wait until next week (hopefully first thing) to try this and let you know whether it works in both my environments.

Hi @jacobtomlinson
Good news. Preliminary testing shows that the problem I described is resolved by this change in my dev environment. Tomorrow I want to also check it in the environment that was already working to verify that still behaves as before.

@jacobtomlinson I can now confirm that your authentication-order fix is working well in all my environments and has not caused any regressions. Thanks again for following up on this so quickly.

Great thanks for confirming @gazzar. I've just released v0.12.6 which contains this fix.

Amazing speed again! Thanks @jacobtomlinson !


proxyMap performance issue

In my application, if there are several hundreds of items in a proxyMap, each item being an object themselves then the performance of the proxyMap decreases severely, especially when writing to it.

#943
There will be a release soon that improves performance on proxyMap and proxySet.

🎉  https://github.com/pmndrs/valtio/releases/tag/v2.1.0

@sethyuan Let us know how it works out for you

It's a lot faster now, what used to take 18s now take 2s~3s. However when I switched to use the native Map class, it took only milliseconds so I think there is room to improvement.

Are you using object keys for the map? Otherwise, if your keys are just strings or numbers, can you try a plain object instead of Map?

My key is a number, potentially a bigint, anyway, I modified the code so it uses now plain object and ran the same test, it now is in milliseconds too, much faster than proxyMap.

Hm, that's weird. I tried a simple benchmark, and the result between plain proxy and proxyMap is less than a double.

diff
diff --git a/tests/proxyMap.bench.ts b/tests/proxyMap.bench.ts
index 8a6bacf..5f60407 100644
--- a/tests/proxyMap.bench.ts
+++ b/tests/proxyMap.bench.ts
@@ -7,7 +7,7 @@ import { proxyMap } from 'valtio/utils'
 function generateTestData(size: number): [number, number][] {
   const data: [any, any][] = []
   for (let i = 0; i < size; i++) {
-    data.push([{ id: i }, { i }])
+    data.push([i, { i }])
   }
   return data
 }
@@ -15,9 +15,23 @@ function generateTestData(size: number): [number, number][] {
 const TEST_SIZES = [1000, 10_000, 100_000]

 TEST_SIZES.forEach((size) => {
-  describe.skip(`Insertion -${size} items`, () => {
+  describe(`Insertion -${size} items`, () => {
     const testData = generateTestData(size)

+    bench('Native Map', () => {
+      const map = new Map<number, number>()
+      testData.forEach(([key, value]) => {
+        map.set(key, value)
+      })
+    })
+
+    bench('proxy', () => {
+      const map = proxy<Record<number, number>>({})
+      testData.forEach(([key, value]) => {
+        map[key] = value
+      })
+    })
+
     bench('proxyMap', () => {
       const map = proxyMap<number, number>()
       testData.forEach(([key, value]) => {
@@ -127,7 +141,7 @@ TEST_SIZES.forEach((size) => {
     })
   })

-  describe(`snapshot & modify -${size} items`, () => {
+  describe.skip(`snapshot & modify -${size} items`, () => {
     const testData = generateTestData(size)
     const oneData = generateTestData(1)[0]!

  Native Map - tests/proxyMap.bench.ts > Insertion -1000 items
    81.30x faster than proxy
    101.51x faster than proxyMap

  Native Map - tests/proxyMap.bench.ts > Insertion -10000 items
    31.13x faster than proxy
    45.70x faster than proxyMap

  Native Map - tests/proxyMap.bench.ts > Insertion -100000 items
    30.77x faster than proxy
    41.79x faster than proxyMap


FzfLua marks jumps to the line, not the position

When using the marks filter with FzfLua it uses ' instead of ` which jumps to the correct line, but not the correct column.
By just changing the ' to ` in the goto_mark-function it jumps correctly:
M.goto_mark = function(selected)
...
  vim.cmd("normal! '" .. mark)
...

M.goto_mark = function(selected)
...
  vim.cmd("normal! `" .. mark)
...

I suggest both options should be available (maybe named "marks" and "marks_line")?

I created a suggestion and made a pull request. If backwards-compatibility is important the line+column mode could be the one with a new name, so that marks just takes you to the start of the line.

Closed via #892, ty @damianoognissanti!


git-hound: 1.4 -> 1.7.2 unbreak

Description of changes
fixes https://github.com/NixOS/nixpkgs/issues/276787

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 276803 run on x86_64-linux 1

1 package built:

git-hound


chore(deps): upgrade dev dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-dev-dependencies-2.x" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 2233

Questions ?
Please refer to the Backport tool documentation


that's interesting work; when will you open the code?

that's interesting work; when will you open the code?

when will you open the code?

Thank you for your feedback! The author is currently engaged in a major research project, and the code is being organized. Updates will be released continuously, so please stay tuned.


Use better direct lowering for pack/unpack operations to AIE DMA operation representation

Currently the iree_linalg_ext.pack operations which map well to the aie dma operations are decomposed to air.dma_memcpy_nd which doesnt seem to model it as well since it doesnt have a concept of wrap or inner tile. We should add a new op in air dialect that maps one to one to iree_linalg_ext.pack-> air.dma_op -> aie.dma_op

@erwei-xilinx , please let me know what your thoughts are on this approach.

@powderluv @nirvedhmeshram I will need more context about this issue. air.dma_memcpy_nd op does have fields which correspond to wraps: the op is essentially a combination one memcpy op plus two subviews (one for src memref and one for dst). Each subview in the op comes with a sizes field which is essentially wraps. The subview also comes with strides and offsets to represent an inner tile.
Currently we do have a workflow which can perform one-to-one mapping between iree_linalg_ext.pack/unpack to air.dma_memcpy_nd. This workflow consists of two passes, where the first one called iree-amdaie-decompose-pack-unpack-to-air which currently is under iree-amd-aie repo; the second one called air-copy-to-dma which is under mlir-air.
From air.memcpy_nd_op down it is more complicated. aie.dma doesn't exist, and mlir-aie dialect represents data movers using DMA BD programs which map to physical AIE DMAs. The pass -air-to-aie in mlir-air currently handles the lowering from logical air.dma_memcpy_nd to physical AIE DMA BDs.

air.dma_memcpy_nd, as a logical representation of data movement, is currently agnostic wrt AIE architecture versions: the op is compatible with both AIE1 and AIE2 architectures. Architecture specific hw constraints are enforced in the downstream passes which lowers the logical data movement to physical resources.
We can squash the whole process and put the pack-to-aie flow all within one single pass, but it may not be one to one, if the later is one single DMA BD, because due to AIE2 BD constraints (wrap cannot go beyond [0, 1023], # dims cannot go beyond 4 for shim/memtile and 3 for tile, only the highest dim wrap can do repeat, etc), one logical pack/unpack data movement may need to lower to a finite-state-machine consisting of multiple BDs.

This makes sense... I think the main idea is to fold the logic of iree-amdaie-decompose-pack-unpack-to-air and what happens in air-copy-to-dma operation. They shouldnt need to be split into two parts. @nirvedhmeshram , you and I maybe need to find some time to look at IR and go a bit deep on this issue.

I think unifying the iree-amdaie-decompose-pack-unpack-to-air and air-copy-to-dma makes sense to me, now that MLIR-AIR is integrating into IREE. The original rationale behind this separation was that MLIR-AIR could use the memref.transpose/subview/expand_shape ops as frontend in addition to iree_linalg_ext.pack/unpack.
Here are some unit tests on iree-amd-aie side for iree-amdaie-decompose-pack-unpack-to-air, and here are some on mlir-air side for air-copy-to-dma.

@erwei-xilinx could you please explain the func1 in the decompose-pack lowering here
I am wondering why we needed an expand shape and not keep the same buffer dimensions from the original pack op?
Secondly, one I try to apply the condensation pattern from air-copy-to-dma (note I didnt apply the entire pass) on the resulting IR, I am getting a crash.

@nirvedhmeshram The memref.expand_shape op expands the shape of the memref to increase the rank of the memref from 4 to 6, then the memref.transpose op performs a permutation, exchanging dim 2 with 4. I don't think memref.transpose can permute a rank 4 memref into a rank 6 memref.
The output of that test is expected to fail in -air-copy-to-dma, because it is just a snapshot of a series of memref layout transformation ops, where I deleted the "layout" field of the input memref type to memref.expand_shape for simplicity. If you look at func6, which is the complete GEMM example, and search for memref.expand_shape, then you will see it's original form of that func1 test, and how the memref.expand_shape input inherits its layout from predecesors.

@erwei-xilinx I was wondering something about how the offset propagation is done when we rank reduce in the current lowering,
lets take this dma operation post condensation
https://gist.github.com/nirvedhmeshram/8e7508a34a5d40b55c1a980c6f93496b#file-condensed_dma-mlir-L33
If you compare it from the starting point here
https://github.com/nod-ai/iree-amd-aie/blob/main/compiler/plugins/target/AMD-AIE/iree-amd-aie/Transforms/test/pack_to_air.mlir#L144
Since offset step can be calculated as size x stride In the final case the the offset step that will be applied is 2x8=16, but the original case the offset step is 1x128=128, so is this technically correct or something our codegen is not getting affected by currently due to these kind of offsets always being 0?

@nirvedhmeshram I think at the moment it's the latter if I understand the question correctly. The pack lowering to AIR hasn't been exercised extensively with offsets not being zero. Once lowered to AIR we have some examples showing how offsets are propagated.

@erwei-xilinx thanks for clarifying, also after sending that message I realized I might not be thinking of offsets correctly, are they just multiplied with the stride, to get the starting element , or are they multiplied with the stride AND the size, in either case I think the lowering is not ending up with the correct offset but understanding the semantics will help us do the correct offset calculation.

@erwei-xilinx thanks for clarifying, also after sending that message I realized I might not be thinking of offsets correctly, are they just multiplied with the stride, to get the starting element , or are they multiplied with the stride AND the size, in either case I think the lowering is not ending up with the correct offset but understanding the semantics will help us do the correct offset lowring in the new pattern.

Are we talking about subview or the DMA op?

@erwei-xilinx thanks for clarifying, also after sending that message I realized I might not be thinking of offsets correctly, are they just multiplied with the stride, to get the starting element , or are they multiplied with the stride AND the size, in either case I think the lowering is not ending up with the correct offset but understanding the semantics will help us do the correct offset lowring in the new pattern.

Are we talking about subview or the DMA op?

I think we are agreeing that The DMA op is not offsetting by same number of elements as the sub view op in the pack representation in the two IR's I linked.

@erwei-xilinx thanks for clarifying, also after sending that message I realized I might not be thinking of offsets correctly, are they just multiplied with the stride, to get the starting element , or are they multiplied with the stride AND the size, in either case I think the lowering is not ending up with the correct offset but understanding the semantics will help us do the correct offset lowring in the new pattern.

Are we talking about subview or the DMA op?

I was asking about DMA, but I could not find documentation for sub view either, although I would assume it is just multiplied to the stride in that case.

@nirvedhmeshram I think it might be useful to capture any offset errors with a unit test, which helps to show whether pack is not generating the memref.subview correctly, or memref.subview is not generating DMA correctly.

@erwei-xilinx thanks for clarifying, also after sending that message I realized I might not be thinking of offsets correctly, are they just multiplied with the stride, to get the starting element , or are they multiplied with the stride AND the size, in either case I think the lowering is not ending up with the correct offset but understanding the semantics will help us do the correct offset lowring in the new pattern.

Are we talking about subview or the DMA op?

I was asking about DMA, but I could not find documentation for sub view either, although I would assume it is just multiplied to the stride in that case.

It's hard to describe in chat. Lets chat tomorrow sometime about it

I have a pattern for pack to dma here
https://github.com/nod-ai/iree-amd-aie/pull/178

Here is how the IR looks like https://gist.github.com/nirvedhmeshram/188a0770133a0e03c40d234776a57080
so the first thing to verify is, if this is the correct lowering, @erwei-xilinx  @stephenneuendorffer  @MaheshRavishankar

also @jtuyls @yzhang93

Hi @nirvedhmeshram , I have a couple of questions:

In func0, why do we have [%c128, %c16, %c16, %c1] here on the src strides? I would have expected [%c128, %c128, %c16, %c1]. Because size is 1 for that dimension, it shouldn't matter, but it looks weird to me that 16 is generated instead of 128.

air.dma_memcpy_nd (%alloc[%c0, %c0, %c0, %c0] [%c1, %c1, %c8, %c16] [%c128, %c128, %c16, %c1], %alloc_0[%c0, %c0, %c0, %c0] [%c1, %c1, %c8, %c16] [%c128, %c16, %c16, %c1]) : (memref<1x1x8x16xi32, 1>, memref<8x16xi32, 1>)

Right now, the pack operation seems to be completely offloaded to the src side of the air.dma_memcpy_nd and dest is always a linear write? Is this on purpose or for simplicity? This does have some implications, for example exceeding BD dimensionality/running out of BDs more quickly due to limited dimensionality of the BDs. Also, in general there is a choice to be made here with multiple solutions and this can affect performance (maybe a consideration for down the road). A different way of converting pack would be to split the transformation across src and dest, for example by handling the tile part on the src side and the transpose part on the dest side. Not sure whether this does matter as this could potentially be rearranged/optimized down the line as well. Any thoughts from you, @erwei-xilinx ?

Hi @nirvedhmeshram , I have a couple of questions:

In func0, why do we have [%c128, %c16, %c16, %c1] here on the src strides? I would have expected [%c128, %c128, %c16, %c1]. Because size is 1 for that dimension, it shouldn't matter, but it looks weird to me that 16 is generated instead of 128.

air.dma_memcpy_nd (%alloc[%c0, %c0, %c0, %c0] [%c1, %c1, %c8, %c16] [%c128, %c128, %c16, %c1], %alloc_0[%c0, %c0, %c0, %c0] [%c1, %c1, %c8, %c16] [%c128, %c16, %c16, %c1]) : (memref<1x1x8x16xi32, 1>, memref<8x16xi32, 1>)

Right now, the pack operation seems to be completely offloaded to the src side of the air.dma_memcpy_nd and dest is always a linear write? Is this on purpose or for simplicity? This does have some implications, for example exceeding BD dimensionality/running out of BDs more quickly due to limited dimensionality of the BDs. Also, in general there is a choice to be made here with multiple solutions and this can affect performance (maybe a consideration for down the road). A different way of converting pack would be to split the transformation across src and dest, for example by handling the tile part on the src side and the transpose part on the dest side. Not sure whether this does matter as this could potentially be rearranged/optimized down the line as well. Any thoughts from you, @erwei-xilinx ?

Great observation about the strides, it took me some time to digest and definitely want to have a discussion about it. So we are setting the strides for memref<1x1x8x16> and lets call this dimensions  here the interesting thing I learned is that the tiledA dimension is still strided as outer to B dimension so the memory layout is actual

iterate A (orignally stride 16, due to tiling stride is = orignal stride x tile size = 16x8 = 128)
iterate tiledA (stride = original outer dim(A) stride = 16)
iterate B (originally stride 1, due to tiling stride is = orginal stride x tile size =1x16 = 16
iterate tiled B (stride = original outer dim(B) stride = 1)
so new stride for  = <128x16x16x1>

I havent yet thought much about this problem but I agree with you that there are hardware implications so its important to explore different code-geneartion methods here. We should be able to do that at the linalg level and enable support down the stack. Lets see what @erwei-xilinx and others think about it.

@nirvedhmeshram and @jtuyls Re 2, the current pack lowering in AIR is generating wraps and strides which operate only on sources and not destinations. Although this doesn't mean that the destination-side DMAs are not doing any data rearrangements (tiling generates memref.subviews on destination-side DMA BDs), there could still be extra hw benefits to offloading some data rearrangements on the destination side. AIR currently isn't able to do that.
In AIR, pack and unpack operations are not the only operations that generate wraps and strides to DMA BDs. The memref.subviews, generated from tiling, also could rely on DMA BDs to do data rearrangements. Given that the pack-to-dma lowering is now direct, there may be the need for an additional mechanism to update the wraps and strides if there's an extra memref.subview.

In AIR, pack and unpack operations are not the only operations that generate wraps and strides to DMA BDs. The memref.subviews, generated from tiling, also could rely on DMA BDs to do data rearrangements. Given that the pack-to-dma lowering is now direct, there may be the need for an additional mechanism to update the wraps and strides if there's an extra memref.subview?

If the subview is consumed by a pack/unpack operation the direct lowering should fold it in, if there are other consumers of the sub-view then yes potentially need a new lowering.

@nirvedhmeshram I did a quick experiment with a Transform Dialect tiling script, and see that at the moment the memref.subview ops (or tensor.extract_slice) do not get automatically folded into pack. See code snippet below:
                %7 = affine.apply #map1()[%arg2]
                %8 = affine.apply #map2()[%arg4]
                %subview_4 = memref.subview %alloc[%7, %8] [32, 32] [1, 1] : memref<64x2048xi32, 1> to memref<32x32xi32, strided<[2048, 1], offset: ?>, 1>
                %alloc_5 = memref.alloc() : memref<4x8x4x8xi32, 2>
                iree_linalg_ext.pack %subview_4 outer_dims_perm = [1, 0] inner_dims_pos = [0, 1] inner_tiles = [4, 8] into %alloc_5 : (memref<32x32xi32, strided<[2048, 1], offset: ?>, 1> memref<4x8x4x8xi32, 2>)

I think your point makes sense, and it should be legal to fold them into pack ops.

@erwei-xilinx I was thinking the folder will happen when doing the pack to dma lowering, I already have an example in the IR linked in this message https://github.com/nod-ai/iree-amd-aie/issues/148#issuecomment-1959952187

I see. I understand now. Thanks!

Great observation about the strides, it took me some time to digest and definitely want to have a discussion about it. So we are setting the strides for accesing in the packed layout defined by memref<1x1x8x16> and lets call these dimensions  here the interesting thing I learned is that the tiledA dimension is still strided as outer to B dimension so the memory layout is ...

Thanks, this makes sense now for the case where tiling size last dimension == last dimension size. It would be useful though for me to add another example/test like func1 but without outer_dims_perm to see how the transformation works on a pack with tiling sizes different from initial sizes, but no permutation happening. Could go directly into the PR though.

@jtuyls I have updated the gist to include the case you suggested https://gist.github.com/nirvedhmeshram/188a0770133a0e03c40d234776a57080
Going to do a clean up/ add tests and mark the PR ready for review soon as well.