Go API

Hi!,
I'm really interested on contribute in my spare time on this project, it  would be great for me port the Python libraries to Go, or help with this task, the problem is that after read the "Contributing guidelines" I still don't know how to do it.
I tried to sing the Individual CLA, but I get an error that says: "You must be an owner of the contributors group in order to submit this CLA.".

Some related projects that I implemented in Go: https://github.com/alonsovidales/pit - This is a distributed recommender system based in the "Adaptive Bootstrapping of Recommender Systems Using Decision Trees" paper by Yahoo. The site of the project: http://pitia.info/
And the Go ML: Some machine learning algorithms in Go: https://github.com/alonsovidales/go_ml I also implemented Neural Networks with CUDA: https://github.com/alonsovidales/go_ml/tree/cuda_implementation
Thanks,

:+1:

I'd be interested in helping @alonsovidales !

Idk if I'm qualified but I'd like to do whatever to help @alonsovidales !

I can help too because I need this library in Go

+1/0

I still have no answer, but we could start working on the libs, and if Google wants, they can use the code, if not we can release the libs as a separate project :) I think that this shouldn't be a problem since the project is licensed using an Apache 2.0 license.
My main concern is that we could be doing the same that they are doing the same in parallel, so, by the moment we can keep reading and trying to understand how the currently available libs works, and if this weekend we don't have any news, we can prepare, plan and distribute the different tasks to do the libraries.
What do you think?
I'm going to send an e-mail to the golang-nuts mail list.

agree! wait till weekend then we can decide.

:+1:

Great move!

The current state of Go is that we have a few people who've expressed interest internally in contributing language bindings, but nothing concrete at the moment. I would suggest that if you got started on this, which would be awesome, you could whip up a proposal (preferably in the form of prototype code) and run the design by the discuss mailing list.

Thanks @vincentvanhoucke , then, let's try to prepare a prototype and let's see how it works :) I had been digging a bit in the Python libs, and it shouldn't be too difficult to prepare the Go libraries.
Btw, which is the discuss mailing list?, I can't find it.
I love the project and it would be awesome for me be able to contribute.

Found the discuss mailing list, sorry :)

Should we keep this issue open to track this?

Sure, better keep this issue open so other people can find it easily.
I think that it would be better to move the conversation about the prototype, etc to another place, we can:

Use the official "discuss mailing list"
Create a new mailing list just for this
Use the issue tracker of a GitHub repo, I created this repo just to start playing around: https://github.com/alonsovidales/tensorflow

Regarding to the prototype, my proposal is to use one of the examples:
http://tensorflow.org/tutorials/mnist/beginners/index.md
And studying how the Python libs works, port just all the involved code to execute the example, since an example like the one for beginners covers most of the functionality, and can give us a great overview of how to port all the libraries.
To reproduce the behaviour of the Python libs, the best option would be to use TDD, we can go method by method and based on the input/output of the python code prepare the test on Go.
We can distribute the tasks to prepare the prototype or work in parallel and put all our ideas together at the end. Since for me the goal here is try to get a general idea about how to implement the final version, tasks that we have to do, etc, I think that work in parallel can be a good approach.
What do you think?

:+1:

Just a little update of what I have been researching this weekend. I couldn't find too much time for this, but I made some small progresses.
I have been researching how the Python libs works internally, and basically, the logic to generate the graph and plan how to execute it, etc is managed from Python that sends all the graph to be executed to the C++ libraries, this libs take the graph and executes all the nodes of the graph. They are working on a pure C++ version of this logic, but it is still not ready.
In order to execute the C++ code from python they use SWIG: www.swig.org and to build the system Bazel: http://bazel.io/ .
The good news is that SWIG also supports Go ( http://www.swig.org/Doc2.0/Go.html ) and Bazel introduces support for Go ( http://bazel.io/docs/be/go.html )  after the 0.1.1 version.
The bad news are that I couldn't build the system using Bazel 0.1.1 by many different problems with the linker, etc that I'm trying to resolve. And the other problem is that we have to create the .i files for SWIG in order to prepare the build for Go, I almost have this part done, but I'm also having a lot of problems.
The main issue is that I have no experience with Bazel and SWIG, so I'm spending most of the time reading documentation and trying to fix stupid problems caused because of I misunderstand something from the docu.
In case of I finally can't prepare a wrapper using SWIG, I think that I'm going to try to prepare the prototype using cgo directly, I already have a small proof of concept that generates the session in Go, but it would be much better if the libs works with SWIG and Bazel as the Python libs does.
I'm not having too much spare time lately, but it is being super interesting for me to research how all this system is done, the code is pretty well documented, and easy to understand.

It seems like Bazel doesn't yet have C/C++ interop for Go so cgo may be the only option unless Bazel adds support for it soon.
http://bazel.io/docs/be/go.html

I think that it would be possible to do a small workaround using a sh_binary to compile run swig like they are doing with the Python libs:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/BUILD#L21
But by the moment for the PoC I'm using an ugly MakeFile:
https://github.com/alonsovidales/tensorflow/blob/master/tensorflow/go/makefile
And setting the flags for the linker here:
https://github.com/alonsovidales/tensorflow/blob/master/tensorflow/go/cgoflags.go
And also a wrapper to access the C++ libraries from C:
https://github.com/alonsovidales/tensorflow/tree/master/tensorflow/go/wrapper
But this is just for the prototype, I plan to move all to Bazel + SWIG :)

I'm still not 100% clear on the long-term delineation between the external library's responsibilities and the underlying TF C++ API.
From http://www.tensorflow.org/api_docs/cc/index.html, this sentence implies that maybe at some point in the future the C++ libraries will add the support to build the graphs: "TensorFlow's public C++ API includes only the API for executing graphs, as of version 0.5."
It seems like the supported way outside of python is to still use python to build the graph, then use tf.train.write_graph() to write the graph to a file, and then load it into a session in your language of choice and then run it using the C++ API.
Are you planning to port the graph building mechanism into Go? It seems like it might be a good idea to get the consensus of the TF dev team before embarking on something as ambitious as that if they're planning on supporting this in the public API at some point in the future.

Well, my idea is to build the graph in the Go libs as it is being builded by the Python libs. They are working on the C++ code, but by the moment it is a WIP and it is, by the moment, unstable.
I'm not sure about their future plans, but from my point of view have this process in Python is not a bad approach since the tensors takes a lot of advantage of the dynamic typing and operators overloading that Python offers, this makes the code more simple to write and to understand, this is being one of the biggest problem I'm having porting the code to Go. I also think that keep the graph building process in Python is a good idea since this process requires almost no time compared with the operations that are implemented in C++, so scarify simplicity in favour of performance don't makes too much sense. I think that they make the good choice here.
I'm planning to port the process to Go by two reasons:

You could create a pure Go application to train your model, that could be interesting for some situations.
I'm working on just a PoC, and I prefer to explore all the code in order to get a global idea of how all the different pieces works together.
And well, port the process is going to be funny for me :)

@alonsovidales any movement on your PoC ?

No :'( I had a personal problem and I couldn't spend a single min on personal projects during last month, I'll try to resume it on next week.

I have a start of a swig-based binding started here: tmc/tensorflow/go_bindings -- mostly cargo-culted and incomplete.

:+1:

:+1:

tmc sent a pull request - https://github.com/tensorflow/tensorflow/pull/1237 - which we should be able to get in with a few iterations.

I finally have the C++ example of the Image Recognition tutorial ported to Go. Implement all the Graph construction, constants and other stuff was relatively easy since there is available the list of operations perfectly defined here:

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/ops/ops.pbtxt

Here you have the Go port of the tuto example:
https://github.com/alonsovidales/tensorflow/blob/go_bindings_tensors/tensorflow/examples/label_image_go/main.go
that corresponds with:
https://github.com/alonsovidales/tensorflow/blob/go_bindings_tensors/tensorflow/examples/label_image/main.cc
This is how looks like running a picture of Ceres my Labrador:
avidales@bigbrain:~/go-src/src/github.com/tensorflow/tensorflow/tensorflow/examples/label_image_go$ go run main.go data/ceres_paris.jpg
Labrador retriever : 0.9383322
American Staffordshire terrier : 0.009385344
Rhodesian ridgeback : 0.007571198
Chesapeake Bay retriever : 0.0027833031
golden retriever : 0.0026763931

I'm going to clean the code and include a section for Go libs on the tutorial.

@dave-andersen What's the status of this?

Closed by #1771.

#1771 appears to be in the 'go' branch but not master -- what's the plan to merge that in?

Realistically, TBD.  I've maybe found someone @ google who might be interested in taking ownership of the bigger picture of TF + go.  I think we're of a mind that the current state isn't what we need yet -- it's not integrated with the normal tests, nor is it go-buildable in the normal way.  Those are the major blockers to getting it into the main branch, but I think there's a further concern that because of the amount of magic that still happens only in Python with shapes & gradients, we're not at a point where we can have the Go bindings we'd like as a first-class supported language.  There's some ongoing work looking at improving our ability to support more languages, and I think some of the momentum on Go is stalled waiting on that.

Fair enough. I personally just wanted a way to execute graphs vs have python-level support.

Let's leave this bug open then, to go along with the bunch of other open language bugs that are much less far along.

I tried the go branch after a rebase and it works on my mac. Now I would like to propose some changes with a PR (https://github.com/Mistobaan/tensorflow/tree/go). But clearly the current go branch is behind respect to master and the PR is huge.
Should we rebase the main go branch or merge it in master and keep continuing the development there?
What is the best feasible strategy to keep patches coming into the go interface?

Hi @Mistobaan , I've been a bit out the last months because I'm moving to Dublin. @dave-andersen mentions on a previous comment that someone at Google could be interested on taking ownership of the Go development, the code was lent in the Go branch waiting for the integration with Jenkins and for other people to play with it and provide some feedback, I'm not sure about what is the current status.
I took a look at your commit and it seems really interesting, I think that you could make a PR to the current branch and we  can care to rebase it in a future :)
I'm using the Go libs on some personal projects and I got feedback from other people that are using them, It would be awesome for us to count with improvements.

@alonsovidales see #3258

Hi @Mistobaan and @alonsovidales, for my own project I have been working on merging the master updates into the go branch here.
TF build is fine, however go test is currently failing:
go git:go✎ go test
# github.com/tensorflow/tensorflow/tensorflow/contrib/go
/var/folders/r3/ndy82hd54c5b2yxqys6mx88w0000gn/T/go-build408123681/github.com/tensorflow/tensorflow/tensorflow/contrib/go/_obj/tensorflow_wrap.cxx:1
850:52: error: elaborated type refers to a typedef
./../../../tensorflow/core/public/tensor_c_api.h:89:3: note: declared here
FAIL    github.com/tensorflow/tensorflow/tensorflow/contrib/go [build failed]

I am new to working with swig. Any ideas here?

Follow-up: I've since resolved that issue (an enum that was improperly typdef'd). I am working out one other type assertion bug, but otherwise looking good. I'll update once all tests pass on my branch

Okay - good to go with all updates from master in go. PR Here.

@Mistobaan see new PR for the same ☝

FYI #3542 resolves a build error I experienced on the Go branch.

I've just published a PR to bump the go branch to the latest master and do some minor fixups in #4223

I'm not sure what's left to do in the go branch to get it merged to master. I might be able to help!

Updated #4223.
I think there was a goal of getting this to build with go get and go install. Currently, the generated protobuf doesn't have the right import paths to build correctly. (It's a version of the problem in https://github.com/golang/protobuf/issues/230)
We're going to have to probably do some sed work to fix it.

Before you do more work on that branch, keep in mind that we have in-progress Go bindings that the TensorFlow team is making :)

That's cool! Didn't know! Maybe missed a comment. Anything I can do to help?

@jhseu , do you have any plan about when we can get the official release?

We've got a handle on them :) For obvious reasons, we're particular about the structure of the Go API and some particulars (like autosubmitting generated protobuf code).
We need some additions to the C API (work-in-progress) to complete them. Should be in about a week or two.

Note that it's here:
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/go
We're making a more reasonable import path for people to use.

Cool! What's the plan for the protobuf generated packages? I'm currently just copying the proto files into my own build to pop out the Go code I want and they just barely don't fit there trivially (because protoc-gen-proto has some rules about where its generated code goes and what import paths they refer to each other with)

Oh, man, after weeks of hunting, I just found https://github.com/golang/protobuf/issues/139 so now I understand how it'll work

@jmhodges : So far whatever functionality the Go API provides  doesn't require protobuf. But of course, the functionality is limited at this point.
If y'all have any feedback on the API as it is today and what you'd find most useful in terms of additional functionality or changes, we'd greatly appreciate that.

Hello folks, the Go API is in a reasonably usable shape at this point. Graphs can be imported, executed, constructed (using primitives or functions generated from the TensorFlow op definitions etc.).
The API is by no means complete, but at this point I'm tempted to close out this generic "Go API" issue with the expectation that as the Go API sees real usage, new issues will be filed with specific feature requests and bug reports.

I created this package to enable tensorboard for tensorflow golang api. Commenting here in case anyone interested https://github.com/helinwang/tfsum

@helinwang, very cool! If you need any help with this I would be happy to contribute

save and load tensorflow tensors using tensorflow golang api: https://gist.github.com/helinwang/7782c6b2815c334c77653fc0e52b6069

Hi Folks, I am just curious why wrapper util functions in "op" folder don't seem to take node name as input argument. This prevents me from calling functions twice in a scope. For instance, I can't define two constants in a graph. There are no errors when I explicitly define constants or placeholders using tf.OpSpec by providing values for the Name field.

@sdeoras : The alternative for now is to use Subscope, which is used in the label image example as well (resulting in constants named make_batch/Const, size/Const etc. in the graph).
The answer to your question though is just that it felt less cumbersome to require a name argument on every op addition when it can often be elided. We'd want it to be an "optional" argument. One option is to do what we do in C++ graph construction API and add a WithOpName to the Scope type.
If you have strong opinions or suggestions, please do share (though perhaps in a separate issue focused on this aspect). In the mean time, hopefully the alternative suggested above lets you make progress.

@asimshankar : Thanks for your quick reply. Since it is easier for me to think in terms of graphs and operations on them, I am currently just rewriting some of the wrapper functions with node name as additional argument to avoid getting errors from underlying C++ code. I do, however, feel the same that providing names for every op declaration is cumbersome and hope that future versions of TF will no longer make this a requirement.

Hi Folks, I am not sure if this thread is the best place to ask TF GO API related questions. If not, please suggest a link to the right forum.
So I can't seem to get following code to work. It complains on second use of op.Reshape at runtime but allows two calls to op.Const... what am I doing wrong?
`	scope := op.NewScope()
x := op.Placeholder(scope, tf.Double)
s := op.Const(scope, []int32{3, 3})
y := op.Reshape(scope, x, s)
yInv := op.MatrixInverse(scope, y)
s2 := op.Const(scope.SubScope("scope.a"), []int32{9})
yInvReshaped := op.Reshape(scope.SubScope("scope.b"), yInv, s2)
graph, err := scope.Finalize()
if err != nil {
	t.Fatal(err)
}

`

Please use stackoverflow for general questions, or file new github issues for bugs/feature requests. In this case I've filed #6799 - I think I know what the problem is, will update there.

Hi Folks, I am running into another issue with shape argument in op.VarHandleOp. Pl. advise if this should be filed as a bug.

package bugs

import (
	tf "github.com/tensorflow/tensorflow/tensorflow/go"
	"github.com/tensorflow/tensorflow/tensorflow/go/op"
	"testing"
)

func TestVarHandleOpShape(t *testing.T) {
	scope := op.NewScope()
	_ = op.VarHandleOp(scope, tf.Int32, []int64{2, 3})
	_, err := scope.Finalize()
	if err != nil {
		t.Fatal(err)
	}
}

/*
=== RUN   TestVarHandleOpShape
--- FAIL: TestVarHandleOpShape (0.03s)
        bugs_test.go:14: failed to add operation "VarHandleOp": AttrValue had value with type 'list(int)' when 'shape' expected
                         for attr 'shape'
                        ; NodeDef: VarHandleOp = VarHandleOp[_class=[], container="", dtype=DT_INT32, shape=[2, 3], shared_name=""](); Op<name=VarHandleOp; signature= -> resource:resource; attr=container:string,default=""; attr=shared_name:string,default=""; attr=dtype:type; attr=shape:shape; is_stateful=true> (Stacktrace: goroutine 5 [running]:
                runtime/debug.Stack(0x0, 0x0, 0x0)
                        /usr/lib/golang/src/runtime/debug/stack.go:24 +0x79
                github.com/tensorflow/tensorflow/tensorflow/go/op.(*Scope).UpdateErr(0xc42000c420, 0x53e4eb, 0xb, 0x7cce00, 0xc420054040)
                        /home/sdeoras/go/src/github.com/tensorflow/tensorflow/tensorflow/go/op/scope.go:113 +0x72
                github.com/tensorflow/tensorflow/tensorflow/go/op.(*Scope).AddOperation(0xc42000c420, 0x53e4eb, 0xb, 0x53e4eb, 0xb, 0x0, 0x0, 0x0, 0xc42000c450, 0xc42000c3f0)
                        /home/sdeoras/go/src/github.com/tensorflow/tensorflow/tensorflow/go/op/scope.go:78 +0xf9
                github.com/tensorflow/tensorflow/tensorflow/go/op.VarHandleOp(0xc42000c420, 0xc400000003, 0xc4200106f0, 0x2, 0x2, 0x0, 0x0, 0x0, 0x171e4d44, 0xc420051f78)
                        /home/sdeoras/go/src/github.com/tensorflow/tensorflow/tensorflow/go/op/wrappers.go:14045 +0x27d
                bitbucket.hgst.com/x/tensorflow.git/tfutil.TestVarHandleOpShape(0xc4200b8180)
                        /home/sdeoras/go/src/bitbucket.hgst.com/x/tensorflow.git/tfutil/bugs_test.go:11 +0x96
                testing.tRunner(0xc4200b8180, 0x54f7f0)
                        /usr/lib/golang/src/testing/testing.go:610 +0x81
                created by testing.(*T).Run
                        /usr/lib/golang/src/testing/testing.go:646 +0x2ec
                )
FAIL
exit status 1
FAIL    bitbucket.hgst.com/x/tensorflow.git/tfutil      0.130s
 */

Yeah, shape attributes handling wasn't finished, but should be easy. Feel free to file a new issue for this (and don't hesitate to file new issues for other things instead of adding to this closed one).

@helinwang Great work with enabling TensorBoard! Could you please include a license file for us to be able to use it for commercial purposes?

@sdeoras Glad to know that the library helps! Added 3-clause BSD.

Thank you @helinwang

Apologies in advance for not following this issue thoroughly, but what's the state of the Go binding? If it's finished can someone point me to tutorial implementations, for example linear regression over the MNIST dataset? Thanks

@4d55397500 I believe the go binding of tensorflow is still under heavy developing. See: https://godoc.org/github.com/tensorflow/tensorflow/tensorflow/go
It is not stable enough and lots of features haven't been implemented.
(I've just compiled the latest version and sadly speaking that it can not pass the builtin test on mac os x. A signal kill occurred when running tests.)

@lipixun @4d55397500 : The Go APIs available today are intended to be usable and tests for both Linux and OS X are included in the continuous builds on Jenkins.
@4d55397500 : The APIs are a little low level though. They should be sufficient to build things like networks for MNIST and linear regression models on top of, but at this time there are no plans for writing convenience higher level APIs.


Pipelines with cron input do not overwrite the contents of their output repo

Each cron tick has a separate datum, and so we never actually overwrite the file for the tick.
This pipeline can be used to reproduce.
{
    "pipeline": {
        "name": "a"
    },
    "description": "Cron squash test",
    "transform": {
        "image": "debian:stable",
        "cmd": ["/bin/bash"],
        "stdin": ["ls -Rv /pfs > /pfs/out/ls.txt"]
    },
    "input": {
        "cron": {
            "name": "tick",
            "spec": "@every 10s"
        }
    }
}

pachctl list file a@master:

Each of the output commits for this pipeline produces a version of ls.txt with a different tag.
We expected to see a single ls.txt  in the commit a@master with a single tag.

@brycemcanally pointed out that this is the documented behavior in 1.x
https://docs.pachyderm.com/latest/concepts/pipeline-concepts/pipeline/cron/
Maybe we should consider changing the default.  In 1.x this pipeline would concatenate to ls.txt but in 2.x it will create a new (path, tag) for every commit.

This seems to be working as intended. cron inputs have an overwrite field which will give you the behavior you're describing. Closing since I don't think there's anything to fix here, feel free to reopen if I missed something.


Add appropriate docs for configuring S3 backend

Need to specify instructions about configuring S3 backend for builder

@krancour @smothiki is this done?


Issues with New Addresses

5: {
    name: "goerli",
    ethUsdPriceFeed: "0xD4a33860578De61DBAbDc8BFdb98FD742fA7028e",
    wethToken: "0xB4FBF271143F4FBf7B91A5ded31805e42b2208d6",
    // This is the AaveV3 contract address
    lendingPoolAddressesProvider: "0xc4dCB5126a3AfEd129BC3668Ea19285A9f56D15D",
    // This is the LINK/ETH feed
    daiEthPriceFeed: "0xb4c4a493AB6356497713A78FFA6c60FB53517c63",
    // This is the LINK token
    daiToken: "0x326C977E6efc84E512bB9C30f76E30c160eD06FB",
},

Hey Patrick!
Why have we updated this lendingPoolAddressesProvider address to aaveV3 when using the old dependencies of aaveV2 "@aave/protocol-v2": "^1.0.1",? This is not working on a Goerli network.
And also the wethToken token address is different.
And also why are we using LINK/ETH feed and LINK token addresses instead of DIA?
I haven't updated the code to AaveV3 because last time (for toolbox dependency) you said you wanted to make it similar to the video code.
Let me know what should we do.

@PatrickAlphaC

Hmm.... We might just have to put a note to skip running the testnet... what do you think?


Decide on Subset of Environmental Variables to Expose

Most environmental variables are settings, so we should only expose ones which are not:

[ ] initial user
[ ] root url
[ ] instance ip

Thanks @geekgonecrazy for the suggestions

Can it access the main process.env e.g. for Heroku environments, there might be a lot of useful stuff in there. Another key one would be local/dev/live environment switches.

Making the list of enabled Rocketlets available as an env var would also be handy for any tricky workarounds once the inevitable conflicts between extensions start to surface.

No, the Rocketlets are going to be in a restricted environment which only has access to provided variables and global properties. Honestly, I'm debating whether or not to even allow environmental variables :thinking:

Well as long as they can access mongo and manage their own settings, I think that would cover most use cases where the Rocketlet's logic might need some wider context about the state of the RC instance. I'm sure there'll be unforeseen but its not worth trying to predict and over-engineer, just have to deal with it as they evolve.
Maybe a permissions layer (which could be quite simple for now) should be considered for this type of thing, to keep instance owners informed of security risks when installing Rocketlets, then we don't have to be overly protective and limiting. Similar to permissions for users, but implemented like adding a facebook app, e.g.
This Rockelet can

modify user roles
modify messages

The above could have some correlation to the internal API endpoints.
I've probably gone off-topic for this issue. 🤷♂

Closing as so far the allowed ones are: 'NODE_ENV', 'ROOT_URL', 'INSTANCE_IP'


conf.py snippet is not enough to use this theme

README instructs users to add the following lines to the project's conf.py:
import sphinx_rtd_theme
html_theme = "sphinx_rtd_theme"
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]

However this only applies to local builds, while RTD builds lack the <link> tags for the theme stylesheets. It looks like the following code is needed as well (see the design-doc conf.py):
if not on_rtd:  # only import and set the theme if we're building docs locally
    html_theme = 'sphinx_italia_theme'
    html_theme_path = ["_themes", ]
else:
    # Override default css to get a larger width for ReadTheDoc build
    html_context = {
        'css_files': [
            '_static/css/theme.css',
            '_static/css/custom.css',
        ],
    }

A sample conf.py should be included in this repository (like in the newer https://github.com/italia/docs-italia-theme) and/or the README should be updated.

Closing as superseeded by new docs-italia-theme


remove tket-opt-level from compilation example

Description
remove tket-opt-level from compilation example notebook to streamline compilation workflow narrative.
Checklist

[x] I have performed a self-review of my code.
[x] I have commented hard-to-understand parts of my code.
[ ] I have made corresponding changes to the public API documentation.
[ ] I have added tests that prove my fix is effective or that my feature works.
[ ] I have updated the changelog with any user-facing changes.

I think this is a good point. There is also another example targeting compilation without Quantinuum API. Should all these compilation examples be collapsed into the submission notebook?

Maybe collapse it into the Quantinuum_circuit_submissions notebook? I think that is probably best place, just include basics of TKET compilation when introducing the user workflow, since there's nothing funky going on with submission anymore.
The examples showing how to use QuantinuumBackend without having to sign in is mainly for MSFT users who wanted to try TKET compilation, but who don't have credentials on user portal. ... in my mind they are distinct for that reason since the rest of user workflow involving retrieving job results, etc. doesn't apply for those users. Thoughts?


The input's JSON structure doesn't have file name or type, just the base64 payloads in an array

streamlit==1.39.0
st-multimodal-chatinput==0.2.1

Python version: 3.12.7
The project's example code implies a certain JSON structure the input supposed to yield, I supposed to get file names and types for example:
chatinput = multimodal_chatinput()

uploaded_files = chatinput["uploadedFiles"] ##list of ALL uploaded files (including images) along with type, name, and content.
uploaded_images = chatinput["uploadedImages"] ## list of base 64 encoding of uploaded images
text = chatinput["textInput"] ##submitted text
for file in uploaded_files:
    filename = file["name"]
    filetype = file["type"] ## MIME type of the uploaded file
    filecontent = file["content"] ## base 64 encoding of the uploaded file

However when I test and perform a print(chatinput), I only have two elements in the JSON, a text and an images which is an array of the base64 payloads. The keys don't match and the data is also different.

My simplified code:
import streamlit as st

from streamlit_chat import message
from st_multimodal_chatinput import multimodal_chatinput

# Initialise session state variables
if "agent_turns" not in st.session_state:
    st.session_state["agent_turns"] = []

if "user_turns" not in st.session_state:
    st.session_state["user_turns"] = []

chat_placeholder = st.empty()

with chat_placeholder.container():
    for i in range(len(st.session_state['agent_turns'])):
        message(st.session_state['user_turns'][i], is_user=True, key=f"{i}_user")
        message(
            st.session_state['agent_turns'][i]['data'],
            key=f"{i}_agent",
            allow_html=True,
            is_table=True if st.session_state['agent_turns'][i]['type']=='table' else False
        )

with st.container():
    chatinput = multimodal_chatinput()

if chatinput:
    print(chatinput)
    # ???


[Flink-8563][Table API & SQL] add unittest for consecutive dot access of composite array element in SQL

What is the purpose of the change
add unittest for consecutive dot access of composite array element in SQL. This depends on https://github.com/apache/flink/pull/5791.

## Brief change log

  -  add unittest

## Verifying this change

This change is already covered by existing tests.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (y no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)

retry build


Path segregation - pipeline middleware - config-driven

@weierophinney
Good morning,
I've an expressive application with a config-driven pipeline. That application exposes different UI levels such as /admin, /reseller and /client, among others...
Right now, I've a specific pipeline middleware that I want to execute only for specific paths:
...
protected function getPipeline()
{
    return [
        [
            'path' => '/admin',
            'middleware' => \Zend\Expressive\Navigation\Middleware\NavigationMiddleware::class
        ],
        [
            'path' => 'reseller',
            'middleware' => \Zend\Expressive\Navigation\Middleware\NavigationMiddleware::class
        ],
        [
            'path' => '/client',
            'middleware' => \Zend\Expressive\Navigation\Middleware\NavigationMiddleware::class
        ],
    ];
}
...

This is working but, I would rather be able to do something like this:
...
protected function getPipeline()
{
    return [
        [
            'paths' => ['/admin', '/reseller', '/client'],
            'middleware' => \Zend\Expressive\Navigation\Middleware\NavigationMiddleware::class
        ],
    }
}
...

Of course, I could use a custom delegator but I think this could be integrated in the default \Zend\Expressive\Container\ApplicationConfigInjectionDelegator delegator as follows:
...
    public static function injectPipelineFromConfig(Application $application, array $config) : void
    {
        if (empty($config['middleware_pipeline'])) {
            return;
        }

        // Create a priority queue from the specifications
        $queue = array_reduce(
            array_map(self::createCollectionMapper(), $config['middleware_pipeline']),
            self::createPriorityQueueReducer(),
            new SplPriorityQueue()
        );

        foreach ($queue as $spec) {
            $paths = $spec['path'] ?? ($spec['paths'] ?? '/');
            foreach((array) $paths as $path) {
                $application->pipe($path, $spec['middleware']);
            }
        }
    }
...

instead of
....
    public static function injectPipelineFromConfig(Application $application, array $config) : void
    {
        if (empty($config['middleware_pipeline'])) {
            return;
        }

        // Create a priority queue from the specifications
        $queue = array_reduce(
            array_map(self::createCollectionMapper(), $config['middleware_pipeline']),
            self::createPriorityQueueReducer(),
            new SplPriorityQueue()
        );

        foreach ($queue as $spec) {
            $path = $spec['path'] ?? '/';
            $application->pipe($path, $spec['middleware']);
        }
    }
....

Thank you.

We don't recommend usage of config-driven routes or pipelines at this point; starting in Expressive 2, we began moving away from using them in examples, and we've discussed deprecating the feature. I know some people swear by them, but it's a complicated feature, and easy to misconfigure. However, if you want to submit a pull request (which will require tests and documentation), I'll review. (We may separate the feature into a standalone package later, so new features will continue to live.)

@weierophinney

some people swear by them

We are one of them. You should really avoid opinionated implementations. This was the mistake you have done with your MVC implementation. Zend Expressive is really cool as micro-framework but if you start forcing developers to follow a single road, that will make it unusable or a bad choice for some developers.
We prefer a config-driven pipeline rather than a pipeline programmatically configured. This is a choice we've made because we want make the configuration close to our modules.
Please don't force us to configure our pipelines and/or routes programatically by deprecating the feature, even through, we could always create our own delegator. If you deprecate that feature which is not so complicated to understand you will really make some of your supporters unhappy.
The major problem with the programmatically approach is that the configuration is not close to the modules. When we setup our OPTIONAL modules (which are close to our application), we want just have to inject their configuration providers, nothing helse. Having to add each module's pipeline/route configurations programatically is really a pain for us and for our end-users.
I'll see if I can contribute a bit more. Right now, I'm developing a Zend helper extension for Plates (extension that compose an helper plugin manager to provide Zend view helpers-like (doctype, navigation ...) in Plates templates).
Thank you.

You should really avoid opinionated implementations. This was the mistake you have made with your MVC implementation. Zend Expressive is really cool as micro-framework and a great alternative to symfony but if you start forcing developers to follow a single road, that will make it unusable or a bad choice for some developers.

We have to choose carefully where we spend our time maintaining code and educating users. While I appreciate that the config-driven approach is useful to a number of people, the number of support questions we had to field based on them was disproportionate, which was why we moved away from the feature.
This sort of feature is quite easy for us to separate into its own package, however, and we could hand off maintenance of that to interested users. This would be the best of both worlds, I feel, as those who want the feature can install the package and use the support, and we can still direct folks to the explicit routing/pipeline approach we document initially.

This repository has been closed and moved to mezzio/mezzio; a new issue has been opened at https://github.com/mezzio/mezzio/issues/4.


[IMPROVEMENT] Improve the speed of replica rebuilding if it is somehow stopped and re-started

Is your improvement request related to a feature? Please describe (👍 if you like this request)
Improve the speed of replica rebuilding if it is somehow stopped (usually caused by unstable network or disk IO starvation) and re-started.
A disk file of the healthy replica is read by direct io (code) in current implementation. Hence, if the replica rebuilding somehow failed and restarted, the data will be read from disk again and cannot benefit from the host's page cache mechanism. Reading a disk file of the healthy replica should be not harmful, because the disk file is opened with RDONLY.
After a quick test, the restarted replica rebuilding can be improved by 50% .
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

On AWS EC2 machines, I cannot see the improvement as well. The improvement on Equinix machines is 35%.
After the rebuilding, the memory usage of the instance-manager-r is (`kubectl -n longhorn-system top pods)
instance-manager-r-78e63e75eb3bedb81a32e48d220dd6e8   5m           31192Mi

This might be scary to users. I plan to revert the improvement.
WDYT? @innobead

Verified on master-head (longhorn-engine 84da610). Following the test steps and observing the memory usage on each step, high memory usage in instance-manager-r like https://github.com/longhorn/longhorn/issues/5130#issuecomment-1427372975 won't happen:
(1) Create a 15 GiB volume with 2 replicas. The original memory usage is:
instance-manager-r-0b165eb6a49550ac97473a12c0045a78   1m           7Mi
instance-manager-r-790a4298a3ab3ecc966c0ab724302cbf   1m           7Mi
instance-manager-r-b951cffeead0a3cbc0b8db8e27a43bac   1m           7Mi

(2) Write 15GiB data and delete one of the 2 replicas. The replica rebuilding is triggered and started.
(3) When the rebuilding progress is 80%, the memory usage is:
instance-manager-r-0b165eb6a49550ac97473a12c0045a78   1m           9Mi
instance-manager-r-790a4298a3ab3ecc966c0ab724302cbf   724m         46Mi
instance-manager-r-b951cffeead0a3cbc0b8db8e27a43bac   800m         62Mi

(3) Delete the same replica again, the replica rebuilding is triggered and started again.
(4) When the rebuilding progress is 80% again, the memory usage is:
instance-manager-r-0b165eb6a49550ac97473a12c0045a78   1m           10Mi
instance-manager-r-790a4298a3ab3ecc966c0ab724302cbf   718m         48Mi
instance-manager-r-b951cffeead0a3cbc0b8db8e27a43bac   505m         61Mi

(5) After rebuilding completed and wait for a while, the memory usage is:
instance-manager-r-0b165eb6a49550ac97473a12c0045a78   1m           9Mi
instance-manager-r-790a4298a3ab3ecc966c0ab724302cbf   1m           35Mi
instance-manager-r-b951cffeead0a3cbc0b8db8e27a43bac   2m           41Mi


export all date_utils functions please

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

We need some of the functions in date_utils to customize the format

Describe the solution you'd like
A clear and concise description of what you want to happen.

export  all the date_utils funcions in index.ts

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Indeed that would be very useful. I'm trying to wrap the components to customise the rendering of all parts to apply tailwind look and feel. Although it is a bit hard to extend some parts. So either making it more extendable (might be very tricky) or export the date utils so we can reuse them if needed. WDYT ?


Error: Group "first" is not registered.

I am getting the error:
if(!group) throw new Error(Group "${command.groupID}" is not registered.);
^
Error: Group "first" is not registered.
What should I be doing differently?

If you need help with discord.js Commando installation or usage, please go to the discord.js Discord server instead:
https://discord.gg/bRCvFy9
This issue tracker is only for bug reports and enhancement suggestions. You won't receive any basic help here.


E-Mail Newsletter

Hey,
My Idea is, wenn you realse a New Papier Version, then Users can get a E-Mail so the knoe there is a now Version
Like a News Letter :)
LG
SGirl

We don't do traditional releases, though. People do utilise things like bots to poll the API and send a message to their Discord. I don't think that email is worth it, especially as the content would be pretty limited, as we're not going to invest time into publishing an actual letter for every build.

Can you tell me how this works whit this Bots and how can i used them?

You would need to self-host a piece of software that will do this or find somebody already hosting a bot that does this. I've seen one or two in various discords that I'm a member of, not sure if those are public, but, I know that people have devised various solutions for this in the past.

generally, just check once a week, thats good enough for most people


"Extra" args which do not matter for job uniqueness

Hi @cypriss,
Thanks for the work on work. 😋
Case in point, in my current setup, I'm loading twitter followers with the job args:
{"twitter_id": 123456, "next_cursor_str": "-1", "ids": []}

If I'm rate limited by twitter API, I schedule a new job from within the current job with the args:
{"twitter_id": 123456, "next_cursor_str": "something", "ids": [1,2,3,4,5]}

(I have to make a query for uniqueness at this time)
If I were to use work for background processing, it seems like embedding partial args like above will have to go unless:
a) There's support to include "extra" args, which do not count towards job uniqueness; or
b) I use a reference to external cache. This sounds reasonably good for my case since the ids part can get pretty large.
With (a) it would be guaranteed that there's only one followers checking job for a user with twitter_id 123456. The next_cursor_str and ids can move into the extra/cache args. What do you think? Are there breaking concerns about how work is implemented, or is this simple enough to do?
I'm willing to take a stab at this with some guidance on what and how this would be possible.

Or a uniqueKey variable, perhaps.

Hi!
It seems like there's a couple options here:
a) The option that modifies gocraft/work to support "extra" args. The best way I can see to do this is to have a magic argument when enqueuing, such as
enqueuer.Enqueue("load_twitter_followers", work.Q{"twitter_id": 123456, ..., "work_unique_key": 123456})

If this unique key is present, we'll use it instead of a full serialization of the args.
b) Persist the ids and next_cursor_str outside of gocraft/work. You could persist them in redis, or perhaps sql (if you use that). May I ask, what do you do with all the ids once you get them? Save them anywhere?
c) Use normal jobs instead of unique jobs, and keep track of the uniqueness yourself (in redis). You could check this unique k/v before enqueuing and clear when done.
--
So, I'm a bit reluctant to just go with (a) because it's a bit magic and not immediately expected/intuitive. I'm also reluctant to go with (a) because I don't know how common this use-case is.
Thoughts?

I believe the uniqueKey should be overridable for when you have relatively large number of args, as the options otherwise are more work (usually premature optimization) than necessary to get the same effect. (a) actually looks good to me
I'm storing/overriding ids in a row dedicated per twitter_id (i.e. keeping only the latest snapshot of all followers).
In case of (c), do normal jobs use args for anything? What can go wrong when {"ids": [has a million int64s]} other than running out of RAM?

All jobs, unique and normal, serialize the args as json and store those bytes in redis. Redis can handle some fairly long values, but obviously you pay linearly for the bytes you use to transfer to redis and to encode/decode.

I would like to use the unique jobs feature as well but also would like to be able to pass an unique ID that does not affect the uniqueness check to the job.

Fixed by PR #110


unsilence logger in ExtractBundlesFromImage

Signed-off-by: Josh Gwosdz<EMAIL_ADDRESS>This will make downstream errors in ExtractBundlesFromImage visible.

@erdii can you please rebase?

done! sorry that it took so long!

Looking at the output, this change may not be such a good idea at all.
Maybe we should buffer log messages from extracting the bundles and only print them in case of an error?

Sure thing. FWIW, the indexExtractor and bundleExtractor are being rewritten to support in-memory caching, sql/file-based catalogs and extract annotations.

After #68 is merged, this will be obsolete. Lets close it.


Update ReadMe

The readme needs to be updated to reflect new changes and addition of heat transfer models.

See most recent pull request.


Fix issues when using use-strings="true"

Fixes #874
Several issues were found while debugging and fixing #874:

Pass just tag name string to callback functions when using use-strings="true"
Call callback functions in a $timeout to ensure model has been appropriately updated with just tag strings before the callback is called.  The model is updated in a watcher so it is asynchronously updated
Use $watchCollection for tags watcher
Do not clobber tags container when adding/removing items

Coverage decreased (-0.4%) to 99.416% when pulling 98e8fdb0a2c0407eb00e59421ef0f1be6685c6b1 on NorthPage:cg-874 into 87d0e6b695fd53d21f73357a1d3377ab569a78d0 on mbenford:master.


build: switching to webpack instead of ncc

v13.0.1 has been released and contains a fix for the issue. Feel free to reopen the issue or to submit a new one if you meet any problems.


performance-table fails: chance_agreement cannot get data["all"]["ground_truths"]

When running the following:
cd dvc/
find training/ -iname "*chk" | xargs morphoclass -vv performance-table --output-dir ../notebooks/results-performance-table

The following error is produced:
15:04:29 morphoclass.console.main (I) Running them morphoclass entrypoint
15:04:29 morphoclass.console.cmd_performance_table (I) Got 409 checkpoints
15:04:29 morphoclass.console.cmd_performance_table (I) Loading libraries
15:04:32 morphoclass.console.cmd_performance_table (I) Creating the performance summary table
15:04:33 morphoclass.console.performance_table (I) Found 409 checkpoint files
15:04:33 morphoclass.console.performance_table (I) Processing checkpoint 1 of 409: '/workdir/dvc/training/checkpoints-lida-alt-neurites/in-merged-all-decision-tree-deepwalk-stratified-k-fold/checkpoint.chk'
15:04:34 morphoclass.console.performance_table (I) Processing checkpoint 2 of 409: '/workdir/dvc/training/checkpoints-lida-alt-neurites/janelia-L5-basal-perslay-tmd-stratified-k-fold/checkpoint.chk'
Traceback (most recent call last):
  File "/home/casalegn/.local/bin/morphoclass", line 33, in <module>
    sys.exit(load_entry_point('morphoclass', 'console_scripts', 'morphoclass')())
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/workdir/src/morphoclass/console/cmd_performance_table.py", line 64, in cli
    make_performance_table(checkpoint_paths, output_dir)
  File "/workdir/src/morphoclass/console/performance_table.py", line 76, in make_performance_table
    table_rows.append(make_report_row(data))
  File "/workdir/src/morphoclass/console/performance_table.py", line 176, in make_report_row
    chance_agreement_score = chance_agreement(data["all"]["ground_truths"])
KeyError: 'ground_truths'

Debugging
This bug is caused by the fact that train_dm_models ( = training of neural nets models: GNN, CNN, PersLay) does not write data["all"]["ground_truths"]
https://github.com/BlueBrain/morphoclass/blob/5eb63eb08e772a0b8a0efc34cc4c50d55caf1b87/src/morphoclass/training/cli.py#L345
This is unlike what happens with train_ml_models, which does write data["all"]["ground_truths"]
https://github.com/BlueBrain/morphoclass/blob/5eb63eb08e772a0b8a0efc34cc4c50d55caf1b87/src/morphoclass/training/cli.py#L293

Proposed solution
We can compute chance_agreement on the concatenation of the "ground_truths" of each split in data["splits"].


Build and publish API and UI images

refactor: Rename Worker Dockerfile and build step
build: Add Dockerfile for API
build: Build and push API image
build: Add Dockerfile for UI
build: Build and push UI image

@haikoschol, this should get you a little further, now it fails at the env validation, that probably should be removed.

@haikoschol, this should get you a little further, now it fails at the env validation, that probably should be removed.

Now it gets stuck again. 😔
[+] Building 1861.5s (13/19)                                                                                docker:orbstack
 => [internal] load build definition from UI.Dockerfile                                                                                0.0s
 => => transferring dockerfile: 1.13kB                                                                                0.0s
 => [internal] load metadata for docker.io/library/node:20.10.0                                                                                1.9s
 => [auth] library/node:pull token for registry-1.docker.io                                                                                0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 220B                                                                                0.0s
 => [base 1/1] FROM docker.io/library/node:20.10.0@sha256:8d0f16fe841577f9317ab49011c6d819e1fa81f8d4af7ece7ae0ac815e07ac84                                                                                0.0s
 => [internal] load build context                                                                                0.8s
 => => transferring context: 1.63MB                                                                                0.8s
 => CACHED [builder 1/4] WORKDIR /app                                                                                0.0s
 => [runner 2/6] RUN addgroup --system --gid 1001 nodejs                                                                                0.8s
 => [builder 2/4] RUN npm i -g turbo                                                                                7.4s
 => [runner 3/6] RUN adduser --system --uid 1001 nextjs                                                                                0.3s
 => [builder 3/4] COPY . .                                                                                4.9s
 => [builder 4/4] RUN turbo prune clearance_ui --docker                                                                                0.8s
 => [installer 2/5] COPY --from=builder /app/out/json/ .                                                                                0.0s
 => [installer 3/5] RUN npm install                                                                                1845.6s
 => => # npm WARN deprecated<EMAIL_ADDRESS>Use your platform's native DOMException instead
 => => # npm WARN deprecated<EMAIL_ADDRESS>Use your platform's native atob() and btoa() methods instead

@haikoschol, this should get you a little further, now it fails at the env validation, that probably should be removed.

Now it gets stuck again. 😔
[+] Building 1861.5s (13/19)                                                                                docker:orbstack
 => [internal] load build definition from UI.Dockerfile                                                                                0.0s
 => => transferring dockerfile: 1.13kB                                                                                0.0s
 => [internal] load metadata for docker.io/library/node:20.10.0                                                                                1.9s
 => [auth] library/node:pull token for registry-1.docker.io                                                                                0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 220B                                                                                0.0s
 => [base 1/1] FROM docker.io/library/node:20.10.0@sha256:8d0f16fe841577f9317ab49011c6d819e1fa81f8d4af7ece7ae0ac815e07ac84                                                                                0.0s
 => [internal] load build context                                                                                0.8s
 => => transferring context: 1.63MB                                                                                0.8s
 => CACHED [builder 1/4] WORKDIR /app                                                                                0.0s
 => [runner 2/6] RUN addgroup --system --gid 1001 nodejs                                                                                0.8s
 => [builder 2/4] RUN npm i -g turbo                                                                                7.4s
 => [runner 3/6] RUN adduser --system --uid 1001 nextjs                                                                                0.3s
 => [builder 3/4] COPY . .                                                                                4.9s
 => [builder 4/4] RUN turbo prune clearance_ui --docker                                                                                0.8s
 => [installer 2/5] COPY --from=builder /app/out/json/ .                                                                                0.0s
 => [installer 3/5] RUN npm install                                                                                1845.6s
 => => # npm WARN deprecated<EMAIL_ADDRESS>Use your platform's native DOMException instead
 => => # npm WARN deprecated<EMAIL_ADDRESS>Use your platform's native atob() and btoa() methods instead

It did the same for me as well, but ctrl+c and trying again got through that.

This works, but can't exit the image with ctrl+c. :D

Ctrl+c works with this, and everything seems good. Didn't try with env variables. https://github.com/doubleopen-project/dos/compare/main...ui-docker-fixes

Both UI and API Docker builds still hang for me during RUN npm ci. Maybe it works in CI. 🤷♂

Closing in favor of #803 to get ownership of the branch.


Cloudfront ACK- ADD OAC

Is your feature request related to a problem?
Not able to create origin access control for cloudfront using ACK

Describe the solution you'd like
Please add a functionality that helps creating OAC
Describe alternatives you've considered
terraform but it will make our job easier if we have within ACK

@bala151187 arriving in v0.0.12!
Thank you @TiberiuGC!
/close


ModuleNotFoundError: No module named 'ragas.cost'

[X] I have checked the documentation and related resources and couldn't resolve my bug.
Description
Tried running the code from the cost calculation page on the documentation. When the interpreter reaches the expression from ragas.cost import get_token_usage_for_openai, I get an ModuleNotFoundError: No module named 'ragas.cost' error.
Ragas version: 0.1.12 (from Conda)
Python version: 3.11.9
Code to Reproduce
# lets import a parser for OpenAI
from ragas.cost import get_token_usage_for_openai

Error trace
ModuleNotFoundError: No module named 'ragas.cost'
Expected behavior
Should import the method as usual.
Additional context
I also tried running this notebook. Returns the same error.

dosu is right - but new release will be out in a couple of hours so that works too

Thanks for the quick response!
It did work, as is, i managed to import and summon it, following the documentation. However, when reaching the evaluation part...
from ragas import evaluate
from ragas.cost import get_token_usage_for_openai

result = evaluate(
    amnesty_qa["eval"],
    metrics=[
        context_precision,
        faithfulness,
        answer_relevancy,
        context_recall,
    ],
    llm=gpt4o,
    token_usage_parser=get_token_usage_for_openai
)

...i get the TypeError: evaluate() got an unexpected keyword argument 'token_usage_parser' error - which is weird, since the constructor appears correct here.

dosu is right - but new release will be out in a couple of hours so that works too. @CaduSantana do share you feedback on the cost module though would really appreciate it 🙂

@jjmachan Unfortunately I ran into another issue. Although I've followed this page, I get the AttributeError: 'str' object has no attribute 'set_run_config' error. As I stumbled on Issue #413, I thought it would be prudent to instead try the examples on the documentation first:
from datasets import Dataset

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on January 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The Super Bowl....season since 1966,','replacing the NFL...in February.'],
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}
dataset = Dataset.from_dict(data_samples)

However, it still returns the same error (AttributeError: 'str' object has no attribute 'set_run_config').

@CaduSantana could you share the trace for the same?

@jjmachan Yes. As it follows:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[8], line 12
      1 from datasets import Dataset
      3 # data_samples = {
      4 #     'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
      5 #     'answer': ['The first superbowl was held on January 15, 1967', 'The most super bowls have been won by The New England Patriots'],
   (...)
      9 # }
     10 # dataset_ = Dataset.from_dict(data_samples)
---> 12 result = evaluate(
     13     dataset_,
     14     llm="gpt-4o-mini",
     15     token_usage_parser=get_token_usage_for_openai,
     16 )

File ~/anaconda3/envs/rag_venv/lib/python3.11/site-packages/ragas/_analytics.py:129, in track_was_completed.<locals>.wrapper(*args, **kwargs)
    126 @wraps(func)
    127 def wrapper(*args: P.args, **kwargs: P.kwargs) -> t.Any:
    128     track(IsCompleteEvent(event_type=func.__name__, is_completed=False))
--> 129     result = func(*args, **kwargs)
    130     track(IsCompleteEvent(event_type=func.__name__, is_completed=True))
    132     return result

File ~/anaconda3/envs/rag_venv/lib/python3.11/site-packages/ragas/evaluation.py:199, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, run_config, token_usage_parser, raise_exceptions, column_map)
    196             reproducable_metrics.append(i)
    198     # init all the models
--> 199     metric.init(run_config)
    201 executor = Executor(
    202     desc="Evaluating",
    203     keep_progress_bar=True,
    204     raise_exceptions=raise_exceptions,
    205     run_config=run_config,
    206 )
    208 # Ragas Callbacks
    209 # init the callbacks we need for various tasks

File ~/anaconda3/envs/rag_venv/lib/python3.11/site-packages/ragas/metrics/base.py:150, in MetricWithLLM.init(self, run_config)
    146 if self.llm is None:
    147     raise ValueError(
    148         f"Metric '{self.name}' has no valid LLM provided (self.llm is None). Please initantiate a the metric with an LLM to run."  # noqa
    149     )
--> 150 self.llm.set_run_config(run_config)

AttributeError: 'str' object has no attribute 'set_run_config'

the code @CaduSantana is wrong
llm="gpt-4o-mini",

that should not be a string but an LLM.
result = evaluate(
     13     dataset_,
     14     metrics=[answer_relevancy, faithfulness, context_recall, context_precision],
     16     token_usage_parser=get_token_usage_for_openai,
     17 )

just do this instead

Oh, so that was the issue.
Thanks for the headsup!
Also

dosu is right - but new release will be out in a couple of hours so that works too. @CaduSantana do share you feedback on the cost module though would really appreciate it 🙂

Bretty gud 👍. It reflected correctly our costs on the OpenAI dashboard. I like how simple and easy it is.
Great project and hope you guys keep growing this!

Close comment.

@CaduSantana awesome! great to hear that 🙂. We will keep growing this too - we want to build an open-source standard for evaluations, lets see how it goes 😅 🤞🏽
btw if you have any feedback or ideas for improvement do share it here on discord too 🙌🏽


Selecting multiple resources on destroy fails

When selecting more than one resource from the list, it fails to destroy as shown below:
My workaround is to select each resource individually then destroy one at a time.
$ tftarget destroy
? Select resources to target destroy: module.Stack.module.Compute.aws_instance.server[0] will be destroyed, module.Stack.module.Compute.aws_instance.server[1] will be destroyed
Enter a value: yes
╷
│ Error: Invalid target "{module.Stack.module.Compute.aws_instance.server[0],module.Stack.module.Compute.aws_instance.server[1]}"
│
│ Must begin with a variable name.
╵

For more help on using this command, run:
  terraform destroy -help
==============🎉 Selected Resources 🎉==============
module.Stack.module.Compute.aws_instance.server[0] will be destroyed
module.Stack.module.Compute.aws_instance.server[1] will be destroyed

$  tftarget --version ; terraform --version
tftarget version 0.0.2 (Built on 2023-05-30 11:50:17.568382767 +0100 BST m=+0.001186340)
Terraform v1.4.6
on linux_arm64
+ provider registry.terraform.io/datadog/datadog v3.23.0
+ provider registry.terraform.io/hashicorp/aws v4.64.0
+ provider registry.terraform.io/hashicorp/local v2.4.0
+ provider registry.terraform.io/hashicorp/null v3.2.1
+ provider registry.terraform.io/hashicorp/random v3.5.1
+ provider registry.terraform.io/hashicorp/tls v4.0.4

@jimsmith
Thank you for your comment. If possible, could you paste the output when you run 'terraform plan' under that configuration setting?

@jimsmith
Okay, I will check this issue. Thank you for your report.

Hi, @jimsmith
please check latest version.
#32


Sort (Order by) error if the collation not default

Description
I am create an InnoDB table with varchar(10) column in local mysql database with utf8mb4_hungarian_ci collation. Some letters correct order example á-b-é-z.
But if I using "Order By ... ASC" command in the column header I got wrong order, example b-z-á-é. (this is same of the result of general collation)
If I refresh the table (F5), while the arrow indicate the ordering in column header, I got the correct order, example á-b-é-z.
But if I using "Order By ... ASC" command in the column header I got again the wrong order, example b-z-á-é.
DBeaver Version
Community <IP_ADDRESS>410201725
Operating System
Win 11 Pro 23H2 22631.4169
Database and driver
MySQL 8.3.0
JDBC:MYSQL
Steps to reproduce
No response
Additional context
No response

@KJoe69
could you provide some sample data to reproduce the issue?
thanks in advance

It's been a while since no update here. If the issue is still actual please let me know and provide additional information.


Add Vindicia gateway

Add Vindicia gateway to the list of gateways.
This is my first Omnipay driver, so any reviews of the code would be greatly appreciated: https://github.com/vimeo/omnipay-vindicia

Hi Nicky,
Thanks for the contribution.
In the PR you have named this gateway "Vimeo" (at the start of the line that you changed) but I think that the correct gateway name is "Vindicia" -- "Vimeo" being your github login.
I will wait until you can fix this before merging.
I will put some code comments in the actual gateway.

Whoops, good catch @delatbabel. Fixed!
Thanks!


Request to rebuild OpenJ9 Windows/Mac with openssl 1.1.1g

Although OpenJ9 does not appear to be affected by the security vulnerability fixed in openssl 1.1.g, @DanHeidinga pointed out that users may scan the Adopt build and find an openssl version that contains a known security vulnerability. They may be restricted from using the binary due to their company policies. To avoid this, Windows and Mac (Java 8, 11, 14) should be rebuilt to include openssl 1.1.1g.

This is in progress thanks to @Willsparker (Installed 1.1.1g on the Windows build machines) and @lumpfish at the following links (build script adjustments in https://github.com/AdoptOpenJDK/openjdk-build/pull/1697)

https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk8-pipeline/1003/
https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/877/
https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk14-pipeline/129/


Handle old style deprecated auth in launch plans

TL;DR
Please replace this text with a description of what this PR accomplishes.
Type

[ ] Bug Fix
[ ] Feature
[ ] Plugin

Are all requirements met?

[ ] Code completed
[ ] Smoke tested
[ ] Unit tests added
[ ] Code documentation added
[ ] Any pending items have an associated Issue

Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
https://github.com/lyft/flyte/issues/
Follow-up issue
NA
OR
https://github.com/lyft/flyte/issues/

also we have hasfield's all over the code base.  worth it to cut a ticket to clean it all up?


cmd/go: fails to perform concurrent compilation on darwin/arm64

What version of Go are you using (go version)?

$ go version
go version devel go1.18-9e60c37147 Mon Sep 20 16:55:47 2021 +0000 darwin/arm64

Does this issue reproduce with the latest release?
Yes, both go1.16.8 and go1.17.1
What operating system and processor architecture are you using (go env)?
go env Output
$ go env

What did you do?
go build -n -a fmt 2>&1 | grep -c -- -c=

What did you expect to see?
grep -c -- -c= should return non-zero.
What did you see instead?
$ go1.16.8 build -n -a fmt 2>&1 | grep -c -- -c=
0
$ go1.17.1 build -n -a fmt 2>&1 | grep -c -- -c=
0
$ go build -n -a fmt 2>&1 | grep -c -- -c=  # Even with the fix from #48490
0

The problem seems that with arm64, there's -shared passed to the compiler:
https://github.com/golang/go/blob/79159f2e83662dac4089e2f5231ec23de9da10b9/src/cmd/go/internal/work/init.go#L173
cause gcBackendConcurrency return 1:
https://github.com/golang/go/blob/79159f2e83662dac4089e2f5231ec23de9da10b9/src/cmd/go/internal/work/gc.go#L214
$ GOOS=linux go1.17.1 build -n -a -x fmt 2>&1 | grep -c -- -c=
35

I've manually reviewed all uses of ctxt.Flag_shared and they all look concurrency-safe to me. And I manually tested with the race detector and found no issues.

Funny enough the M1 shows all 4 fat cores at 100% when compiling gotip.

@andig that probably due to process-level concurrency.

Maybe. For what its worth: recompiling gotip with this CL applied does not provide a speedup afaikt:
❯ time GOROOT_BOOTSTRAP=/Users/andig/htdocs/golang ./make.bash
Building Go cmd/dist using /Users/andig/htdocs/golang. (devel go1.18-579ff8b131 Mon Oct 4 19:51:37 2021 +0000 darwin/arm64)
Building Go toolchain1 using /Users/andig/htdocs/golang.

Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/arm64.
---
Installed Go for darwin/arm64 in /Users/andig/htdocs/golang2
Installed commands in /Users/andig/htdocs/golang2/bin
GOROOT_BOOTSTRAP=/Users/andig/htdocs/golang ./make.bash  166.35s user 19.85s system 421% cpu 44.204 total

Yes, make.bash has ample process level parallelism. That is not true of all build graphs.


Add kustomization.yml example to the docs

There is support to avoid manual downloading of manifest simply by using Kustomize resources. All we need is to add following example to the documentation to allow others to quickly start deploying and modifying their operator.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yml
- github.com/VictoriaMetrics/operator/config/default?ref=v0.2.0

namespace: monitoring
images:
- name: victoriametrics/operator
  newTag: v0.2.0

Please let me know what do you think and I can send patch right away.
As bonus point it would be great if we could bake in version into git revision so one can skip image patching, but I would expect that as separate step as it would probably require some changes in release process.

Hello,
To be honest, i'm not quite familiar with kustomize and your suggestion looks great for me. Feel free to add pr.
I'll take a look at github actions, probably it's not hard to add git revision baking.

Hi,
yes I'm big fan of kustomize way of installation which is basically fetch existing manifests and then patch it. Way better than hey how about learn new DSL language.. I'm looking at you  helm 👀
Please check #73, we can work with it bit more I was basing it on metallb installation guide https://metallb.universe.tf/installation/#installation-with-kustomize hope it make sense and it is not overwhelming.
I would suggest to do also following steps:

Add kind: Namespace to base manifests as usually you most times want to create new namespace for monitoring operator so it make sense to make it part of base manifests
Add image version to base manifests as discussed
Maybe move operator-settings patching to some example folder to not overwhelm first timers

Please let me know what do you think and which steps you want to also include in this MR and we can discuss rest of them as separate MRs.
Thanks

Thanks for docs, it looks good, i've added few remarks. Also i was able to spot one bug with rbac, when was testing your pr, thx )
I'm not happy with helm, but it has some advantages and became some kind of standard.

Yep, sounds reasonable. Lets add it.
version backing for release is a bit tricky. Releases use git tag under hood, so you cannot modify file content, after release was created. So the options are:

point to master branch at documentation and advice to change version to latest release  manually (you have to create kustomization file any way, so its not a big deal). And bake new version to release assets.
changed release creation process, for instance, we cannot use New Release button at github any more. Instead we have to merge code to some release branch and create new release with github actions. Its complicated and error prone. I wouldn't like to go that way.
draft new release locally and push tag to github.

I like first path)

Well, i think, that you shouldn't modify operator-settings, it has optimal defaults.  So its better to move operator-settings to separate examples folder, with some tips ( How to disable prometheus api conversion for example).

Yeah after 4 years of yamling I don't beleve in ultimate template for all users 😅

It's there, but I would suggest to change it to something by more usable like monitoring-system also with this we can close https://github.com/VictoriaMetrics/operator/issues/62
I understand that it's bit tricky, problem with pointing to master is same as docker:latest image you never know where it will break your CI/CD pipeline. Release branches are not fun so I agree to skip it for now so please check how I'm dealing with it and do let me know if it's ok.
Done, I hope location is ok, I was about to add it to /config folder, but there are already different examples, so I choose docs as better fit.


Suspended transactions that is never terminated

A server uses odbc driver for inserting/retrieving data into/from mssql database
Also server "forks" new processes via WinAPI (CreateProcess)
Sometimes a process can "hang" on an execution of a statemen. It happens, when another one seizes the same table for writing/reading and etc. So, as I can see in mssql, the transaction is suspended, however, I do not understand what a stage is it awaiting on.
When I terminate process, transactions proceed "hanging" (why are not they terminated?). So, a process is terminated but the transaction is still alive but suspended (but should no longer exist)
a) What should I do in this cases?
b) I am not sure that WinAPI uses the same idea with processes as in Linux, but still:
For example, some process is trying to commit an execution and it dies (for some reason) and transaction left. Cannot I replace the receiver of a result of this query? May be I can do it intentionally... May it be a security vulnerability?

@Ayrtat -- thank you for your questions.  Are you reading a particular docs.microsoft.com article?  Meanwhile, please see these resources:

DBA Stack Exchange
Stack Overflow


Updated for Pandas 0.18, added optional thousands separator

also removed deprecation warnings for as_money(), which is more flexible than the Babel as_currency() as precision and thousands separator can be controlled.

I'm going to give this a look over and update tests for this today

Cool, thanks!  I haven't allowed for different thousands separators (comma is assumed), and I think some places separate hundreds rather than thousands... I guess Babel might handle all of that, but not the precision issue...
All of the commits before 78d815c are from my other PR #31, but I didn't know how to isolate these changes... sorry!  If I can do anything else let me know

I need to give the Babel documentation another read anyways. It seemed to be breaking the ability to specify precision which I would like to fix.

I think you've since covered the pandas compatibility issue in master.  Closing as the other stuff needs to be rebased.  Thanks!


Multiple session.send()

Hi,
I try to do in a function of a waterfall:
session.send('how')
session.send('are')
session.send('you')

But only the first one is sent, is there some kind of restriction ?
I thought next() was "flushing" the session
Should I concatenate my string ?

I'm assuming your using the emulator... The node version has an issue with sending multiple messages to the emulator. I have a fix coming very shortly. You can look at examples\TestBot\BotConnectorBot.js for a temporary workaround.

The latest version so fix this for the emulator. Please re-open if the issue is still persisting.

I update to latest library and latest emulator
It seems the bug came back

Closing this issue due to age. @JpEncausse - please open a new issue if this issue is not fixed and needs further discussion. @Stevenic - reopen if necessary. Thanks!


Updating new docker images

:tada: This PR is included in version 3.1.6 :tada:
Your semantic-release bot :package::rocket:


fix typo in a tutorial

Coverage remained the same at 94.298% when pulling 85b5c2338ac638eb567d7be2dc43355a5bdf6248 on wojdyr:patch-1 into 93963a5971c8b7cc9c7b252f3454cfbc5aaddc56 on rasbt:master.

by the way, having atoms in two separate frames is rather not a good idea.
At first glance it may look like the protein chains are all ATOM, but wwPDB uses different criterium:
only natural amino-acids (and nucleic acids) are marked as ATOM, and the modified ones are marked as HETATM.
So MET is ATOM but MSE is HETATM.
If you keep them both separately such an example:
ppdb.df['ATOM']['b_factor'].plot(kind='hist')

won't work as expected - it may skip some residues

Thanks for the PR and the comment. Let me open a new issue to continue the discussion.


fix: Added missing zones

:tada: This PR is included in version 4.2.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


fix Disable sql_limit variable when query rewrite by materialize view

Proposed changes
In BI or some other SQL client，it will set sql_select_limit or default_order_by_limit session variables automatically.
This will make mistake when query rewrite by materialized view, Because after query rewrite by materialized view, the rewritten plan would be optimized by rules again which should add limit on SQL.


@conda-forge-admin, please add user @linuxrider

Hi! This is the friendly automated conda-forge-webservice.
I just wanted to let you know that I added user @linuxrider in conda-forge/svgdigitizer-feedstock#4.


getModelForGuard(): Argument #1 ($guard) must be of type string, null given

I am getting an exception when viewing both the Role and Permission resource:

getModelForGuard(): Argument #1 ($guard) must be of type string, null given, called in /Users/fredrik/Documents/code/xxx/vendor/vyuldashev/nova-permission/src/Role.php on line 92

It's referencing this line:
https://github.com/kiritokatklian/nova-permission/blob/master/src/Role.php#L99-L101
Specifically is the $this->guard_name null for some reason. When I look in the DB it's set to web and when I hardcode the $userResource = \App\Nova\User::class instead it doesn't throw an exception but it doesn't show the User. Very strange.

Nvm, updated the package and this is no longer the issue. Creating another one.


Cannot read properties of undefined (reading 'originalError')

Issue workflow progress

Progress of the issue based on the
Contributor Workflow

[ ] 1. The issue provides a reproduction available on
Github,
Stackblitz
or
CodeSandbox

Make sure to fork this template and run yarn generate in the terminal.
Please make sure Mesh package versions under package.json matches yours.

[ ] 2. A failing test has been provided
[ ] 3. A local solution has been provided
[ ] 4. A pull request is pending review

Describe the bug
I think this happens if one of subschemas return error.
Will try to create reproducable repo.
Bad thing is that error contains no stack or whatsoever.
To Reproduce Steps to reproduce the behavior:
TODO
Expected behavior
To forward error without creating new one on top of it.
Environment:

OS:
@graphql-mesh/*": 0.96.0:
NodeJS: 18

Additional context

Thank you for creating the issue! it'd be better to see a reproduction :)

To forward error without creating new one on top of it.

This doesn't work because we need to update paths and mask the error if it is an internal one, etc.

Closing the issue due to the lack of response.


Fix assemble icon for dark mode

Hello there,
Since BoxBuddy uses a custom hammer icon for the distrobox assemble button it only comes with a dark variant.
This one does not line up with the dark mode of Gnome.
By this PR I introduced a util function is_dark_mode() which uses the Adwaita StyleManager to check if the desktop is running with a dark or a light theme.
In addition I provided a light version of the hammer icon which is then used for the button instead of the dark variant.
Edit: Forgot to mention this.
I also added:
    debug_assert!({
        return format!("icons/{}", icon);
    });

for the icon path for development environments.
This code is only executed if BoxBuddy is compiled without optimizations.
In order for developers not needing to uncomment the icon path for development purpose.
For further details I leave the link to the official documentation here
Kind regards,
V.

I just pushed a little addition.
The hammer icon now also reacts on theme changes.
Before only at start up the dark mode would be evaluated and the icon set accordingly. But not while the app was running and the user might have changed the mode.

Good catch, I only use light mode myself!


Shuttle deployment status badge

It would be nice to support a "project health" / "deployment status" dynamic badge for Shuttle deployments.
Some examples:

Might be related: #840
I can also submit this to shields.io probably.

hmmm, i am new to shuttle, but maybe i can work to do this, @orhun 👍

Feel free!

also i never used dynamic badges so lol

I would say this is blocked until we have a proper solution for #840


[iframe-plugin] Crashes on Android when using certain content.

I swear that I have completed these tasks before submitting:

[ x] I have read the README
[x ] I have checked that the issue has not been reported yet
[ x] I have prefixed this issue title with the plugin suffix (e.g: [iframe-plugin]) depending on the affected plugin

Bug Report
Environment

React Native

  expo-env-info 1.0.5 environment info:
    System:
      OS: macOS 13.0.1
      Shell: 5.8.1 - /bin/zsh
    Binaries:
      Node: 16.18.1 - /usr/local/bin/node
      Yarn: 1.22.19 - /usr/local/bin/yarn
      npm: 8.19.3 - /Volumes/Data/Datenprojekte/VVP/vvp_App/node_modules/.bin/npm
      Watchman: 2<IP_ADDRESS> - /usr/local/bin/watchman
    Managers:
      CocoaPods: 1.11.2 - /usr/local/bin/pod
    SDKs:
      iOS SDK:
        Platforms: DriverKit 22.1, iOS 16.1, macOS 13.0, tvOS 16.1, watchOS 9.1
    IDEs:
      Android Studio: 2021.3 AI-213.7172.25.2113.9123335
      Xcode: 14.1/14B47b - /usr/bin/xcodebuild
    npmPackages:
      expo: ~47.0.8 => 47.0.8
      react: 18.1.0 => 18.1.0
      react-native: 0.70.5 => 0.70.5
    npmGlobalPackages:
      eas-cli: 2.9.0
      expo-cli: 6.0.8
    Expo Workflow: bare

Libraries

react-native-render-html: "^6.3.4",
@native-html/iframe-plugin: "^2.6.1"
react-native-webview: "11.23.1"

Devices

Happens on multiple (>6) different Android Devices, so it's definitely an Android Issue.
I Am using Expo App & build App in Simulator
Reproduction

Use twit frame to generate iFrame for a tweet: https://twitframe.com
Render Iframe using:

const renderers = {
   iframe: IframeRenderer
};
const customHTMLElementModels = {
   iframe: iframeModel
};

Open Project in Expo App or build for Simulator.

Description
The Expo App crashes (without any warning or error messages)
Solution
What solved this for me was wrapping the Webview into a:
<View renderToHardwareTextureAndroid={true} >...</View>
This seems to be primarily a web view issue, however this fix could also be implemented in this plugin in the meantime.

I am having a similar issue when end scrolling on the view.
Even the crash report from Crashlytics is not helpful.
Although @pkreissel 's fix is working.
The way I implemented it is:
const renderers: CustomTagRendererRecord = {
  iframe: (props) => (
    <View renderToHardwareTextureAndroid={true}>
      <IframeRenderer {...props} />
    </View>
  ),
};

@pkreissel I'll cheers to you tonight. This bug has been plaguing our android app for months and it's been tough to trace down and resolve. Your fix immediately worked. Thank you so, so much.

I also faced this issue but above solution work for me @pkreissel Thanks dude

This seems to be like a very important change to implement in the library (or at the very least, document)


Consumer Group does not terminate when KafkaSource is Not Ready

Describe the bug
Warning  InternalError    18s (x19 over 22m)     kafkasource-controller  failed to create consumer group default/47aa8fcb-04a4-450a-bcad-6095302859c1: create not allowed while custom resource definition is terminating

Expected behavior
When KafkaSource is deleted, consumer group must be finalized.
To Reproduce

Install KafkaSource that has an issue causing Ready to be False
ConsumerGroup resource is created
Delete KafkaSource
ConsumerGroup can't be finalized
Re-install the KafkaSource
Controller throws error above because ConsumerGroup still exists

Knative release version
1.7.1
Additional context
Give up after trying a couple of times to delete consumer group offset on Kafka similar to KafkaSource in eventing-kafka
Broker bug reference: https://github.com/knative-sandbox/eventing-kafka-broker/commit/37c43784d7144fb343aaa3a3c74a437b60e06f06

/remove-lifecycle stale

/remove-lifecycle stale


Bumped development RELEASE for 1.22-13

Bumped development RELEASE for 1.22-13
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm
/approve


[Feature Request] Check typing.Protocol Annotation Types

The Problem
Currently, beartype checks typing.Protocol keys, but not the corresponding types:
from dataclasses import dataclass
from typing import Protocol, runtime_checkable

from beartype.door import is_bearable

@dataclass
class Example:
    x: str

@runtime_checkable
class WrongName(Protocol):
    wrong_name: str

@runtime_checkable
class WrongType(Protocol):
    x: int

if __name__ == "__main__":
    example = Example("triceratops")

    print(is_bearable(example, WrongName))  # False, should be False
    print(is_bearable(example, WrongType))  # True, should be False (?)

Ideally, the second call to is_bearable would return False, since "triceratops" is not an int.
A Potential Workaround (?)
For now, a hacky way to circumvent this is to use a metaclass with __instancecheck__:
import inspect
from dataclasses import dataclass
from typing import Any

from beartype.door import is_bearable

@dataclass
class Example:
    x: str

class HackMeta(type):
    def __instancecheck__(self, instance: Any) -> bool:
        for key, value in inspect.get_annotations(self).items():
            # Check if the key is missing.
            if not hasattr(instance, key):
                return False

            # Check if the type is wrong.
            if not isinstance(getattr(instance, key), value):
                return False

        return True

class Hack(metaclass=HackMeta):
    x: int

if __name__ == "__main__":
    print(is_bearable(Example("triceratops"), Hack))  # False, should be False
    print(is_bearable(Example(1), Hack))  # True, should be True

This feels wrong though... Are there any plans to make beartype check both the keys and values/types for typing.Protocol annotations?

I will defer to @leycec to respond directly, of course, but it may help to know that runtime checking of Protcol attributes is already lacking in core Python.  (Which means if this problem is to be solved, it's probably going to be a real bear. :smirk:)
typing.Protocol currently derives from something called _ProtocolMeta, which Python devs have signaled some desire to retire, but it apparently does not want to go gentle into that good night. You can see its __instancecheck__ implementation at Lib/typing.py#L1885-L1916. In short, it basically suffers from the same issue that you raise, namely that it merely checks for the presence of attributes, not their types.
Beartype provides an override Protocol that does little else other than cache runtime instance checks for performance reasons, although the implementation is a beast.1 See: https://github.com/beartype/beartype/blob/3a5b216c78034359d2c634307440adb04ffe5a39/beartype/typing/_typingpep544.py#L117-L388
If you wanted to experiment with your approach, you might consider tinkering with that? It would require you to derive your protocols from beartype.typing.Protocol, though.
Possibly related: #184, #237

1 Whoever wrote that pile ought to consider a career in either the food service or housekeeping industries.
As an aside, Python kind of does something crudely similar, but limited to ABCMeta classes in Lib/_py_abc.py#L92-L106. That may seem like a non sequitur, but note that ABCMeta.__isinstance__ is called from _ProtocolMeta.__instancecheck__ (masquerading as _abc_instancecheck) where the proprietary/internal _is_protocol attribute is missing, but it's unclear when this code path applies. (To say Python's typing code is spaghetti-esque is to insult spaghetti, I think.)

Oh, @posita. You sweet summer bear. I'd hug you, but I live in a wasteland of ice surrounded by man-eating predators with a taste for high IQ. Instead, I'll just restate everything @posita said while pretending I'm adding something of value to the conversation. My ego is fragile.
typing.Protocol: It Pretty Much Sucks, Bro.
PEP 544 standardized typing.Protocol. PEP 544 doesn't do much. Therefore, typing.Protocol doesn't do much either. Sucks, huh? We know. We feel that searing pain in our intestines, too.
Specifically, typing.Protocol only checks that methods of the same names exist. That's it. Srsly. That's all typing.Protocol does. Things that typing.Protocol does not do include:

Everything else.
Everything that you want to do, @dcharatan.
Anything involving type hints.
Anything involving method signatures, including:

Method parameter names.
Method parameter order.
Method parameter types (e.g., keyword-only, positional-only, variadic positional, variadic keyword).
Calling convention (e.g., async versus non-async, generator versus non-generator).

Sadly, @beartype is standards-based. Sucks, huh? Usually, being standards-based is good. But when the standards suck, so too does @beartype.
Thankfully, you've already gone this far. You're already poised on the precipice of madness. If you'd like to go a bit further, do it do it no peer pressure or anything proposing this to CPython's official typing special interest group (SIG) is absolutely the best way to make this happen for everybody. Everybody includes @beartype.
But don't worry! You don't need to actually do anything extreme like author a PEP or whatevahs. Just briefly propose the idea and suggest that, as a prehistoric aquatic dinosaur studying at MIT, your work resurrecting Megalodon from its ancient watery grave as humanity's newest reason to never go scuba-diving vitally depends on typing.Protocol doing something useful with its life for once.
Thus spake Bearthustra. Oh, and...
You Probably Already Know This, But...
class HackMeta(type):
    def __instancecheck__(self, instance: Any) -> bool:
        for key, value in inspect.get_annotations(self).items():
            # Check if the key is missing.
            if not hasattr(instance, key):
                return False

            # Check if the type is wrong.
            #FIXME: isinstance() fails to support most type hints. So,
            #you probably instead want to do this. All hail West Texas.  # <-- wut
            if not is_bearable(getattr(instance, key), value):  # <-- totally works, in general
            #if not isinstance(getattr(instance, key), value):  # <-- totally doesn't work, in general
                return False

        return True

\o/

This issue deeply saddens me. I :heart_eyes_cat: Megalodon and would :heart_eyes_cat: to make this happen for everybody. But this has yet to be standardized. Our paws are tied. Somebody who is not me, please standardize this for everybody.
Until then, with many apologies, I quietly close this. Our issue tracker is exploding. Desperate times call for desperation. :face_with_head_bandage:

Thank you @leycec and @posita for the detailed responses and the tip on using is_bearable!


DeveloperForce 1.3.2 Incompatible with .Net 4.6.1

Reproduced using a new 4.6.1 project:
stacktrace.txt
This fails:
ForceClient client = new ForceClient(auth.InstanceUrl, auth.AccessToken, auth.ApiVersion);
Because of this error:
"Method not found: 'Void Salesforce.Force.ForceClient..ctor(System.String, System.String, System.String, System.Net.Http.HttpClient)'."
The System.Net.Http version in 4.6.1 is <IP_ADDRESS>.  I've tried an assembly binding redirect, but the
packages wants a specific version.

Update: confirmed that this is fixed in the 1.4 beta.
Unfortunately, my team spent two days on this issue before discovering this.

Wow...wow...wow, thanks David.  We too burned a lot of time on this one.  Your comment validated what I suspected.  Almost one year later and v1.4 remains in beta, but at least it fixed the issue!

Apologies.
I've already ported everything to net core/standard and will push new packages soon.


[SUPPORT] Payload logic need comfirm whether changed in master branch

Describe the problem you faced
Payload changed not work in master branch，which can be work in 0.14.x，currently master can not work，https://github.com/apache/hudi/pull/10857
Expected behavior
A clear and concise description of what you expected to happen.
Environment Description

Hudi version : master branch

Spark version : 3.2.0

Hive version :

Hadoop version : 3.2.0

Storage (HDFS/S3/GCS..) : HDFS

Running on Docker? (yes/no) :

Additional context
Add any other context about the problem here.
Stacktrace
Add the stacktrace of the error.

@jonvex @ad1happy2go @danny0405 Could you look into it or give me some message about Payload changed in master branch.

Hi, I think the reason is because the new FG reader uses the native record formats when merging. Try with hoodie.file.group.reader.enabled=false and see if it works then. #9894 I think we might be moving away from merge logic in the payload, but @yihua would probably have a more definitive answer.

@jonvex yes，it works. thanks, close the issue


fix(helm): running linting using golangci-lint

Signed-off-by: Constantin Bugneac<EMAIL_ADDRESS>
Fixes #11707
What this PR does / why we need it:
Fixes security vulnerability.
Applies lint formatting using golangci-lint.
Special notes for your reviewer:
If applicable:

[ ] this PR contains documentation
[ ] this PR contains unit tests
[ ] this PR has been tested for backwards compatibility

Closing


Support bulkload and manual compaction

In our scenario, about 1 billion k-v will be inserted into pika during initialization stage, the inserting operation will trigger compaction when storage uses up to ~70GB, thus the init stage will cost much more time than expected.
After going through the source code, rocksdb::Options supports bulkload scenario, but the options could not be passed through pika -> nemo -> nemo-rocksdb init chain.
Is it possible to add new bulkload options and enable triggering manual compaction?
related code:
  // Set appropriate parameters for bulk loading.
  // The reason that this is a function that returns "this" instead of a
  // constructor is to enable chaining of multiple similar calls in the future.
  //

  // All data will be in level 0 without any automatic compaction.
  // It's recommended to manually call CompactRange(NULL, NULL) before reading
  // from the database, because otherwise the read can be very slow.
  Options* PrepareForBulkLoad();

emmm... ping this @Axlgrep
since blackwidow is now the current engine, is there any possibility that exposes bulkload ?