[BEAM-6908] Support Python3 performance benchmarks - part 2

Support Python3 performance benchmarks - part 2
Keep use of Perfkit performance infrastructure for Python3 benchmarking due to exist integration with Python pipeline and automated metrics publishing. Part 1 changes are on PerfkitBenchmarker in https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1895
Changes include in this PR:

Move python performance test parameters to new class PythonPerformanceTestConfiguration to make task construction clear and easier to maintain.
Create enablePythonPerformanceTest in BeamModulePlugin so that people can enable python performance test in different projects by simply calling this function.
Enable performance tests in Python3 project sdks/python/test-suites/dataflow/py3
Fix a bug in run_integration_test.sh due to wrong check order.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

+R: @tvalentyn

run python postcommit

Thank you. postcommit run failed consistently for a while and got fixed recently (today?). I'll rebase and rerun them.

run python postcommit

Perfkit corresponding changes are merged in https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1900/commits/d0ad984e2ae4d836fc7d6440e44104d2b469ec98. We now can merge this PR after tests pass.

Run Java PreCommit

Run Java PreCommit

Rebased this PR against master.

Run Python_PVR_Flink PreCommit


Ambiguity Performance Measure

Please add a summary in data.respy.info after simulation. Now, I always need to go back to the the very large ambiguity log file.

... moved to https://github.com/structAmbiguity/robupy/issues/2


Relative symlink support

The integration test fixtures use absolute symlinks. What about relative symlinks? Are these supported?

Should add test to this. Maybe it works ?


Add error handling and fallback for .filter

I use fuzzy on a backend project where the pattern and the arr are sometimes undefined or empty. Thought it'll be good to have some error handling and fallbacks. Without it, when you pass undefined, it breaks:
# When pattern is undefined
TypeError: Cannot read property 'toLowerCase' of undefined`
# When arr is undefined
TypeError: Cannot read property 'reduce' of undefined

Awesome! Thanks for the PR

all fixed! :+1:


Fix Crash on IDA 7.1

It seems that there is a naming clash in IDA 7.1, and DecodeInstruction() as-is was returning an int. Be more specific in the module name (idautils) so we know we're gettning an insn_t.

Thank you, the conflict should already be fixed now. I'm closing this PR unless you're still getting this issue.


Merge trivially mergeable intersection types for identity comparison

Please verify that:

[ ] There is an associated issue in the Backlog milestone (required) - No one has created an issue for this specifically, but it is a subset of a larger set of problems discussed in https://github.com/microsoft/TypeScript/issues/27024 and is similar to https://github.com/microsoft/TypeScript/pull/60417.
[x] Code is up-to-date with the main branch
[x] You've successfully run hereby runtests locally
[x] There are new or updated unit tests validating the change

Does not explicitly fix any particular issues I can find, but addresses use cases in the comments of https://github.com/microsoft/TypeScript/issues/27024

TL;DR: this PR makes {a: 1} & {b: 2} compare "equal" to {a: 1; b: 2}.
Use case:
Type testing libraries want to be able to compare types that are intuitively equal. Examples of such libraries include expect-type and type-plus. These try to leverage type comparisons that use the underlying isTypeIdenticalTo function in checker.ts. isTypeIdenticalTo has stronger guarantees than mutual assignability, like ensuring {a?: 1} != {} and {readonly a: 1} != {a: 1}.
Lack of these guarantees leads to deficiences like
import {z} from 'zod';
const mismatch1: z.ZodType<{a?: number}> = z.object({});  // type error desirable
const mismatch2: z.ZodType<{a: number}> = z.object({a: z.number()}).readonly();  // type error desirable

It's increasingly important to be able to trust explicit type annotations under isolated declarations.
Approach:
Unlike https://github.com/microsoft/TypeScript/pull/60417, this PR does not achieve equality by changing how types are normalized. That's because we don't actually want to change how the type is represented. For example, if we were to normalize {a: 1} & {b: 2} to {a: 1; b: 2} the following code would break:
function example<T extends {} | 1>(t: T) {
    // Ok because {a: 1} relates to {a: number} and {b: 2} relates to {b: number}
    Object.assign(t, {a: 1}, {b: 2}) satisfies {a: number} & {b: number}

    // @ts-expect-error Neither {a: 1} nor {b: 2} relates to {a: number; b: number}
    Object.assign(t, {a: 1}, {b: 2}) satisfies {a: number; b: number}
}

Instead this approach updates the logic of isRelatedTo to directly handle this case.
Caveats
This PR only attempts to make "trivial cases" work. At a high level this includes only intersections of plain object literals. I'll try to explain why other cases are not covered. Also consult the new test cases I added as reference.
Supported - All intersection constituents are plain object literals
{a: 1} & {b: 2} & {c: 3} == {a: 1; b: 2; c: 3}
Supported - Recursive comparisons
{a: {x: 1}} & {a: {y: 2}} == {a: {x: 1; y: 2}}
Unsupported - Any objects have call or construct signatures
{a: 1} & {(): void} != {a: 1; (): void}
This is not supported right now because merging call signatures is more complicated. You would have to ensure that the ordering is preserved since the intersection of call signatures is equivalent to function overloading, where order matters. A subset of simpler call signature cases could be supported, but it doesn't seem worth the complexity for only a half-baked solution.
Unsupported - Any objects have index signatures
{a: 1} & {[key: number]: 2} != {a: 1; [key: number]: 2}
The main reason to not support this now is because recursive merging can be more complicated, e.g.
You can write {[key: number]: {x: 1}} & {[key: number]: {z: 1}} but writing {[key: number]: {x: 1}; [key: number]: {z: 1}} isn't allowed!
Unsupported - Intersection includes non-plain objects, notably generic type variables
T & {a: 1} & {b: 2} != T & {a: 1} & {b: 2}
The main reason to not support this is because the implementation just becomes complicated and causes more types to be instantiated. Also consider the type {a: 1} & T & {b: 2}. At first glance, you might think that it's ok to make this equal to T & {a: 1} & {b: 2} and hence equal to T & {a: 1} & {b: 2}, but it's not actually safe to reorder constituents of intersections. As mentioned above, changing the order of call signatures changes the semantics of the type:
function example<T extends {a: () => void}>(t: T, x: {a: () => 1}, y: {a: () => 2}) {
   return Object.assign(x, t, y);
   // => {a: () => 1} & T & {a: () => 2}
   // => {a: () => 1} & T & {a: () => void} & {a: () => 2}
}

Like other cases, we could try to handle a simpler subset of cases, but it doesn't seem worth it.
Testing
This patch has been tested in Airtable's codebase and does not introduce any visible regressions. I added several test cases in this PR.
Note
This PR does not automatically make libraries like expect-type work. For example, this will still not work:
import {expectTypeOf} from 'expect-type'
expectTypeOf<{a: 1} & {b: 2}>().toEqualTypeOf<{a: 1; b: 2}>()

Why? Because of the way that expect-type implements its equality check.

export type StrictEqualUsingTSInternalIdenticalToOperator<L, R> =
  (<T>() => T extends (L & T) | T ? true : false) extends <T>() => T extends (R & T) | T ? true : false
    ? IsNever<L> extends IsNever<R>
      ? true
      : false
    : false

Note the (L & T) and (R & T). These intersections contain generic type variables and hence fall under the caveat above. I do not know why expect-type implements type equality this way. I mean I know that it probably copy-pasted the recipe from this thread, but it's unclear why the recipe is defined that way. My proposal for libraries like expect-type to benefit from this PR is to simplify their implementation to:
export type StrictEqualUsingTSInternalIdenticalToOperator<L, R> =
  (<T>() => T extends L ? true : false) extends <T>() => T extends R ? true : false
    ? IsNever<L> extends IsNever<R>
      ? true
      : false
    : false

I don't know of a case that the former definition satisfies that the latter does not, so if anyone knows, feel free to chime in.

@typescript-bot test it

Hey @RyanCavanaugh, the results of running the DT tests are ready.
There were interesting changes:
Branch only errors:
Package: hasura Error:
Error:
/mnt/vss/_work/1/DefinitelyTyped/types/hasura/hasura-tests.ts
  44:23  error  TypeScript@local compile error:
Object literal may only specify known properties, and '_eq' does not exist in type 'WhereBoolExp<ScalarJSON<{ foo: string; }>>'                        @definitelytyped/expect
  67:5   error  TypeScript@local compile error:
Type 'string' is not assignable to type 'OrderBy<ScalarJSON<{ foo: string; }>>'.
  Type 'string' is not assignable to type '{ [key: string]: any; }'   @definitelytyped/expect
  68:5   error  TypeScript@local compile error:
Type 'string' is not assignable to type 'OrderBy<ScalarJSONB<{ foo: string; }>>'.
  Type 'string' is not assignable to type '{ [key: string]: any; }'  @definitelytyped/expect

✖ 3 problems (3 errors, 0 warnings)

    at combineErrorsAndWarnings (/mnt/vss/_work/1/DefinitelyTyped/node_modules/.pnpm/@definitelytyped+dtslint@0.2.28_typescript@5.8.0-dev.20241212/node_modules/@definitelytyped/dtslint/dist/index.js:194:28)
    at runTests (/mnt/vss/_work/1/DefinitelyTyped/node_modules/.pnpm/@definitelytyped+dtslint@0.2.28_typescript@5.8.0-dev.20241212/node_modules/@definitelytyped/dtslint/dist/index.js:186:20)

You can check the log here.

Will take a look at this this week

Can't promise we'll get to this before the holidays, but we'll definitely discuss

I don't know of a case that the former definition satisfies that the latter does not, so if anyone knows, feel free to chime in.

@MichaelMitchell-at Looks like it's because of the two cases mentioned here.

Very interesting! Is it be possible to pack this patch for testing, please?
I am interested to try it with TSTyche which is using .isTypeRelatedTo() to compare types.

I don't know of a case that the former definition satisfies that the latter does not, so if anyone knows, feel free to chime in.

@MichaelMitchell-at Looks like it's because of the two cases mentioned here.

Thank you for sharing those cases @som-sm. One observation is that while the longer definition of IsEqual does satisfy:
IsEqual<{a: 1} & {a: 1}, {a: 1}>
IsEqual<{a: 1} | {a: 1}, {a: 1}>

it works for types at the top level and fails to work if there is there is any nesting, e.g.
IsEqual<[{a: 1} & {a: 1}], [{a: 1}]>
IsEqual<[{a: 1} | {a: 1}], [{a: 1}]>

and IsEqual<T & T, T> doesn't always work for all T, e.g.
IsEqual<({a: 1} | {b: 2}) & ({a: 1} | {b: 2}), {a: 1} | {b: 2}>

So my recommendation still stands to use the shorter definition of IsEqual since this PR makes IsEqual<{a: 1} & {a: 1}, {a: 1}> work with the shorter definition (when T is an object literal type). It's unfortunate that IsEqual<{a: 1} | {a: 1}, {a: 1}> wouldn't work so here are two alternative options:

A) Change the definition to:

type IsWeaklyEqual<A, B> = [A, B] extends [B, A] ? true : false;
type IsEqual<A, B> = IsWeaklyEqual<
    A extends unknown ? (<G>() => G extends A ? 1 : 2) : never,
    B extends unknown ? (<G>() => G extends B ? 1 : 2) : never
>

B) Wait and see if https://github.com/microsoft/TypeScript/pull/60868 is accepted, which I just wrote :)


Inability to approve certain Kata due to 500 error

Yesterday I was just searching for beta Kata that were "Awaiting Moderator Approval" and approving them and I ran into this issue - although some Kata are "Awaiting Moderator Approval" (and perhaps for a long time), when I tried to approve them by clicking on "Approve/Edit Kata", a 500 error occurred and I cannot edit/approve the Kata.
I also noticed that this 500 error (when trying to edit certain beta Kata) is Kata-specific, that is, once a beta Kata is plagued with this bug, no matter how may times I click on "Edit/Approve Kata", the server will always respond with a 500 error rendering that particular beta Kata dead (even if it was very well-written and no issues were raised).
Some examples and screenshots:
http://www.codewars.com/kata/filling-an-array-part-1/

http://www.codewars.com/kata/sum-and-length/

http://www.codewars.com/kata/sum-of-positive/

I'm not sure but I think it might happen if "Allow Contributors" has been turned off by the kata author because it's quite likely there are such katas but I've never seen a proper error message stating that contributors are not allowed..

I'm not sure if it's related (I'm guessing it might be?) but when I try to just click Edit Kata on previously ranked katas, I'm getting the same 500 error. I wasn't planning on actually editing the ranked katas but just wanted to look inside a couple of them.
I've tried it on both preview.codewars.com and codewars.com with the same result. I was just about to make an issue but this one seemed to be so close I thought I would comment here first.

Another kata with the same problem. :(

I saw some talk about this same issue in the discourse for a kata recently, and wondered whether this is why my "Resistor Color Codes" kata has been in the "Awaiting moderator approval" state for some weeks now: https://www.codewars.com/kata/search/my-languages?q=needs_approval&xids=not_completed

It looks like this issue is related to author's unselecting the "allow contributors" checkbox. After a certain period of inaction by the author to respond to a new issue, this checkbox doesn't have an affect - which is why it looks like eventually these kata were able to be approved.

As above, the first kata I wrote is still awaiting approval, but my second kata was approved with no problem quite a while ago. With both of them I have the "allow contributors" checkbox unchecked, but one difference between them that I suppose could possibly be affecting things is that with my approved one I had approved some translations by other people before it was approved, whereas my presumably-stuck one has not has any translations submitted by others yet.


Request for upgrade rouge-score on pypi

@peterjliu Hi, I see the rouge has been updated many times in the repository. So may you upgrade it on pypi for researchers' convenient use.
Thanks!

Indeed, a release would be much appreciated!

Thanks!!!

There seems to be an issue with the release: https://github.com/google-research/google-research/issues/1212

Whoops, try 0.0.8

Should be fixed in 0.1.2.


IGNITE-14133 Document cluster API implementation for CPP

Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
The Contribution Checklist

[ ] There is a single JIRA ticket related to the pull request.
[ ] The web-link to the pull request is attached to the JIRA ticket.
[ ] The JIRA ticket has the Patch Available state.
[ ] The pull request body describes changes that have been made.
The description explains WHAT and WHY was made instead of HOW.
[ ] The pull request title is treated as the final commit message.
The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
[ ] A reviewer has been mentioned through the JIRA comments
(see the Maintainers list)
[ ] The pull request has been checked by the Teamcity Bot and
the green visa attached to the JIRA ticket (see TC.Bot: Check PR)

Notes

How to Contribute
Coding abbreviation rules
Coding Guidelines
Apache Ignite Teamcity Bot

If you need any help, please email<EMAIL_ADDRESS>or ask anу advice on http://asf.slack.com #ignite channel.

@dmagda @Mmuzaf Could you please review the PR?
The changes included are approved by @isapego

@Nikita-tech-writer, thanks! Merging to the master. @Mmuzaf please take care of 2.10.


LanCache Aynology

2

3

4

5

6

7

8

DSM 6
Am Fr., 27. Jan. 2023 um 10:11 Uhr schrieb MicroshopHelp <
@.***>:

Thanks for the guide, what version of DSM is this? because mine looks
different
—
Reply to this email directly, view it on GitHub
https://github.com/tpill90/lancache-prefill-common/issues/1#issuecomment-1406213406,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC3CGB7F6WH6MPDEDN3CJADWUOGLTANCNFSM6AAAAAAUFPUPTU
.
You are receiving this because you authored the thread.Message ID:
@.***>


[BUG] unable to commission lighting-app demo on esp32 with iOS -- mbedTLS error: ECP - The signature is not valid

Reproduction steps
reproduction steps:
-clone this repo
-install build environments
-build examples/esp32/lighting-app/esp32
-flash example on ESPWROOM32 dev board
-start commissioning from Home app on iOS 16.4.1
-get Home error message: Unable to add accessory, pairing with accessory failed
see errors in accessory log attached
log.txt
Bug prevalence
always
GitHub hash of the SDK that was being used
2e4064e17b2bfa8394100483279a4bbaf9d40b42
Platform
esp32
Platform Version(s)
ESPWROOM32, iOS 16.4.1
Anything else?
No response

@albrownwood Is this reliably reproducible?  If so, could you please can you please follow instructions below to file a feedback assistant ticket which would include logs from both the esp32 device and the iOS device?
Feedback assistant ticket:

File a feedback assistant ticket.
https://developer.apple.com/bug-reporting/#feedback-assistant
Click on the link below using your account:
https://feedbackassistant.apple.com/new-form-response
Choose iOS/iPadOS to file an issue
Choose the area as Home App and fill in the details of the type of issue and the description

Technical Information
Please provide the following
a) Sysdiagnose logs for iPhone / iPad with the timestamp / timeframe of the issue
b) Sysdiagnose logs for all the Home Hubs (HomePod mini / HomePod / Apple TV) with the timestamp / timeframe of the issue
c) Accessory logs with the timestamp / timeframe of the issue
d) mDNS network logs with timestamp / timeframe of the issue
e) Video or screenshot describing the issue (optional but beneficial)
Please make sure you download and install the profiles for iOS and tvOS as provided here. The instructions are also in these links.
iOS:

https://developer.apple.com/bug-reporting/profiles-and-logs/?platform=ios&name=homekit
https://developer.apple.com/bug-reporting/profiles-and-logs/?platform=ios&name=network
https://developer.apple.com/bug-reporting/profiles-and-logs/?platform=ios&name=mdns

tvOS:

https://developer.apple.com/bug-reporting/profiles-and-logs/?platform=tvos&name=homekit
https://developer.apple.com/bug-reporting/profiles-and-logs/?platform=tvos&name=network
https://developer.apple.com/bug-reporting/profiles-and-logs/?platform=tvos&name=mdns
For installation instructions, refer to https://developer.apple.com/bug-reporting/profiles-and-logs.
Note: Once you have the profile on the iPhone, it will ask you if you want to install it on other devices. Follow the prompts to install on the HomePod, Apple TV, etc., where you have the described issue and reproduce it.

@albrownwood And if you do file the feedback assistant ticket, please mention the ticket number here so I can make sure it gets looked at.

closing as I am not able to reproduce that issue


Hide Mesh

Great work with this module!
Is there any way of toggling the visibility of a mesh?  I'm trying to incorporate this with ipytree, but there's no callback or trait that lets me toggle the global visibility of a mesh.  Is this a low hanging PR?

Thanks for the kind words!

I'm trying to incorporate this with ipytree

I originally created ipytree with this exact purpose in mind!

but there's no callback or trait that lets me toggle the global visibility of a mesh. Is this a low hanging PR?

It's unfortunately not possible as of today. And it requires changes in https://github.com/QuantStack/GanyJS. It would need adding a setter for visible just like for the defaultColor in here: https://github.com/QuantStack/GanyJS/blob/master/src/Block.ts#L192 (this requires adding a setter for visible in this class as well that does this.mesh.visible = visibility).
Then we need to update ipygany to expose the visible trait to Python.


Expand/Collapse Button

I have some parents with children and some without. But only the one with children shall have a button. how can I do this?

@XxGoliathusxX when binding the parent you have the parent object and therefore know whether it has children or not. Based on that information you can toggle the visibility of your button.


First run feedback

I confirm that I've successfully run your example:
import requests
resp = requests.post('http://localhost:5000/api/statistics/isi', json={'spiketrain': [1,3,6,10]})
resp.json()
# [2, 3, 4]

An unordered list of issues encountered, suggestions proposed:

the basic functional - /api/module/call - is exactly what I wanted: simple and neat. I find get_arguments a bit lengthy, but we'll improve it in the future to handle all peculiarities of elephant input-output data relationships. For now, it's great! I suppose, you just copied this function from NEST-Desktop - NEST integration.
we're not going to support /exec endpoint. Whatever benefits it brings, I'm not signing up for insecure exec() function in Elephant. When you remove exec-related functionality, the code simplifies considerably.
offtopic: I noticed that you use a handwritten stdout capture class. Consider this method: https://stackoverflow.com/a/40417352/2840134. I mean, for NEST, not Elephant.
we need to be python-friendly in terms of a naming convention. Please call functions lowercase only. For example, consider renaming SpikeTrain -> to_spiketrain(). This is my part, I'll do it once we move your code to a place where we can collaborate.
we need to talk through the serialize function. Currently, it captures XX percent of elephant usage. I'm not sure that I want to have a really large and general function for the serialization. An alternative would be using Cristiano's DataObject classes (the output of each elephant function; currently in the experimental branch), where serialize is implemented. Then the serialization is object-specific and handled transparently for the user: result = data_object.serialize(). The same interface for deserializing.

You don't need to edit your code right now. I suggest we talk through it first.

Thank you for prompt response.
I agree most points and have already changed the codes.

[ ] get_argument function
[x] remove exec endpoint
[x] remove stdout capture (it is used for exec)
[x] python-friendly name convention (SpikeTrain -> to_spike_train)
[ ] serialize function

For other aspects we can discuss in VC.


Static phone number

is it possible to have only one (static) phone number? So anyone who wanna use video chat can chose only one phone number.  Every time page loads there is different number so where in code is possibility to make it static. Thank you for answer.

Yes!  You are correct.  You can pick the static phone number.  You can either set one per user, or you can use localStorage.

Thank you for your answer. Can you tell me where in code i need to set up static phone number?

Static Number Example
// create static number
let number = localStorage.getItem('static-number') || (''+Math.random()).split('.').join('');
localStorage.setItem( 'static-number', number );

var phone = PHONE({
    number        : number,
    publish_key   : 'pub-c-561a7378-fa06-4c50-a331-5c0056d0163c',
    subscribe_key : 'sub-c-17b7db8a-3915-11e4-9868-02ee2ddab7fe',
    ssl           : true
});


centrifuge build version

Hi, Julian and I spent the better part of today trying to figure out why his plascope did something different compared to mine. The outcome: I had an older build version of centrifuge.
Proposed solution: hardcode the build version of centrifuge in the command installing plascope/centrifuge in the plasmidEC_plascope environment, or (in the case this is handled by plascope) update it after installing plascope.
Code that solved my issue:
conda install centrifuge=1.0.3=py36pl5.22.0_3 -c bioconda

Hi Jesse,
I thought that the plascope conda environment would include a specific version of centrifuge, but this is apparently not the case.
I will implement your solution.

Closed by 8913a0d9c8cd9b013d42452ca141e9cc3067d350


ux: can you show download info like npmjs.com

Duplicate of https://github.com/jsr-io/jsr/issues/14


adding breakout port testing for singleton testcas

CL approved and passed go readability - https://critique.corp.google.com/cl/489594630
testing pass - ran test for arista 7280 - http://sponge2/6bddfb57-0fae-4bc3-a312-f10a3f46539d
execution link from command line for 7808 - https://paste.googleplex.com/6547896437571584

/gcbrun


Post table sync SQL execution

Suggested by Simbiotica guys

To have a PG function (or arbitrary SQL code) execute after every successful sync of a given table
As an "advanced" option, out of the UI, only for command-line based imports
This would help a lot in some scenarios

e.g.
UPDATE xxx SET the_geom=ST_MAKEVALID(the_geom)

There has been no activity on this issue for more several months. We are closing it. If you think this still needs to be addressed please open a new issue.


Define Java mirror url

It's need to download the older versions.

Can you explain a bit what you are trying to achieve? I think with the current version it should be possible to download older versions, as long as Oracle has not pulled them from their website.

You can't download jdk 1.8.0_65 from console because Oracle removed old versions from /otn-pub/.
Now you can download it only manually. But you can download it to corporate storage and install it from there.

Good idea, thank you!


updating the link to the aro ocp docs

@sakthi-vetrivel quick update for ARO doc links on the Red Hat side.

Hi @sakthi-vetrivel - Could you review this PR? Thanks.


blobFUSE flexVolume README example problems

I followed the README instructions using a PV/PVC, but the flexVolume won't attach to the pod in the end. Even using the example files (with required modifications eg container). kubectl describe pod nginx-flex-blobfuse-pvc gives:
MountVolume.SetUp failed for volume "pv-blobfuse-flexvol" : mount command failed, status: Failure, reason: validation failed, error log:accountname is empty
It does seem to work when it's a dynamic claim (eg nginx-flex-blobfuse.yaml), but that example doesn't let you set the size of the claim. It defaults to 30Gi.
The secret does exist, and I can validate that it's correct with kubectrl describe secret blobfusecreds. Not sure how to proceed debugging here.

error logs says accountname is empty, are you using accountname keyword in kubectl create secret command?
kubectl create secret generic blobfusecreds --from-literal accountname=ACCOUNT-NAME --from-literal accountkey="ACCOUNT-KEY" --type="azure/blobfuse"

You may provide kubectrl describe secret blobfusecreds output.

Besides that, there is no size concept for azure storage, it has only a size limit which is very big, size in PVC is only a tag, cannot prevent you to exceed that size.

Thanks for the fast response!
$ kubectl describe secret blobfusecreds

Name:         blobfusecreds
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  azure/blobfuse

Data
====
accountkey:   88 bytes
accountname:  23 bytes

Looks OK to me. I can also use kubectl get secret blobfusecreds -o yaml and validate that the accountkey and accountname are correct (base64 encoded versions of the raw values I input). Not to mention, those creds HAVE to be OK for it to work in the dynamic claim.
Regarding the size, what I mean is that the mounted device in the container is seen (by df -h) as 30G in size. In the PV/C method in the README, you get to explicitly set the size of the volume and claim.
Can you reproduce using the demo files?

 I re-created everything today and it DOES mount now.
BUT df -h still reports the funny device size. Actually I noticed that it's reporting the same used and total as the root device. So I'm not at all certain that it's actually using the blobstore.
$ df -h

Filesystem                                                                                Size  Used Avail Use% Mounted on
overlay                                                                                30G  9.4G   20G  33% /
tmpfs                                                                                3.5G     0  3.5G   0% /dev
tmpfs                                                                                3.5G     0  3.5G   0% /sys/fs/cgroup
/dev/sda1                                                                                30G  9.4G   20G  33% /etc/hosts
shm                                                                                64M     0   64M   0% /dev/shm
blobfuse                                                                                30G  9.4G   20G  33% /var/www/html/data
//nextcloudstorageaccount.file.core.windows.net/kubernetes-dynamic-pvc-4a1d1fbb-c36a-11e8-a7a2-2aa91002eab5   20G  184M   20G   1% /var/www/html/config
tmpfs                                                                                3.5G   12K  3.5G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                                3.5G     0  3.5G   0% /sys/firmware

For reference, here is the config:
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-blobfuse-flexvol
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  flexVolume:
    driver: "azure/blobfuse"
    secretRef:
      name: blobfusecreds
    options:
      container: myblobcontainer
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-blobfuse-flexvol
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  volumeName: pv-blobfuse-flexvol
  storageClassName: ""
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-nextcloud
  labels:
    app: azure-nextcloud
spec:
  selector:
    matchLabels:
      app: azure-nextcloud
      tier: frontend
  template:
    metadata:
      labels:
        app: azure-nextcloud
        tier: frontend
    spec:
      initContainers:
        - name: config-clone
          image: alpine/git
          args:
            - clone
            - --single-branch
            - --
            - https://github.com/ohthehugemanatee/nextcloud-config
            - /config
          securityContext:
            runAsUser: 1
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          volumeMounts:
            - name: config-data
              mountPath: /config
      containers:
      - name: azure-nextcloud
        image: nextcloud
        volumeMounts:
          - name: flexvol-mount
            mountPath: /var/www/html/data
          - name: persistent-config
            mountPath: /var/www/html/config
          - name: config-data
            mountPath: /var/www/html/config/custom.config.php
            subPath: custom.config.php
        ports:
        - containerPort: 80
          name: nextcloud
      volumes:
        - name: flexvol-mount
          persistentVolumeClaim:
            claimName: pvc-blobfuse-flexvol
        - name: persistent-config
          persistentVolumeClaim:
            claimName: azurefile
        - name: config-data
          emptyDir: {}

How can I validate that it's mounting from the blobstore? Is it possible that it's failing over to loopback-mounting the root drive or something?

I tested creating a 512MB file in the "blobstore" mounted directory, and can confirm that it changes the df -h values for everything based on the root device. So I guess it's actually using the root device.
Here's the describe output for the PV and PVC. They're supposed to be 1TB in this case.
$ kubectl describe pv pv-blobfuse-flexvol

Name:            pv-blobfuse-flexvol
Labels:          <none>
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"pv-blobfuse-flexvol","namespace":""},"spec":{"accessModes":["ReadWrit...
                 pv.kubernetes.io/bound-by-controller=yes
Finalizers:      []
StorageClass:
Status:          Bound
Claim:           default/pvc-blobfuse-flexvol
Reclaim Policy:  Retain
Access Modes:    RWX
Capacity:        1Ti
Node Affinity:   <none>
Message:
Source:
    Type:       FlexVolume (a generic volume resource that is provisioned/attached using an exec based plugin)
    Driver:     azure/blobfuse
    FSType:
    SecretRef:  &{blobfusecreds }
    ReadOnly:   false
    Options:    map[container:nextcloud]
Events:         <none>

$ kubectl describe pvc pvc-blobfuse-flexvol

Name:          pvc-blobfuse-flexvol
Namespace:     default
StorageClass:
Status:        Bound
Volume:        pv-blobfuse-flexvol
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"pvc-blobfuse-flexvol","namespace":"default"},"spec":{"accessMode...
               pv.kubernetes.io/bind-completed=yes
Finalizers:    []
Capacity:      1Ti
Access Modes:  RWX
Events:        <none>

Your oginial issue is due to this issue: https://github.com/Azure/kubernetes-volume-drivers/issues/4, fixed in blobfuse-flexvol-installer:1.0.7.
Could you paste logs from /var/log/blobfuse-driver.log under aks-nodepool1-40115916-0? I doubt whether blobfuse is aleady installed on your aks node, you could check by running blobfuse directly.
To verify whether blobfuse mount successfully, you could check by
azureuser@aks-nodepool1-20449952-1:~$ mount | grep blobfuse
blobfuse on /var/lib/kubelet/pods/b31cc112-d5bd-11e8-85db-fe56e25f3d37/volumes/azure~blobfuse/pv-blobfuse-flexvol type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
blobfuse on /var/lib/kubelet/pods/b31cc112-d5bd-11e8-85db-fe56e25f3d37/volumes/azure~blobfuse/pv-blobfuse-flexvol type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072)

$ kubectl exec -it azure-nextcloud-7fc9dbb8cc-jhhj7 -- bash

root@azure-nextcloud-7fc9dbb8cc-jhhj7:/var/log# ls /var/log
alternatives.log  apache2  apt	btmp  dpkg.log	faillog  lastlog  wtmp
root@azure-nextcloud-7fc9dbb8cc-jhhj7:/var/log# mount |grep blobfuse
blobfuse on /var/www/html/data type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
root@azure-nextcloud-7fc9dbb8cc-jhhj7:/var/log# blobfuse
bash: blobfuse: command not found

yep looks like it's not installed. That's odd; kubectl version says I'm on 1.9.9:
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.9", GitCommit:"57729ea3d9a1b75f3fc7bbbadc597ba707d47c8a", GitTreeState:"clean", BuildDate:"2018-06-29T01:07:01Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

And I definitely ran kubectl apply -f https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/flexvolume/blobfuse/deployment/blobfuse-flexvol-installer-1.9.yaml. I can see the installer container running with kubectl describe node aks-nodepool1-40115916-0:
Name:               aks-nodepool1-40115916-0
Roles:              agent
Labels:             agentpool=nodepool1
                    beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=Standard_DS2_v2
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=westeurope
                    failure-domain.beta.kubernetes.io/zone=0
                    kubernetes.azure.com/cluster=MC_myAKSCluster_myAKSCluster_westeurope
                    kubernetes.io/hostname=aks-nodepool1-40115916-0
                    kubernetes.io/role=agent
                    storageprofile=managed
                    storagetier=Premium_LRS
Annotations:        node.alpha.kubernetes.io/ttl=0
                    volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp:  Fri, 28 Sep 2018 23:13:02 +0200
Taints:             <none>
Unschedulable:      false
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Fri, 28 Sep 2018 23:13:54 +0200   Fri, 28 Sep 2018 23:13:54 +0200   RouteCreated                 RouteController created a route
  OutOfDisk            False   Tue, 23 Oct 2018 23:54:34 +0200   Fri, 28 Sep 2018 23:13:02 +0200   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure       False   Tue, 23 Oct 2018 23:54:34 +0200   Fri, 28 Sep 2018 23:13:02 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Tue, 23 Oct 2018 23:54:34 +0200   Fri, 28 Sep 2018 23:13:02 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  Ready                True    Tue, 23 Oct 2018 23:54:34 +0200   Fri, 28 Sep 2018 23:13:52 +0200   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  <IP_ADDRESS>
  Hostname:    aks-nodepool1-40115916-0
Capacity:
 cpu:     2
 memory:  7137172Ki
 pods:    110
Allocatable:
 cpu:     1940m
 memory:  5567380Ki
 pods:    110
System Info:
 Machine ID:                 5909d081b6474f5888c11ef872d63236
 System UUID:                1C5E687E-BE10-F84A-880A-0F85B7AA2363
 Boot ID:                    b76d5969-21c5-4a61-8894-52e8786c940d
 Kernel Version:             4.15.0-1021-azure
 OS Image:                   Ubuntu 16.04.5 LTS
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://1.13.1
 Kubelet Version:            v1.9.9
 Kube-Proxy Version:         v1.9.9
PodCIDR:                     <IP_ADDRESS>/24
ProviderID:                  azure:///subscriptions/95c5fc65-9db4-4f8c-886a-82d49474fba8/resourceGroups/MC_myAKSCluster_myAKSCluster_westeurope/providers/Microsoft.Compute/virtualMachines/aks-nodepool1-40115916-0
Non-terminated Pods:         (12 in total)
  Namespace                  Name                                     CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                                     ------------  ----------  ---------------  -------------
  default                    azure-mariadb-7cf9dd5785-kq7l6           0 (0%)        0 (0%)      0 (0%)           0 (0%)
  default                    azure-nextcloud-7fc9dbb8cc-jhhj7         0 (0%)        0 (0%)      0 (0%)           0 (0%)
  flex                       blobfuse-flexvol-installer-5jdz6         0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                heapster-6f4d8b48bc-f5mcd                130m (6%)     130m (6%)   230Mi (4%)       230Mi (4%)
  kube-system                kube-dns-v20-7d874cb9b6-7n878            110m (5%)     0 (0%)      120Mi (2%)       220Mi (4%)
  kube-system                kube-dns-v20-7d874cb9b6-kv9n8            110m (5%)     0 (0%)      120Mi (2%)       220Mi (4%)
  kube-system                kube-proxy-j58z5                         100m (5%)     0 (0%)      0 (0%)           0 (0%)
  kube-system                kube-svc-redirect-vflkt                  10m (0%)      0 (0%)      34Mi (0%)        0 (0%)
  kube-system                kubernetes-dashboard-68f8cc4d8c-4qw9n    100m (5%)     100m (5%)   50Mi (0%)        300Mi (5%)
  kube-system                omsagent-55zvm                           50m (2%)      150m (7%)   200Mi (3%)       750Mi (13%)
  kube-system                omsagent-rs-66bddff988-zf5kl             50m (2%)      150m (7%)   100Mi (1%)       500Mi (9%)
  kube-system                tunnelfront-776d87969b-8vhph             10m (0%)      0 (0%)      64Mi (1%)        0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource  Requests     Limits
  --------  --------     ------
  cpu       670m (34%)   530m (27%)
  memory    918Mi (16%)  2220Mi (40%)
Events:     <none>

I just tried recreating it with kubectl apply and had no unexpected problems.
So why isn't blobfuse available on my node?

kubectl apply -f https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/flexvolume/blobfuse/deployment/blobfuse-flexvol-installer-1.9.yaml suppose blobfuse binary is already installed on VM, and on latest AKS node, blobfuse binary is already installed by default.
You may install blobfuse manually on your agent node:
https://github.com/Azure/azure-storage-fuse/wiki/1.-Installation#option-1---install-from-aptyum-package-repositories

BTW, your AKS cluster may be old, on the newly created AKS cluster, blobfuse binary is already pre-installed on every node. Yyou need to install blobfuse binary manually on your current node.

Hm. I created the AKS cluster in the last week of September. Is that old enough not to have blobfuse? Or maybe it's the region I'm in?

which region? I can help you check.

westeurope. I just tried removing and re-creating the entire cluster, then running the readme again. Still:
$ kubectl exec -it nginx-flex-blobfuse -- bash

root@nginx-flex-blobfuse:/# mount |grep blobfuse
blobfuse on /data type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
root@nginx-flex-blobfuse:/# blobfuse
bash: blobfuse: command not found
root@nginx-flex-blobfuse:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          30G  7.7G   22G  27% /
tmpfs           3.5G     0  3.5G   0% /dev
tmpfs           3.5G     0  3.5G   0% /sys/fs/cgroup
blobfuse         30G  7.7G   22G  27% /data
/dev/sda1        30G  7.7G   22G  27% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           3.5G   12K  3.5G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           3.5G     0  3.5G   0% /sys/firmware
root@nginx-flex-blobfuse:/#

Thank you again for all your support.

@ohthehugemanatee I created an AKS cluster in west europe, by running following command, it works well.
kubectl create secret generic blobfusecreds --from-literal accountname=ACCOUNT-NAME --from-literal accountkey="ACCOUNT-KEY" --type="azure/blobfuse"
wget https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/flexvolume/blobfuse/pv-blobfuse-flexvol.yaml
vi pv-blobfuse-flexvol.yaml
kubectl create -f pv-blobfuse-flexvol.yaml
kubectl create -f https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/flexvolume/blobfuse/pvc-blobfuse-flexvol.yaml
kubectl create -f https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/flexvolume/blobfuse/nginx-flex-blobfuse-pvc.yaml

azureuser@aks-nodepool1-57931585-1:~$ mount | grep blobfuse
blobfuse on /var/lib/kubelet/pods/2100d18c-db49-11e8-bc12-6a4c34652008/volumes/azure~blobfuse/pv-blobfuse-flexvol type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
azureuser@aks-nodepool1-57931585-1:~$ blobfuse
Error: No config file provided and --container-name is not set.
Usage: blobfuse <mount-folder> --tmp-path=</path/to/fusecache> [--config-file=</path/to/config.cfg> | --container-name=<containername>]    [--use-https=true] [--file-cache-timeout-in-seconds=120] [--log-level=LOG_OFF|LOG_CRIT|LOG_ERR|LOG_WARNING|LOG_INFO|LOG_DEBUG]
...

Could you ssh to the agent VM which is running nginx-flex-blobfuse and get the /var/log/blobfuse-driver.log, the VM should be under resource group name: MC_{RESOUCE-GROUP-NAME}{CLUSTER-NAME}{REGION} and blobfuse binary is already built-in in AKS node.

I notice your example is logging into the nodepool - I was logging into the pods!
When I log into the nodepool, the blobfuse binary is available, and the mount looks right:
blobfuse on /var/lib/kubelet/pods/e89cf20c-d967-11e8-b7ee-5e8d71815678/volumes/azure~blobfuse/pv-blobfuse-flexvol type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072)

Here's /var/log/blobfuse-driver.log.
$ cat /var/log/blobfuse-driver.log
Fri Oct 26 21:31:08 UTC 2018 INFO: {"status": "Success", "capabilities": {"attach": false}}
Fri Oct 26 21:41:34 UTC 2018 INFO: tmp-path not specified, use default path: /tmp/blobfuse/
Fri Oct 26 21:41:34 UTC 2018 EXEC: mkdir -p /var/lib/kubelet/pods/e89cf20c-d967-11e8-b7ee-5e8d71815678/volumes/azure~blobfuse/pv-blobfuse-flexvol
Fri Oct 26 21:41:34 UTC 2018 INF: export storage account - export AZURE_STORAGE_ACCOUNT=nextcloudstorageaccount AZURE_STORAGE_ACCESS_KEY=...
Fri Oct 26 21:41:34 UTC 2018 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/e89cf20c-d967-11e8-b7ee-5e8d71815678/volumes/azure~blobfuse/pv-blobfuse-flexvol --container-name=nextcloud --tmp-path=/tmp/blobfuse/ -o allow_other
Fri Oct 26 21:41:34 UTC 2018 INFO: {"status": "Success" , "message":"log:Fri Oct 26 21:41:34 UTC 2018 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/e89cf20c-d967-11e8-b7ee-5e8d71815678/volumes/azure~blobfuse/pv-blobfuse-flexvol --container-name=nextcloud --tmp-path=/tmp/blobfuse/ -o allow_other " }

So it's getting to the nodepool OK. Why does the pod seem to use the pod's own root filesystem?

@ohthehugemanatee it has mounted correctly. You could write a file under /data and check that file under your storage account. The file system size info in docker volume mapping for blobfuse dir seems incorrect, anyway, it's a minor issue.

Thanks again. Confirmed that it's working.

@andyzhangx ,

Cluster's Location: westus2

We have 2 cluster with different scenarios

we recently upgraded the cluster from 1.17.X to 1.18.X
Pod can start, but the mounted volume is empty.

pod failed to start on 1.20.5 with the following error
Warning  FailedMount  105s (x13 over 12m)  kubelet            MountVolume.SetUp failed for volume 'test' : invalid character 'R' looking for beginning of value
Warning  FailedMount  55s (x5 over 10m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[test], unattached volumes=[test default-token-wqghg]: timed out waiting for the condition

We able to fix the issue by manually upgrading the blobfuse drivers in each individual nodes.
We would like to know why it is happen. the binaries should get upgrade right after upgrading the cluster.
Thank you in advance.


Don't recognize 5k-20k

Describe the bug
Don't recognize 5k-20k
To Reproduce
Steps to reproduce the behavior:

code

        Gson gson = new GsonBuilder().create();
        System.out.println("====A");
        String input = "5k-20k";
        System.out.println("input:" + input);

        System.out.println("====B");
        List<ModelResult> results = NumberRecognizer.recognizeNumber(input, Culture.Chinese);
        results.forEach(e -> {
            System.out.println(gson.toJson(e));
        });

        System.out.println("====C");
        results = NumberRecognizer.recognizeNumberRange(input, Culture.Chinese);
        results.forEach(e -> {
            System.out.println(gson.toJson(e));
        });

2.result
====A
input:5k-20k
====B
WARN: Look-behind groups with no maximum length not supported. Java version <= 8.
====C
{"text":"5k-20k","start":0,"end":6,"typeName":"numberrange","resolution":{}}
3.error

recognizeNumber don't recognize
recognizeNumberRange recognized,but resolution is null

Expected behavior

recognizeNumber:5000,20000
recognizeNumberRange:5000 - 20000

Platform (please complete the following information):

Platform:  Java
Environment: package & console
Version of package : master

debug info:
class:BaseNumberExtractor
method:extract(String source)
line:Match[] matches = RegExpUtility.getMatches(k, source);
k=((?<!(\d{1,10}\s{0,10})|[-－])[-－])?[\d１２３４５６７８９０]+\s*(K|k|M|G|T|B|b)
source=5k-20k
result:
5k
-20k
so modify code:
class:ChineseNumeric
filed:NegativeNumberTermsRegexNum
public static final String NegativeNumberTermsRegexNum = "((?<!(\\d+\\s*)|[-－])[-－])";
should be:
public static final String NegativeNumberTermsRegexNum = "((?<!(\\d+\\s*)|[-－]))";
test again :
I found the following entities (2):
{
"text" : "5k",
"start" : 0,
"end" : 2,
"typeName" : "number",
"resolution" : {
"value" : "5000"
}
}
{
"text" : "20k",
"start" : 3,
"end" : 6,
"typeName" : "number",
"resolution" : {
"value" : "20000"
}
}

another badcase:
====A
input:5k20k
====B
WARN: Look-behind groups with no maximum length not supported. Java version <= 8.
====C
debug info:
class:BaseNumberExtractor
method:extract(String source)
line:Match[] matches = RegExpUtility.getMatches(k, source);
result:
5
20
5k
20k
But maybe something is wrong, in BaseNumberExtractor.extract(String source)。
The filter is pass every matched result:
Optional srcMatches = matchSource.keySet().stream().filter(o -> o.index == finalStart && o.length == finalLength).findFirst();

@tellarin, can this issue be closed? Or additional work is required?

@LionbridgeCSII this issue was originally reported in Java, but the issue is fixed only on .NET, so I don't think it can be closed. But Java is outside your scope.

Closing now as we've validated it works in Java too.


[#186] Display channel diffs

Fixes #186

@TomasHofman if you've got a moment could you take a look?

Example output:
[thofman@rama prospero]$ ./prospero history --dir wfhist --revision 390de405
Updates:
  [Updated] org.jboss.universe.producer:wildfly-producers:		1.3.2.Final ==> 1.3.1.Final
  [Updated] com.carrotsearch:hppc:		0.8.1 ==> 0.8.0

Configuration changes:
  [Added] wfcore:
    manifest:		[] ==> file:/home/thofman/tmp/wildfly-core-manifest.yaml
    repositories:
      [] ==> temp-repo-0::https://repo1.maven.org/maven2/

What about adding nouns to explain what the reported items are? E.g. instead of [Added] / [Updated] there could be  [Updated artifact] or [Added channel]. I think it may not be clear what the configuration change represent currently.

What about adding nouns to explain what the reported items are? E.g. instead of [Added] / [Updated] there could be [Updated artifact] or [Added channel]. I think it may not be clear what the configuration change represent currently.

I like that 👍  let me see if I can add it

Observation, if I modify the installer-config.yaml, I cause an un-versioned change (prospero history --revision <latest hash> would report my change). This change will remain un-versioned until I do some CLI operation that modifies the config too (like adding / removing a channel).
Should there be a CLI command that would version my manual changes?

Should there be a CLI command that would version my manual changes?

hm I'm not sure. At the moment the idea is that installation-config.yaml changes should only be done via the CLI/API operations.


How to use Dinov2 for self-supervised training?

Hello, I would like to ask, how to use Dinov2 for self-supervised learning of unlabeled images, get pre-trained weights, and then load the obtained pre-trained weights for supervised learning with labeled data (for image classification)?

Hi team,
Thanks @zhanglaoban-kk brought this up, I have the same problem.

For using unlabeled images

Is that like what #142 @TheoMoutakanni mentioned that the label in training/eval/testing are all just for evaluation, and all we need just set the label as something like '0' or 'np.nan', then mocking the ImageNet dataset?

For loading pre-trained weights, I would like to use some unlabeled data to do post-pretrain based on the vanilla DINOv2 checkpoint, I'm using dinov2-vit-b14 checkpoint, and load in dinov2/configs/ssl_default_config.yaml line 2

MODEL:   WEIGHTS: 'https://dl.fbaipublicfiles.com/dinov2/dinov2_vitb14/dinov2_vitb14_pretrain.pth'
However, it reports error
  ... File "/hpcfs/users/a1781032/default_studio/dinov2/dinov2/train/train.py", line 153, in do_train     start_iter = checkpointer.resume_or_load(cfg.MODEL.WEIGHTS, resume=resume).get("iteration", -1) + 1 ...     checkpoint_state_dict = checkpoint.pop("model") KeyError: 'model'
which the dinov2_vitb14_pretrain.pth does not have key model but the architecture keys: 'cls_token', 'pos_embed', 'mask_token', 'patch_embed.proj.weight', 'patch_embed.proj.bias', 'blocks.0.norm1.weight', 'blocks.0.norm1.bias', 'blocks.0.attn.qkv.weight', 'blocks.0.attn.qkv.bias', 'blocks.0.attn.proj.weight', 'blocks.0.attn.proj.bias' ...
How can we do to load the provided pre-train checkpoint into the model before training?
Best,
Steve

Hi team,
I had some new findings. After I looked through #154 @surajyakoa explained that he/she wanted to fine-tune DINOv2 with some unlabeled data and @qasfb has explained that it requires the full checkpoint not just the backbone, and since they hadn't release them so it's not practical.
Is that correct?
Steve

Hi team,
I had some new findings. After I looked through #154 @surajyakoa explained that he/she wanted to fine-tune DINOv2 with some unlabeled data and @qasfb has explained that it requires the full checkpoint not just the backbone, and since they hadn't release them so it's not practical.
Is that correct?
Steve

Sad moment, so it is not posible to fine-tune current model from hub?

Any news here?

Any news here?

Nope for now, someone that know how to make self-distillation using the train file in the repo ?

Hi,
My advice would be to not load the weights using the checkpointer resume function, as it also expects the optimizer buffer etc. However you can launch a normal training from scratch, and load these pretrained weights before the first iteration with torch.load and model.load_state_dict. Make sure you both load the weights to the student and the teacher.
You may need to slightly rename the checkpoint keys: blocks.0.attn.qkv.weight may be named something like module.backbone.blocks.0.0.attn.qkv.weight or whatever. I don't remember the exact mapping, but it should not be hard to find if you compare the keys on both sides
You might also need to use strict=False, because the checkpoint does not include the head weights. However, make sure you print out the load message, so that you know if the loading failed or not. That is a common source of bugs.
The only real question is: how well will the training work with a pretrained backbone and a random init head. I think it may be beneficial to freeze the backbone and train only the heads for a few thousand iterations, so it has a bit of time to fit. Then unfreeze the whole and start the real training.
Best of luck,
Tim

Hi team,
Thanks @zhanglaoban-kk brought this up, I have the same problem.

For using unlabeled images

Is that like what #142 @TheoMoutakanni mentioned that the label in training/eval/testing are all just for evaluation, and all we need just set the label as something like '0' or 'np.nan', then mocking the ImageNet dataset?

For loading pre-trained weights, I would like to use some unlabeled data to do post-pretrain based on the vanilla DINOv2 checkpoint, I'm using dinov2-vit-b14 checkpoint, and load in dinov2/configs/ssl_default_config.yaml line 2

MODEL: WEIGHTS: 'https://dl.fbaipublicfiles.com/dinov2/dinov2_vitb14/dinov2_vitb14_pretrain.pth'
However, it reports error  ... File "/hpcfs/users/a1781032/default_studio/dinov2/dinov2/train/train.py", line 153, in do_train start_iter = checkpointer.resume_or_load(cfg.MODEL.WEIGHTS, resume=resume).get("iteration", -1) + 1 ... checkpoint_state_dict = checkpoint.pop("model") KeyError: 'model'
which the dinov2_vitb14_pretrain.pth does not have key model but the architecture keys: 'cls_token', 'pos_embed', 'mask_token', 'patch_embed.proj.weight', 'patch_embed.proj.bias', 'blocks.0.norm1.weight', 'blocks.0.norm1.bias', 'blocks.0.attn.qkv.weight', 'blocks.0.attn.qkv.bias', 'blocks.0.attn.proj.weight', 'blocks.0.attn.proj.bias' ...
How can we do to load the provided pre-train checkpoint into the model before training?
Best, Steve

Hi Steve,
I have the same require as yours. I also want to train dinov2 using unlabeled images.
Does it work on [imagenet] mocked dataset ?
Could you offer some detail to me?
Best, Luo


Allow domains in liveReloadBaseUrl

#28 breaks live-reload injection because it does not make the difference between baseUrl/rootUrl and liveReloadBaseUrl.
It happens in var baseURL = options.liveReloadBaseUrl || options.rootURL || options.baseURL;.
Then app.use(baseURL + 'ember-cli-live-reload.js', ...  does not work anymore since app.use won't accept an url with domain.

@jgoclawski - Can you review?

@rwjblue Is it just me or is this still a problem? Can't do live reload cross domain because EMBER_CLI_INJECT_LIVE_RELOAD_BASEURL needs to include domain:port in contentFor and can't include domain in serverMiddleware.
I used to rewrite the live-reload.js src on the server in dev, but now I have two ember apps on different ports and can't really do that anymore. Happy to help if this pr isn't 🎯

It looks like #36 somewhat overlaps with this branch, but fixes it more generally. Does that PR supersede this one?

@aprescott Indeed, it does. I'm closing this one in favor of #38


helm-charts/ray-cluster: Allow multiple workers

Why are these changes needed?
We can not set additional workers with the current helm chart.
There are some PRs that aim to fix this but they appear to be stale https://github.com/ray-project/kuberay/pull/266

We wanted to keep untouched the default worker that is already included in the helm chart but allow the addition of new workers.
The changes are adding the following logic:

Allow to disable the default worker group, if needed, via worker.disabled = true
Allow to set additional workers by using the extraWorkers map

The extraWorkers map has a different structure compared to the default worker because I wanted to add additional flexibility. Due to that, some variables in the extraWorkers have different naming (which matches the name of the related CRD attributed)
Note: PR was done to support additional workers in the infrastructure of Metrika.co

Closing this, will open anew from master


Set Password white sending a Create user Request via the Provisioning API in ENTRA ID

Hi Team,
I am trying to set a random password while creating new account in Entra ID for users and send them the credentials via email.
The HTTP call says success and the account gets created but the Password does not work. It says that the username or email is incorrect.  Can you please help me in setting a password via the API-Driven Inbound User Provisioning?

@ShinraTenseiii
It would be great if you could add a link to the documentation you are following for these steps? This would help us redirect the issue to the appropriate team. Thanks!!

@ShinraTenseiii
Since we have not heard back from you, we will now proceed to close this thread.If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


Is there any way to export data with charts to excel files?

Like inserting a distribution radar map in the end of the file.

This library does not allow that.
For more complex usage you would need another Excel library.

👌 Thanks!


transport: add support for not primitive types

Add ability to bind and scan:

List
Map
Set
UDT
Tuple

Related to #152


Updated API functionality

Hello lox,
I implemented some of the new methods available in the newer version of the Localbitcoins API.

How about adding invoicing support?


add support for scram-sha-256 with channel binding

Setting password_encryption to scram-sha-256 in postgresql.conf will
encrypt the password with SCRAM-SHA-256. The default is md5, which will
encrypt the password with MD5.
If the password was encrypted with SCRAM-SHA-256 in postgres (by
setting password_enc56 in postgresql.conf), then postgres clients can
authenticate to the postgres server by setting the ff the password was
encrypted with SCRAM-SHA-256 in postgres (by setting
password_encryption to scram-sha-256 in postgresql.conf), then postgres
cl ients can authenticate to the postgres server by setting the
authentication method specification to scram-sha-256 in pg_hba.conf.
The problem is that the pgmoon client doesn't support the SCRAM-SHA-256
authentication method. So the postgres server supports SCRAM-SHA-256,
but the client doesn't. If the password is encrypted with SCRAM-SHA-256
(by setting password_encryption to scram-sha-256 in postgresql.conf)
and if the postgres method authentication is set to SCRAM-SHA-256 (by
setting the authentication method specification to scram-sha-256 in
pg_hba.conf) then pgmoon won't be able to authenticate to postgres
because the password will be sent as SCRAM-SHA-256 encrypted and not
MD5 or plaintext (the ones supported by pgmoon).
Here we are implementing the client-side support for SCRAM-SHA-256 as
defined by [3]. Per the postgres docs [2] only SASL is supported as per
the RFC 7677 [3]. So the client implementation should be conformant to
that RFC only, not for the HTTP version of SCRAM-SHA-256 (RFC 7804).
Implementation details:

Support for channel binding is implemented as per RFC 5929 [4]

Only 'tls-server-end-point' is implemented because it's the only
channel binding type used by postgres
Requires a patch to the LuaSec project for the socket type
luasocket [5]
Requires a patch to the lua-resty-openssl project for the socket
type nginx [6]

There is no support for authzid since postgres doesn't support it [1]
Right now we are relying on luaossl to provide all the crypto things

It would be nice if in the future we create a compatibility layer to use both the luaossl and resty-openssl libraries

[1] https://github.com/postgres/postgres/blob/REL_13_1/src/backend/libpq/auth-scram.c#L1011-L1013
[2] https://www.postgresql.org/docs/13/auth-password.html
[3] https://www.rfcreader.com/#rfc7677
[4] https://www.rfcreader.com/#rfc5929
[5] Implement getsignaturename in LuaSec
[6] This branch: https://github.com/fffonion/lua-resty-openssl/tree/ssl

Failing tests reason: the feat/scram-sha-256 branch exists on top of feat/mtls. This latter branch has the same failing tests on it, so we are unfortunately inheriting failing tests from feat/mtls.
You can see on my fork that the tests are passing with this PR's commit: https://github.com/murillopaula/pgmoon

Failing tests reason: the feat/scram-sha-256 branch exists on top of feat/mtls. This latter branch has the same failing tests on it, so we are unfortunately inheriting failing tests from feat/mtls.
You can see on my fork that the tests are passing with this PR's commit: https://github.com/murillopaula/pgmoon


Ruby version at http://ruby.thoughtbot.com/latest

Hey guys,
http://ruby.thoughtbot.com/latest returns 2.2.2.
Suspenders requires 2.2.3
https://github.com/thoughtbot/suspenders/blob/master/.ruby-version
Have you forgotten to update this value on your server? If laptop is run and then Suspenders updated, it fails as laptop sets ruby to 2.2.2...

@rikkipitt it is up to date now.

Awesome, thanks @thiagogabriel!

@rikkipitt thanks @croaky :+1:

Sorry for the delay!


Kne link flap

feature/gribi/otg_tests/ipv4_entry_test/ipv4_entry_test.go has been changed. Due to addition of the feature kne link flap this test needs to be changed. Main chaange is there should not be any set config/push config call after link flap has been triggered.

Pull Request Functional Test Report for #2542 / eb5a3a6564613753605cf1eb81629b31ff50ce65
Virtual Devices

Device
Test
Test Documentation
Job
Raw Log

Arista cEOS

TE-2.1: gRIBI IPv4 Entry

Cisco 8000E

TE-2.1: gRIBI IPv4 Entry

Cisco XRd

TE-2.1: gRIBI IPv4 Entry

Juniper cPTX

TE-2.1: gRIBI IPv4 Entry

Juniper ncPTX

TE-2.1: gRIBI IPv4 Entry

Nokia SR Linux

TE-2.1: gRIBI IPv4 Entry

Openconfig Lemming

TE-2.1: gRIBI IPv4 Entry

Hardware Devices

Device
Test
Test Documentation
Raw Log

Arista 7808

TE-2.1: gRIBI IPv4 Entry

Cisco 8808

TE-2.1: gRIBI IPv4 Entry

Juniper PTX10008

TE-2.1: gRIBI IPv4 Entry

Nokia 7250 IXR-10e

TE-2.1: gRIBI IPv4 Entry

Help

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Warning: This coverage report may be inaccurate.
We've detected an issue with your CI configuration that might affect the accuracy of this pull request's coverage report.
To ensure accuracy in future PRs, please see these guidelines.
A quick fix for this PR: rebase it; your next report should be accurate.

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 52.141%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1668

Relevant Lines:
3199

💛  - Coveralls

Closed in favor of https://github.com/openconfig/featureprofiles/pull/2573


building from source failed

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS7
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): source
TensorFlow version: latest master from github
Python version: 3.6.8
Installed using virtualenv? pip? conda?: virtualenv
Bazel version (if compiling from source): 0.25.3
GCC/Compiler version (if compiling from source): gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC)
CUDA/cuDNN version: CUDA 10/ cuDNN 7
GPU model and memory:V100 32G

Describe the problem
when I run: bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package, I get error:
WARNING: The following configs were expanded more than once: [cuda, using_cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=129
INFO: Reading rc options for 'build' from /home/minds/isaac/tensorflow/.bazelrc:
  'build' options: --apple_platform_type=macos --define framework_shared_object=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone --strategy=Genrule=standalone -c opt --announce_rc --define=grpc_no_ares=true --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include
INFO: Reading rc options for 'build' from /home/minds/isaac/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/home/minds/.virtualenvs/venv_tf/bin/python --action_env PYTHON_LIB_PATH=/home/minds/.virtualenvs/venv_tf/lib/python3.6/site-packages --python_path=/home/minds/.virtualenvs/venv_tf/bin/python --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda --action_env TF_CUDA_COMPUTE_CAPABILITIES=7.0,7.0 --action_env GCC_HOST_COMPILER_PATH=/bin/gcc --config=cuda --action_env TF_CONFIGURE_IOS=0
INFO: Found applicable config definition build:cuda in file /home/minds/isaac/tensorflow/.bazelrc: --config=using_cuda --define=using_cuda_nvcc=true
INFO: Found applicable config definition build:using_cuda in file /home/minds/isaac/tensorflow/.bazelrc: --define=using_cuda=true --action_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain
INFO: Found applicable config definition build:opt in file /home/minds/isaac/tensorflow/.tf_configure.bazelrc: --copt=-march=native --copt=-Wno-sign-compare --host_copt=-march=native --define with_default_optimizations=true
INFO: Found applicable config definition build:cuda in file /home/minds/isaac/tensorflow/.bazelrc: --config=using_cuda --define=using_cuda_nvcc=true
INFO: Found applicable config definition build:using_cuda in file /home/minds/isaac/tensorflow/.bazelrc: --define=using_cuda=true --action_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain
INFO: An error occurred during the fetch of repository 'io_bazel_rules_docker'
INFO: Call stack for the definition of repository 'io_bazel_rules_docker':
 - /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/bazel_toolchains/repositories/repositories.bzl:37:9
 - /home/minds/isaac/tensorflow/WORKSPACE:29:1
ERROR: error loading package '': Encountered error while reading extension file 'repositories/repositories.bzl': no such package '@io_bazel_rules_docker//repositories': Traceback (most recent call last):
	File "/home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/bazel_tools/tools/build_defs/repo/git.bzl", line 234
		_clone_or_update(ctx)
	File "/home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/bazel_tools/tools/build_defs/repo/git.bzl", line 74, in _clone_or_update
		fail(("error cloning %s:\n%s" % (ctx....)))
error cloning io_bazel_rules_docker:
+ cd /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external
+ rm -rf /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker
+ git clone '' https://github.com/bazelbuild/rules_docker.git /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker
Too many arguments.

usage: git clone [options] [--] <repo> [<dir>]

    -v, --verbose         be more verbose
    -q, --quiet           be more quiet
    --progress            force progress reporting
    -n, --no-checkout     don't create a checkout
    --bare                create a bare repository
    --mirror              create a mirror repository (implies bare)
    -l, --local           to clone from a local repository
    --no-hardlinks        don't use local hardlinks, always copy
    -s, --shared          setup as shared repository
    --recursive           initialize submodules in the clone
    --recurse-submodules  initialize submodules in the clone
    --template <template-directory>
                          directory from which templates will be used
    --reference <repo>    reference repository
    -o, --origin <name>   use <name> instead of 'origin' to track upstream
    -b, --branch <branch>
                          checkout <branch> instead of the remote's HEAD
    -u, --upload-pack <path>
                          path to git-upload-pack on the remote
    --depth <depth>       create a shallow clone of that depth
    --single-branch       clone only one branch, HEAD or --branch
    --separate-git-dir <gitdir>
                          separate git dir from working tree
    -c, --config <key=value>
                          set config inside the new repository

+ git clone https://github.com/bazelbuild/rules_docker.git /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker
+ git -C /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker reset --hard 251f6a68b439744094faff800cd029798edf9faa
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
+ git -C /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker fetch '' origin 251f6a68b439744094faff800cd029798edf9faa:251f6a68b439744094faff800cd029798edf9faa
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
+ git -C /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker fetch origin 251f6a68b439744094faff800cd029798edf9faa:251f6a68b439744094faff800cd029798edf9faa
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
ERROR: error loading package '': Encountered error while reading extension file 'repositories/repositories.bzl': no such package '@io_bazel_rules_docker//repositories': Traceback (most recent call last):
	File "/home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/bazel_tools/tools/build_defs/repo/git.bzl", line 234
		_clone_or_update(ctx)
	File "/home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/bazel_tools/tools/build_defs/repo/git.bzl", line 74, in _clone_or_update
		fail(("error cloning %s:\n%s" % (ctx....)))
error cloning io_bazel_rules_docker:
+ cd /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external
+ rm -rf /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker
+ git clone '' https://github.com/bazelbuild/rules_docker.git /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker
Too many arguments.

usage: git clone [options] [--] <repo> [<dir>]

    -v, --verbose         be more verbose
    -q, --quiet           be more quiet
    --progress            force progress reporting
    -n, --no-checkout     don't create a checkout
    --bare                create a bare repository
    --mirror              create a mirror repository (implies bare)
    -l, --local           to clone from a local repository
    --no-hardlinks        don't use local hardlinks, always copy
    -s, --shared          setup as shared repository
    --recursive           initialize submodules in the clone
    --recurse-submodules  initialize submodules in the clone
    --template <template-directory>
                          directory from which templates will be used
    --reference <repo>    reference repository
    -o, --origin <name>   use <name> instead of 'origin' to track upstream
    -b, --branch <branch>
                          checkout <branch> instead of the remote's HEAD
    -u, --upload-pack <path>
                          path to git-upload-pack on the remote
    --depth <depth>       create a shallow clone of that depth
    --single-branch       clone only one branch, HEAD or --branch
    --separate-git-dir <gitdir>
                          separate git dir from working tree
    -c, --config <key=value>
                          set config inside the new repository

+ git clone https://github.com/bazelbuild/rules_docker.git /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker
+ git -C /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker reset --hard 251f6a68b439744094faff800cd029798edf9faa
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
+ git -C /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker fetch '' origin 251f6a68b439744094faff800cd029798edf9faa:251f6a68b439744094faff800cd029798edf9faa
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
+ git -C /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/io_bazel_rules_docker fetch origin 251f6a68b439744094faff800cd029798edf9faa:251f6a68b439744094faff800cd029798edf9faa
Unknown option: -C
usage: git [--version] [--help] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]
INFO: Elapsed time: 7.123s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

Provide the exact sequence of commands / steps that you executed before running into the problem
followed this guide, until I faced the issue
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

What is the output of git --version?

git version <IP_ADDRESS>

I understand. Could you help me out?

Can't reproduce. Can you try recloning the repo on a different directory and recompiling from scratch?

I have tried building again and failed with a different error:
ERROR: /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/protobuf_archive/BUILD:91:1: undeclared inclusion(s) in rule '@protobuf_archive//:protobuf_lite':
this rule is missing dependency declarations for the following files included by 'external/protobuf_archive/src/google/protobuf/stubs/statusor.cc':
  '/include/c++/4.8.5/new'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/c++config.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/os_defines.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/cpu_defines.h'
  '/include/c++/4.8.5/exception'
  '/include/c++/4.8.5/bits/atomic_lockfree_defines.h'
  '/include/c++/4.8.5/bits/exception_ptr.h'
  '/include/c++/4.8.5/bits/exception_defines.h'
  '/include/c++/4.8.5/bits/nested_exception.h'
  '/include/c++/4.8.5/string'
  '/include/c++/4.8.5/bits/stringfwd.h'
  '/include/c++/4.8.5/bits/memoryfwd.h'
  '/include/c++/4.8.5/bits/char_traits.h'
  '/include/c++/4.8.5/bits/stl_algobase.h'
  '/include/c++/4.8.5/bits/functexcept.h'
  '/include/c++/4.8.5/bits/cpp_type_traits.h'
  '/include/c++/4.8.5/ext/type_traits.h'
  '/include/c++/4.8.5/ext/numeric_traits.h'
  '/include/c++/4.8.5/bits/stl_pair.h'
  '/include/c++/4.8.5/bits/move.h'
  '/include/c++/4.8.5/bits/concept_check.h'
  '/include/c++/4.8.5/type_traits'
  '/include/c++/4.8.5/bits/stl_iterator_base_types.h'
  '/include/c++/4.8.5/bits/stl_iterator_base_funcs.h'
  '/include/c++/4.8.5/debug/debug.h'
  '/include/c++/4.8.5/bits/stl_iterator.h'
  '/include/c++/4.8.5/bits/postypes.h'
  '/include/c++/4.8.5/cwchar'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/stdarg.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/stddef.h'
  '/include/c++/4.8.5/cstdint'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/stdint.h'
  '/include/c++/4.8.5/bits/allocator.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/c++allocator.h'
  '/include/c++/4.8.5/ext/new_allocator.h'
  '/include/c++/4.8.5/bits/localefwd.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/c++locale.h'
  '/include/c++/4.8.5/clocale'
  '/include/c++/4.8.5/iosfwd'
  '/include/c++/4.8.5/cctype'
  '/include/c++/4.8.5/bits/ostream_insert.h'
  '/include/c++/4.8.5/bits/cxxabi_forced.h'
  '/include/c++/4.8.5/bits/stl_function.h'
  '/include/c++/4.8.5/backward/binders.h'
  '/include/c++/4.8.5/bits/range_access.h'
  '/include/c++/4.8.5/bits/basic_string.h'
  '/include/c++/4.8.5/ext/atomicity.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/gthr.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/gthr-default.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/atomic_word.h'
  '/include/c++/4.8.5/initializer_list'
  '/include/c++/4.8.5/ext/string_conversions.h'
  '/include/c++/4.8.5/cstdlib'
  '/include/c++/4.8.5/cstdio'
  '/include/c++/4.8.5/cerrno'
  '/include/c++/4.8.5/bits/functional_hash.h'
  '/include/c++/4.8.5/bits/hash_bytes.h'
  '/include/c++/4.8.5/bits/basic_string.tcc'
  '/include/c++/4.8.5/utility'
  '/include/c++/4.8.5/bits/stl_relops.h'
  '/include/c++/4.8.5/algorithm'
  '/include/c++/4.8.5/bits/stl_algo.h'
  '/include/c++/4.8.5/bits/algorithmfwd.h'
  '/include/c++/4.8.5/bits/stl_heap.h'
  '/include/c++/4.8.5/bits/stl_tempbuf.h'
  '/include/c++/4.8.5/bits/stl_construct.h'
  '/include/c++/4.8.5/ext/alloc_traits.h'
  '/include/c++/4.8.5/bits/alloc_traits.h'
  '/include/c++/4.8.5/bits/ptr_traits.h'
  '/include/c++/4.8.5/random'
  '/include/c++/4.8.5/cmath'
  '/include/c++/4.8.5/limits'
  '/include/c++/4.8.5/bits/random.h'
  '/include/c++/4.8.5/vector'
  '/include/c++/4.8.5/bits/stl_uninitialized.h'
  '/include/c++/4.8.5/bits/stl_vector.h'
  '/include/c++/4.8.5/bits/stl_bvector.h'
  '/include/c++/4.8.5/bits/vector.tcc'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/opt_random.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/x86intrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/ia32intrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/mmintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/xmmintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/mm_malloc.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/emmintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/pmmintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/tmmintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/smmintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/popcntintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/wmmintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/immintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/avxintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/avx2intrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/lzcntintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/bmiintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/bmi2intrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/fmaintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/f16cintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/rtmintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/xtestintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/rdseedintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/prfchwintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/fxsrintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/xsaveintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/xsaveoptintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/adxintrin.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/pkuintrin.h'
  '/include/c++/4.8.5/bits/random.tcc'
  '/include/c++/4.8.5/numeric'
  '/include/c++/4.8.5/bits/stl_numeric.h'
  '/include/c++/4.8.5/functional'
  '/include/c++/4.8.5/typeinfo'
  '/include/c++/4.8.5/tuple'
  '/include/c++/4.8.5/array'
  '/include/c++/4.8.5/stdexcept'
  '/include/c++/4.8.5/bits/uses_allocator.h'
  '/include/c++/4.8.5/iostream'
  '/include/c++/4.8.5/ostream'
  '/include/c++/4.8.5/ios'
  '/include/c++/4.8.5/bits/ios_base.h'
  '/include/c++/4.8.5/bits/locale_classes.h'
  '/include/c++/4.8.5/bits/locale_classes.tcc'
  '/include/c++/4.8.5/streambuf'
  '/include/c++/4.8.5/bits/streambuf.tcc'
  '/include/c++/4.8.5/bits/basic_ios.h'
  '/include/c++/4.8.5/bits/locale_facets.h'
  '/include/c++/4.8.5/cwctype'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/ctype_base.h'
  '/include/c++/4.8.5/bits/streambuf_iterator.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/ctype_inline.h'
  '/include/c++/4.8.5/bits/locale_facets.tcc'
  '/include/c++/4.8.5/bits/basic_ios.tcc'
  '/include/c++/4.8.5/bits/ostream.tcc'
  '/include/c++/4.8.5/istream'
  '/include/c++/4.8.5/bits/istream.tcc'
  '/include/c++/4.8.5/map'
  '/include/c++/4.8.5/bits/stl_tree.h'
  '/include/c++/4.8.5/bits/stl_map.h'
  '/include/c++/4.8.5/bits/stl_multimap.h'
  '/include/c++/4.8.5/memory'
  '/include/c++/4.8.5/bits/stl_raw_storage_iter.h'
  '/include/c++/4.8.5/ext/concurrence.h'
  '/include/c++/4.8.5/bits/unique_ptr.h'
  '/include/c++/4.8.5/bits/shared_ptr.h'
  '/include/c++/4.8.5/bits/shared_ptr_base.h'
  '/include/c++/4.8.5/backward/auto_ptr.h'
  '/include/c++/4.8.5/set'
  '/include/c++/4.8.5/bits/stl_set.h'
  '/include/c++/4.8.5/bits/stl_multiset.h'
  '/include/c++/4.8.5/cstddef'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/limits.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/syslimits.h'
  '/include/c++/4.8.5/unordered_map'
  '/include/c++/4.8.5/bits/hashtable.h'
  '/include/c++/4.8.5/bits/hashtable_policy.h'
  '/include/c++/4.8.5/bits/unordered_map.h'
  '/include/c++/4.8.5/unordered_set'
  '/include/c++/4.8.5/bits/unordered_set.h'
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 17.327s, Critical Path: 1.23s
INFO: 4 processes: 4 local.
FAILED: Build did NOT complete successfully

It seems that your toolchain is incomplete

I will suggest you to use a Docker container to build TF.

I have used docker but that failed as well.

I have seen this problem when the compiler changes after running ./configure.
Could you run bazel clean --expunge, then retry building?

@gunan
I have done so:
bazel clean --expunge
./configure
bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

but it failed again. I wonder if it is because I am using default compiler -- gcc -- as my nvcc compiler.
Error message
ERROR: /home/minds/.cache/bazel/_bazel_minds/7c1f2bd7870d1ecc8754a91ebff0746c/external/com_googlesource_code_re2/BUILD:26:1: undeclared inclusion(s) in rule '@com_googlesource_code_re2//:re2':
this rule is missing dependency declarations for the following files included by 'external/com_googlesource_code_re2/re2/unicode_groups.cc':
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/stdint.h'
  '/include/c++/4.8.5/string'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/c++config.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/os_defines.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/cpu_defines.h'
  '/include/c++/4.8.5/bits/stringfwd.h'
  '/include/c++/4.8.5/bits/memoryfwd.h'
  '/include/c++/4.8.5/bits/char_traits.h'
  '/include/c++/4.8.5/bits/stl_algobase.h'
  '/include/c++/4.8.5/bits/functexcept.h'
  '/include/c++/4.8.5/bits/exception_defines.h'
  '/include/c++/4.8.5/bits/cpp_type_traits.h'
  '/include/c++/4.8.5/ext/type_traits.h'
  '/include/c++/4.8.5/ext/numeric_traits.h'
  '/include/c++/4.8.5/bits/stl_pair.h'
  '/include/c++/4.8.5/bits/move.h'
  '/include/c++/4.8.5/bits/concept_check.h'
  '/include/c++/4.8.5/type_traits'
  '/include/c++/4.8.5/bits/stl_iterator_base_types.h'
  '/include/c++/4.8.5/bits/stl_iterator_base_funcs.h'
  '/include/c++/4.8.5/debug/debug.h'
  '/include/c++/4.8.5/bits/stl_iterator.h'
  '/include/c++/4.8.5/bits/postypes.h'
  '/include/c++/4.8.5/cwchar'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/stdarg.h'
  '/lib/gcc/x86_64-redhat-linux/4.8.5/include/stddef.h'
  '/include/c++/4.8.5/cstdint'
  '/include/c++/4.8.5/bits/allocator.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/c++allocator.h'
  '/include/c++/4.8.5/ext/new_allocator.h'
  '/include/c++/4.8.5/new'
  '/include/c++/4.8.5/exception'
  '/include/c++/4.8.5/bits/atomic_lockfree_defines.h'
  '/include/c++/4.8.5/bits/exception_ptr.h'
  '/include/c++/4.8.5/bits/nested_exception.h'
  '/include/c++/4.8.5/bits/localefwd.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/c++locale.h'
  '/include/c++/4.8.5/clocale'
  '/include/c++/4.8.5/iosfwd'
  '/include/c++/4.8.5/cctype'
  '/include/c++/4.8.5/bits/ostream_insert.h'
  '/include/c++/4.8.5/bits/cxxabi_forced.h'
  '/include/c++/4.8.5/bits/stl_function.h'
  '/include/c++/4.8.5/backward/binders.h'
  '/include/c++/4.8.5/bits/range_access.h'
  '/include/c++/4.8.5/bits/basic_string.h'
  '/include/c++/4.8.5/ext/atomicity.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/gthr.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/gthr-default.h'
  '/include/c++/4.8.5/x86_64-redhat-linux/bits/atomic_word.h'
  '/include/c++/4.8.5/initializer_list'
  '/include/c++/4.8.5/ext/string_conversions.h'
  '/include/c++/4.8.5/cstdlib'
  '/include/c++/4.8.5/cstdio'
  '/include/c++/4.8.5/cerrno'
  '/include/c++/4.8.5/bits/functional_hash.h'
  '/include/c++/4.8.5/bits/hash_bytes.h'
  '/include/c++/4.8.5/bits/basic_string.tcc'
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 26.030s, Critical Path: 2.50s
INFO: 15 processes: 15 local.
FAILED: Build did NOT complete successfully

@meteorcloudy @hlopko in case they have any ideas.
@av8ramit in case he ran into similar issues on centos.

My issue was slightly different, it couldn't find the linker so I needed to manually provide the crosstool.

I am waiting to make a PR but but I cant test out my code. It would be really great if we I can find a way to build from source or using docker.
Thanks again!

Are you by any chance on a 32 bit system?

No

Cannot reproduce at all, sorry

As far as I can see, this is a purely bazel issue, where bazel has a problem autodetecting the toolchain on the system.
However, I have no direct solution for it.
Please try reaching out to bazel team, or the sommunity on stackoverflow may have some ideas.

Closing the issue, as this is a bazel toolchain autodetection problem, on an operating system not supported by us.
You can reach out to bazel team, or try getting help through stackoverflow.


How are you? Hope everyone in Ukraine is good

Hello,
My all good wishes with you and your family . I am happy to see you contributing your repos.Every Ukrainian is suffering the unjustified war on them.

Thank you, my family and I are fine so far.


Allow extra fields in the config

Right now we cannot add extra fields to the config without it not being backwards compatible.  Should follow the rules of protobuf here where you can add new fields but they must be optional and Astacus should be able to cope with these fields missing.
This will be required for the SQLite work where a new field on the node config is being added.
Also since fastapi is available we can use jsonable_encoder to make jsondict a bit less awkward.

Okay I think in the config allow_extra might make sense if we want to prevent typos.
But right now with genbumps we run multiple Astacus versions in the same cluster and communication is happening with Pydantic models that do not allow additional fields.  This means we can never add a field to the IPC models.
It is impossible for the following 3 to hold simultaneously.

Allow for smooth genbump where multiple versions are running in the same cluster.
Do not allow for additional fields when parsing the IPC requests.
Allow the protocol to ever evolve.

So I think that at least the IPC models (and probably any models in the coordinator API although I don't think there are any right now) we should reconsider this design choice.  There is prior art here in gRPC and other RPC frameworks allowing additional fields to be present.
I will put this in a different PR so since it is no longer about the config.

There is a system of Features between nodes: nodes can declare which features they support and coordinators can send a different IPC message if the related feature is supported.
It's a bit more tedious than ignoring optional fields, on the other hand this allows the coordinator to be more intentional when sending a message to older nodes: the non-optional values can be different if that helps, and the responses can also be handled differently.
(So far we've used features twice, we never did more than send a message with more parameters, but we can :slightly_smiling_face:)


Updating database when a product is marked favourite

I want to work on this issue. I am a Gssoc23 contributor, Please assign me!