AzCopy sync failes to write hash data when using a readonly VSS shadow copy as the source

Which version of the AzCopy was used?
10.23.0
Which platform are you using?
Windows
What command did you run?
sync #SourcePath# #BlobStoragePath# --recursive=true --exclude-pattern=*humbs.db --exclude-attributes=S;H --put-md5 --delete-destination=true --compare-hash=md5 --cap-mbps=900
What problem was encountered?
Seeing this in the output:
INFO: One or more hash storage operations (read/write) have failed. Check the scanning log for details.
Seeing this in the logs:
failed to write hash data for #RelativePath#/#Filename#: failed to open data stream: open \?\C:\Backup\SymbolicLink#RelativePath##Filename#:.azcopysyncmeta: The media is write protected.
How can we reproduce the problem in the simplest way?

Create a VSS snapshot of a volume using diskshadow.exe
Create a Symbolic Link to the shadow copy
cmd /c mklink /d $SymbolicLinkPath "$ShadowDevicePath"
Run AzCopy sync using the Symbolic Link as the source

Have you found a mitigation/solution?
No
The sync works but I am not clear if this will cause issues long term. It seems AzCopy saves the hash of every file as an Alternate Data Stream on the source files however this is now failing because the shadow copy is read only.
I ran the sync command directly on the source files before I started using VSS snapshots. Will this now cause problems because the read only snapshot copy has an out of date hash that can no longer be updated? I am seeing a lot of these messages:
File #FilePath# was skipped because the source has the same hash

(also, please re-open this issue if this doesn't entirely solve your issue" ^)

I am trying this but it is failing with the following error (for every file):
failed to write hash data for Folder1/Filename.doc: failed to open hash meta file: open T:\AzCopyMetdata\ContainerA\Folder1.Filename.doc.azcopysyncmeta: The system cannot find the path specified.
The arguments I passed were the following:
--compare-hash=md5 --local-hash-storage-mode=HiddenFiles --hash-meta-dir=T:\AzCopyMetdata\ContainerA
To confirm, T:\AzCopyMetdata\ContainerA is an empty directory. Should AzCopy create all the directories and files in here automatically?

Typically, the meta directory root should already be created. In hindsight, I think what I'll do is re-open this issue and file a quick PR to add some automatic directory creation.

@adreed-msft thank you!
While waiting for a fix for this I used robocopy to mirror the directory structure to the metdata folder so I could test. This did allow it to go further and it creates all the metadata files now. However I am still seeing the following error in the scanning log duplicated for every file:
failed to write hash data for Folder1/Filename.doc: failed to hide file: The media is write protected.
failed to write hash data for Folder1/Filename2.doc: failed to hide file: The media is write protected.
failed to write hash data for Folder2/FilenameABC.doc: failed to hide file: The media is write protected.

I checked the code and it seems the SetHashData function in hash_data_adapter_hidden_files.go is trying to hide the data file instead of the metaFile which seems odd. Is this a bug?

@adreed-msft
I am using azcopy on two different volumes. Both use to work fine but now one of them is crashing:
Capturing this error is dificult as it only shows up in console. No errors are logged to the log files.
75364 Files Scanned at Source, 0 Files Scanned at Destination
81855 Files Scanned at Source, 0 Files Scanned at Destination
88484 Files Scanned at Source, 0 Files Scanned at Destination
94584 Files Scanned at Source, 0 Files Scanned at Destination
102581 Files Scanned at Source, 0 Files Scanned at Destination
104073 Files Scanned at Source, 12514 Files Scanned at Destination
104073 Files Scanned at Source, 22916 Files Scanned at Destination
104073 Files Scanned at Source, 41081 Files Scanned at Destination
104073 Files Scanned at Source, 64449 Files Scanned at Destination
104073 Files Scanned at Source, 84358 Files Scanned at Destination
runtime: VirtualAlloc of 1048576 bytes failed with errno=1455
fatal error: out of memory

runtime stack:
runtime.throw({0x10d495e?, 0xc036f71000?})
	C:/hostedtoolcache/windows/go/1.19.12/x64/src/runtime/panic.go:1047 +0x65 fp=0x2f3cfce0 sp=0x2f3cfcb0 pc=0x43a5c5
runtime.sysUsedOS(0xc036ec8000, 0x100000)
	C:/hostedtoolcache/windows/go/1.19.12/x64/src/runtime/mem_windows.go:83 +0x1c9 fp=0x2f3cfd40 sp=0x2f3cfce0 pc=0x4190e9
runtime.sysUsed(0x19ca3b0?, 0x80?, 0x41a436?)
	C:/hostedtoolcache/windows/go/1.19.12/x64/src/runtime/mem.go:77 +0x25 fp=0x2f3cfd60 sp=0x2f3cfd40 pc=0x418be5
runtime.(*mheap).allocSpan(0x19ca3a0, 0x80, 0x0, 0x1)
	C:/hostedtoolcache/windows/go/1.19.12/x64/src/runtime/mheap.go:1342 +0x619 fp=0x2f3cfdf8 sp=0x2f3cfd60 pc=0x429659
runtime.(*mheap).alloc.func1()
	C:/hostedtoolcache/windows/go/1.19.12/x64/src/runtime/mheap.go:918 +0x65 fp=0x2f3cfe40 sp=0x2f3cfdf8 pc=0x428c45
runtime.systemstack()
	C:/hostedtoolcache/windows/go/1.19.12/x64/src/runtime/asm_amd64.s:492 +0x4e fp=0x2f3cfe48 sp=0x2f3cfe40 pc=0x4682ee

goroutine 37047 [running]:
runtime.systemstack_switch()
	C:/hostedtoolcache/windows/go/1.19.12/x64/src/runtime/asm_amd64.s:459 fp=0xc00474d578 sp=0xc00474d570 pc=0x468280
runtime.(*mheap).alloc(0x100000?, 0x80?, 0x70?)
	C:/hostedtoolcache/windows/go/1.19.12/x64/src/runtime/mheap.go:912 +0x65 fp=0xc00474d5c0 sp=0xc00474d578 pc=0x428b85
runtime.(*mcache).allocLarge(0xc002ec0cb0?, 0x100000, 0x1)
	C:/hostedtoolcache/windows/go/1.19.12/x64/src/runtime/mcache.go:233 +0x85 fp=0xc00474d610 sp=0xc00474d5c0 pc=0x417c85
runtime.mallocgc(0x100000, 0xeccba0, 0x1)
	C:/hostedtoolcache/windows/go/1.19.12/x64/src/runtime/malloc.go:1029 +0x57e fp=0xc00474d688 sp=0xc00474d610 pc=0x40df9e
runtime.makeslice(0x10?, 0xc000313400?, 0xc00228cea0?)
	C:/hostedtoolcache/windows/go/1.19.12/x64/src/runtime/slice.go:103 +0x52 fp=0xc00474d6b0 sp=0xc00474d688 pc=0x44f4f2
github.com/Azure/azure-storage-azcopy/v10/common.(*multiSizeSlicePool).RentSlice(0xc00000000c?, 0xfae4d)
	D:/a/1/s/common/multiSizeSlicePool.go:175 +0x14e fp=0xc00474d710 sp=0xc00474d6b0 pc=0xcc7e4e
github.com/Azure/azure-storage-azcopy/v10/common.(*singleChunkReader).blockingPrefetch(0xc003982b60, {0x12cea00, 0xc000a28c20}, 0x0)
	D:/a/1/s/common/singleChunkReader.go:241 +0x202 fp=0xc00474d830 sp=0xc00474d710 pc=0xcd04a2
github.com/Azure/azure-storage-azcopy/v10/common.(*singleChunkReader).BlockingPrefetch(0xc003982b60, {0x12cea00, 0xc000a28c20}, 0xc0?)
	D:/a/1/s/common/singleChunkReader.go:211 +0x87 fp=0xc00474d890 sp=0xc00474d830 pc=0xcd01a7
github.com/Azure/azure-storage-azcopy/v10/ste.scheduleSendChunks({0x12e53c0, 0xc0004b5570}, {0xc002ec0cb0, 0x65}, {0x12d2648?, 0xc000a28c20}, 0xfae4d, {0x12d8b60, 0xc002bf2a50}, 0xc0039db0f8, ...)
	D:/a/1/s/ste/xfer-anyToRemote-file.go:432 +0x5b8 fp=0xc00474da98 sp=0xc00474d890 pc=0xd98438
github.com/Azure/azure-storage-azcopy/v10/ste.anyToRemote_file({0x12e53c0, 0xc0004b5570}, 0xc000399a20, {0x2b1b6028, 0xc00030a4c0}, 0x1174b98, 0x1174bb8)
	D:/a/1/s/ste/xfer-anyToRemote-file.go:370 +0xe0c fp=0xc00474dd80 sp=0xc00474da98 pc=0xd9786c
github.com/Azure/azure-storage-azcopy/v10/ste.anyToRemote({0x12e53c0, 0xc0004b5570}, {0x2b1b6028, 0xc00030a4c0}, 0xc00474dea8?, 0xd6f275?)
	D:/a/1/s/ste/xfer-anyToRemote-file.go:204 +0x2ff fp=0xc00474de20 sp=0xc00474dd80 pc=0xd968df
github.com/Azure/azure-storage-azcopy/v10/ste.computeJobXfer.func5({0x12e53c0?, 0xc0004b5570?}, {0x2b1b6028?, 0xc00030a4c0?})
	D:/a/1/s/ste/xfer.go:75 +0x3a fp=0xc00474de60 sp=0xc00474de20 pc=0xda339a
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobPartMgr).StartJobXfer(0x19505e0?, {0x12e53c0?, 0xc0004b5570?})
	D:/a/1/s/ste/mgr-JobPartMgr.go:472 +0x42 fp=0xc00474de90 sp=0xc00474de60 pc=0xd6ace2
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobPartTransferMgr).StartJobXfer(0xc0004b5570?)
	D:/a/1/s/ste/mgr-JobPartTransferMgr.go:245 +0x32 fp=0xc00474deb8 sp=0xc00474de90 pc=0xd6bdd2
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).transferProcessor.func1({0x12e53c0, 0xc0004b5570})
	D:/a/1/s/ste/mgr-JobMgr.go:1093 +0x199 fp=0xc00474df28 sp=0xc00474deb8 pc=0xd68119
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).transferProcessor(0xc000580400, 0xc005806c60?)
	D:/a/1/s/ste/mgr-JobMgr.go:1105 +0xbd fp=0xc00474dfc0 sp=0xc00474df28 pc=0xd67ebd
github.com/Azure/azure-storage-azcopy/v10/ste.NewJobMgr.func2()
	D:/a/1/s/ste/mgr-JobMgr.go:200 +0x2a fp=0xc00474dfe0 sp=0xc00474dfc0 pc=0xd6268a
runtime.goexit()
	C:/hostedtoolcache/windows/go/1.19.12/x64/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00474dfe8 sp=0xc00474dfe0 pc=0x46a5e1
created by github.com/Azure/azure-storage-azcopy/v10/ste.NewJobMgr
	D:/a/1/s/ste/mgr-JobMgr.go:200 +0xc17

This is just the first two stack traces. The full output is over 5mb.

@ktrace2
This issue should be resolved with our next release since the PR above has been merged. Closing this as a fix has been merged to main.

I checked the code and it seems the SetHashData function in hash_data_adapter_hidden_files.go is trying to hide the data file instead of the metaFile which seems odd. Is this a bug?

@gapra-msft What about this issue? Looks like it is still an issue.


build netcdf with hdf5-1.12.0 beta release fails in h5_test directory

Well hdf5-1.12.0 beta has been released! How exciting.

Sadly we fail immediately:
tst_h_files4.c: In function ‘op_func’:
tst_h_files4.c:54:43: error: ‘const union <anonymous>’ has no member named ‘address’
    if ((id = H5Oopen_by_addr(g_id, info->u.address)) < 0) ERR;
                                           ^
In file included from /usr/local/hdf5-1.12.0-beta/include/H5public.h:32:0,
                 from /usr/local/hdf5-1.12.0-beta/include/hdf5.h:22,
                 from tst_h_files4.c:11:
tst_h_files4.c: In function ‘main’:
/usr/local/hdf5-1.12.0-beta/include/H5version.h:759:30: error: too few arguments to function ‘H5Oget_info_by_idx3’
   #define H5Oget_info_by_idx H5Oget_info_by_idx3
                              ^
tst_h_files4.c:189:7: note: in expansion of macro ‘H5Oget_info_by_idx’
   if (H5Oget_info_by_idx(grpid, ".", H5_INDEX_CRT_ORDER, H5_ITER_INC,
       ^~~~~~~~~~~~~~~~~~
In file included from /usr/local/hdf5-1.12.0-beta/include/H5Apublic.h:22:0,
                 from /usr/local/hdf5-1.12.0-beta/include/hdf5.h:23,
                 from tst_h_files4.c:11:
/usr/local/hdf5-1.12.0-beta/include/H5Opublic.h:190:15: note: declared here
 H5_DLL herr_t H5Oget_info_by_idx3(hid_t loc_id, const char *group_name,
               ^~~~~~~~~~~~~~~~~~~
Makefile:933: recipe for target 'tst_h_files4.o' failed
make[2]: *** [tst_h_files4.o] Error 1
make[2]: Leaving directory '/home/ed/noaa/netcdf-c/h5_test'
Makefile:1402: recipe for target 'check-am' failed
make[1]: *** [check-am] Error 2
make[1]: Leaving directory '/home/ed/noaa/netcdf-c/h5_test'
Makefile:705: recipe for target 'check-recursive' failed
make: *** [check-recursive] Error 1

@epourmal did you change the signature of H5Oget_info_by_idx3?

@edwardhartnett Hi Ed,
to address the main problem here, it looks like the netCDF code uses the HDF5 compatibility macro for H5Oget_info_by_idx. In the 1.10 series, we versioned H5Oget_info(_by_name/_by_idx) into version 1 and version 2 and added a parameter to the version 2 functions, but the new compatibility macro we added pointed to the version 1 function, so netCDF code wouldn't have had any problems at that point in time. With the 1.12 release, this macro now points to the new version 3 H5Oget_info(_by_name/_by_idx), which contain the extra parameter from the version 2 counterparts. The signature for H5Oget_info_by_idx now looks like:
H5_DLL herr_t H5Oget_info_by_idx3(hid_t loc_id, const char *group_name,
H5_index_t idx_type, H5_iter_order_t order, hsize_t n, H5O_info2_t *oinfo,
unsigned fields, hid_t lapl_id);
where the next-to-last parameter 'fields' was added. If the netCDF code should be updated to be compatible with the latest API versions, H5O_INFO_ALL can simply be passed for this parameter. Otherwise, see the next-to-last paragraph about HDF5's compatibility macros for info about compiling HDF5 in 1.10 release-compatible mode, where no netCDF code changes should be needed.
Now, for the larger work done for the 1.12 release, we wanted to better accommodate HDF5 VOL connectors by replacing the notion of "object addresses in a file" (which may not make sense to some VOL connectors) with a more generic "object token" type, H5O_token_t. To this end, we deprecated H5Oopen_by_addr() and added a new H5Oopen_by_token() routine, which takes a location ID for the first parameter (the same as H5Oopen_by_addr()) and an H5O_token_t object for the second parameter. This H5O_token_t object is essentially meant to be read-only from the application side and is returned from several HDF5 API calls where an haddr_t was previously returned. It's suggested that H5Oopen_by_addr() calls be replaced by calls to H5Oopen_by_token(), but the former is still available.
We also changed several API calls to switch to using these new object tokens instead of object addresses. These are:

H5Lget_info(_by_idx) - These routines are now versioned into version 1 and version 2, with version 2 being preferred. The H5L_info_t struct they used is also now versioned, with the haddr_t type contained within the union having been replaced with an H5O_token_t object; version 2 of this struct is preferred.

H5Literate(_by_name)/H5Lvisit(_by_name) - Now versioned into version 1 and version 2, with version 2 being preferred. The H5L_iterate_t callback function is now also versioned, with the H5L_info_t parameter being versioned such that H5L_iterate1_t expects an H5L_info1_t and H5L_iterate2_t expects an H5L_info2_t.

The H5O routines are a little bit more complex:

H5Oget_info(by_name/by_idx) - A version 3 of these routines was added and is now preferred. The change here is that the H5O_info_t struct parameter is now versioned into version 1 and version 2, with the version 2 having some fields removed and the previous haddr_t field being replaced with an H5O_token_t object field. The version 1 and 2 H5Oget_info routines use the H5O_info1_t struct and the version 3 H5Oget_info routines use the H5O_info2_t struct. However, our compatibility macros should match those up for you when using the plain H5Oget_info_ symbol without a version number.

H5Oget_native_info(_by_name/_by_idx) - These routines were added in order to separate the native HDF5 file format-specific information in the H5O_info_t struct (such as object header and B-tree info) away from the more generic data model-specific information (such as the number of attributes attached to an object).

H5Ovisit(by_name) - A version 3 of these routines was added and is now preferred. The H5O_iterate_t callback function is now also versioned, such that H5O_iterate1_t expects an H5O_info1_t and H5O_iterate2_t expects an H5O_info2_t. The version 1 and 2 H5Ovisit routines use the H5O_iterate1_t callback and the version 3 H5Ovisit_ routines use the H5O_iterate2_t callback. However, our compatibility macros should match those up for you when using the plain H5Ovisit_ symbol without a version number.

Unfortunately, the nature of the 1.12 release means that we didn't get this information out to users early enough before making 1.12 available as a beta release. For netCDF, these changes mean that either the netCDF code can:

specify certain flags at HDF5 build time to force the compatibility macros to use a specific API version
or
the HDF5 API calls mentioned here can be upgraded to directly use the latest versions as specified, e.g. by directly calling H5Oget_info_by_idx3

To force HDF5's compatibility macros to use a specific API version, please see HDF5's API Compatibility Macros. If I understand the netCDF code correctly, specifying the --with-default-api-version=v110 flag at build time should force all of HDF5's compatibility macros to use 1.10 release-compatible API versions and should clear up any build problems without needing to update the netCDF code.
Please let me know if there is anything I can do to help the netCDF team with transitioning to the new release.

I will upgrade to H5Oget_info_by_idx3 - it's only a couple of places in the code. I will take a look at this after the szip stuff is complete. ;-)

I got the same build error as Ed with unmodified release version netcdf-c 4.7.3, and HDF5 1.12.0-rc1 on Mac OS.  However, the build succeeded by enabling 1.8 Compatibility Macros as suggested by @jhendersonHDF.  Specifically I added -DH5_USE_18_API to CFLAGS for configure.
Then I added --disable-dap-remote-tests to configure, to avoid unrelated OpenDAP test problems.  With that, make check completed all remaining tests.
FWIW, this appears to be a fully working build of netcdf-c 4.7.3 with HDF5 1.12.0-rc1.
mac56:~/temp/netcdf/0227.dap-tests 8> ldd $prg
/Users/dallured/netcdf/4.7.3.h5-120rc1/bin/ncdump:
  /Users/dallured/netcdf/4.7.3.h5-120rc1/lib/libnetcdf.15.dylib (compatibility version 18.0.0, current version 18.1.0)
  /Users/dallured/3rd/hdf5/1.12.0-rc1/lib/libhdf5_hl.200.dylib (compatibility version 201.0.0, current version 201.0.0)
  /Users/dallured/3rd/hdf5/1.12.0-rc1/lib/libhdf5.200.dylib (compatibility version 201.0.0, current version 201.0.0)
  /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
  /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
  /opt/local/lib/libgcc/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

mac56:~/netcdf/4.7.3.h5-120rc1 57> ncdump -s src/examples/C/simple_xy_nc4.nc
netcdf simple_xy_nc4 {
dimensions:
  x = 6 ;
  y = 12 ;
variables:
  int data(x, y) ;
    data:_Storage = "chunked" ;
    data:_ChunkSizes = 1, 3 ;
    data:_DeflateLevel = 1 ;
    data:_Shuffle = "true" ;
    data:_Endianness = "little" ;

// global attributes:
    :_NCProperties = "version=2,netcdf=4.7.3,hdf5=1.12.0," ;
    :_SuperblockVersion = 0 ;
    :_IsNetcdf4 = 1 ;
    :_Format = "netCDF-4" ;
data:

 data =
  0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11,
  12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,
  24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
  36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
  48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59,
  60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71 ;
}

We actually have a PR ready with a fix which makes 1.12.0 compile and run fine.
There's a bit of a back-up of pull requests at the moment, when it gets cleared out I will put the PR back up.

Thanks to @brtnfld for providing the code to resolve this issue in #1640!
I have a branch ready to go with these changes, merged to current master. I'll put it up as soon as #1663 gets merged.

THis is being held up by the CMake build, which does not work. See #1665

@edwardhartnett I did not try with CMake


Transaction API

In Rust we can leverage the borrow-checker to ensure that unsafe mid-change operations don't compile, for instance calling merge or apply_changes whilst we have an open change. This was originally being done manually by using ensure_transaction_closed in each function call. This means that methods that don't mutate the document no longer need to take a &mut self, rather just a &self.
This PR also adds a version of the original Automerge, using an internally stored transaction for use with interop such as wasm/js (AutoTxn, happy for suggestions for the name).
Overall, these changes make the library more idiomatically Rust, whilst still retaining the convenient automatic transactions behaviour from before, allowing more things to be built on-top of the library.

Man - this is great.  I really hated those ensure_transaction_closed() statements.   As best as I can tell there are three ways to make an op now.  Explicitly make a transaction and then perform an op against it.  Have AutoTxn which behaves like the old Automerge.  And then there is now a third way where the base Automerge object makes a one op transaction for each op.  I really would like to avoid this third scenario.  Given how costly each transaction is I don't want it to be easy to commit each op on its own.  I would much rather we just rip out all of the write functions from Automerge and make you either pick explicit transaction or AutoTxn.   Im curious as to why you added this behavior?  Was it to make the tests simpler?  Maybe instead switch all the tests to AutoTxn::init() and support the old api only there?
Also I noticed there are a few of the underscore functions still laying around - such as fn _get_changes - given these changes I'd love to see these all taken out - I only added them to avoid calling ensure_transaction_closed over and over in calls that needed these methods.

Yep, you're right with those three ways of making an op, and yes, I was basically having the top level set etc for ease of testing. I've switch most over to the AutoTxn now, using the explicit tx for others.
Also removed the underscore functions.
I've updated the commit API to be more Rusty too as it is annoying having to pass (None, None) in every time. This leaves the js interop the old way though, not sure if that might be better realised as a dict of optional things?
Another addition is the transact and transact_with functions (and associated types) so that Rust users don't have to forget to manually commit the transaction.
I've also added in a rollback on drop for the explicit transactions to ensure we don't break the document (this is skipped if manually calling commit or rollback).

Awesome - love all these changes - Nothing else jumps out at me.  I'd like to get @alexjg's input here as he's a consumer of the rust api as well.  What did the benchmarks say as far as the performance impact of all this?

In general this looks really nice. I'll review in detail over the weekend.

What did the benchmarks say as far as the performance impact of all this?

About 25ms difference on my machine (474ms for manual, 499 for auto)


Publish v0.1.1 to pip

I think we should publish all the additions we made last fall to pip - what do you think, @jessecusack @hdrake ? Unless you think there is anything we should fix before then, I would offer to publish a new release. Then we could also have the documentation point to v0.1.1 (#66) so everyone can benefit from the great additions to the docs.

Agreed, a version 0.1.1 would be a good next step. I'll try and find some time in the next month to clear up the remaining issues for 0.1.2 :)

Okay, I will get this out to pip when I have a moment. Should we set up a milestone for a 0.1.2 version to collect the issues that we think are most pressing? Or maybe work towards a 0.2 version and leave 0.1.2 open in case we find any bugs to fix until then?

I agree a 0.1.1 release makes sense! I had some ideas for what I'd like to see in v0.1.2 and v0.2.0 in this comment: https://github.com/modscripps/mixsea/issues/60#issuecomment-950923051
Unfortunately I don't have much time to contribute right now, but hopefully will have time to circle back in the coming months.

Thanks @hdrake and no worries about being short on time at the moment, I just mentioned you to acknowledge how much effort you put into this last year =)
I will start a 0.2.0 milestone where we can add #60 and whatever else we think would be good in there. This leaves 0.1.2 open for any fixes with more immediate needs.

Yes, lets collect together issues into milestones. I'll work through the ones related to Thorpe scales.
If fixing the most pressing issues leads to some breaking changes (e.g. #60), I would vote for the milestone being 0.2.0

I'll work through the ones related to Thorpe scales.

Feel free to assign me as a reviewer if you want a second set of eyes on anything.

Yes, I think #60 will be a breaking change and am in favor of 0.2.0


Change default for proposal status enum to Accepted

Motivation
Currently, every review or project goes into a proposed stage before the accept_review extrinsic is called. Since we don't really have a governance system in place, it would be more user friendly to have all content start off as accepted, and then have it switch to other states depending on how the content evaluation pallet (when included) views it.
Suggested Solution
Change impl block for default on Status enum to Accepted and check create_review and create_project for compliance.
Additional Information

This might break the ad-hoc rewarding logic for reviews since it depends on switching status to "accepted". A fix for that could be to leave the current status enum as is and have the switch be done automatically by the content evaluation pallet on passing, or just leave the rewarding aspect alone to be done by the content evaluation pallet by tweaking accept_review.

This also means that the current genesis config doesn't have to worry about state of reviews, so it can be simplified further.

Note: This also means that the current genesis config doesn't have to worry about state of reviews/projects, so it can be simplified further.

Note: This also means that the current genesis config doesn't have to worry about state of reviews/projects, so it can be simplified further.

Hi. I would like to contribute to this issue. can I?

Hi @Immanuel-john , you're welcome to contribute to this issue!
Though this description of the issue is a bit outdated as I've moved the pallets over to the parachain repo.
The most relevant issue on that repo to this one would be https://github.com/chocolatenetwork/chocolate-parachain/issues/19
Thanks!


Add promise package with defer

Codecov Report

Merging #18 (26e7bb8) into main (f156bbf) will decrease coverage by 2.12%.
The diff coverage is 75.00%.

@@             Coverage Diff             @@
##              main      #18      +/-   ##
===========================================
- Coverage   100.00%   97.87%   -2.13%
===========================================
  Files            8        9       +1
  Lines           86       94       +8
  Branches        10       10
===========================================
+ Hits            86       92       +6
- Misses           0        2       +2

Impacted Files
Coverage Δ

packages/promise/src/defer.ts
75.00% <75.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


PageLayout: Fix layout shift for divider=line

Closes https://github.com/primer/react/issues/3331

Problem
When using <PageLayout.Pane divider={{regular: 'line', narrow: 'none'}}>, we need to figure out which responsive value to use (regular, narrow or wide)
The hook useResponsiveValue we use renders the default (no divider) before the passed variant (like line) is rendered. This causes some layout shift (see video below)
https://github.com/primer/react/assets/1863771/6ba438ad-3050-4b00-9705-693c4c9a6e19
This shift is slightly worse when the page is rendered on the server because this hook is not SSR compatible:
https://github.com/primer/react/assets/1863771/4e450df2-d6f1-4d1b-8892-f80d77841173
Note: This shift is also visible even when we have the same variant across screen sizes (<PageLayout.Pane divider="line">)

Changed
The ideal solution here would be to make this change css only, and not need any client side javascript. This however isn't very trivial and requires some time.
This PR has a pragmatic solution just for PageLayout to eliminate layout shift for the most common use case of <PageLayout.Pane divider="line"> by rendering a transparent divider for divider=none. This will hopefully buy us more time to fix the actual cause.

Before: There is no divider rendered at first, and then it renders and pushes the content away causing a layout shift
https://github.com/primer/react/assets/1863771/25dc12a9-c25f-47ab-a85f-598e64a5ff97
After: There is a transparent divider rendered at first, which does not cause any layout shift
https://github.com/primer/react/assets/1863771/506f3dd4-1cbd-4ee8-b113-38bc8e2ab78f

Rollout strategy

[x] Patch release

Testing & Reviewing

Merge checklist

[x] Added/updated tests
~- [ ] Added/updated documentation~
~- [ ] Added/updated previews (Storybook)~
[x] Changes are SSR compatible
[x] Tested in Chrome
~- [ ] Tested in Firefox~
~- [ ] Tested in Safari~
~- [ ] Tested in Edge~
[ ] (GitHub staff only) Integration tests pass at github/github (Learn more about how to run integration tests)

@joshblack Nah, those are all wrong 😢 Converted back to drafts

This was bigger than I initially anticipated, abandoning this


allow clients to request unparsed json

Getting concerned there are no tests here and you're adding more features. Can we get some testing going?

agreed! with the platform announcement having gone out this morning, the goal was to quickly get things up to snuff and backfill in tests later. I am testing this functionality pretty extensively on my machine with files that I'm not checking in :+1:


Factor out HtmlTreeBranch and related code from template parser.

This code is not directly related to the template parser, so it
can safely live in its own file.
Fixes: #1774.

Automated message from Dropbox CLA bot
@GPHemsley, it looks like you've already signed the Dropbox CLA. Thanks!


Add userhigh script

Add script to list a user's high links.
$ kpu script userhigh mlsteele
{"seqnos":[1,15,17,19,20,26,28,54,60,62,69,73,74,78,79,107,113,116,125,127,129,133,136]}

Results of spot check in CORE-8856
One failure.

@github there are no conflicts...


axios response data incorrect

Was hoping i can get some assistance on this.
I am calling another API which returned an attachment to me.
If i was to call that API normally by using postman or anything i can download the ZIP successfully and unzip the contents.
But when i implement it to express my data is not right.
As seen below the issue is because when i try writing the returned data to a file it is not right.
The content-length is actually supposed to be 24109 but when i look at the response data length i only have 23062.
Is this because of some sort of body parser that i need to implement?
I have the following code:
const express = require('express')
const axios = require('axios');
const app = express()

const generate = () => {
  return axios.post('http://localhost:8050/api/json', {
      host: 'lightapi.net',
      service: 'codegen',
      action: "generate",
      version: '0.0.1',
      data: [{"framework":"swagger"}]
  }, {
      responseType: 'blob',
  });
};

app.get('/', (req, res) => {
  generate().then((response) => {
    console.log('########### HEADERS: ', response.headers);
    console.log('########### AXIOS: ', response.data.length);
  });
  res.send('Hello World!');
});

app.listen(9000, () => console.log('Example app listening on port 9000!'))

and i get the following response:
########### HEADERS:  { connection: 'close',
  server: 'L',
  'content-type': 'application/zip',
  'content-length': '24109',
  'content-disposition': 'attachment',
  date: 'Fri, 27 Apr 2018 16:48:05 GMT' }
########### AXIOS:  23062

Thanks for the report! I can take a look into the issue, and thanks for the code above! Can you provide the following additional information?

Version of Node.js
Version of express and all other used modules
The exact steps to see the issue. I tried to just run it, but I don't know how to reproduce whatever http://localhost:8050/api/json is responding with to be able to reproduce and debug.

yes. i removed the part which creates the zip file. i can add that in if you would like and this way you can get a zip file back rather then just a hello world?
The main issue was the zip file will not get created right because i am not seeing the right amount of data in the response.
as seen on the server:
expected length: 24109
actual length: 23062

Can you paste me the output if you try running the following script?
const axios = require('axios');

const generate = () => {
  return axios.post('http://localhost:8050/api/json', {
      host: 'lightapi.net',
      service: 'codegen',
      action: "generate",
      version: '0.0.1',
      data: [{"framework":"swagger"}]
  }, {
      responseType: 'blob',
  });
};

  generate().then((response) => {
    console.log('########### HEADERS: ', response.headers);
    console.log('########### AXIOS: ', response.data.length);
  });
});

sorry i am not to familiar with JS so do i just make a new file and execute the above?
What i am doing is i have a website front end and it talks to the express server for any API calls i make.
The site will reach out to the server and then get the response and create a zip file.
Currently i have all my APIs going to the express server but not this one call because of the zip file.
To make the zip file on the UI i do the following where generate is the axios command without express going straight to the API:
const generatedResponse = await generate();
    const blob = new Blob([generatedResponse.data]);
    const filename = framework + '.zip';
    if (typeof window.navigator.msSaveBlob !== 'undefined') {
        window.navigator.msSaveBlob(blob, filename);
    } else {
        const url = window.URL.createObjectURL(blob);
        const link = document.createElement('a');
        link.href = url;
        link.setAttribute('download', filename);
        document.body.appendChild(link);
        link.click();
    }
    dispatch(codegenisSuccessful(true));
    dispatch(codegenisLoading(false));

with above where the call is nowhere near express, i can get a successful zip file since the data is the same length as the content-length.
My end goal is for UI to contact express server which contacts the API and then returns the information back to the UI.
When i do this the express server i noticed is when i get a response back whether axios or any other method, the content-length and actualy response.data.length start differing so when i create the zip file i get an error unzipping saying cannot unzip.

that is why i was assuming something is wrong with the actual response because i was expecting a certain amount of data length back but when i look at it i get less then expected. which creates a incorrect file. that is why then i assumed maybe some sort of body-parser or something

sorry i am not to familiar with JS so do i just make a new file and execute the above?

exactly

... more responses

I can help you, but there is too much assumptions here. I'm not sure how to get everything running. For example, pretend like you're on a new machine without any of your stuff you're talking about setup. Please give me step-by-step all the commands to run to see the issue you're describing from that machine so I can also get everything setup and see the issue. Without clear instructions on how I can reproduce the issue you're seeing, I'm not sure how to diagnose it.

get the api running which will provide a zip file:

git clone git@github.com:auahmed/api.git
cd api
java -Dlight-4j-config-dir=./config -Dlogback.configurationFile=./logback.xml -cp "./service/*" com.networknt.server.Server

start the server:

git clone git@github.com:auahmed/test-server.git
cd test-server
npm install
node app.js

Call the server using your preferred method:

curl http://localhost:9000

Issue is that the data is not expected to create a successful zip file.
If my web application does not go through express, the content-length and response.data.length are both 24109. But when i send it through express, the response.data.length appears to be a little less then expected. to create a successful zip file the response.data.length that is showing to be 23062 should be 24109.

sorry accidentally closed it.
and the file how would i execute a js file like that?

the response.data.length should be 24109 just like the content-length in header to create a successful zip file.

If this is the case, it looks like it's an issue with either (a) axios or (b) how you're using axios. I can see the different values without involving Express.js in any way:
$ java -Dlight-4j-config-dir=./config -Dlogback.configurationFile=./logback.xml -cp "./service/*" com.networknt.server.Server
13:46:39,911 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [./logback.xml] at [file:/express-3636/api/./logback.xml]
13:46:39,957 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
13:46:39,971 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
13:46:39,973 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [stdout]
13:46:39,973 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
13:46:40,011 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.FileAppender]
13:46:40,012 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [log]
13:46:40,014 |-WARN in ch.qos.logback.core.FileAppender[log] - This appender no longer admits a layout as a sub-component, set an encoder instead.
13:46:40,014 |-WARN in ch.qos.logback.core.FileAppender[log] - To ensure compatibility, wrapping your layout in LayoutWrappingEncoder.
13:46:40,014 |-WARN in ch.qos.logback.core.FileAppender[log] - See also http://logback.qos.ch/codes.html#layoutInsteadOfEncoder for details
13:46:40,014 |-INFO in ch.qos.logback.core.FileAppender[log] - File property is set to [target/test.log]
13:46:40,015 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
13:46:40,015 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [audit]
13:46:40,016 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
13:46:40,016 |-WARN in ch.qos.logback.classic.encoder.PatternLayoutEncoder@71e7a66b - As of version 1.2.0 "immediateFlush" property should be set within the enclosing Appender.
13:46:40,016 |-WARN in ch.qos.logback.classic.encoder.PatternLayoutEncoder@71e7a66b - Please move "immediateFlush" property into the enclosing appender.
13:46:40,017 |-WARN in ch.qos.logback.classic.encoder.PatternLayoutEncoder@71e7a66b - Setting the "immediateFlush" property of the enclosing appender to true
13:46:40,021 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@2ac1fdc4 - Will use zip compression
13:46:40,023 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[audit] - Active log file name: target/audit.log
13:46:40,023 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[audit] - File property is set to [target/audit.log]
13:46:40,023 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to DEBUG
13:46:40,023 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [stdout] to Logger[ROOT]
13:46:40,024 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.networknt] to DEBUG
13:46:40,024 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [log] to Logger[com.networknt]
13:46:40,024 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [Audit] to ERROR
13:46:40,024 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [Audit] to false
13:46:40,024 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [audit] to Logger[Audit]
13:46:40,024 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
13:46:40,025 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@5f150435 - Registering current configuration as safe fallback point

13:46:40.224 [main]   INFO  com.networknt.config.Config checkCacheExpiration - daily config cache refresh
13:46:40.230 [main]   INFO  com.networknt.config.Config getConfigStream - Config loaded from externalized folder for server.yml in ./config
13:46:40.254 [main]   INFO  com.networknt.config.Config getConfigStream - Config loaded from externalized folder for secret.yml in ./config
13:46:40.257 [main]   INFO  com.networknt.server.Server main - server starts
13:46:40.257 [main] com.networknt.portal.hybrid.query-1.0.0  WARN  com.networknt.server.Server loadConfig - Warning! No light-env has been passed in from command line. Default to dev
13:46:40.257 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.server.Server loadConfig - light-config-server-uri is missing in the command line. Use local config files
13:46:40.258 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from externalized folder for service.yml in ./config
13:46:40.274 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Unable to load config from externalized folder for rpc-router.yml in ./config
13:46:40.275 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Trying to load config from classpath directory for file rpc-router.yml
13:46:40.277 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from default folder for rpc-router.yml
13:46:41.121 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Unable to load config from externalized folder for eventuate-client.yml in ./config
13:46:41.122 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Trying to load config from classpath directory for file eventuate-client.yml
13:46:41.122 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from default folder for eventuate-client.yml
13:46:41.222 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Unable to load config from externalized folder for exception.yml in ./config
13:46:41.222 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Trying to load config from classpath directory for file exception.yml
13:46:41.223 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from default folder for exception.yml
13:46:41.226 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Unable to load config from externalized folder for metrics.yml in ./config
13:46:41.226 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Trying to load config from classpath directory for file metrics.yml
13:46:41.226 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from default folder for metrics.yml
13:46:41.236 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Unable to load config from externalized folder for traceability.yml in ./config
13:46:41.236 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Trying to load config from classpath directory for file traceability.yml
13:46:41.236 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from default folder for traceability.yml
13:46:41.237 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Unable to load config from externalized folder for correlation.yml in ./config
13:46:41.238 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Trying to load config from classpath directory for file correlation.yml
13:46:41.238 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from default folder for correlation.yml
13:46:41.239 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from externalized folder for cors.yml in ./config
13:46:41.242 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from externalized folder for security.yml in ./config
13:46:41.244 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Unable to load config from externalized folder for audit.yml in ./config
13:46:41.244 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Trying to load config from classpath directory for file audit.yml
13:46:41.244 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from default folder for audit.yml
RpcStartupHookProvider: handlers size 4
RpcStartupHookProvider add lightapi.net/codegen/listFramework/0.0.1 to com.networknt.codegen.handler.FrameworkListHandler13:46:41.317 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Unable to load config from externalized folder for codegen-web.yml in ./config
13:46:41.318 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Trying to load config from classpath directory for file codegen-web.yml
13:46:41.318 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from default folder for codegen-web.yml
RpcStartupHookProvider add lightapi.net/codegen/generate/0.0.1 to com.networknt.codegen.handler.GeneratorServiceHandlerRpcStartupHookProvider add lightapi.net/codegen/getSchema/0.0.1 to com.networknt.codegen.handler.SchemaGetHandlerRpcStartupHookProvider add lightapi.net/codegen/validateUploadFile/0.0.1 to com.networknt.codegen.handler.ValidateUploadFileHandler13:46:41.330 [main] com.networknt.portal.hybrid.query-1.0.0  DEBUG c.n.rpc.router.AbstractRpcHandler <clinit> - schema file = jar:file:/express-3636/api/service/codegen-web-1.5.12.jar!/schema.json
13:46:41.399 [main] com.networknt.portal.hybrid.query-1.0.0  DEBUG c.n.rpc.router.AbstractRpcHandler <clinit> - schema = {"lightapi.net/codegen/generate/0.0.1":{"schema":{"title":"Service","type":"object","properties":{"generators":{"type":"array","items":{"properties":{"model":{"type":"object"},"config":{"type":"object"},"framework":{"type":"string"}},"required":["config","framework"]},"maxItems":5,"minItems":1}}},"scope":"codegen.w"},"lightapi.net/codegen/validateUploadFile/0.0.1":{"schema":{},"scope":"codegen.r"},"lightapi.net/codegen/getSchema/0.0.1":{"schema":{"title":"Service","type":"object","properties":{"framework":{"type":"string"}},"required":["framework"]},"scope":"codegen.r"},"lightapi.net/codegen/listFramework/0.0.1":{"schema":{},"scope":"codegen.r"}}
13:46:41.403 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.server.Server start - Plugin: com.networknt.audit.AuditHandler
13:46:41.404 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.server.Server start - Plugin: com.networknt.rpc.security.JwtVerifyHandler
13:46:41.405 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.server.Server start - Plugin: com.networknt.cors.CorsHttpHandler
13:46:41.405 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from externalized folder for cors.yml in ./config
13:46:41.406 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.server.Server start - Plugin: com.networknt.correlation.CorrelationHandler
13:46:41.407 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Unable to load config from externalized folder for correlation.yml in ./config
13:46:41.407 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Trying to load config from classpath directory for file correlation.yml
13:46:41.408 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from default folder for correlation.yml
13:46:41.409 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.server.Server start - Plugin: com.networknt.traceability.TraceabilityHandler
13:46:41.409 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Unable to load config from externalized folder for traceability.yml in ./config
13:46:41.410 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Trying to load config from classpath directory for file traceability.yml
13:46:41.410 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from default folder for traceability.yml
13:46:41.411 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.server.Server start - Plugin: com.networknt.metrics.MetricsHandler
13:46:41.411 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.server.Server start - Plugin: com.networknt.exception.ExceptionHandler
13:46:41.411 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Unable to load config from externalized folder for exception.yml in ./config
13:46:41.411 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Trying to load config from classpath directory for file exception.yml
13:46:41.412 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.config.Config getConfigStream - Config loaded from default folder for exception.yml
13:46:41.435 [main] com.networknt.portal.hybrid.query-1.0.0  DEBUG org.jboss.logging logProvider - Logging Provider: org.jboss.logging.Slf4jLoggerProvider found via system property
13:46:41.438 [main] com.networknt.portal.hybrid.query-1.0.0  DEBUG io.undertow start - starting undertow server io.undertow.Undertow@ea6147e
13:46:41.444 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  org.xnio <clinit> - XNIO version 3.3.8.Final
13:46:41.450 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  org.xnio.nio <clinit> - XNIO NIO Implementation Version 3.3.8.Final
13:46:41.551 [XNIO-1 I/O-1]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-1', selector sun.nio.ch.KQueueSelectorImpl@730ba98c
13:46:41.551 [XNIO-1 I/O-3]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-3', selector sun.nio.ch.KQueueSelectorImpl@6057a30f
13:46:41.551 [XNIO-1 I/O-4]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-4', selector sun.nio.ch.KQueueSelectorImpl@4bd6949f
13:46:41.551 [XNIO-1 I/O-5]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-5', selector sun.nio.ch.KQueueSelectorImpl@4c78c6e4
13:46:41.551 [XNIO-1 I/O-2]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-2', selector sun.nio.ch.KQueueSelectorImpl@18670c05
13:46:41.551 [XNIO-1 I/O-6]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-6', selector sun.nio.ch.KQueueSelectorImpl@321eb716
13:46:41.551 [XNIO-1 I/O-7]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-7', selector sun.nio.ch.KQueueSelectorImpl@48fbc376
13:46:41.552 [XNIO-1 I/O-9]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-9', selector sun.nio.ch.KQueueSelectorImpl@1596f8ed
13:46:41.552 [XNIO-1 I/O-8]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-8', selector sun.nio.ch.KQueueSelectorImpl@77cdd8cb
13:46:41.552 [XNIO-1 I/O-10]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-10', selector sun.nio.ch.KQueueSelectorImpl@71eabef3
13:46:41.552 [XNIO-1 I/O-11]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-11', selector sun.nio.ch.KQueueSelectorImpl@776fe4
13:46:41.556 [XNIO-1 I/O-14]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-14', selector sun.nio.ch.KQueueSelectorImpl@a89028d
13:46:41.556 [XNIO-1 I/O-13]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-13', selector sun.nio.ch.KQueueSelectorImpl@5b16e4fb
13:46:41.556 [XNIO-1 I/O-12]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-12', selector sun.nio.ch.KQueueSelectorImpl@1961f4d5
13:46:41.556 [XNIO-1 I/O-15]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-15', selector sun.nio.ch.KQueueSelectorImpl@3533281c
13:46:41.556 [XNIO-1 I/O-16]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 I/O-16', selector sun.nio.ch.KQueueSelectorImpl@459d7c1f
13:46:41.556 [XNIO-1 Accept]   DEBUG org.xnio.nio run - Started channel thread 'XNIO-1 Accept', selector sun.nio.ch.KQueueSelectorImpl@143e6443
13:46:41.557 [main] com.networknt.portal.hybrid.query-1.0.0  DEBUG io.undertow start - Configuring listener with protocol HTTP for interface <IP_ADDRESS> and port 8050
HOST IP null
Http Server started on ip:<IP_ADDRESS> Port:8050
13:46:41.596 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.server.Server bind - Http Server started on ip:<IP_ADDRESS> Port:8050
Https port disabled.
13:46:41.597 [main] com.networknt.portal.hybrid.query-1.0.0  INFO  com.networknt.server.Server bind - Https port disabled.

13:46:53.015 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG io.undertow.request match - Matched prefix path /api/json for path /api/json
13:46:53.020 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.rpc.router.JsonHandler lambda$handleRequest$0 - Post method with message = {"host":"lightapi.net","service":"codegen","action":"generate","version":"0.0.1","data":[{"framework":"swagger"}]}
13:46:53.022 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.rpc.router.JsonHandler processRequest - serviceId = lightapi.net/codegen/generate/0.0.1
13:46:53.023 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG c.n.c.h.GeneratorServiceHandler validate - serviceId = lightapi.net/codegen/generate/0.0.1 serviceMap = {"schema":{"title":"Service","type":"object","properties":{"generators":{"type":"array","items":{"properties":{"model":{"type":"object"},"config":{"type":"object"},"framework":{"type":"string"}},"required":["config","framework"]},"maxItems":5,"minItems":1}}},"scope":"codegen.w"}
13:46:53.023 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG c.n.c.h.GeneratorServiceHandler validate - Skipping validation on generator request for now.
13:46:53.039 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA INFO  c.f.rocker.runtime.RockerRuntime <init> - Rocker version 0.22.0
13:46:53.040 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA INFO  c.f.rocker.runtime.RockerRuntime setReloading - Rocker template reloading not activated
com.jsoniter.spi.JsonException: Value not found: failed to get key paths from null
	at com.jsoniter.any.NotFoundAny.<init>(NotFoundAny.java:25)
	at com.jsoniter.any.Any.get(Any.java:202)
	at com.networknt.codegen.rest.SwaggerGenerator.injectEndpoints(SwaggerGenerator.java:318)
	at com.networknt.codegen.rest.SwaggerGenerator.generate(SwaggerGenerator.java:137)
	at com.networknt.codegen.handler.GeneratorServiceHandler.handle(GeneratorServiceHandler.java:91)
	at com.networknt.rpc.router.JsonHandler.processRequest(JsonHandler.java:96)
	at com.networknt.rpc.router.JsonHandler.lambda$handleRequest$0(JsonHandler.java:52)
	at io.undertow.io.AsyncReceiverImpl.receiveFullString(AsyncReceiverImpl.java:140)
	at io.undertow.io.AsyncReceiverImpl.receiveFullString(AsyncReceiverImpl.java:84)
	at com.networknt.rpc.router.JsonHandler.handleRequest(JsonHandler.java:46)
	at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:94)
	at com.networknt.audit.AuditHandler.handleRequest(AuditHandler.java:150)
	at com.networknt.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:73)
	at com.networknt.correlation.CorrelationHandler.handleRequest(CorrelationHandler.java:70)
	at com.networknt.traceability.TraceabilityHandler.handleRequest(TraceabilityHandler.java:64)
	at com.networknt.exception.ExceptionHandler.handleRequest(ExceptionHandler.java:72)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:336)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
13:46:53.074 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA ERROR c.n.c.h.GeneratorServiceHandler handle - Exception:
com.jsoniter.spi.JsonException: Value not found: failed to get key paths from null
	at com.jsoniter.any.NotFoundAny.<init>(NotFoundAny.java:25)
	at com.jsoniter.any.Any.get(Any.java:202)
	at com.networknt.codegen.rest.SwaggerGenerator.injectEndpoints(SwaggerGenerator.java:318)
	at com.networknt.codegen.rest.SwaggerGenerator.generate(SwaggerGenerator.java:137)
	at com.networknt.codegen.handler.GeneratorServiceHandler.handle(GeneratorServiceHandler.java:91)
	at com.networknt.rpc.router.JsonHandler.processRequest(JsonHandler.java:96)
	at com.networknt.rpc.router.JsonHandler.lambda$handleRequest$0(JsonHandler.java:52)
	at io.undertow.io.AsyncReceiverImpl.receiveFullString(AsyncReceiverImpl.java:140)
	at io.undertow.io.AsyncReceiverImpl.receiveFullString(AsyncReceiverImpl.java:84)
	at com.networknt.rpc.router.JsonHandler.handleRequest(JsonHandler.java:46)
	at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:94)
	at com.networknt.audit.AuditHandler.handleRequest(AuditHandler.java:150)
	at com.networknt.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:73)
	at com.networknt.correlation.CorrelationHandler.handleRequest(CorrelationHandler.java:70)
	at com.networknt.traceability.TraceabilityHandler.handleRequest(TraceabilityHandler.java:64)
	at com.networknt.exception.ExceptionHandler.handleRequest(ExceptionHandler.java:72)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:336)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
13:46:53.097 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.utility.NioUtils preVisitDirectory - Creating directory %s

13:46:53.099 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.utility.NioUtils preVisitDirectory - Creating directory %s

13:46:53.104 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.utility.NioUtils preVisitDirectory - Creating directory %s

13:46:53.105 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.utility.NioUtils preVisitDirectory - Creating directory %s

13:46:53.105 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.utility.NioUtils preVisitDirectory - Creating directory %s

13:46:53.106 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.utility.NioUtils preVisitDirectory - Creating directory %s

13:46:53.107 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.utility.NioUtils preVisitDirectory - Creating directory %s

13:46:53.107 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.utility.NioUtils preVisitDirectory - Creating directory %s

13:46:53.108 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.utility.NioUtils preVisitDirectory - Creating directory %s

13:46:53.109 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.utility.NioUtils preVisitDirectory - Creating directory %s

/tmp/3-UCxUk2Rnq--82PZOpuPg/src/test/resources/logback-test.xml
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/test/resources/config/server.yml
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/test/resources/config/client.yml
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/test/resources/config
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/test/resources
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/test
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/main/resources/logback.xml
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/main/resources/config/service.yml
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/main/resources/config/server.yml
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/main/resources/config/security.yml
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/main/resources/config/secret.yml
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/main/resources/config/oauth/secondary.crt
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/main/resources/config/oauth/primary.crt
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/main/resources/config/oauth
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/main/resources/config
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/main/resources
/tmp/3-UCxUk2Rnq--82PZOpuPg/src/main
/tmp/3-UCxUk2Rnq--82PZOpuPg/src
/tmp/3-UCxUk2Rnq--82PZOpuPg/pom.xml
/tmp/3-UCxUk2Rnq--82PZOpuPg/docker/Dockerfile-Redhat
/tmp/3-UCxUk2Rnq--82PZOpuPg/docker/Dockerfile
/tmp/3-UCxUk2Rnq--82PZOpuPg/docker
/tmp/3-UCxUk2Rnq--82PZOpuPg/build.sh
/tmp/3-UCxUk2Rnq--82PZOpuPg/README.md
/tmp/3-UCxUk2Rnq--82PZOpuPg/LICENSE
/tmp/3-UCxUk2Rnq--82PZOpuPg/.project
/tmp/3-UCxUk2Rnq--82PZOpuPg/.gitignore
/tmp/3-UCxUk2Rnq--82PZOpuPg/.classpath
/tmp/3-UCxUk2Rnq--82PZOpuPg
13:46:53.126 [XNIO-1 task-1]  RO8dPW9hTOqZ4XChVJ5agA DEBUG com.networknt.rpc.router.JsonHandler processRequest - java.nio.DirectByteBuffer[pos=0 lim=24109 cap=24109]

$node foo.js
########### HEADERS:  { connection: 'close',
  server: 'L',
  'content-type': 'application/zip',
  'content-length': '24109',
  'content-disposition': 'attachment',
  date: 'Fri, 27 Apr 2018 17:46:53 GMT' }
########### AXIOS:  23355

$ cat foo.js
const axios = require('axios');

const generate = () => {
  return axios.post('http://localhost:8050/api/json', {
      host: 'lightapi.net',
      service: 'codegen',
      action: "generate",
      version: '0.0.1',
      data: [{"framework":"swagger"}]
  }, {
      responseType: 'blob',
  });
};

  generate().then((response) => {
    console.log('########### HEADERS: ', response.headers);
    console.log('########### AXIOS: ', response.data.length);
  });

awesome thanks! i'll look at axios then.

So it may help here, but I have no idea how axios works, so cannot say what this does or if this is right. If I change responseType: 'blob' to responseType: 'arraybuffer' I see both report the same 24109 value.

I can't find their docs on this option, but just saw the enumeration here: https://github.com/axios/axios/blob/dd16944ecf70d8939ce77d79b57463e193d023a6/index.d.ts#L37 and tried a few until they lengths matched.

awesome!!! youre a life saver.


[DOCS] Specifies the title of the getting started section in Azure Marketplace and ARM template

This PR amends the getting started section titles in the Azure Marketplace related docs. The amended titles specify the subject of the getting started section – in line with the efforts of the documentation team to make getting started guides and tutorials more exact.

@szabosteve does this change need to be ported to 7.2, 7.1, 7.0, 6.7, 6.6, 6.5 and 6.4?

@szabosteve does this change need to be ported to 7.2, 7.1, 7.0, 6.7, 6.6, 6.5 and 6.4?

Yes, I'll take care of this.


Incorrect order of TLS1.3 ciphers

I came across a bug where the list of TLS ciphers for version 1.3 always comes in the same order. Moreover, this problem is present on macOS, but everything is correct on Windows.
For example, I am sending such ja3: 771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,65037-17513-65281-18-11-0-45-16-23-5-35-51-27-10-43-13,25497-29-23-24,0.
ja3 response (from https://tls.peet.ws/api/tls) comes like this: 771,4867-4865-4866-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,65037-17513-65281-18-11-0-45-16-23-5-35-51-27-10-43-13,25497-29-23-24,0.
As you can see, the first three TLS 1.3 ciphers did not return in the order in which I sent them when calling fetch.

Fixed in v1.4.7, major patches have been uploaded to the patches directory.

Fixed in v1.4.7, major patches have been uploaded to the patches directory.

Thanks. With your built addon everything works fine.
But if I compile the Node.js addon myself, I get an error: [1]    80384 segmentation fault.
Maybe there are some differences in the libcurl.a files etc between the ones the addon published on npm was built with?
For reference, I'm using macOS on ARM (M1 Max).

Fixed in v1.4.7, major patches have been uploaded to the patches directory.

Thanks. With your built addon everything works fine. But if I compile the Node.js addon myself, I get an error: [1] 80384 segmentation fault.
Maybe there are some differences in the libcurl.a files etc between the ones the addon published on npm was built with? For reference, I'm using macOS on ARM (M1 Max).

I used docker image mcandre/snek tag x86_64-apple-darwin arm64-apple-darwin to build the library for macOS. Since compilation is susceptible to environment variables and compilation options, I may upload the docker image for compilation in the future.

I may upload the docker image for compilation in the future.

Yes, it would be nice for me to be able to build a correct .node addon myself. At the moment there seems to be some issues with the build process.

major patches have been uploaded to the patches directory

I'm having trouble applying the patches :(
Could you please share the full sources of a project where I can build a modified curl?

major patches have been uploaded to the patches directory

I'm having trouble applying the patches :( Could you please share the full sources of a project where I can build a modified curl?

some Dockerfile are pushed in patches directory.

some Dockerfile are pushed in patches directory.

Thank you very much :)


Add enterprise language

This adds the enterprise language for Tidelift with the nyc tracking code.
I think we should use this in the base of the istanbuljs repo too.
@coreyfarrell @bcoe Any objections to me adding a "For Enterprise" page to the website?

Thanks @phated.  I have no objection to adding something about enterprise support to the webpage (I assume you are talking about istanbul.js.org).  Not sure how/where that should go?


Bug: Deserializing object with multiple private fields with the same name fails.

Describe the bug
Deserialization fails when processing an object with multiple private fields with the same name.
    public class Base
    {
        private bool _dirty;
    }

    public class Child : Base
    {
        private bool _dirty;
    }

    [Test]
    public void InhitanceWorks()
    {
        var config = new SerializerConfig {DefaultTargets = TargetMember.AllFields, VersionTolerance = {Mode = VersionToleranceMode.Standard}};
        var ceras = new CerasSerializer(config);
        var data = ceras.Serialize(new Child());
        var result = ceras.Deserialize<Child>(data);
    }

Additional info
Message:
    System.ArgumentException : An item with the same key has already been added.
  Stack Trace:
    ThrowHelper.ThrowArgumentException(ExceptionResource resource)
    Dictionary`2.Insert(TKey key, TValue value, Boolean add)
    Dictionary`2.Add(TKey key, TValue value)
    DynamicFormatter`1.GenerateDeserializer(CerasSerializer ceras, Schema schema, Boolean isSchemaFormatter, Boolean isStatic) line 251
    SchemaDynamicFormatter`1.ActivateSchema(Schema schema) line 150
    ISchemaTaintedFormatter.OnSchemaChanged(TypeMetaData meta) line 83
    CerasSerializer.ActivateSchemaOverride(Type type, Schema schema) line 1029
    SchemaDynamicFormatter`1.Deserialize(Byte[] buffer, Int32& offset, T& value) line 67
    ReferenceFormatter`1.Deserialize(Byte[] buffer, Int32& offset, T& value) line 288
    CerasSerializer.Deserialize[T](T& value, Byte[] buffer, Int32& offset, Int32 expectedReadLength) line 524
    CerasSerializer.Deserialize[T](Byte[] buffer) line 478
    SerializationTests.InhitanceWorks() line 47

Platform

Which version of Ceras are you using? net4.5

IIRC this bug comes from an oversight in the design of Ceras. (read: I just didn't think of that case :P)
But it is possible to fix it in multiple ways.
You could rename the field by using attributes. But I imagine that would become a hassle pretty fast.
It would probably be better to use config.OnConfigNewType, then go through each member, and if you ever encounter a member named _dirty, you'd rename it to something like "ContainingClass._dirty", for example:  Base._dirty or Child._dirty.

It's a pretty rare edge case and so far have only seen it a couple of times
in projects I control and can change. Happy to close this as a workaround
is available should I come across it again in code I can't modify.
Appreciate your help. S
On Sun, 11 Apr 2021, 10:24 Moritz Staudinger, @.***>
wrote:

IIRC this bug comes from an oversight in the design of Ceras. (read: I
just didn't think of that case :P)
But it is possible to fix it in multiple ways.
You could rename the field by using attributes. But I imagine that would
become a hassle pretty fast.
It would probably be better to use config.OnConfigNewType, then go through
each member, and if you ever encounter a member named _dirty, you'd
rename it to something like "ContainingClass._dirty", for example:
Base._dirty or Child._dirty.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/rikimaru0345/Ceras/issues/91#issuecomment-817277010,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADGXYEZXUQENSILXR4IT7DDTIFTD3ANCNFSM4WYXUQHA
.


doc: fix python version in github flow

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thank you!


LIVY-804: Add session ID to LineBufferedStream logs

If multiple sessions are started at once, it can be difficult to corelate the stdout logs with the sessions that started the spark-submit process. This PR adds the session ID as a prefix to the LineBufferedStream logs.
https://issues.apache.org/jira/browse/LIVY-804
Manual Log Test

@jerryshao @mgaido91 Can you please review this?

@jerryshao @mgaido91 Can you please review this?

Codecov Report

Merging #315 (de1aa21) into master (4d8a912) will decrease coverage by 0.04%.
The diff coverage is 80.00%.

@@             Coverage Diff              @@
##             master     #315      +/-   ##
============================================
- Coverage     68.50%   68.45%   -0.05%
+ Complexity      841      839       -2
============================================
  Files           103      103
  Lines          5940     5947       +7
  Branches        898      898
============================================
+ Hits           4069     4071       +2
- Misses         1312     1314       +2
- Partials        559      562       +3

Impacted Files
Coverage Δ
Complexity Δ

...e/livy/server/interactive/InteractiveSession.scala
69.76% <0.00%> (ø)
51.00 <0.00> (ø)

...ala/org/apache/livy/utils/LineBufferedStream.scala
81.81% <80.00%> (+1.33%)
5.00 <1.00> (ø)

...la/org/apache/livy/utils/LineBufferedProcess.scala
88.88% <85.71%> (+3.17%)
8.00 <1.00> (ø)

...la/org/apache/livy/server/batch/BatchSession.scala
86.73% <100.00%> (ø)
14.00 <0.00> (ø)

...la/org/apache/livy/utils/SparkProcessBuilder.scala
54.44% <100.00%> (ø)
11.00 <0.00> (ø)

core/src/main/scala/org/apache/livy/Logging.scala
75.00% <0.00%> (-8.34%)
0.00% <0.00%> (ø%)

...ain/java/org/apache/livy/rsc/driver/RSCDriver.java
78.75% <0.00%> (-2.09%)
43.00% <0.00%> (-3.00%)

...ain/scala/org/apache/livy/utils/SparkYarnApp.scala
73.75% <0.00%> (-1.25%)
40.00% <0.00%> (ø%)

...main/scala/org/apache/livy/server/LivyServer.scala
31.25% <0.00%> (-0.45%)
11.00% <0.00%> (ø%)

...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala
65.11% <0.00%> (+2.32%)
0.00% <0.00%> (ø%)

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d8a912...de1aa21. Read the comment docs.

Codecov Report

Merging #315 (de1aa21) into master (4d8a912) will decrease coverage by 0.04%.
The diff coverage is 80.00%.

@@             Coverage Diff              @@
##             master     #315      +/-   ##
============================================
- Coverage     68.50%   68.45%   -0.05%
+ Complexity      841      839       -2
============================================
  Files           103      103
  Lines          5940     5947       +7
  Branches        898      898
============================================
+ Hits           4069     4071       +2
- Misses         1312     1314       +2
- Partials        559      562       +3

Impacted Files
Coverage Δ
Complexity Δ

...e/livy/server/interactive/InteractiveSession.scala
69.76% <0.00%> (ø)
51.00 <0.00> (ø)

...ala/org/apache/livy/utils/LineBufferedStream.scala
81.81% <80.00%> (+1.33%)
5.00 <1.00> (ø)

...la/org/apache/livy/utils/LineBufferedProcess.scala
88.88% <85.71%> (+3.17%)
8.00 <1.00> (ø)

...la/org/apache/livy/server/batch/BatchSession.scala
86.73% <100.00%> (ø)
14.00 <0.00> (ø)

...la/org/apache/livy/utils/SparkProcessBuilder.scala
54.44% <100.00%> (ø)
11.00 <0.00> (ø)

core/src/main/scala/org/apache/livy/Logging.scala
75.00% <0.00%> (-8.34%)
0.00% <0.00%> (ø%)

...ain/java/org/apache/livy/rsc/driver/RSCDriver.java
78.75% <0.00%> (-2.09%)
43.00% <0.00%> (-3.00%)

...ain/scala/org/apache/livy/utils/SparkYarnApp.scala
73.75% <0.00%> (-1.25%)
40.00% <0.00%> (ø%)

...main/scala/org/apache/livy/server/LivyServer.scala
31.25% <0.00%> (-0.45%)
11.00% <0.00%> (ø%)

...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala
65.11% <0.00%> (+2.32%)
0.00% <0.00%> (ø%)

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d8a912...de1aa21. Read the comment docs.

not sure how convenient this change is. There may be different solutions, such as retrieving the logs of spark apps directly from spark, where they are separated. I'll leave it to other commuters if they judge this change is helpful.

@mgaido91 Thanks for the review.
When submitting in cluster mode in YARN, from Spark we get driver logs only after application master is launched successfully. Prior to this spark-submit needs to submit the application to YARN. The spark-submit process's logs are printed using LineBufferedStream. When submitting multiple sessions to Livy, these spark-submit processes can error out whose logs can't be correlated to session IDs.
This has helped us in debugging issues like these at Microsoft.


Laravel 10.8.0 Shift

This pull request includes updates for the recent minor version release of Laravel as well as updates to your package dependencies. Feel free to commit any additional changes to the shift-ci-v10.8.0 branch.
Before merging, you need to:

Checkout the shift-ci-v10.8.0 branch
Review all pull request comments for additional changes
Run composer update
Thoroughly test your application (no tests?, no CI?)

:x: Shift was unable to run composer update. Please be sure to run composer update before merging to ensure your dependency changes are fully compatible and reflected in your composer.lock file.


Prepare @plone/volto-slate for release, add a bit more of docs, and ESlint

@stevepiercy You can review this one later, but this README is not that important.


Break up sharding and master/slave handling

The intent of this is to split master/slave logic from sharding logic allowing either to be understood more easily.
We introduce a BaseConfig module which must be included before the sharding (ConnectionSwitching) and master/slave (SlaveDb) modules. This ensures that the connection_config method will work if either module is missing and regardless of the order in which the latter two modules are included.

:+1: on green.

the defined? check was for silencing warnings
Thread.current.key? would be the same ... but not needed here
On Mon, Nov 6, 2017 at 7:54 AM, Benjamin Quorning<EMAIL_ADDRESS>wrote:

@bquorning commented on this pull request.
In lib/active_record_shards/slave_db.rb
https://github.com/zendesk/active_record_shards/pull/155#discussion_r149118305
:

 if self == ActiveRecord::Base || !switch_to_slave

   yield

 else

   readonly.scoping(&block)

 end

ensure

 @disallow_slave -= 1 if which == :master

 switch_connection(slave: old_options)

end

def current_slave_selection=(on_slave)

 Thread.current[:slave_selection] = on_slave

end

def current_slave_selection

 Thread.current[:slave_selection] ||= false

That’s true. With instance variable memorization you’d check if defined?
before memorizing potentially falsey values. Is there something similar for
thread variables?
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
https://github.com/zendesk/active_record_shards/pull/155#discussion_r149118305,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAsZzN_4ujIpyxymR7zVa0Jp4YUbsRPks5szyuxgaJpZM4QTIPC
.

the defined? check was for silencing warnings

I was referring to this pattern: https://www.justinweiss.com/articles/4-simple-memoization-patterns-in-ruby-and-one-gem#what-about-nil
Unrelated: Your Github thread comments seems to somehow break out of the thread…


Strategy - Edit Price validation

When editing prices of a strategy, there is a price validation process which prevents to save the changes if one of the order has prices set to 0.
current:
conditions that are applied: IF budget >0, force price >0.

IF both prices are indicating "enter price", the CTA is enabled

updated:

ignore the budget requirement
allow 1 order to be set to price 0 (but only 1 can be set to 0)
following point 2, IF both prices are null in a state of "enter price", disable the CTA
when editing prices,

IF buy.price = 0 AND sell.price = 0, show current warning "Price must be greater than 0" and disable CTA.
Else IF buy.price = 0 OR sell.price = 0:
-- display warning "Buy prices are set to zero" / "Sell prices are set to zero"
-- display checkbox "I've reviewed the warning(s) but choose to proceed."
-- disable CTA until checkbox is checked.

will be fixed with #1253


Add Extension icon

An icon for each extension shows in list of installed extensions and on the VS Code Extension Marketplace. It'd be nice to add one for vscode-lowercase.
If you'd like me to create a quick one and open a PR with it - let me know. :-)

I don't feel particularly creative at the moment :stuck_out_tongue: but I guess that might be an interesting idea. Yes, feel free to open the PR!


Changed logger signature to be sf 5.4 compatible

Codecov Report
All modified lines are covered by tests :white_check_mark:

Comparison is base (9adabb6) 84.54% compared to head (f35419a) 84.54%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff            @@
##               main      #24   +/-   ##
=========================================
  Coverage     84.54%   84.54%
  Complexity      376      376
=========================================
  Files            25       25
  Lines           893      893
=========================================
  Hits            755      755
  Misses          138      138

Files
Coverage Δ

src/Logger/TestLogger.php
83.33% <100.00%> (ø)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Ops Server scripts not honoring pauses in build process

Install scripts are not honoring pauses and are not waiting for human input
Affected OS: Windows 7 64-bit.
VM created using internal ECS system.

Please be specific in where in the install process the pause is not be honored. Thanks.
-------- Original message --------
From: Avi Cueva
Date:08/21/2015 9:59 AM (GMT-08:00)
To: Esri/ops-server-config
Cc: Eric Linz
Subject: [ops-server-config] Ops Server scripts not honoring pauses in build process (#940)
Install scripts are not honoring pauses and are not waiting for human input
Affected OS: Windows 7 64-bit.
VM created using internal ECS system.
—
Reply to this email directly or view it on GitHubhttps://github.com/Esri/ops-server-config/issues/940.


Invalid SignalRTemplate Error while using Python

This is for language: Python
https://docs.microsoft.com/en-us/azure/azure-signalr/signalr-quickstart-azure-functions-python
I'm trying to create "negotiate" function as mentioned in doc using below command:
> func new -n negotiate -t SignalRNegotiateHTTPTrigger

This has resulted in an error saying:
Can't find template "SignalRNegotiateHTTPTrigger" in "Python"
In one of the community forums I have seen that we need to use another template name as : SignalRTrigger
https://issueexplorer.com/issue/MicrosoftDocs/azure-docs/82501
> func new -n negotiate -t SignalRTrigger

Even this has resulted in similar error saying:
Can't find template "SignalRTrigger" in "Python"
Please reach out ASAP.
Thanks :)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f04b2376-8901-ee7a-86a7-500b09cfcf94
Version Independent ID: 01198991-2c3b-9873-3b2c-233b7b9a0ff0
Content: Azure SignalR Service serverless quickstart - Python
Content Source: articles/azure-signalr/signalr-quickstart-azure-functions-python.md
Service: signalr
GitHub Login: @anthonychu
Microsoft Alias: antchu

@bnvsp Thank you for bringing this to our attention. I was able to reproduce the same error.

As a result, I have tagged this as a doc bug and assigned it directly to the doc author.

@zackliu @JialinXin Can you please take a look?

@bnvsp Python is missing SignalRNegotiateHTTPTrigger template. Negotiation function is actually using HttpTrigger, so use func new -n negotiate -t HttpTrigger and others are the same. I'm updating the document.

@zackliu Thanks for clarification, will use it as suggested.

Thanks for providing feedback that helps improve our documentation. This issue has been resolved and we are closing the issue. #please-close


import module fails in ipython console; works when running a file

From<EMAIL_ADDRESS>on 2014-12-03T14:40:23Z
Spyder Version:  2.3.1
Python Version:  2.7.8
Qt Version    :  4.8.6, PyQt4 (API v2) 4.10.4 on Windows
pyflakes >=0.5.0:  0.8.1 (OK)
pep8 >=0.6      :  1.5.7 (OK)
IPython >=0.13  :  2.3.1 (OK)
pygments >=1.6  :  2.0.1 (OK)
pandas >=0.13.1 :  0.14.1 (OK)
sphinx >=0.6.6  :  1.2.3 (OK)
rope >=0.9.2    :  0.9.4 (OK)
matplotlib >=1.0:  1.4.2 (OK)
sympy >=0.7.0   :  0.7.5 (OK)
pylint >=0.25   :  None (OK)
What steps will reproduce the problem?
Open up an ipython console; try to type 'import' (without quotes) and as soon as you add a space to put the name of the library, you get an error.
What is the expected output? What do you see instead?
Traceback (most recent call last):
File "C:\Anaconda\lib\site-packages\IPython\qt\console\frontend_widget.py", line 60, in highlightBlock
super(FrontendHighlighter, self).highlightBlock(string)
File "C:\Anaconda\lib\site-packages\IPython\qt\console\pygments_highlighter.py", line 115, in highlightBlock
for token, text in self._lexer.get_tokens(string):
File "C:\Anaconda\lib\site-packages\pygments\lexer.py", line 192, in streamer
for i, t, v in self.get_tokens_unprocessed(text):
File "C:\Anaconda\lib\site-packages\IPython\qt\console\pygments_highlighter.py", line 30, in get_tokens_unprocessed
for item in action(self, m):
TypeError: 'NoneType' object is not callable
Traceback (most recent call last):
File "C:\Anaconda\lib\site-packages\IPython\qt\console\frontend_widget.py", line 60, in highlightBlock
super(FrontendHighlighter, self).highlightBlock(string)
File "C:\Anaconda\lib\site-packages\IPython\qt\console\pygments_highlighter.py", line 115, in highlightBlock
for token, text in self._lexer.get_tokens(string):
File "C:\Anaconda\lib\site-packages\pygments\lexer.py", line 192, in streamer
for i, t, v in self.get_tokens_unprocessed(text):
File "C:\Anaconda\lib\site-packages\IPython\qt\console\pygments_highlighter.py", line 30, in get_tokens_unprocessed
for item in action(self, m):
TypeError: 'NoneType' object is not callable
Traceback (most recent call last):
File "C:\Anaconda\lib\site-packages\IPython\qt\console\frontend_widget.py", line 60, in highlightBlock
super(FrontendHighlighter, self).highlightBlock(string)
File "C:\Anaconda\lib\site-packages\IPython\qt\console\pygments_highlighter.py", line 115, in highlightBlock
for token, text in self._lexer.get_tokens(string):
File "C:\Anaconda\lib\site-packages\pygments\lexer.py", line 192, in streamer
for i, t, v in self.get_tokens_unprocessed(text):
File "C:\Anaconda\lib\site-packages\IPython\qt\console\pygments_highlighter.py", line 30, in get_tokens_unprocessed
for item in action(self, m):
TypeError: 'NoneType' object is not callable
Please provide any additional information below
. When you run a script, this works.
Original issue: http://code.google.com/p/spyderlib/issues/detail?id=2073

From ccordoba12 on 2014-12-04T13:07:16Z
Please update to 2.3.2. This was already fixed :-)
Status: Duplicate
Mergedinto: 2042

From<EMAIL_ADDRESS>on 2014-12-03T11:42:18Z
Note that this fails in Spyder, but not when running ipython from a Windows 7 console.


Check .ruby-version for TargetRubyVersion

Feature: Expected behavior
Check .ruby-version for TargetRubyVersion once it's not defined in .rubocop.yml
Actual behavior
Uses 1.9 as a default of TargetRubyVersion
Steps to reproduce the problem
Do not define TargetRubyVersion.
Use ruby named parameters def method(param1:, param2:)
And you'll get offence
... E: unexpected token tCOMMA
(Using Ruby 2.0 parser; configure using TargetRubyVersion parameter, under AllCops)
  def def method(param1:, param2:)

RuboCop version
rubocop -V
0.38.0 (using Parser <IP_ADDRESS>, running on ruby 2.2.2 x86_64-linux)

PR welcome! :-)

I'd actually prefer this not be a feature. There are a half dozen (at least) different ways that different tools try to guess ruby versions (Gemfile, gemset, .ruby-version, etc). A required and explicit ruby version in the config file is the simplest solution here. If rubocop tries to guess ruby versions I think it will probably lead to some confused issue reports down the road too.

@backus agree

@backus so let make sure I'm following your thinking: Because we have too many ways to define the Ruby version targeted by a project, we should add another one?

This is especially important for people who have rubocop config gems or consultants who use the same rubocop config for every project.  With an explicit TargetRubyVersion now certain projects need to have the TargetRubyVersion overridden in a custom (for the project) .rubocop.yml when before the entire Rubocop config could have just been used verbatim.
If the TargetRubyVersion was pulled from .ruby-version or another standard location, then we're back to being able to share the entire config with no overrides.

@thegranddesign My point is that magically trying to determine the ruby version maybe saves you ten seconds of time that would have otherwise gone into dealing with a very obvious and explicit error message:

If you're a contractor who really spreads himself thin across different client projects then my most generous estimate of total time lost might be a minute or two per week.
Having a hidden file determine rubocop's parser is unnecessary if all it is doing is saving a few seconds for users who work on many projects. The downside is confusing users who have to track down what is going on and learn that rubocop added this feature.

@thegranddesign Also if you really need this in your case, why don't you just add some script like this to your project?
# Get ruby version
File.read('.ruby-version')
target_ruby_version = File.read('.ruby-version').to_f

# Write ruby version to rubcop config
config = YAML.load_file(".rubocop.yml")
config['AllCops'] ||= {}
config.fetch('AllCops')['TargetRubyVersion'] = target_ruby_version

# Write updated settings
File.write('.rubcop.yml', all_cops)

If you use rake then you can just make this a prerequisite for the rubocop task. Otherwise you could just put this somewhere like bin/style which would just run this code and then invoke rubocop.

@backus

Having a hidden file determine rubocop's parser is unnecessary

Firstly, of course it's unnecessary.  No one said the user had to use a .ruby-version file in order for Rubocop to get the Ruby version.  All I said was that if it's there, and the user has specified that they want the value pulled from it, that it should be used.
Secondarily, the user put that hidden file there presumably because they wanted to lock the Ruby version.  It's not like that file just magically appears one day and they're like "Gee all of a sudden my Rubocop is parsing with a different Ruby version.  I wonder why."
In fact, if Rubocop didn't auto-detect the Ruby version, that is the surprising behavior.  Am I going to be surprised that I locked my project to Ruby 2.3 and Rubocop is using a 2.0 parser?  Absolutely.  Am I going to be surprised that I locked my project to 2.0 and Rubocop is complaining about optional keyword arguments?  Not at all.

If you use rake then you can just make this a prerequisite for the rubocop task.

All of the steps you outlined are true.  One could do that, but it doesn't make them any less of a hack.

that would have otherwise gone into dealing with a very obvious and explicit error message

I don't understand how having Rubocop auto-detect the Ruby version, or have an explicit source specified like this:
AllCops:
  TargetRubyVersion: 'gemfile'

Which would cause an error like:
example.rb:1:9: E: unexpected token tIDENTIFIER
(Using Ruby 2.0 parser set in your Gemfile; configure by changing the TargetRubyVersion
parameter under AllCops or by editing your Gemfile)
puts 0.1r

(or some other error message) is in the least bit confusing.

Eh I disagree on your point about what is surprising behavior and what isn't. Regardless, this issue is about automatically detecting the ruby version in the absence of TargetRubyVersion in the config file. The TargetRubyVersion: 'gemfile' example you gave is less offensive IMO.
This is turning too much into https://xkcd.com/386/ for me so I'm going to unsubscribe from this issue. I'm just some guy who disagreed by the way, I have zero influence on rubocop. In fact it looks like @bbatsov said a PR is welcome so you could probably just implement what you want and get it merged.

I vote for relying on .ruby-version as the default if present, and falling back to a default if not. It's simple, specific, and well-used by other tools. Parsing the Gemfile would be unnecessarily complicated, but pulling from the .ruby-version is easy-peasy.
Personally, I find that rubocop doesn't do this surprising. And requiring separate declaration is not DRY, which itself leads to maintenance overhead / bugs.

Then PR it

I'm not aware of many other tools that use .ruby-version so I'd be curious of what you're referring to besides maybe bundler and rails.

Personally, I find that rubocop doesn't do this surprising.

If you think it is an obvious feature then that is fine but I don't think a lack of magic is surprising.

1 minute per week * N users * M years = several lifetimes.

Heh well personally I find that a very liberal calculation of the pay off here. I will reiterate though that @bbatsov said PRs welcome so you are welcome to implement this I think.

.ruby-version was created in 2012 https://gist.github.com/fnichol/1912050 to consolidate the various ruby version manager configuration files: rvm (.rvmrc), rbenv (.rbenv-version), rbfu (.rbfu-version), uru. Some other tools have picked up on it, e.g. Circle CI https://circleci.com/docs/unrecognized-ruby-version/
The only other way to declare the version, to my knowledge, is the Gemfile ruby declaration. That's helpful too, because bundler enforces it, but you can consolidate that simply: ruby File.read('.ruby-version').strip  whereas accessing the reverse requires being able to host the Gemfile dsl, or doing something worse like using regex :P https://github.com/aripollak/rbenv-bundler-ruby-version/blob/master/bin/rbenv-bundler-ruby-version

As originally proposed by @astrauka and properly argumented by @thegranddesign & @Empact, I think we should go ahead and (as @bbatsov suggests) submit a PR with this change (check .ruby-version file to set the TargetRubyVersion setting).
Is someone feeling like giving it a stab? ;)

Thanks @mikegee @pclalv @backus!

I had this same issue and determined that rubocop wasn't giving a clear definition to the problem. It had nothing to do with my ruby version. It had to do with a mistake in my code. Once I cleared up the mistake, the rubocop warning went away.

I had this same issue and determined that rubocop wasn't giving a clear definition to the problem. It had nothing to do with my ruby version. It had to do with a mistake in my code. Once I cleared up the mistake, the rubocop warning went away.

I keep getting this message even with defining ruby version. I guess I should look deeper into the code, who knows what rubocop complain about :) But it's not about ruby version, even if it say so...

I keep getting this message even with defining ruby version. I guess I should look deeper into the code, who knows what rubocop complain about :) But it's not about ruby version, even if it say so...

@alx3dev I had something similar and I found a weird "fix" to it: define an empty TargetRubyVersion. Without it, it reverts to 2.5; with it, it reads .ruby-version. It's most probably a bug, since with 0.66.0 it works, but not with 1.39.0; but in that later project, it solved my problem.
AllCops:
  TargetRubyVersion:


yarn iOS fails for open source - due to fonts and GoogleService-Info.plist

while yarn start works
yarn ios fails:
The following build commands failed:
	CpResource DERIVED/Build/Products/Debug-iphonesimulator/Artsy.app/Unica77LL-MediumItalic.otf WORKSPACE/eigen/ios/Artsy/View_Controllers/Live_Auctions/Resources/Artsy/fonts/Unica77LL-MediumItalic.otf (in target 'Artsy' from project 'Artsy')
	CpResource DERIVED/Build/Products/Debug-iphonesimulator/Artsy.app/Unica77LL-Regular.otf WORKSPACE/eigen/ios/Artsy/View_Controllers/Live_Auctions/Resources/Artsy/fonts/Unica77LL-Regular.otf (in target 'Artsy' from project 'Artsy')
	CpResource DERIVED/Build/Products/Debug-iphonesimulator/Artsy.app/Unica77LL-Medium.otf WORKSPACE/eigen/ios/Artsy/View_Controllers/Live_Auctions/Resources/Artsy/fonts/Unica77LL-Medium.otf (in target 'Artsy' from project 'Artsy')
	CpResource DERIVED/Build/Products/Debug-iphonesimulator/Artsy.app/Unica77LL-Italic.otf WORKSPACE/eigen/ios/Artsy/View_Controllers/Live_Auctions/Resources/Artsy/fonts/Unica77LL-Italic.otf (in target 'Artsy' from project 'Artsy')
	CopyPlistFile DERIVED/Build/Products/Debug-iphonesimulator/Artsy.app/GoogleService-Info.plist WORKSPACE/eigen/ios/Artsy/GoogleService-Info.plist (in target 'Artsy' from project 'Artsy')

where

WORKSPACE - source clone directory
DERIVED - Xcode derived data

thank you!


Back-off pulling image "alpine:3.10"

What happened?
What command can be used to modify the image name in initContainers?

What did you expect to happen?
modify the image name in initContainers
Environment
Kubernetes version:
$ kubectl version

Training Operator version:
$ kubectl get pods -n kubeflow -l control-plane=kubeflow-training-operator -o jsonpath="{.items[*].spec.containers[*].image}"
kubeflow/training-operator:v1-855e096

Training Operator Python SDK version:
$ pip show kubeflow-training

Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍

Modify training-operator config
set --pytorch-init-container-image=xxx in commd

Modify training-operator config set --pytorch-init-container-image=xxx in commd

Yes, you can pass an arbitrary initContainer image via the operator command: https://github.com/kubeflow/training-operator/blob/ff19a10842deba1fdb553a8b89e332731bf4f3ad/cmd/training-operator.v1/main.go#L98-L99
/remove-kind bug
/kind question

training-operator config

training-operator config   Where is it?  I found deploymenty.yaml 。But I don't know how to set pytorch-init-container-image

Modify training-operator config set --pytorch-init-container-image=xxx in commd

/remove-kind bug

Modify training-operator config set --pytorch-init-container-image=xxx in commd

Yes, you can pass an arbitrary initContainer image via the operator command:
https://github.com/kubeflow/training-operator/blob/ff19a10842deba1fdb553a8b89e332731bf4f3ad/cmd/training-operator.v1/main.go#L98-L99
/remove-kind bug /kind question

How to use this command? I only found deploymenty.yaml.

I couldn't find the training operator config, use : kubectl get cm -A
Can you explain it in detail? Thank you

Modify training-operator config set --pytorch-init-container-image=xxx in commd

/remove-kind bug

Modify training-operator config set --pytorch-init-container-image=xxx in commd

Yes, you can pass an arbitrary initContainer image via the operator command:
https://github.com/kubeflow/training-operator/blob/ff19a10842deba1fdb553a8b89e332731bf4f3ad/cmd/training-operator.v1/main.go#L98-L99
/remove-kind bug /kind question

How to use this command? I only found deploymenty.yaml.

I couldn't find the training operator config, use : kubectl get cm -A
Can you explain it in detail? Thank you

@lizu18xz

@lizu18xz
kubectl edit deploy training-operator -n kubeflow
    spec:
      containers:
      - command:
        - /manager
        - --pytorch-init-container-image=your-image

/remove-label lifecycle/needs-triage

Feel free to re-open this issue.


Update readme with a bit of sales

Closes #235
I pulled the internal instructions out to a separate document, since that was causing some confusion/distraction for external users in testing.
I also added a bit more content to the opening paragraph... a very low-key explanation of the main goals right at the top.
I thought it made sense to do this before we got the next release out!

Added a small update so gha stops complaining.


Apache Drill vs textQL

Hi
Seems that all or most of the capabilities of your tool being already covered by the Apache Drill + in addition Apache Drill can do a join between csv table to oracle or any other data source that you will define.
Regards,
Dor

Duplicate of #71


trouble building on Ubuntu 18.04.2 LTS

We ran into a little bit of trouble building the source on an AWS EC2 instance.
we needed to

run sudo apt update else gcc would not install
use sudo to run app-get

Attached is a shell script that automates the download and build. github does not allow files with '.sh' suffix to be uploaded
Kind regards
Andy
marginPolish.sh.txt

Thank you for reporting this Andy! We will update the installation commands during our next round of update.