uploadFromStream failed

I am not familar with Java code,  sorry for the inconvenience
Environment:
windows 10 22H2
docker desktop minio
docker destktop databend
mysql 5.7.37
sync just one table, 200M
Error Message:
uploadFromStream failed
{"timestamp":"2023-10-10T13:36:01.414+08:00","sequence":134,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.pipeline.ChangeEventSourceCoordinator","level":"WARN","message":"Coordinator didn't stop in the expected time, shutting down executor now","threadName":"pool-7-thread-1","threadId":31,"mdc":{},"ndc":"","hostName":"desktop-bpsk61g","processName":"io.debezium.server.Main","processId":22964}
{"timestamp":"2023-10-10T13:36:06.185+08:00","sequence":135,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.pipeline.source.AbstractSnapshotChangeEventSource","level":"WARN","message":"Snapshot was interrupted before completion","threadName":"debezium-mysqlconnector-sales-change-event-source-coordinator","threadId":39,"mdc":{"dbz.taskId":"0","dbz.connectorName":"sales","dbz.connectorType":"MySQL","dbz.connectorContext":"snapshot"},"ndc":"","hostName":"desktop-bpsk61g","processName":"io.debezium.server.Main","processId":22964}
{"timestamp":"2023-10-10T13:36:06.187+08:00","sequence":136,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.pipeline.ChangeEventSourceCoordinator","level":"WARN","message":"Change event source executor was interrupted","threadName":"debezium-mysqlconnector-sales-change-event-source-coordinator","threadId":39,"mdc":{"dbz.taskId":"0","dbz.connectorName":"sales","dbz.connectorType":"MySQL","dbz.connectorContext":"snapshot"},"ndc":"","hostName":"desktop-bpsk61g","processName":"io.debezium.server.Main","processId":22964,"exception":{"refId":1,"exceptionType":"java.lang.InterruptedException","message":null,"frames":[{"class":"java.lang.Object","method":"wait0"},{"class":"java.lang.Object","method":"wait","line":366},{"class":"io.debezium.connector.base.ChangeEventQueue","method":"doEnqueue","line":204},{"class":"io.debezium.connector.base.ChangeEventQueue","method":"enqueue","line":169},{"class":"io.debezium.pipeline.EventDispatcher$BufferingSnapshotChangeRecordReceiver","method":"changeRecord","line":440},{"class":"io.debezium.pipeline.EventDispatcher$1","method":"changeRecord","line":166},{"class":"io.debezium.relational.RelationalChangeRecordEmitter","method":"emitReadRecord","line":90},{"class":"io.debezium.relational.RelationalChangeRecordEmitter","method":"emitChangeRecords","line":50},{"class":"io.debezium.pipeline.EventDispatcher","method":"dispatchSnapshotEvent","line":155},{"class":"io.debezium.relational.RelationalSnapshotChangeEventSource","method":"createDataEventsForTable","line":407},{"class":"io.debezium.relational.RelationalSnapshotChangeEventSource","method":"createDataEvents","line":316},{"class":"io.debezium.relational.RelationalSnapshotChangeEventSource","method":"doExecute","line":132},{"class":"io.debezium.pipeline.source.AbstractSnapshotChangeEventSource","method":"execute","line":76},{"class":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"doSnapshot","line":155},{"class":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"executeChangeEventSources","line":137},{"class":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"lambda$start$0","line":109},{"class":"java.util.concurrent.Executors$RunnableAdapter","method":"call","line":577},{"class":"java.util.concurrent.FutureTask","method":"run","line":317},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","line":1144},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","line":642},{"class":"java.lang.Thread","method":"run","line":1589}]}}

@peleusj Sorry for late reponse, I will look into this issue.

Hi @peleusj , there is Change event source executor was interrupted log and it seems your debezium mysql source connector may config error. Would like to share your conf/application.properities?

Thanks for your detailed response.
My configuration files are on my work computer, and I'll do some tests tomorrow following your suggestions.
If there are still issues, I'll do my best to provide more logs.
Thanks again for your time.

@hantmac Sorry for the delay, didn't have time to run the test last Friday.
I run test again this morning, uploadFromStream failed didn't occur as I run databend on 'fs', but Change event source executor was interrupted still occured.
I double checked conf/application.properities, it seems no problem, and the local databend database successfully retrive the table structure of the source mysql database table, just no data flushed in.
The source mysql database is a RDS of ctyun.cn.
Logs:
{"timestamp":"2023-10-16T13:13:21.263+08:00","sequence":129,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.apache.kafka.connect.runtime.WorkerConfig","level":"WARN","message":"The worker has been configured with one or more internal converter properties ([internal.key.converter, internal.value.converter]). Support for these properties was deprecated in version 2.0 and removed in version 3.0, and specifying them will have no effect. Instead, an instance of the JsonConverter with schemas.enable set to false will be used. For more information, please visit http://kafka.apache.org/documentation/#upgrade and consult the upgrade notesfor the 3.0 release.","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":27399}
{"timestamp":"2023-10-16T13:13:21.275+08:00","sequence":130,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.apache.kafka.connect.runtime.WorkerConfig","level":"WARN","message":"Variables cannot be used in the 'plugin.path' property, since the property is used by plugin scanning before the config providers that replace the variables are initialized.

In the directory which you run debezium-server-databend, it will generate a data dir.

@hantmac Thanks. I remove the data dir and run test again, here is the logs:
{"timestamp":"2023-10-16T14:32:06.305+08:00","sequence":129,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.apache.kafka.connect.runtime.WorkerConfig","level":"WARN","message":"The worker has been configured with one or more internal converter properties ([internal.key.converter, internal.value.converter]). Support for these properties was deprecated in version 2.0 and removed in version 3.0, and specifying them will have no effect. Instead, an instance of the JsonConverter with schemas.enable set to false will be used. For more information, please visit http://kafka.apache.org/documentation/#upgrade and consult the upgrade notesfor the 3.0 release.","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":31916}
{"timestamp":"2023-10-16T14:32:06.312+08:00","sequence":130,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.apache.kafka.connect.runtime.WorkerConfig","level":"WARN","message":"Variables cannot be used in the 'plugin.path' property, since the property is used by plugin scanning before the config providers that replace the variables are initialized. The raw value 'null' was used for plugin scanning, as opposed to the transformed value 'null', and this may cause unexpected results.","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":31916}
{"timestamp":"2023-10-16T14:32:20.35+08:00","sequence":131,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.connector.mysql.MySqlSnapshotChangeEventSource","level":"WARN","message":"Tables were locked explicitly, but to get a consistent snapshot we cannot release the locks until we've read all tables.","threadName":"debezium-mysqlconnector-from_mysql-change-event-source-coordinator","threadId":26,"mdc":{"dbz.taskId":"0","dbz.connectorName":"from_mysql","dbz.connectorType":"MySQL","dbz.connectorContext":"snapshot"},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":31916}
{"timestamp":"2023-10-16T14:32:23.732+08:00","sequence":132,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.server.databend.tablewriter.RelationalTable","level":"WARN","message":"Loaded Databend table debezium.debezium.from_mysql_scp_sale_t_xs_stock \nColumns:{contract_amount=DatabendRawType{type=contract_amount, isNullable=false}, avble_amount=DatabendRawType{type=avble_amount, isNullable=false}, owner_id=DatabendRawType{type=owner_id, isNullable=false}, weight_range_max=DatabendRawType{type=weight_range_max, isNullable=false}, stock_zone_name=DatabendRawType{type=stock_zone_name, isNullable=false}, tolerance_range=DatabendRawType{type=tolerance_range, isNullable=false}, avble_pound_weight=DatabendRawType{type=avble_pound_weight, isNullable=false}, contract_pound_weight=DatabendRawType{type=contract_pound_weight, isNullable=false}, stock_zone_id=DatabendRawType{type=stock_zone_id, isNullable=false}, prod_area_id=DatabendRawType{type=prod_area_id, isNullable=false}, lading_amount=DatabendRawType{type=lading_amount, isNullable=false}, workgroup_name=DatabendRawType{type=workgroup_name, isNullable=false}, product_texture_id=DatabendRawType{type=product_texture_id, isNullable=false}, plan_pound_weight=DatabendRawType{type=plan_pound_weight, isNullable=false}, weight_range_min=DatabendRawType{type=weight_range_min, isNullable=false}, sale_return_amount=DatabendRawType{type=sale_return_amount, isNullable=false}, expand1=DatabendRawType{type=expand1, isNullable=false}, product_brand_name=DatabendRawType{type=product_brand_name, isNullable=false}, sku_id=DatabendRawType{type=sku_id, isNullable=false}, expand2=DatabendRawType{type=expand2, isNullable=false}, create_user_code=DatabendRawType{type=create_user_code, isNullable=false}, version=DatabendRawType{type=version, isNullable=false}, price_pound=DatabendRawType{type=price_pound, isNullable=false}, manual_manager_weight=DatabendRawType{type=manual_manager_weight, isNullable=false}, warehouse_name=DatabendRawType{type=warehouse_name, isNullable=false}, product_texture_name=DatabendRawType{type=product_texture_name, isNullable=false}, supply_meter_weight=DatabendRawType{type=supply_meter_weight, isNullable=false}, sale_lock_flag=DatabendRawType{type=sale_lock_flag, isNullable=false}, lock_pound_weight=DatabendRawType{type=lock_pound_weight, isNullable=false}, purchase_date=DatabendRawType{type=purchase_date, isNullable=false}, average_pound_cost=DatabendRawType{type=average_pound_cost, isNullable=false}, meter_weight=DatabendRawType{type=meter_weight, isNullable=false}, move_amount=DatabendRawType{type=move_amount, isNullable=false}, resource_management=DatabendRawType{type=resource_management, isNullable=false}, demand_manager_weight=DatabendRawType{type=demand_manager_weight, isNullable=false}, price_manager=DatabendRawType{type=price_manager, isNullable=false}, version_price=DatabendRawType{type=version_price, isNullable=false}, stock_room_name=DatabendRawType{type=stock_room_name, isNullable=false}, move_manager_weight=DatabendRawType{type=move_manager_weight, isNullable=false}, pieces=DatabendRawType{type=pieces, isNullable=false}, spec3=DatabendRawType{type=spec3, isNullable=false}, sale_return_pound_weight=DatabendRawType{type=sale_return_pound_weight, isNullable=false}, spec4=DatabendRawType{type=spec4, isNullable=false}, process_pound_weight=DatabendRawType{type=process_pound_weight, isNullable=false}, org_name=DatabendRawType{type=org_name, isNullable=false}, spec1=DatabendRawType{type=spec1, isNullable=false}, spec2=DatabendRawType{type=spec2, isNullable=false}, weight_range=DatabendRawType{type=weight_range, isNullable=false}, purchase_return_pound_weight=DatabendRawType{type=purchase_return_pound_weight, isNullable=false}, __deleted=DatabendRawType{type=__deleted, isNullable=false}, lading_manager_weight=DatabendRawType{type=lading_manager_weight, isNullable=false}, ship_type=DatabendRawType{type=ship_type, isNullable=false}, stock_id=DatabendRawType{type=stock_id, isNullable=false}, warehouse_id=DatabendRawType{type=warehouse_id, isNullable=false}, stock_room_id=DatabendRawType{type=stock_room_id, isNullable=false}, tolerance_range_min=DatabendRawType{type=tolerance_range_min, isNullable=false}, create_user_name=DatabendRawType{type=create_user_name, isNullable=false}, lock_manager_weight=DatabendRawType{type=lock_manager_weight, isNullable=false}, org_short_name=DatabendRawType{type=org_short_name, isNullable=false}, product_brand_id=DatabendRawType{type=product_brand_id, isNullable=false}, ten_meter_weight=DatabendRawType{type=ten_meter_weight, isNullable=false}, plan_manager_weight=DatabendRawType{type=plan_manager_weight, isNullable=false}, sale_area_id=DatabendRawType{type=sale_area_id, isNullable=false}, steel_coil_no=DatabendRawType{type=steel_coil_no, isNullable=false}, create_date=DatabendRawType{type=create_date, isNullable=false}, tolerance=DatabendRawType{type=tolerance, isNullable=false}, del_flag=DatabendRawType{type=del_flag, isNullable=false}, owner_name=DatabendRawType{type=owner_name, isNullable=false}, stock_amount=DatabendRawType{type=stock_amount, isNullable=false}, average_manager_price=DatabendRawType{type=average_manager_price, isNullable=false}, average_pound_price=DatabendRawType{type=average_pound_price, isNullable=false}, update_user_name=DatabendRawType{type=update_user_name, isNullable=false}, tolerance_range_max=DatabendRawType{type=tolerance_range_max, isNullable=false}, contract_manager_weight=DatabendRawType{type=contract_manager_weight, isNullable=false}, update_date=DatabendRawType{type=update_date, isNullable=false}, demand_pound_weight=DatabendRawType{type=demand_pound_weight, isNullable=false}, org_id=DatabendRawType{type=org_id, isNullable=false}, sale_area_name=DatabendRawType{type=sale_area_name, isNullable=false}, quantity_type=DatabendRawType{type=quantity_type, isNullable=false}, process_manager_weight=DatabendRawType{type=process_manager_weight, isNullable=false}, manual_pound_weight=DatabendRawType{type=manual_pound_weight, isNullable=false}, sku_code=DatabendRawType{type=sku_code, isNullable=false}, price_discount_pound=DatabendRawType{type=price_discount_pound, isNullable=false}, product_class_name=DatabendRawType{type=product_class_name, isNullable=false}, profitLoss_amount=DatabendRawType{type=profitLoss_amount, isNullable=false}, sale_return_manager_weight=DatabendRawType{type=sale_return_manager_weight, isNullable=false}, purchase_return_manager_weight=DatabendRawType{type=purchase_return_manager_weight, isNullable=false}, process_amount=DatabendRawType{type=process_amount, isNullable=false}, owner_code=DatabendRawType{type=owner_code, isNullable=false}, update_user_code=DatabendRawType{type=update_user_code, isNullable=false}, demand_amount=DatabendRawType{type=demand_amount, isNullable=false}, plan_amount=DatabendRawType{type=plan_amount, isNullable=false}, sys_id=DatabendRawType{type=sys_id, isNullable=false}, specification2=DatabendRawType{type=specification2, isNullable=false}, price_discount_manager=DatabendRawType{type=price_discount_manager, isNullable=false}, specification1=DatabendRawType{type=specification1, isNullable=false}, workgroup_id=DatabendRawType{type=workgroup_id, isNullable=false}, avble_manager_weight=DatabendRawType{type=avble_manager_weight, isNullable=false}, profitLoss_pound_weight=DatabendRawType{type=profitLoss_pound_weight, isNullable=false}, lading_pound_weight=DatabendRawType{type=lading_pound_weight, isNullable=false}, move_pound_weight=DatabendRawType{type=move_pound_weight, isNullable=false}, product_class_id=DatabendRawType{type=product_class_id, isNullable=false}, average_manager_cost=DatabendRawType{type=average_manager_cost, isNullable=false}, purchase_return_amount=DatabendRawType{type=purchase_return_amount, isNullable=false}, length=DatabendRawType{type=length, isNullable=false}, prod_area_name=DatabendRawType{type=prod_area_name, isNullable=false}, specification=DatabendRawType{type=specification, isNullable=false}, profitLoss_manager_weight=DatabendRawType{type=profitLoss_manager_weight, isNullable=false}, stock_pound_weight=DatabendRawType{type=stock_pound_weight, isNullable=false}, stock_manager_weight=DatabendRawType{type=stock_manager_weight, isNullable=false}, manual_amount=DatabendRawType{type=manual_amount, isNullable=false}, platform_id=DatabendRawType{type=platform_id, isNullable=false}, lock_amount=DatabendRawType{type=lock_amount, isNullable=false}, stock_room_type=DatabendRawType{type=stock_room_type, isNullable=false}} \nPK:{}","threadName":"pool-7-thread-1","threadId":18,"mdc":{},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":31916}
{"timestamp":"2023-10-16T14:32:34.861+08:00","sequence":133,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.pipeline.source.AbstractSnapshotChangeEventSource","level":"WARN","message":"Snapshot was interrupted before completion","threadName":"debezium-mysqlconnector-from_mysql-change-event-source-coordinator","threadId":26,"mdc":{"dbz.taskId":"0","dbz.connectorName":"from_mysql","dbz.connectorType":"MySQL","dbz.connectorContext":"snapshot"},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":31916}
{"timestamp":"2023-10-16T14:32:34.863+08:00","sequence":134,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.pipeline.ChangeEventSourceCoordinator","level":"WARN","message":"Change event source executor was interrupted","threadName":"debezium-mysqlconnector-from_mysql-change-event-source-coordinator","threadId":26,"mdc":{"dbz.taskId":"0","dbz.connectorName":"from_mysql","dbz.connectorType":"MySQL","dbz.connectorContext":"snapshot"},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":31916,"exception":{"refId":1,"exceptionType":"java.lang.InterruptedException","message":"Interrupted while snapshotting table scp_sale.t_xs_stock","frames":[{"class":"io.debezium.relational.RelationalSnapshotChangeEventSource","method":"createDataEventsForTable","line":383},{"class":"io.debezium.relational.RelationalSnapshotChangeEventSource","method":"createDataEvents","line":316},{"class":"io.debezium.relational.RelationalSnapshotChangeEventSource","method":"doExecute","line":132},{"class":"io.debezium.pipeline.source.AbstractSnapshotChangeEventSource","method":"execute","line":76},{"class":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"doSnapshot","line":155},{"class":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"executeChangeEventSources","line":137},{"class":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"lambda$start$0","line":109},{"class":"java.util.concurrent.Executors$RunnableAdapter","method":"call","line":515},{"class":"java.util.concurrent.FutureTask","method":"run","line":264},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","line":1128},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","line":628},{"class":"java.lang.Thread","method":"run","line":829}]}}

Change event source executor was interrupted indicates that the processing was interrupted. It seems that the MySQL has some unknown problems. Would u like to share the config file?

@hantmac OK.  and I just find out the source database is a read-only replica of production RDS of ctyun.cn. Does this matter?
debezium.sink.type=databend
debezium.sink.databend.upsert=true
debezium.sink.databend.upsert-keep-deletes=false
debezium.sink.databend.database.databaseName=debezium
debezium.sink.databend.database.url=jdbc:databend://localhost:8000
debezium.sink.databend.database.username=databend
debezium.sink.databend.database.password=databend
# debezium.sink.databend.database.primaryKey=id
# debezium.sink.databend.database.tableName=products
# additional databend parameters
debezium.sink.databend.database.param.ssl=false

# enable event schemas
debezium.format.value.schemas.enable=true
debezium.format.key.schemas.enable=true
debezium.format.value=json
debezium.format.key=json

# mysql source, related docs: https://docs.confluent.io/kafka-connectors/debezium-mysql-source/current/mysql_source_connector_config.html
debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.offset.flush.interval.ms=60000

debezium.source.database.hostname=
debezium.source.database.port=3306
debezium.source.database.user=
debezium.source.database.password=
debezium.source.database.dbname=scp_sale
debezium.source.database.server.name=from_mysql
debezium.source.include.schema.changes=false
debezium.source.table.include.list=scp_sale.t_xs_stock
# debezium.source.database.ssl.mode=required
# Run without Kafka, use local file to store checkpoints
debezium.source.database.history=io.debezium.relational.history.FileDatabaseHistory
debezium.source.database.history.file.filename=data/status.dat
# do event flattening. unwrap message! delete mode must open there conf
debezium.transforms=unwrap
debezium.transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
#debezium.transforms.unwrap.add.fields=op,table,source.ts_ms,db
debezium.transforms.unwrap.delete.handling.mode=rewrite
# make tombstones as true to soft delete
debezium.transforms.unwrap.drop.tombstones=true

# ############ SET LOG LEVELS ############
quarkus.log.console.json=true
# Ignore messages below warning level from Jetty, because it's a bit verbose
quarkus.log.category."org.eclipse.jetty".level=WARN
quarkus.log.file.path=./logs/debezium.log
quarkus.log.file.rotation.max-file-size=5M
quarkus.log.file.rotation.file-suffix=.yyyy-MM-dd.gz
quarkus.log.file.rotation.max-backup-index=3
quarkus.log.level=WARN
quarkus.log.file.enable=true
quarkus.http.port=8080

@peleusj BTW, has you enable the binlog for the RDS of ctyun.cn? Please follow this doc https://debezium.io/documentation/reference/stable/connectors/mysql.html#enable-mysql-binlog to check it first.

@hantmac OK, I will check that with my workmate asap.

@hantmac Sorry for the delay.
I have communicate with workmate, and use comand show variables like 'log_bin' check the binlog status, the result indicates it is on and it matches status you mentioned:
log_bin           = mysql-bin
binlog_format     = ROW
binlog_row_image  = FULL
expire_logs_days  = 10

Also, I tried Airbyte this morning, it can successfully sync the same database table to destination database, like Porstgres or ClickHouse.
Hope these information may help you finding out what's going on.

OK. I have a laptop with Ubuntu 22.04 installed, I will test on that machine and pay attention to JDK version.

@hantmac  I have run a fresh test on Ubuntu 22.04, with Java 11, here is the logs:
docker run \
    -p 8000:8000 \
    -v ${HOME}/tmp/databend/meta:/var/lib/databend/meta \
    -v ${HOME}/tmp/databend/query:/var/lib/databend/query \
    -v ${HOME}/tmp/databend/log:/var/log/databend \
    -e QUERY_DEFAULT_USER=databend \
    -e QUERY_DEFAULT_PASSWORD=databend \
    datafuselabs/databend

{"timestamp":"2023-10-18T15:34:18.855+08:00","sequence":129,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.apache.kafka.connect.runtime.WorkerConfig","level":"WARN","message":"The worker has been configured with one or more internal converter properties ([internal.key.converter, internal.value.converter]). Support for these properties was deprecated in version 2.0 and removed in version 3.0, and specifying them will have no effect. Instead, an instance of the JsonConverter with schemas.enable set to false will be used. For more information, please visit http://kafka.apache.org/documentation/#upgrade and consult the upgrade notesfor the 3.0 release.","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":148989}
{"timestamp":"2023-10-18T15:34:18.862+08:00","sequence":130,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.apache.kafka.connect.runtime.WorkerConfig","level":"WARN","message":"Variables cannot be used in the 'plugin.path' property, since the property is used by plugin scanning before the config providers that replace the variables are initialized.
I copy my application.properties and use there config to test it again:
debezium.sink.type=databend
debezium.sink.databend.upsert=true
debezium.sink.databend.upsert-keep-deletes=false
debezium.sink.databend.database.databaseName=debezium
debezium.sink.databend.database.url=jdbc:databend://localhost:8000
debezium.sink.databend.database.username=databend
debezium.sink.databend.database.password=databend
debezium.sink.databend.database.primaryKey=id
debezium.sink.databend.database.tableName=products
debezium.sink.databend.database.param.ssl=false
# additional databend parameters

# enable event schemas
debezium.format.value.schemas.enable=true
debezium.format.key.schemas.enable=true
debezium.format.value=json
debezium.format.key=json

# mysql source
debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.offset.flush.interval.ms=60000

debezium.source.database.hostname=localhost
debezium.source.database.port=3306
debezium.source.database.user=root
debezium.source.database.password=123456
debezium.source.database.dbname=mydb
debezium.source.database.server.name=from_mysql
debezium.source.include.schema.changes=false
debezium.source.table.include.list=mydb.products
debezium.source.max.batch.size=200000
debezium.source.max.queue.size=800000
# debezium.source.database.ssl.mode=required
# Run without Kafka, use local file to store checkpoints
debezium.source.database.history=io.debezium.relational.history.FileDatabaseHistory
debezium.source.database.history.file.filename=data/status.dat
# do event flattening. unwrap message!
# https://debezium.io/documentation/reference/1.2/configuration/event-flattening.html#extract-new-record-state-drop-tombstones
debezium.transforms=unwrap
debezium.transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
#debezium.transforms.unwrap.add.fields=op,table,source.ts_ms,db
# soft delete
#debezium.transforms.unwrap.delete.handling.mode=rewrite
#debezium.transforms.unwrap.drop.tombstones=false
# hard delete
debezium.transforms.unwrap.delete.handling.mode=none
debezium.transforms.unwrap.drop.tombstones=false

# ############ SET LOG LEVELS ############
quarkus.log.console.json=true
# Ignore messages below warning level from Jetty, because it's a bit verbose
quarkus.log.category."org.eclipse.jetty".level=WARN
quarkus.log.file.path=./logs/debezium.log
quarkus.log.file.rotation.max-file-size=5M
quarkus.log.file.rotation.file-suffix=.yyyy-MM-dd.gz
quarkus.log.file.rotation.max-backup-index=3
quarkus.log.level=INFO
quarkus.log.file.enable=true

@ I use your conf and just change the source mysql info, the logs seems have something different, it is too long, I save it to gist:
https://gist.github.com/peleusj/addb82f0d9496e0888df95ec3ef1bd73

@hantmac I used your conf and just change the source mysql info, the logs seems have something different, it is too long, I save it to gist:
https://gist.github.com/peleusj/addb82f0d9496e0888df95ec3ef1bd73

Great! I found some very useful error message:
"message":"java.sql.SQLException: Query failed (#a722e8a1-a14a-4b73-b1b2-f888d2e44191): storage doesn't support presign operation"

This error was caused by your minio does not support presign. You must config your minio to support it(https://min.io/docs/minio/linux/integrations/presigned-put-upload-via-browser.html). Databend use this fucntion to write data into object storage.
I believe that it will success after handling this problem.
BTW,  change this config to your tables: debezium.source.table.include.list=mydb.products.

@hantmac  Glad to have progrss!
I have changed debezium.source.table.include.list in last test.
I have a little confusion, does this mean I have to start a minio container to storage configure the presign, which in last test I just run  a databen container which is default config with fs backend, my little confusion is why minio is needed here.
Any way, I will try to start a minio docker container.

@hantmac Glad to have progrss!
I have changed debezium.source.table.include.list in last test.
I have a little confusion, does this mean I have to start a minio container to storage data, and configure the presign of minio, which in last test I just run a databend docker container which is default config with fs backend. My little confusion is why minio is needed here.
Any way, I will try to start a minio docker container and test it again.

@peleusj
Databend use presign to write data into object storage. The fs storage is just for testing and can't be used in production.
You can see this doc about presign https://databend.rs/doc/contributing/rfcs/presign.
Just use this config in databend-query.toml:
[storage]
type = "s3"
[storage.s3]
bucket = "databend"
endpoint_url = "http://<IP_ADDRESS>:9000"
access_key_id = "minioadmin"
secret_access_key = "minioadmin"

@hantmac  I try to use your databend-query.toml, but get some errors, I guess it need to set regions, but I checked some docs, could not find how to configure it:
Databend Query start failure, cause: StorageOther. Code: 4000, Text = other error (ConfigInvalid (permanent) at Builder::build, context: { service: s3 } => region is missing. Please find it by S3::detect_region() or set them in env.).

@hantmac I try to use your databend-query.toml, but get some errors, I guess it need to set regions, but I checked some docs, could not find how to configure it:
Databend Query start failure, cause: StorageOther. Code: 4000, Text = other error (ConfigInvalid (permanent) at Builder::build, context: { service: s3 } => region is missing. Please find it by S3::detect_region() or set them in env.).

Are u uring aws s3 storage? You can find more error message in databend-query.

@hantmac I am just using a docker-compose.yml and databend-query.toml with your suggest configs:
services:
  databend:
    image: datafuselabs/databend
    volumes:
      - /home/albert/test/databend/conf/databend-query.toml:/etc/databend/query.toml
    environment:
      QUERY_CONFIG_FILE: /etc/databend/query.toml
      QUERY_DEFAULT_USER: databend
      QUERY_DEFAULT_PASSWORD: databend
      MINIO_ENABLED: 'true'
    ports:
      - '8000:8000'
      - '9000:9000'
      - '3307:3307'
      - '8124:8124'

[query]
max_active_sessions = 256
wait_timeout_mills = 5000

flight_api_address = "<IP_ADDRESS>:9090"
admin_api_address = "<IP_ADDRESS>:8080"
metric_api_address = "<IP_ADDRESS>:7070"

mysql_handler_host = "<IP_ADDRESS>"
mysql_handler_port = 3307

clickhouse_http_handler_host = "<IP_ADDRESS>"
clickhouse_http_handler_port = 8124

http_handler_host = "<IP_ADDRESS>"
http_handler_port = 8000

flight_sql_handler_host = "<IP_ADDRESS>"
flight_sql_handler_port = 8900

tenant_id = "default"
cluster_id = "default"

[log]

[log.stderr]
level = "WARN"
format = "text"

[log.file]
level = "INFO"
dir = "/var/log/databend"

[meta]
endpoints = ["<IP_ADDRESS>:9191"]
username = "root"
password = "root"
client_timeout_in_second = 60

[storage]
type = "s3"

[storage.s3]
bucket = "databend"
endpoint_url = "http://<IP_ADDRESS>:9000"
access_key_id = "minioadmin"
secret_access_key = "minioadmin"

And whre should I look after for databend-query you metioned?

@hantmac I try to use your databend-query.toml, but get some errors, I guess it need to set regions, but I checked some docs, could not find how to configure it:
Databend Query start failure, cause: StorageOther. Code: 4000, Text = other error (ConfigInvalid (permanent) at Builder::build, context: { service: s3 } => region is missing. Please find it by S3::detect_region() or set them in env.).

Are u uring aws s3 storage? You can find more error message in databend-query.

@Xuanwo Hi, would you like to help figure out this problem? 😄

I guess it need to set regions, but I checked some docs, could not find how to configure it:

You can set region this way:
[storage.s3]
bucket = "databend"
endpoint_url = "http://<IP_ADDRESS>:9000"
access_key_id = "minioadmin"
secret_access_key = "minioadmin"
+ region = "abc"

For minio, you can use us-east-1 if no other region has been set for minio.

Connection refused

This error typically suggests improper storage setup. For instance, port 9000 might not be open or listening on <IP_ADDRESS>. Please review your settings to ensure correct minio configuration. To verify, run curl http://<IP_ADDRESS>:9000 in the same databend container.
And given you are using a non-https endpoint like http://xxx, you will need to add an extra param in databend query to allow it:
[storage]
type = "s3"
+ allow_insecure = true

@Xuanwo Sorry for the bothering.
My Docker knowledge is limited, and I'm unsure about how to run this test.

To verify, run curl http://<IP_ADDRESS>:9000 in the same databend container.

I can only test through browser with minio console port <IP_ADDRESS>:9090 in my configuration.
I've attached my complete configuration.
Could you please help me check it to find out the wrong configuration?
Thanks a lot!
docker-compose.yaml
services:
  minio:
    image: quay.io/minio/minio
    volumes:
      - /home/albert/tmp/minio:/data
    environment:
      - "MINIO_ROOT_USER=test"
      - "MINIO_ROOT_PASSWORD=test1234"
    ports:
      - "9000:9000"
      - "9090:9090"
    command: [ "server", "/data", "--console-address", ":9090" ]

  databend:
    image: datafuselabs/databend
    volumes:
      - /home/albert/test/databend/conf/databend-query.toml:/etc/databend/query.toml
    environment:
      QUERY_CONFIG_FILE: /etc/databend/query.toml
      QUERY_DEFAULT_USER: databend
      QUERY_DEFAULT_PASSWORD: databend
    ports:
      - '8000:8000'
      - '3307:3307'
      - '8124:8124'

databend-query.toml
# Usage:
# databend-query -c databend-query.toml

[query]
max_active_sessions = 256
wait_timeout_mills = 5000

# Internal flight rpc for cluster communication.
flight_api_address = "<IP_ADDRESS>:9091"

# Admin REST API.
admin_api_address = "<IP_ADDRESS>:8080"

# Metrics REST API.
metric_api_address = "<IP_ADDRESS>:7070"

# Query Handler: MySQL
mysql_handler_host = "<IP_ADDRESS>"
mysql_handler_port = 3307

# Query Handler: Clickhouse HTTP
clickhouse_http_handler_host = "<IP_ADDRESS>"
clickhouse_http_handler_port = 8124

# Query Handler: HTTP API
http_handler_host = "<IP_ADDRESS>"
http_handler_port = 8000

# Query Handler: Experimental Arrow Flight SQL API
flight_sql_handler_host = "<IP_ADDRESS>"
flight_sql_handler_port = 8900

tenant_id = "default"
cluster_id = "default"

table_engine_memory_enabled = true

# [[query.users]]
# name = "root"
# auth_type = "no_password"

# [[query.users]]
# name = "databend"
# auth_type = "double_sha1_password"
# # echo -n "databend" | sha1sum | cut -d' ' -f1 | xxd -r -p | sha1sum
# auth_string = "3081f32caef285c232d066033c89a78d88a6d8a5"

# [[query.users]]
# name = "datafuselabs"
# auth_type = "sha256_password"
# #  echo -n "datafuselabs" | sha256sum
# auth_string = "6db1a2f5da402b43c066fcadcbf78f04260b3236d9035e44dd463f21e29e6f3b"

[log]

[log.file]
level = "WARN"
format = "text"
dir = "/var/log/databend"

[meta]
# It is a list of `grpc_api_advertise_host:<grpc-api-port>` of databend-meta config
endpoints = ["<IP_ADDRESS>:9191"]
username = "root"
password = "root"
client_timeout_in_second = 60
auto_sync_interval = 60

# Storage config.
[storage]
# fs | s3 | azblob | gcs | oss | cos | hdfs | webhdfs
type = "s3"
allow_insecure = true

# Set a local folder to store your data.
# Comment out this block if you're NOT using local file system as storage.
# [storage.fs]
# data_path = "/var/lib/databend/data"

# To use an Amazon S3-like storage service, uncomment this block and set your values.
[storage.s3]
bucket = "databend"
endpoint_url = "http://<IP_ADDRESS>:9000"
access_key_id = "test"
secret_access_key = "test1234"
region = "us-east-1"
# enable_virtual_host_style = false

# To use Azure Blob Storage, uncomment this block and set your values.
# [storage.azblob]
# endpoint_url = "https://<your-storage-account-name>.blob.core.windows.net"
# container = "<your-azure-storage-container-name>"
# account_name = "<your-storage-account-name>"
# account_key = "<your-account-key>"

# To use Google Cloud Storage, uncomment this block and set your values.
# [storage.gcs]
# bucket = "<your-bucket-name>"
# endpoint_url = "<your-endpoint>"
# credential = "<your-credentials>"

# To use Alibaba Cloud OSS, uncomment this block and set your values.
# [storage.oss]
# bucket = "<your-bucket-name>"
# endpoint_url = "<your-endpoint>"
# access_key_id = "<your-key-id>"
# access_key_secret = "<your-account-key>"

# To use Tencent Cloud Object Storage, uncomment this block and set your values.
# [storage.cos]
# bucket = "<your-bucket-name>"
# endpoint_url = "<your-endpoint>"
# secret_id = "<your-secret-id>"
# secret_key = "<your-secret-key>"

# To use HDFS, uncomment this block and set your values.
# [storage.hdfs]
# name_node = "<your-name-node>"

# To use WebHDFS, uncomment this block and set your values.
# [storage.webhdfs]
# endpoint_url = "<your-endpoint>"

# Cache config.
[cache]
# Type of storage to keep the table data cache
#
# available options: [none|disk]
# default is "none", which disable table data cache
# use "disk" to enabled disk cache
data_cache_storage = "none"

[cache.disk]
# cache path
path = "/var/lib/databend/cache"
# max bytes of cached data 20G
max_bytes =<PHONE_NUMBER>0

@Xuanwo I successfully deploed a standalone databend in my latop, follow these articles:
https://databend.rs/doc/deploy/deploying-databend
https://www.bilibili.com/read/cv25407731/
and it does not need to set the region in databend-query.toml as deploy with Docker.
@hantmac I finally succfully start the run.sh, and get part of my data to databend tables, but there is some error:
{"timestamp":"2023-10-20T12:35:07.165+08:00","sequence":1090,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.jdbc.JdbcConnection","level":"INFO","message":"Connection gracefully closed","threadName":"pool-10-thread-1","threadId":31,"mdc":{},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":26324}
{"timestamp":"2023-10-20T12:35:07.183+08:00","sequence":1091,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.apache.kafka.connect.storage.FileOffsetBackingStore","level":"INFO","message":"Stopped FileOffsetBackingStore","threadName":"pool-7-thread-1","threadId":18,"mdc":{},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":26324}
{"timestamp":"2023-10-20T12:35:07.184+08:00","sequence":1092,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.server.ConnectorLifecycle","level":"INFO","message":"Connector completed: success = 'false', message = 'Stopping connector after error in the application's handler method: Java heap space', error = '{}'","threadName":"pool-7-thread-1","threadId":18,"mdc":{},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":26324,"exception":{"refId":1,"exceptionType":"java.lang.OutOfMemoryError","message":"Java heap space","frames":[{"class":"java.util.LinkedHashMap","method":"newNode","line":256},{"class":"java.util.HashMap","method":"putVal","line":627},{"class":"java.util.HashMap","method":"put","line":608},{"class":"com.fasterxml.jackson.databind.node.ObjectNode","method":"replace","line":585},{"class":"com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer","method":"_deserializeContainerNoRecursion","line":594},{"class":"com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer","method":"deserialize","line":98},{"class":"com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer","method":"deserialize","line":23},{"class":"com.fasterxml.jackson.databind.deser.DefaultDeserializationContext","method":"readRootValue","line":323},{"class":"com.fasterxml.jackson.databind.ObjectMapper","method":"_readTreeAndClose","line":4772},{"class":"com.fasterxml.jackson.databind.ObjectMapper","method":"readTree","line":3138},{"class":"io.debezium.server.databend.DatabendChangeConsumer","method":"lambda$handleBatch$0","line":200},{"class":"io.debezium.server.databend.DatabendChangeConsumer$$Lambda$802/0x000000084061f840","method":"apply"},{"class":"java.util.stream.ReferencePipeline$3$1","method":"accept","line":195},{"class":"java.util.ArrayList$ArrayListSpliterator","method":"forEachRemaining","line":1655},{"class":"java.util.stream.AbstractPipeline","method":"copyInto","line":484},{"class":"java.util.stream.AbstractPipeline","method":"wrapAndCopyInto","line":474},{"class":"java.util.stream.ReduceOps$ReduceOp","method":"evaluateSequential","line":913},{"class":"java.util.stream.AbstractPipeline","method":"evaluate","line":234},{"class":"java.util.stream.ReferencePipeline","method":"collect","line":578},{"class":"io.debezium.server.databend.DatabendChangeConsumer","method":"handleBatch","line":207},{"class":"io.debezium.embedded.ConvertingEngineBuilder","method":"lambda$notifying$2","line":83},{"class":"io.debezium.embedded.ConvertingEngineBuilder$$Lambda$375/0x0000000840321c40","method":"handleBatch"},{"class":"io.debezium.embedded.EmbeddedEngine","method":"run","line":822},{"class":"io.debezium.embedded.ConvertingEngineBuilder$2","method":"run","line":192},{"class":"io.debezium.server.DebeziumServer","method":"lambda$start$1","line":152},{"class":"io.debezium.server.DebeziumServer$$Lambda$415/0x0000000840371440","method":"run"},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","line":1128},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","line":628},{"class":"java.lang.Thread","method":"run","line":829}]}}
{"timestamp":"2023-10-20T12:35:07.212+08:00","sequence":1093,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.server.DebeziumServer","level":"INFO","message":"Received request to stop the engine","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":26324}
{"timestamp":"2023-10-20T12:35:07.212+08:00","sequence":1094,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.embedded.EmbeddedEngine","level":"INFO","message":"Stopping the embedded engine","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":26324}
{"timestamp":"2023-10-20T12:35:07.255+08:00","sequence":1095,"loggerClassName":"org.jboss.logging.Logger","loggerName":"io.quarkus","level":"INFO","message":"debezium-server-databend-dist stopped in 0.068s","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"thinkpad-x250","processName":"io.debezium.server.Main","processId":26324}

does this mean the data transfer is working, but my old laptop Thinkpad X250 8G memory is not enough to complete the task?

'Stopping connector after error in the application's handler method: Java heap space

@peleusj Congratulation!

Yes, because of this error 'Stopping connector after error in the application's handler method: Java heap space , it means you have no enough memory for the task.
Change event source executor was interrupted was caused by error mysql debezium settings.
Open issue any time if you have any questions!

Lastly, thanks @Xuanwo very much!

So I closed this issue since solved. Open the issue any time if you have any questions.

@hantmac Sorry for bothering again.
Today, I rerun  previous test on a machine with more memory.  All the configurations were the same as the previous test.
It only synchronized approximately 25,000 records before it was interrupted. The source was around 60,000 records.
Below is the log information:
{"timestamp":"2023-10-25T15:22:48.426+08:00","sequence":1102,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.connector.mysql.MySqlStreamingChangeEventSource","level":"INFO","message":"Stopped reading binlog after 0 events, last recorded offset: {transaction_id=null, ts_sec=1698218534, file=mysql-bin.011044, pos=212595309, gtids=c2932ac0-66e5-11ed-abba-fa163ed055f0:1-1295929881, server_id=1695037098, event=1}","threadName":"blc-<IP_ADDRESS>:3306","threadId":55,"mdc":{"dbz.taskId":"0","dbz.connectorName":"from_mysql","dbz.connectorType":"MySQL","dbz.connectorContext":"binlog"},"ndc":"","hostName":"localhost.localdomain","processName":"io.debezium.server.Main","processId":27392}
{"timestamp":"2023-10-25T15:22:48.434+08:00","sequence":1103,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.jdbc.JdbcConnection","level":"INFO","message":"Connection gracefully closed","threadName":"pool-10-thread-1","threadId":58,"mdc":{},"ndc":"","hostName":"localhost.localdomain","processName":"io.debezium.server.Main","processId":27392}
{"timestamp":"2023-10-25T15:22:48.451+08:00","sequence":1104,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.apache.kafka.connect.storage.FileOffsetBackingStore","level":"INFO","message":"Stopped FileOffsetBackingStore","threadName":"pool-7-thread-1","threadId":27,"mdc":{},"ndc":"","hostName":"localhost.localdomain","processName":"io.debezium.server.Main","processId":27392}
{"timestamp":"2023-10-25T15:22:48.452+08:00","sequence":1105,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.server.ConnectorLifecycle","level":"INFO","message":"Connector completed: success = 'false', message = 'Stopping connector after error in the application's handler method: java.sql.SQLException: failed to upload input stream', error = '{}'","threadName":"pool-7-thread-1","threadId":27,"mdc":{},"ndc":"","hostName":"localhost.localdomain","processName":"io.debezium.server.Main","processId":27392,"exception":{"refId":1,"exceptionType":"java.lang.RuntimeException","message":"java.sql.SQLException: failed to upload input stream","frames":[{"class":"io.debezium.server.databend.tablewriter.UpsertTableWriter","method":"deleteUpsert","line":80},{"class":"io.debezium.server.databend.tablewriter.UpsertTableWriter","method":"addToTable","line":46},{"class":"io.debezium.server.databend.DatabendChangeConsumer","method":"handleBatch","line":212},{"class":"io.debezium.embedded.ConvertingEngineBuilder","method":"lambda$notifying$2","line":83},{"class":"io.debezium.embedded.EmbeddedEngine","method":"run","line":822},{"class":"io.debezium.embedded.ConvertingEngineBuilder$2","method":"run","line":192},{"class":"io.debezium.server.DebeziumServer","method":"lambda$start$1","line":152},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","line":1128},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","line":628},{"class":"java.lang.Thread","method":"run","line":829}]}}
{"timestamp":"2023-10-25T15:22:48.485+08:00","sequence":1106,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.server.DebeziumServer","level":"INFO","message":"Received request to stop the engine","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"localhost.localdomain","processName":"io.debezium.server.Main","processId":27392}
{"timestamp":"2023-10-25T15:22:48.485+08:00","sequence":1107,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.embedded.EmbeddedEngine","level":"INFO","message":"Stopping the embedded engine","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"localhost.localdomain","processName":"io.debezium.server.Main","processId":27392}
{"timestamp":"2023-10-25T15:22:48.525+08:00","sequence":1108,"loggerClassName":"org.jboss.logging.Logger","loggerName":"io.quarkus","level":"INFO","message":"debezium-server-databend-dist stopped in 0.071s","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"localhost.localdomain","processName":"io.debezium.server.Main","processId":27392}

@peleusj Hi bro, there is an error message : "message":"java.sql.SQLException: failed to upload input stream", it indicates that the debezium-server can't connect with your oss, there has a network error.

@hantmac Thanks for your quick responsing.
I deploy minio and databend both locally, and my connect to my RDS seems solid.
Also do you have any insights with the target coloum data type issues, The decimal became varchar, the datetime became bigint.
Thanks again.


Horovod test fix

Adding the correct versions for the framework versions as they will be varying on the Dataproc version being used.
Ignoring the tests run for Dataproc versions 2.1 and 2.2 because it is not supported with the current framework versions being suggested and downgrading doesn't allow the frameworks to be installed.

/gcbrun

/gcbrun

/gcbrun


Server Access Request

Please answer the following questions
What is your experience playing modded Minecraft?
I've played Minecraft for a couple of years now. When it comes to modded Minecraft, I've done a playthrough of Blightfall and some other modpacks.
Have you successfully been able to play this modpack in single-player?
I have not be able to play the modpack in single-player.
What is your Minecraft username?
Shiridari
Please re-state the server rules in your own words
The server doesn't allow pvp combat that's uninvited. Don't disturb things made by, or look like they may have been made by another player. Don't build anything in view of another person's land without asking permission first. Don't make it difficult for other people to be able to tell that you made something. Don't tear up the land without actually doing something with it; Its not cool walking around and seeing blown-up holes in the ground everywhere.
Next Steps
Please be patient while waiting for your request to be considered!  Access requests are considered once per day, so you should generally receive a response in less than 24h.  In the meantime, you can set up your Minecraft instance to access the server at:
istari.tungstonminer.com:25654
Finally, it is highly recommended that you read the wiki for the modpack.  There are a lot of changes to world generation, mob spawning, ore placement, recipes and general game mechanics which will be very confusing otherwise.

Sounds good! Welcome to the server! If you run into any problems, feel free to ask for help on the Discord channel.


Job list

Given https://github.com/sbt/sbt/issues/2806
steps

Start sbt on a multi-project build: for instance echo-server and echo-client
Run echo-server/bgrun to start background "run" task in the background
Run jobList

expectation
jobList displays the list of all jobs under control by sbt's command engine.
Each job is associated with:

job id
command line
initiating channel
(optional) subproject scoping

In addition I can run jobStop 2 to stop job id 2.

I think this is done with background run in https://github.com/sbt/sbt/pull/2936. It's called ps.


Removed Unused Setters and Getters also Added Lombok to add Setters a…

I have used Lombok which will help in generation of setters and getters so code can be optimized
Also I have removed unused setter and getters from Entity classes
Code Refractored.

@shimingxy  Review my PR and suggest me should I do the same changes in all Entity Classes or not ?

Thank you @shimingxy
Is there anything on which I can work which is beginner friendly?


new language notification

When a new language is added and a user already defaulted to english / other because it wasn't available, there's no easy way to check if it's been added/available besides going into settings.
We would probably need to store available languages somewhere, and then compare browser language / selected language. Not sure we can add any metadata to the language options themselves.

We could also just force it in certain regions when it becomes available also.


URI exception while running device simulation

Not able to start the device simulation app. getting below error at line
var openIdConfig = await this.openIdCfgMan.GetConfigurationAsync(token)
As mentioned updated the connection string, still getting error.
iothub_connstring
System.UriFormatException
HResult=0x80131537
Message=Invalid URI: The format of the URI could not be determined.
Source=System.Private.Uri
StackTrace:
at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)

@AfreenShaha - Please reach out if you are still seeing this issue.  I'll go ahead and close for now until we have more feedback.
Thanks,
-Tom


Add Paint.NET

Package Requested
Paint.NET is a very popular image editor, much like Notepad++. Both scoop and chocolatey have it. Download link: https://www.dotpdn.com/files/paint.net.4.2.16.install.zip
Chocolatey link: https://community.chocolatey.org/packages/paint.net

[ ] I would like someone else to build the manifest.
[x] I would like help so I can submit the manifest.
[x] I have performed a search and couldn't find this package.
[ ] I think there is a new version available and I have provided the URL.

Manifest:
Fill out as much of the manifest metadata you can:
PackageIdentifier:   # publisher.package format (example: "Microsoft.WindowsTerminal")
PackageVersion:      # version number format x.y.z.a (example: "1.6.10571.0")
PackageLocale:       # meta-data locale (example: "en-US")
Publisher:           # publisher name (example: "Microsoft Corporation")
PackageName:         # package name (example: "Windows Terminal")
License:             # package license (example: "MIT")
ShortDescription:    # package description (example: "The new Windows Terminal")
Installers:
 - Architecture:     # installer architecture (example: "x64")
   InstallerType:    # installer type (example: "msix")
   InstallerUrl:     # installer URL (should be https://)
   InstallerSha256:  # SHA256 hash calculated from installer
ManifestType: singleton
ManifestVersion: 1.0.0

It's not possible yet due to issue #174

Oh, so I should I close this, or should we keep it open to track Paint.NET package?

Leave it open in case it works out in the future.

It's on the store if that helps?
https://www.microsoft.com/en-au/p/paintnet/9nbhcs1lx4r0#activetab=pivot:overviewtab

It's on the store if that helps?
https://www.microsoft.com/en-au/p/paintnet/9nbhcs1lx4r0#activetab=pivot:overviewtab

I don't think that helps much. How should winget-cli handle paid Microsoft Store applications?
Also, winget-cli has no logic to detect Microsoft Store UWP apps. Not Sin32 apps either is Microsoft Store id is used, for example Microsoft PowerToys.

I asked the developers if they can load to github in compatible format

Until winget supports MSI inside ZIP:

https://github.com/o-l-a-v/winget-intune-win32/tree/main/Not yet supported by Winget/System - Install and upgrade/Paint.NET

Not tested it from Intune yet. But should work. :)

Made a request on paint.net's forums to release as .exe or .msi for winget integration.
Link

Seems like the developers have no plans to make .exe or .msi this leave this blocked until winget has zip support as linked.

Seems like the developers have no plans to make .exe or .msi this leave this blocked until winget has zip support as linked.

Indeed. The paint.net dev responded to my request on their forum and stated as such.

Here's my actual quote on the matter. Basically, winget can do the work once, rather than requiring every app developer out there to change how they do releases. (every app developer who releases ZIPs, that is)

I think winget just needs to add ZIP support. I'm already publishing numerous package types, with long builds times and gigabytes to upload. It's already a lot to keep track of. All they have to do is unzip the EXE and do what they'd normally do for an EXE.

https://forums.getpaint.net/topic/118574-please-add-paintnet-to-the-available-packages-for-windows-package-manager-winget/?do=findComment&comment=590802

Looks like the issue for ZIP support has been moved to the winget-cli repo instead for anyone looking for link to blocker
https://github.com/microsoft/winget-cli/issues/140


Limit Castle.Core to <5.0 to avoid breaking changes

With the release of<EMAIL_ADDRESS>there are a number of breaking changes (for example https://github.com/castleproject/Core/pull/563) that affect Windsor.
Example code that blows up:
var container = new WindsorContainer().Install(FromAssembly.Named("Does.Not.Matter"));
Gives an exceptions along the lines:
System.TypeLoadException : Could not load type 'Castle.Core.Internal.PermissionUtil' from assembly 'Castle.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=407dd0808d44fbdc'.

System.TypeLoadException : Could not load type 'Castle.Core.Pair`2' from assembly 'Castle.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=407dd0808d44fbdc'.

This PR limits the upper version of Castle.Core to < 5.0, so that these changes don't bite consumers at runtime, until Windsor is updated to support the new version.
Took the opportunity to remove unnecessary references to Castle.Core in Facility projects that transitively include it already by referencing the base Castle.Windsor project.

There are a bunch of unreleased changes in master including some changes to target frameworks:
https://github.com/castleproject/Windsor/compare/v5.1.1...master
Let's leave that in master so we can do what's relevant for today, I've created a branch off v5.1.1 to work in.

@Jevonius Your changes look good. Can you rebase off the releases/5.1 branch to remove the changes we don't want to ship.

@Jevonius Your changes look good. Can you rebase off the releases/5.1 branch to remove the changes we don't want to ship.

I think I've done this now (haven't done a huge amount of rebasing before).


Underscore sample IDs

Main discussion here: https://github.com/biocore/deblur/issues/157

Closing in favor of qiime2/qiime2#334.
The specific line of issue, though it is operating on MIENS complaint assumptions.


create users with different objectclass-es

Laravel Version: 5.5
Adldap2-Laravel Version: 4.0.7
PHP Version: 7.0.7
LDAP Type: OpenLDAP

Create users with different objectclasses!
Hi again!
I want to ask, that is there any best solution that during the creation I can set the objectclass?
My problem, that every usertype has some unique objectclass, what others have not. I can make a schema for every user, but its not so pretty solution.
Thanks in advance!
Greg

Hi @GergelyLakatos, just set the objectclass attribute on the user when you are making one:
$user = Adldap::make()->user();

$user->objectclass = [
    'top',
    'person',
    'organizationalPerson',
    'user',
];

$user->save();

Hi Steve,
When I try to create user with defined object class, I get
ErrorException
ldap_add(): Add: Object class violation
vendor/adldap2/adldap2/src/Connections/Ldap.php:333
`$cn = 'John Doe';
$user = Adldap::make()->user([
'cn' => 'John Doe',
'dn' => "cn=John Doe,ou=users,dc=0xm,dc=net",
'sn' => 'Grant',
'mail' =><EMAIL_ADDRESS>]);
$user->objectclass = [
'top',
'posixAccount',
'inetOrgPerson',
];
$user->save();`


Entity and numeric character references

@lostenderman Please, provide a reference to the issues that are related and/or will be closed by the pull request.


Balance (left/right)

Is it possible to adjust the balance through the webinterface?

Afraid not. The webinterface mainly just exposes Mopidy settings which doesn't include this.

Okay, created an issue there: https://github.com/mopidy/mopidy/issues/1227
Will update this when I get feedback.

Maybe my issue  can help you somehow


Allow singular variables in list locations

https://github.com/facebook/graphql/pull/509

This one is done but we have to review if tests exist.


adaptive concurrency: Fix min_concurrency behavior when gradient shrinks

The min_concurrency setting dictated the concurrency limit when in a
min_rtt calculation window. However, if the gradient caused the
concurrency limit to shrink (due to increased latency), it was possible
for the limit to drop below this configured minimum. This patch fixes
this behavior so that the calculated concurrency limit is always >= the
configured minimum.
In addition, a minor change to the stat indicating whether the min_rtt
calculation window is active. When active, the stat was being set to the
concurrency limit; however, according to the documentation this should
be either 0 or 1. This patch also fixes this behavior.
Risk Level: Low, this filter is still experimental.
Testing: Adaptive concurrency tests

LGTM thanks. Can you merge master?
/wait


DM username online indicator still discolored

use the theme pinned on my profile for a fix
i'll definitely get banned from this org


弃用警告：从collections中导入ABCs已被弃用，并在python3.8中将停止工作，可使用collections.abc代替它进行使用

PR types

PR changes

Description

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.DamonBit seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

@Damonpkl 感谢贡献！请签署下CLA https://github.com/PaddlePaddle/PaddleNLP/pull/1707#issuecomment-1048971657 。另外其他的地方是不是还有这种情况呢？比如：https://github.com/PaddlePaddle/PaddleNLP/blob/develop/paddlenlp/transformers/funnel/tokenizer.py#L18

已签署CLA
弃用警告：从collections中导入ABCs已被弃用，并在python3.8中将停止工作，可使用collections.abc代替它进行使用

@Damonpkl CLA 还没签署成功哈，请再次check一下😊

现在呢😊


Failure to connnect to azure active directory

Hello support,
Application legacy is no longer available, i can't connect to azure ad via my mobile client
when i sign in using UWP platform, i had error message "user cancelled authentication. on an android device this could be due to lack of capabilities, such as custom tabs....." . before this update, the connection using Azure AD work fine ?
I deleted my native application and i floowed the new steps to connect to azure, but i have same error
i did like this :

Can you help please
Cordially

after many search, i found the problem, Microsoft.IdentityModel.Clients.ActiveDirectory.AdalServiceException, i think that this expetion due of the upgrading of azure because application legacy is not available and i can't create native application :/

@XamarinTounsi Thanks for your feedback! We will investigate and update as appropriate.

@XamarinTounsi  can you please clarify which doc you're referring to?

In addition to that can you please add the actual error you're getting as well as getting a fiddler trace of the error occurring?
@XamarinTounsi

Hello thanks for you answer, I used ADAL, in UWP platform can’t connect to azure AD, error message
« USER CANCEL AUTHENTIFICATION, on an android device ..... »

Hey @XamarinTounsi  please file the github issue against the respective library.
This is not the right area for support on this issue.
I am now closing out this issue. Thanks,


Specifying normalization layers.

Is your feature request related to a problem? Please describe.
Currently, the processor is implemented with LayerNormalization. I would like to use other normalization layers (https://pytorch.org/docs/stable/nn.html#normalization-layers) including custom normalization layers.
Describe the solution you'd like
I would like to specify the normalization layer of the processor in the config, e.g. transformer.yaml:
layer_norm:  # This needs to be a partial instantiation since it is used in multiple places
  _target_: torch.nn.LayerNorm
  _partial_: True
  normalized_shape: ${model.num_channels}

processor:
  _target_: anemoi.models.processor.TransformerProcessor
  _convert_: all
  activation: ${model.activation}
  num_layers: 16
  num_chunks: 2
  mlp_hidden_ratio: 4 # GraphTransformer or Transformer only
  num_heads: 16 # GraphTransformer or Transformer only
  window_size: 512
  dropout_p: 0.0 # GraphTransformer
  layer_norm: ${model.layer_norm} # (Optional) Default nn.LayerNorm

Describe alternatives you've considered
No response
Additional context
No response
Organisation
No response

Cathal already did experiments with RMSNorm (but from TransformerEngine, I think). It might have been hard coded but good to coordinate.
CC: @cathalobrien

Hey, yeah i have this PR https://github.com/ecmwf/anemoi-models/pull/35 . I put it on ice a while back bc I didnt thought it would cause problems in inference if we have arbitrary functions in the checkpoint file.
but now that the checkpoints are weights only, it should be fine. I can refresh it next week

I see, this is related but I was thinking of something more general. I would like to be able to write custom normalization layers, e.g.
class TransformerProcessorBlock(BaseBlock):
    """Transformer block with MultiHeadSelfAttention and MLPs."""

    def __init__(
        self,
        num_channels: int,
        hidden_dim: int,
        num_heads: int,
        activation: str,
        window_size: int,
        dropout_p: float = 0.0,
        layer_norm: Optional[dict] = None,
    ):
        super().__init__()

        try:
            act_func = getattr(nn, activation)
        except AttributeError as ae:
            LOGGER.error("Activation function %s not supported", activation)
            raise RuntimeError from ae

        # Instantiate normalization layers using Hydra
        self.layer_norm1 = layer_norm() if layer_norm else nn.LayerNorm(num_channels)
        self.layer_norm2 = layer_norm() if layer_norm else nn.LayerNorm(num_channels)
        ...

    def forward(
        self,
        x: Tensor,
        shapes: list,
        batch_size: int,
        model_comm_group: Optional[ProcessGroup] = None,
        **layer_kwargs,
    ) -> Tensor:
        # Need to be out of place for gradient propagation
        x = x + self.attention(self.layer_norm1(x, **layer_kwargs), shapes, batch_size, model_comm_group=model_comm_group)
        x = x + self.mlp(self.layer_norm2(x, **layer_kwargs))
        return x

Ah I see, yeah I think this should work.
I already have this implemented
    LayerNorm:
      #_target_: "torch.nn.LayerNorm" #the default PyTorch implementation
      _target_: "liger_kernel.transformers.rms_norm.LigerRMSNorm" # my desired layernorm
      _partial_: True

I havent tried with a handwritten layernorm, but i assume as long as the import in target points to the right place it should be fine.
I like your idea of passing **layer_kwargs directly to the instantiated layer_norm, i was wondering how to handle arbitrary parameters at the time.

I like your idea of passing **layer_kwargs directly to the instantiated layer_norm, i was wondering how to handle arbitrary parameters at the time.

On a second thought, I believe it should be only **kwargs. In the future someone wants to do something else in the forward function.

Yes, e.g. cross attention or some fancy bias terms for the attention could also be passed.


Parking Lot

Post any questions you may have here during the training.

📷 Recordings

[ ] Is there a way to sort by name automatically in repo list (or at org level)?


cannot open procedure inside package in oracle database

System information:

kDE Neon 5.22.2
DBeaver 21.1.1
Additional extensions

Connection specification:

oracle
oracle jdbc<IP_ADDRESS>
Do you use tunnels or proxies (SSH, SOCKS, etc)? no

Describe the problem you're observing:
When trying to open procedure in oracle in the latest dbeaver it gives the following error
There is no handler to execute for command org.jkiss.dbeaver.ext.oracle.code.package.navigate
Steps to reproduce, if exist:
Install the latest dbeaver version
Try to open a procedure in existing oracle connection.
Include any warning/errors/backtraces from the logs

@faroukelabady ,
unfortunately, cannot reproduce it.
could you provide a video with the issue?

Okay, after reinstalling, seems the issue is gone now, thanks a lot for your support.


'svgo' breaks some logos in Firefox

Chrome looks ok though. Tried with this logo:
https://commons.wikimedia.org/wiki/File:American_Express_logo.svg
https://upload.wikimedia.org/wikipedia/commons/3/30/American_Express_logo.svg
Seems to be a problem with the radial gradient?

@sebaranowski I'm pretty sure this is a problem with SVGO resp. one of its (sometimes very aggressive) plugins. Please try to disable some of them (or SVGO altogether) to see if this helps.


nimble/porting: Explicitly NULL terminate string buffer after using strncpy.

os_mempool.c: Using strncpy, buffer will not be NULL terminated if size of source string is same as the buffer size. Hence relpace strncpy with strlcpy.

this should be fixed in mynewt-core and then ported to mynewt-nimble

@andrzej-kaczmarek this was fixed in mynewt-core quite some time ago apache/mynewt-core#1865

this should be fixed in mynewt-core and then ported to mynewt-nimble

@andrzej-kaczmarek this was fixed in mynewt-core quite some time ago apache/mynewt-core#1865

ok, I did not know that. so file in mynewt-nimble should be just updated to match mynewt-core instead of making different fix here.


mksquashsfs just got reproducible

Dear Michal,
how does this:
https://github.com/plougher/squashfs-tools/commit/e0d74d07bb350e24efd3100d3798f4f6d893a3d9
and the zstd support included in a "soon" to be released release improve the situation? Because I'm scratching my head how to provide incremental updates to our https://github.com/fsfw-dresden/usb-live-linux in the future, instead of just replacing the whole image...

Yep, that sounds really cool! That said, I don't think squashdelta needs any changes for that, as it doesn't invoke mksquashfs. If you build reproducible images, it's just going to be happy about them.
That said, if that option affects compressors somehow, then it would be something for squashdelta to use.


Broken link in take home projects

Describe your problem and how to reproduce it:
In some of the take home projects there is a broken link in the description - https://learn.freecodecamp.org/challenges/get-set-for-our-dynamic-web-application-projects - which leads to a 404.
Add a Link to the page with the problem:
https://learn.freecodecamp.org/coding-interview-prep/take-home-projects/chart-the-stock-market/

Here is a breakup of the action items (can be done in parallel, by multiple contributors):

[ ] The original challenge in the needs to be ported over to Glitch. @vadimdez we were hoping you could point us in the right direction by helping us with the source code?
[ ] This Glitch project can then be published and the challenge updated with the link and the verbiage.
[ ] Finally we would also need to point the link for getting started use our standard intro for other projects.

@Alex-Cannon, @rishikesh1410, please let me know if you are still interested in making a pull request for this issue.

@Alex-Cannon @rishikesh1410 @ahmadabdolsaheb @raisedadead
Please inform on the status of the work on this issue, otherwise I would like to have a go at it.

I would like to know the status. Do we have the correct link available or not?

@Alex-Cannon What is the original challenge link which https://learn.freecodecamp.org/challenges/get-set-for-our-dynamic-web-application-projects attempts to link to?

@RandellDawson please let me know once the actual link is confirmed. I would like to work on this issue.

@RandellDawson I'm not sure where the original link is. I thought it was in Apis And Microservices in the legacy site, but I can't find it. We may have to re-create this page? It is supposed to let users know how to create a repository and deploy to heroku. For some of these apps, users can also just use glitch.

@Alex-Cannon It has to be somewhere in the history on Github.  When using Git, it is not possible to lose content.

@Alex-Cannon I'm not sure this the very latest instance of the file, but it's the best I could find.  If there's a way to find the latest version, I'd love to know how!
Regardless, you might be able to get some inspiration from there, but unfortunately the specifics are all out of date.

Yes please.


Difficulty importing large datasets

I'm attempting to import a whole statewide database and I've run into a few issues.

Importing script fails without an error message if it aborted due to being out of memory or it otherwise did not complete.
Importing seems to get exponentially longer the larger the database import.

Smaller db - 640 organizations, 1250 locations, 2440 services / ~30 minutes to import

adding taxonomy and service areas / ~60 minutes to import

Large db - 5110 organizations, 9920 locations, 20,000 services / 36 hours and counting...

My VM is still seems to be working on importing, ruby is using between 2.5-4G memory, 99% cpu.  I had thought it might finish at about 24 hours, but now I'm not sure it will even complete.  Watching top shows the postgres process running only a moment every 5-10 minutes.
I'm not sure what my next step needs to be.  I'm wondering about either needing to set up a vagrant 64-bit box or attempting to build and merge dumps like described at http://blog.fespinozacast.com/2014/01/04/merge-two-backups-of-a-postgresql-database/ .
Any thoughts or suggestions?

I haven't tested with such a large dataset, but it makes sense that the larger the data, the longer it will take, especially if the script has to geocode addresses. The machine's or VM's capabilities will play a major role in the speed as well. I would try updating the Vagrant box to a 64-bit machine and assigning it as much memory as possible (at least 4GB). I'll update the ohana-api-dev-box to do the same.
What kind of machine are you running the script on? I'm assuming a Windows machine. Would you be able to try it on a Mac with at least 8GB of RAM? If not, would you be able to share the CSV files with me so I can try it on my Mac?

Last night I put together an x64 Vagrant machine.  Pc specs are at. https://github.com/codeforamerica/ohana-api-dev-box/issues/12
All the addresses I geocoded beforehand as Google has a 2500 limit per day.
On the x32 I had upped my VM memory size to 5.5G as ruby crashed when set to 4G.  After having upped to 5.5G the x32 VM would only use up to 4.2G, and would never pass importing services.csv after 48 hours.
So far the x64 seems to be working. I set the memory for the VM to 6.6G, and I'v seen it use as much as 6.2G. This morning my import is finally past services.csv and is currently working on phones.csv (my last table). I expect it to finish at about 14 or 16 hours, (so long as my ups keeps up with the power failures today)

Thanks for the details. Have you tried running the script directly on your Linux box (not inside a VM)?
I suspect the script might be loading all CSV contents in memory. I will look into it and see if it can be optimized.

@monfresh The script loads the whole file into memory with File.read, then loads all the CSV entries into memory with CSV.new(...).entries, then creates an array of Organization objects before saving them all.
I briefly played with using Enumerator::Lazy to optimize parts of it when I saw this thread, but it's going to be quite a bit of work (but well worth it I'd say).

Using CSV.new(...).each instead of #entries would be a start, and I got most of the OrganizationImporter spec working with that change, but I didn't go all the way through the code to make it fully lazy.

Thanks for looking into this, Mike. If you have time to keep working on it and feel like it, that would be great. I'm not sure I have much time to devote to this at the moment.
Cory, please let us know whether your latest attempt was successful. If not, for a short term fix, I would either try to find more RAM, or split your CSV files into smaller chunks.

find more RAM, or split your CSV files into smaller chunks.

I just upgraded my nachine to 8GB, DDR2 is old and pricey so I am limited unless I build a new machine.
What do I need to do to split the files?

To split the files, I was thinking you could take your organizations.csv and save it as a new file with half (or a quarter) of the rows, then repeat with all the other CSVs, while making sure the foreign keys between the CSVs match each other. Alternatively, if you have a script that generated the CSVs in the first place, run it again but with only a subset of the data at a time.

Thanks @monfresh, I will try that as well and let you kniw how it goes.

@cderenburger Could you try this branch with your smaller dataset and see if it improves the speed? https://github.com/codeforamerica/ohana-api/tree/361-csv-memory
If that doesn't work, I'm thinking of looking into https://github.com/tilo/smarter_csv or https://github.com/fiksu/rcsv

I've also started working on some code here: https://github.com/appropriate/ohana-api/tree/import-incremental

More work is needed to update LocationImporter, remove File.read calls from FileChecker and AddressExtractor. Probably some more tweaks as well.

Do you think it might be easier to use smarter_csv? It provides the ability to process the CSV in chunks. Also, how are you testing the improvements? Do you have a large dataset to test with?

@monfresh I don't think the CSV processor is the issue, particularly since the chunking would have to be smart enough to work across files as you said in another comment. Using a faster or less memory-intensive CSV processor wouldn't hurt, but I think the problem here is mainly that multiple copies of the same data is being retained in memory (leading to high memory requirements and GC pressure).
I see two main things that should be done to optimize the processing of large CSV files:

Make sure that everything is processing files, CSV rows, and objects in a streaming fashion
Aiming for constant memory use by persisting things to the DB as soon as possible

The work I did in my branch is a start for achieving number 1 (the changes you made partially address that as well). I also started to address the second point, but I think the whole thing would need to be wrapped in a transaction as well to avoid committing partial results (or to only commit partial results in a controlled manner).

BTW, the memory use won't be constant to the extent that errors are being stored in memory, but I'd probably address that by having a maximum error count after which all processing is aborted. The DB memory use may also not be constant due to the open transaction, but DBs are usually better at managing that memory pressure and putting things on disk temporarily than any code you or I would write.

Also, I haven't tested the improvements directly. I'm just making sure the specs pass at this point and doing general cleanups based on things that I've already found loading the whole dataset into memory.
I did do some standalone tests to make sure that CSV.new(...).lazy.each is not leaking when reading from an infinite stream (using the yes command).

@monfresh Sounds good to me :+1:
You may still want to look at my code for how I was able to strip most of the code out of the EntityImporter subclasses. I think it would make the whole thing easier to improve in the future.

Also, I haven't made any changes to LocationImporter or the related classes for locations and addresses yet, so a comparison is probably premature.

I'm having an issue with the merge in my dev branch
running script.import:
rake aborted!
NameError: uninitialized constant FileChecker::SmarterCSV
/media/sf_vagrant/ohana-api/lib/file_checker.rb:64:in csv_entries' /media/sf_vagrant/ohana-api/lib/file_checker.rb:26:in empty?'
/media/sf_vagrant/ohana-api/lib/file_checker.rb:34:in required_but_empty?' /media/sf_vagrant/ohana-api/lib/file_checker.rb:38:in missing_or_empty?'
/media/sf_vagrant/ohana-api/lib/file_checker.rb:9:in validate' /media/sf_vagrant/ohana-api/lib/entity_importer.rb:9:in check_and_import_file'
/media/sf_vagrant/ohana-api/lib/tasks/import.rake:9:in block (2 levels) in <top (required)>' -e:1:in '
Tasks: TOP => import:all => import:organizations
I'm busy at work, but trying to figure out where my issue lies.

@cderenburger You'll need to run bundle install to pick up the smarter_csv gem.

I did pick up the gem, but perhaps I need to clear them out and grab them all again.  Had a similar issue with puma earlier.
I cloned a new directory from the master repo to test.  https://github.com/codeforamerica/ohana-api/tree/361-csv-memory seems to work great.  Far faster import, took 17 minutes for an import that usually take 45-60minutes.  Memory usage is around 1.75G

Re-running the tests with clean master copy:
master: Time: 44m  Memory: ~1.61G max
361-csv-memory: 44m  Time:  Memory: ~1.57G max
When I get home I'll upload a copy of my sample data for you to test with.  It will require settings.yml as I have service areas set.

@cderenburger Thanks!
@md5 I saw that DRYing up. It looks nice.

You can find my small data set at https://github.com/cderenburger/ohana-api/tree/smaller-dataset
I've been testing and retesting checking memory usage, each time the results are inconsistent.  If I have the Virtualbox Manager open resident memory is about 1.5G, virtual mem about 2.5G, with Virtualbox Manager closed resident memory is 150M, virtual mem is 3G.  Yet in both tests overall system memory usage tops out 3.9G.  I've been doing a vagrant halt and vagrant up between each test.
Previous test results were from watching top mem used.

I finally installed the all the dependencies to run outside of the Vagrant VM.  Running without the Vagrant machine saves nearly a Gig of memory and cut import time 39 minutes.

Thanks for the data. Is that the one that is taking 44 minutes for you? On my 2012 Macbook Air, it takes 31 minutes. The script takes so long because of the various touches that are required to populate the full-text search index. To speed up the import, which should only be a one-time task, you can do a search for , touch: true and remove them temporarily while you import. In addition, you can remove or comment out the after_add and after_remove calls in app/models/service.rb, as well as this line:
after_save :update_location_status, if: :status_changed?

With these changes, the script ran in 8 minutes.
Once the script is done, you can then populate the search index by running Location.find_each(&:touch) from the Rails console (rails c). That takes an additional 3 minutes. At this point, your DB should be ready to export via script/export, at which point you'll have a Postgres .dump file which you can then use to restore the DB via script/restore, which will only take seconds.
Finally, you'll need to add the code that you removed back.
The easiest solution to this speed problem is to update the installation instructions based on what I mentioned above. Off the top of my head, I can't think of an easy way to programmatically bypass the touches during import.

It looks like they're working on making the touch overhead less for Rails 5, but there doesn't appear to be a built-in solution for Rails 4. Here's the relevant PR: https://github.com/rails/rails/pull/19324
In the original issue (https://github.com/rails/rails/issues/18606, Limit touching to once per transaction), this gem is mentioned: https://github.com/godaddy/activerecord-delay_touching/
I believe that both the built-in solution for Rails 5 and the activerecord-delay_touching gem expect or possibly require the whole thing to be wrapped in a transaction. I suspect that wrapping it all in a transaction may also provide a performance gain all on its own, though you shouldn't expect more than a modest improvement.

Is that the one that is taking 44 minutes for you?

Outside of the VM it now takes 38 minutes for me to import.  Memory usage between the master and 361-csv-memory branch seem to take the same amount of time to import and seemed to use the same amount of memory.  I just now saw your latest commit, I haven't merged it in yet to test, will try shortly.
In Vagrant

master: Time: 44m Memory: ~1.62G max
361-csv-memory: 44m Time: Memory: ~1.57G max

Most recent test:
361-csv-memory:  Time:  48m  Memory: ~2.1G
each test in the VM had very inconsistent memory usage from one test to the next
Outside Vagrant VM
master: Time: 38m  Memory: 1G
361-csv-memory:  Time 38m  Memory: 1G
Sorry all, seems that many of the issues I've been having are apparently related to Vagrant and VirtualBox.  I thought it was great to have a dev box I could build and destroy as needed, but the memory usage ended up being an issue.  When I start a test the VM uses ~500MB more, which is expected, but as the testing goes it balloons to needing 1G more memory than the same test outside the VM.  This doubled my memory needs for this smaller data set.  I'll have to see what the effect is on the large data set.

It turns out Rails has a no_touching method, and wrapping all the save calls in a ActiveRecord::Base.no_touching do block does the trick. The import now takes less than 11 minutes total.
I've updated master with this latest code. Please pull it down again and give it a try. You won't need to change any code manually as I had mentioned in my previous comment.

It seems like it might be good to still touch all the affected objects at the end of an import (though that may lead to cascading touches through the belongs_to relations). Keeping track of the affected ids and arel instead of ActiveRecord to do a bulk update of the updated_at column should be pretty fast though.

Yep. That's what this does, although it's not fast because each location's `tsv_body

:+1:

My one concern here would be that a partial import will still update the updated_at column of all locations, not just the ones that were imported.
Thanks for pointing me to the right part of the diff.

Fantastic
I finally got a chance to pull the latest master and the import with the smaller data set took me 15 minutes to import.  This is so much faster now.
  It took about 45 minutes to load the data, then ran for the next 3 hours updating the full-text search.  I after the import script had gone to the "Updating full-text-search" I attempted to cancel the import.  I could not abort the process using Ctrl + c.  Later in the night it must have aborted at some point.  This morning I was unable to script/reset, I ended up having to sudo -u postgres dropdb ohana_api_development to finally drop the database.  Probably user error on my part.
I'll attempt the full import again today.  If the current master dropped my import time on the smaller data set to 1/3 the time, perhaps this will take about 8 hours.
I really appreciate the time and effort you have spent to help me on this issue.  Thanks @monfresh and @md5!

@md5 Good point. I'll open a new issue for that.
@cderenburger You're welcome. Thanks for helping us improve this project!


address tasks.register lazyness

The serenity gradle plugin relies on global state. This can cause several problems, in particular when applied to multiple modules and when parallelism is used.

Closing in favour of a follow-up PR with test coverage.


cUrl final url issue fixed, thanks to @alisherdavronov for support

@simialbi i think i have solved with this.

I solved it with a commit in your repo


refactor ui_draw_driver_view

I used draw_rect to fix this gap.
That one pixel gap has always existed before,it was covered by draw_rect.maybe we need to find out why this happens next time.

I used draw_rect to fix this gap.
That one pixel gap has always existed before,it was covered by draw_rect.maybe we need to find out why this happens next time.


Reduce path checking

doing this merge as a pull request with the thought that it might be good to have Perry or Daniel look at the changes to checkConjugacy before we merge to devel.

Changes reviewed, appear correct with basic testing.  Merging in.
Thanks for the improvement, @paciorek


Reconsider the behavior when a variable is double-clicked

Right now, when a variable is double-clicked, the rename dialog pops up. While renaming a variable is a commonly used feature, we should reconsider the situation since we also have stack view and variables view. Sometimes I actually wish to view the variable in stack/variables view rather than renaming it.
Since we can still press N to rename a variable, it would not be too inconvenient if we change the default behavior of double-clicking. On the other hand, there is no easy way to navigate to a variable in the stack/variables view, it will save the users some time.
I think we might default to navigate to the variable in the stack view, and make a configurable option to let the user choose from: 1. navigate to var in stack view, 2. navigate to var in variables list, 3. rename the var.
However, we might want to introduce some notification system for this because the debugger might also be interested in doing something when a variable is clicked. For example, when the target is running, the debugger can navigate to the variable on the stack/heap. We can first implement the easier solution mentioned above and deal with this more powerful implementation later.

Also, besides the double-click behavior, we might want to add a context menu that lets people directly navigate to variables list/stack view, or rename

A bit of context on this: I originally added the behavior of renaming variable when double-clicked. It received mixed feedback from the user and the team has internally discussed the change before. This issue is created to track the discussion and potential change on the behavior.

Related to #2712

I am closing this in favor of https://github.com/Vector35/binaryninja-api/issues/5269, which has a more specific title


failed to initialize origin northstar lsx running

Bug
crashes on launch, tried to put -northstar in both launch options and ns_startup_args, worked once and then after pc was turned off and back on it broke, tried replacing the files with older versions, verifying integrity, even uninstalling and reinstalling the main game twice to try and change the files back. nothing worked
Specifications

Northstar version: 1.11.1
Platform: Steam

Try to launch Origin/EA-App before launching Northstar.


File "brownie/project/sources.py", line 135, in get_fn IndexError: list index out of range

brownie Version: brownie 1.0.0b7
solc Version: 0.5.7
Python Version: 3.5
OS: osx

Contact that works on Brownie v1.0.0b2 does not work on Brownie v1.0.0b7. The error I am having when I try to compile using brownie compile.
$ brownie compile
Using solc version v0.5.7
Brownie v1.0.0b7 - Python development framework for Ethereum

Using solc version v0.5.7
Compiling contracts...
Optimizer: Enabled  Runs: 200
 - eBlocBroker.sol...
 - Lib.sol...
 - eBlocBrokerInterface.sol...
  File "brownie/cli/__main__.py", line 66, in main
    importlib.import_module("brownie.cli."+args['<command>']).main()
  File "brownie/cli/compile.py", line 24, in main
    project.load(project_path)
  File "brownie/project/__init__.py", line 91, in load
    Build()._load()
  File "brownie/project/build.py", line 67, in _load
    build_json = compiler.compile_contracts(changed_paths)
  File "brownie/project/compiler.py", line 59, in compile_contracts
    return _compile_and_format(input_json)
  File "brownie/project/compiler.py", line 80, in _compile_and_format
    compiled = _generate_pcMap(compiled)
  File "brownie/project/compiler.py", line 183, in _generate_pcMap
    'fn': sources.get_fn(contract, last[0], last[0]+last[1])
  File "brownie/project/sources.py", line 135, in get_fn
    if start < offsets[-1][1]:
IndexError: list index out of range

=> How could I fix this error?

Sorry about that. Try deleting the build folder within your project.  This will force a complete recompile of the project which (I think) should fix the issue.

After deleting the build folder still I am facing with the same error.

I was able to replicate the issue by compiling a contract without any functions.
This issue has been fixed in the upcoming release.  I'm still a couple of days away from pushing to pypi but you can clone the 1.0.0b8 branch and install it using setuptools.

I have a .sol file that contains an interface.
pragma solidity ^0.5.7;

interface eBlocBrokerInterface {
      event hello(uint val);
      ...
}


RecordType entries should be further explained

Maybe you think RecordType entries are self explanatory but I do not feel like that 😄 .  Some times I have serious doubts of which one I should use. A couple of examples:

is SharePointFileOperation also useful for folder activities?
for searching who deleted a file in a Microsoft 365 group, which RecordType should I use? OneDrive or SharePoint?

Of course I do not mean for you to answer these questions here. I think that a second page explaining the use of them as you did with the Operations parameter should be appreciated. Or maybe the same linked page would be enough.
Thank you very much.
Best regards.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 75b757c9-b775-8294-91f7-870bead6a907
Version Independent ID: 54ebab36-97bb-cd9f-6eb2-12198af885e2
Content: Search-UnifiedAuditLog (ExchangePowerShell)
Content Source: exchange/exchange-ps/exchange/Search-UnifiedAuditLog.md
Service: exchange-powershell
GitHub Login: @chrisda
Microsoft Alias: chrisda

@qosobrin, I replaced the static list of values in the RecordType or RecordTypes parameter description with:
For details about the available values, see the RecordType property description in Detailed properties in the audit log.
In the following cmdlet reference topics:

New-ActivityAlert
Set-ActivityAlert
Search-UnifiedAuditLog
Get-UnifiedAuditLogRetentionPolicy
New-UnifiedAuditLogRetentionPolicy
Set-UnifiedAuditLogRetentionPolicy

Thanks for the feedback.
Closing.

Sorry to insist after closing the issue. With these changes now we do not have a reference of the available RecordTypes anywhere. I agree with your removing of the static list from the commandLet reference page but I think that data should be moved to the linked page, not completely removed.
Thank you very much.
Best regards.

@qosobrin, believe it or not, those numbers in that topic are values for the RecordType parameter (you can use a numerical value or the text value). Coincidentally, I was speaking to the owner of that topic, and he suggested an even better target, which I used instead: https://docs.microsoft.com/office/office-365-management-api/office-365-management-activity-api-schema#auditlogrecordtype.
I was thinking of doing something like this a long time ago. It's difficult if not impossible to keep a long list of values like that synchronized between 6 different cmdlet reference topics. The target topic is now the one source of truth. In fact, the writer of https://docs.microsoft.com/microsoft-365/compliance/detailed-properties-in-the-office-365-audit-log is going to update the description for RecordType so it also points to https://docs.microsoft.com/office/office-365-management-api/office-365-management-activity-api-schema#auditlogrecordtype.
Also, be advised there are lots of available RecordType values, but many of them aren't hooked up, always return empty value, require external connectors, etc.

Thank you very much. It is much better now.

Thanks for the feedback.
Closing


Always On Top

Description of the new feature / enhancement
The ability to show an application on top of others.
Scenario when this would be used?
Working away on my study in a Word Document and I wanted to watch a movie at the same time in VLC Player.
I would like to Pin VLC Player to always be placed on top of other applications.
Supporting information
No response

Issue is a duplicate
Thank you for your issue! We've linked your report against another issue #13
Thanks for helping us to make PowerToys a better piece of software.

Duplicate of #13

This is in progress too :D
Hope we can ship this soon.

This is a necessity!! Can't stress the importance of this feature enough in saving lots of unnecessary clicks especially now that Taskbar drag and drop file support is gone.

goal is Dec release


Ability to save dashboard table column filters for later re-use.

A dashboard table can only have a single filter. If you want a different filter you need to delete/overwrite the exiting filter.
It would be useful to be able to save (at least a few) filters on dashboard columns so you can easily swap between different views.

I assume this is similar to the concept of fieldsets?

I guess it depends how you approach the fieldset stuff. I was thinking of raising  a separate Issue# for fieldsets but happy for it all to be wrapped into one thing.
My perception (and maybe this isn't normal convention) of a fieldset is a predefined set of index fields and custom expressions that define the columns I have presented to me in the dashboard table. If I select a different fieldset then the column view of my dashboard table changes.
Column Filters to me is another layer across the top of fieldsets that further constrains the data presented in any given column in the table.
Need to decide whether such filters and fieldsets are available at a dashboard global level and available to be selected in multiple dashboards, or whether it is at a per dashboard level. I tend to favour the global approach to minimise re-work but if this seriously inflates the difficulty of making it happen then evn per dashboard would be welcome enhancement.
Also moving the fields around within a dashboard is cumbersome and and being able to move relative column positions with a dialog box and using say up/down arrows would be great.
I see this particular request Pete raised in another channel more eloquently than me here ("Pop-up dialog for visually sorting columns via drag/drop. Currently, moving a newly-inserted column to the beginning is a pain, due to having to fight the horizontal scrollbar. OR: add heading context items like "Move to Start" and "Move to End" (or similar)"
I don't know that all the other ehancement requests that Pete raised then (12th Nov in the main channel) have been captured as an issue requested.
Shall I just capture them here before they are lost as they were all dashboard enhancement requests??


Write documentation and tag 1.0 release

Documentation:

Update readme with basic info, install instructions
Remove mentions of "experimental" or "early work"
Document known issues in readme

Probably need two other doc pages:

one for config (defaults, examples)
some recommendations for compatible code (not everything you can do in Kirby or PHP will carry over nicely to a static build!)

Documentation still needs some work.

Now in pretty good shape.