Elastic Search Error - Invalid UTF-8 byte (demo data) #498

patrickhoefer · 2018-09-13T09:04:46Z

Gentics Mesh Version, operating system, or hardware.

v0.24.0 (with demo data)

Operating System

Windows

JVM

Java(TM) SE Runtime Environment (build 1.8.0_151-b12)

Problem

Elastic Search error while indexing some documents containing strange characters.

Exception:

[2018-09-13T10:42:39,974][DEBUG][o.e.a.b.TransportShardBulkAction] [mesh-node-217f8c981ada4642bf8c981adaa642c3-879033dba4874e459033dba4870e45e7-b9ed88d0f8114cc3ad88d0f811acc3ab-draft][1] failed to execute bulk item (index) BulkShardRequest [[mesh-node-217f8c981ada4642bf8c981adaa642c3-879033dba4874e459033dba4870e45e7-b9ed88d0f8114cc3ad88d0f811acc3ab-draft][1]] containing [index {[mesh-node-217f8c981ada4642bf8c981adaa642c3-879033dba4874e459033dba4870e45e7-b9ed88d0f8114cc3ad88d0f811acc3ab-draft][default][adaf48da8c124049af48da8c12a0493e-en], source[{"uuid":"adaf48da8c124049af48da8c12a0493e","editor":{"uuid":"e5861ba26b914b21861ba26b91ab211a"},"edited":"2018-07-25T12:04:29Z","creator":{"uuid":"e5861ba26b914b21861ba26b91ab211a"},"created":"2018-07-25T12:04:29Z","project":{"name":"demo","uuid":"217f8c981ada4642bf8c981adaa642c3"},"tags":{"name":["Gasoline","Plutonium"],"uuid":["1c51487f8e8043c291487f8e8053c2c1","1d0f65dced8048538f65dced80c853ea"]},"tagFamilies":{"Fuels":{"uuid":"f36400b436d244c3a400b436d244c3f4","tags":[{"name":"Gasoline","uuid":"1c51487f8e8043c291487f8e8053c2c1"},{"name":"Plutonium","uuid":"1d0f65dced8048538f65dced80c853ea"}]}},"_roleUuids":["cc9999a4194d4f069999a4194d1f06d7","ddcddceb33d648318ddceb33d618314f","be703d9cccdf4771b03d9cccdff771c1"],"parentNode":{"uuid":"ca6c7df3f45b48d4ac7df3f45ba8d42f"},"language":"en","schema":{"name":"vehicle","uuid":"2aa83a2b3cba40a1a83a2b3cba90a1de","version":"1.0"},"fields":{"vehicleImage":"3db965753d9d41f0b965753d9da1f019","price":1000000,"name":"DeLorean DMC-12","weight":1230,"description":"The DeLorean DMC-12 (commonly referred to simply as \"the DeLorean,\" as it was the only model ever produced by the company) is a sports car manufactured by John DeLorean's DeLorean Motor Company for the American market from 1981 83. The car features gull-wing doors and an innovative fiberglass body structure with a steel backbone chassis, along with external brushed stainless steel body panels. It became widely known and iconic for its appearance, and a modified DMC-12 was immortalized as the DeLorean time machine in the Back to the Future media franchise..","SKU":6,"stocklevel":50,"slug":"delorean-dmc-12","niceUrl":"/cars/outatime"},"displayField":{"key":"name","value":"DeLorean DMC-12"},"version":"bb170ef3"}]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [fields.description]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:302) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:607) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:407) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:482) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:500) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:394) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:67) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:261) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:708) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:686) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:667) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:548) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequest(TransportShardBulkAction.java:140) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:236) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:123) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:110) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:72) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1033) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1011) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:358) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:298) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:974) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:971) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2211) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:983) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:97) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:319) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:294) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:281) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.1.2.jar:6.1.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0x96
 at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@29f6de67; line: 1, column: 1245]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.10.jar:2.8.10]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.10.jar:2.8.10]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3544) ~[jackson-core-2.8.10.jar:2.8.10]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3538) ~[jackson-core-2.8.10.jar:2.8.10]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2543) ~[jackson-core-2.8.10.jar:2.8.10]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2469) ~[jackson-core-2.8.10.jar:2.8.10]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:315) ~[jackson-core-2.8.10.jar:2.8.10]
        at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:237) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.TextFieldMapper.parseCreateField(TextFieldMapper.java:345) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:297) ~[elasticsearch-6.1.2.jar:6.1.2]
        ... 42 more

Reproducer

start mesh-demo.jar and check logs

The text was updated successfully, but these errors were encountered:

Jotschi · 2019-06-12T14:19:29Z

I assume this error happens since the input string is not converted to UTF-8. If there is content which does not contain UTF-8 (e.g. windows-1252, ISO8859-1) than the parser in ES will fail because it can only handle UTF-8. Possible options:

Sanitize/convert content before storing it in Gentics Mesh
Sanitze/convert content before sending it to ES

Jotschi · 2019-06-18T12:36:36Z

@patrickhoefer I was not able to reproduce the issue. Can you please POST a request (incl. headers) that produces the issue?

Jotschi · 2019-06-18T13:48:47Z

Not reproducible.

kaplanke · 2019-06-20T13:47:43Z

Text fields with Turkish characters due to elastic search integration.

Os:
Windows 10 Pro

Java:
C:\dev\getmesh\a>java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

Run:
java -jar mesh-demo-0.35.0.jar

Reproduce:
Try to create a vehicle with Turkish characters in name and slug.

Error:

16:40:36.335 [Arrogant Mightyena] INFO  [vert.x-worker-thread-3] - 0:0:0:0:0:0:0:1 - - [Thu, 20 Jun 2019 13:40:36 GMT] "GET /api/v2/demo/nodes/ca6c7df3f45b48d4ac7df3f45ba8d42f?lang=en,de HTTP/1.1" 200 1960 "http://localhost:8080/mesh-ui/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
16:40:36.346 [Arrogant Mightyena] INFO  [vert.x-worker-thread-1] - 0:0:0:0:0:0:0:1 - - [Thu, 20 Jun 2019 13:40:36 GMT] "GET /api/v2/demo/nodes/7402161d8cbc4ad282161d8cbc6ad22b?expandAll=true&lang=en,de HTTP/1.1" 200 1989 "http://localhost:8080/mesh-ui/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
16:40:36.352 [Arrogant Mightyena] INFO  [vert.x-worker-thread-6] - Flush request received
[2019-06-20T16:40:36,357][DEBUG][o.e.a.b.TransportShardBulkAction] [mesh-node-217f8c981ada4642bf8c981adaa642c3-0d0bef188fc24e078bef188fc24e0774-f04dcf1a697545fc8dcf1a697505fcd8-draft][2] failed to execute bulk item (index) BulkShardRequest [[mesh-node-217f8c981ada4642bf8c981adaa642c3-0d0bef188fc24e078bef188fc24e0774-f04dcf1a697545fc8dcf1a697505fcd8-draft][2]] containing [index {[mesh-node-217f8c981ada4642bf8c981adaa642c3-0d0bef188fc24e078bef188fc24e0774-f04dcf1a697545fc8dcf1a697505fcd8-draft][default][7402161d8cbc4ad282161d8cbc6ad22b-en], source[{"uuid":"7402161d8cbc4ad282161d8cbc6ad22b","editor":{"uuid":"3daea7737a584f51aea7737a58df51da"},"edited":"2019-06-20T13:40:36.302Z","creator":{"uuid":"3daea7737a584f51aea7737a58df51da"},"created":"2019-06-20T13:40:36.299Z","project":{"name":"demo","uuid":"217f8c981ada4642bf8c981adaa642c3"},"tags":{"name":[],"uuid":[]},"tagFamilies":{},"_roleUuids":["83c166d4617947fd8166d4617987fdec","ddcddceb33d648318ddceb33d618314f","1b59a678013d45d899a678013d45d8dc"],"parentNode":{"uuid":"ca6c7df3f45b48d4ac7df3f45ba8d42f"},"language":"en","schema":{"name":"vehicle","uuid":"2aa83a2b3cba40a1a83a2b3cba90a1de","version":"1.0"},"fields":{"name":"?��","slug":"?�?","niceUrl":"?�?"},"displayField":{"key":"name","value":"?��"},"version":"afde8fa1"}]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [fields.name]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:302) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:607) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:407) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:482) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:500) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:394) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:67) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:261) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:708) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:686) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:667) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:548) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequest(TransportShardBulkAction.java:140) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:236) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:123) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:110) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:72) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1033) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1011) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:104) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:358) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:298) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:974) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:971) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2211) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:983) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:97) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:319) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:294) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:281) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:652) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:637) [elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.1.2.jar:6.1.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_181]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_181]
Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0xfc
 at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@a534be4; line: 1, column: 637]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.10.jar:2.8.10]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.10.jar:2.8.10]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3544) ~[jackson-core-2.8.10.jar:2.8.10]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3538) ~[jackson-core-2.8.10.jar:2.8.10]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2543) ~[jackson-core-2.8.10.jar:2.8.10]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2469) ~[jackson-core-2.8.10.jar:2.8.10]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:315) ~[jackson-core-2.8.10.jar:2.8.10]
        at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:237) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.TextFieldMapper.parseCreateField(TextFieldMapper.java:345) ~[elasticsearch-6.1.2.jar:6.1.2]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:297) ~[elasticsearch-6.1.2.jar:6.1.2]
        ... 42 more
16:40:36.368 [Arrogant Mightyena] ERROR [OkHttp http://localhost:9200/...] - Could not store document {mesh-node-217f8c981ada4642bf8c981adaa642c3-0d0bef188fc24e078bef188fc24e0774-f04dcf1a697545fc8dcf1a697505fcd8-draft:7402161d8cbc4ad282161d8cbc6ad22b-en} - mapper_parsing_exception : failed to parse [fields.name]
16:40:36.369 [Arrogant Mightyena] ERROR [OkHttp http://localhost:9200/...] - Request failed {Storing document batch.}
16:40:36.369 [Arrogant Mightyena] ERROR [OkHttp http://localhost:9200/...] - com.gentics.mesh.search.impl.ElasticsearchBulkResponseError: failed to parse [fields.name]
16:40:36.369 [Arrogant Mightyena] ERROR [OkHttp http://localhost:9200/...] - Error after sending request to Elasticsearch
com.gentics.mesh.search.impl.ElasticsearchBulkResponseError: failed to parse [fields.name]
        at com.gentics.mesh.search.impl.ElasticSearchProvider.lambda$processBulk$23(ElasticSearchProvider.java:348)
        at io.reactivex.internal.operators.single.SingleFlatMap$SingleFlatMapCallback.onSuccess(SingleFlatMap.java:76)
        at io.reactivex.internal.operators.single.SingleCreate$Emitter.onSuccess(SingleCreate.java:68)
        at com.gentics.elasticsearch.client.okhttp.ElasticsearchOkClient$1.onResponse(ElasticsearchOkClient.java:178)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
16:40:36.370 [Arrogant Mightyena] ERROR [OkHttp http://localhost:9200/...] - Not retrying because it is an error inside elasticsearch.
16:40:36.370 [Arrogant Mightyena] INFO  [OkHttp http://localhost:9200/...] - Flushing requests

POST data: (To show that the posted data is in proper UTF-8 format)

C:\dev\getmesh\a>CHCP 65001
Active code page: 65001

C:\dev\getmesh\a>\dev\netcat\ncat.exe -kvl 8080
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Listening on :::8080
Ncat: Listening on 0.0.0.0:8080
Ncat: Connection from ::1.
Ncat: Connection from ::1:52134.
POST /api/v2/demo/nodes/7402161d8cbc4ad282161d8cbc6ad22b?expandAll=true&lang=en,de HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 1523
Accept: application/json, text/plain, */*
Origin: http://localhost:8080
Anonymous-Authentication: disable
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36
Content-Type: application/json;charset=UTF-8
Referer: http://localhost:8080/mesh-ui/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8,tr;q=0.7
Cookie: refresh_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX25hbWUiOiJhZG1pbkBib3J1c2FuLmNvbSIsInNjb3BlIjpbInJlYWQiLCJ3cml0ZSIsIndlYiJdLCJhdGkiOiI4MWQwMjMxZC0wNTEwLTQyZmQtYWI5Mi1mZjY4NmEyZmRkOTYiLCJleHAiOjE1NjMzNDM3OTUsImF1dGhvcml0aWVzIjpbIlVTRVIiXSwianRpIjoiNzcyNDFmYTQtYjIzMi00Zjg4LThkZDgtNjBjNjE5Y2UwZjU0IiwiY2xpZW50X2lkIjoid2ViX2NsaWVudCJ9.Ghy3DhMACApn9dHiad3kwnsznhz1GeWK36jClyp61j0; access_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NjA3OTQ5OTUsInVzZXJfbmFtZSI6ImFkbWluQGJvcnVzYW4uY29tIiwiYXV0aG9yaXRpZXMiOlsiVVNFUiJdLCJqdGkiOiI4MWQwMjMxZC0wNTEwLTQyZmQtYWI5Mi1mZjY4NmEyZmRkOTYiLCJjbGllbnRfaWQiOiJ3ZWJfY2xpZW50Iiwic2NvcGUiOlsicmVhZCIsIndyaXRlIiwid2ViIl19.6iPB-Wu0PuplhwMayMErbIFaqKQ9Uv8mghyeZ3KIrEA; mesh.token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyVXVpZCI6IjNkYWVhNzczN2E1ODRmNTFhZWE3NzM3YTU4ZGY1MWRhIiwiaWF0IjoxNTYxMDM4MDg5LCJleHAiOjE1NjEwNDE2ODl9.aU5ukjuHObNZZRzDPZGEaXpsI3C8JJ_914mvQpLcpp0

{"uuid":"7402161d8cbc4ad282161d8cbc6ad22b","creator":{"uuid":"3daea7737a584f51aea7737a58df51da"},"created":"2019-06-20T13:40:36.299Z","editor":{"uuid":"3daea7737a584f51aea7737a58df51da"},"edited":"2019-06-20T13:41:26.888Z","language":"en","availableLanguages":{"en":{"published":false,"version":"0.2"}},"parentNode":{"projectName":"demo","uuid":"ca6c7df3f45b48d4ac7df3f45ba8d42f","displayName":"Automobiles","schema":{"name":"category","uuid":"2ca2362b041247c4a2362b041227c4da"}},"tags":[],"project":{"name":"demo","uuid":"217f8c981ada4642bf8c981adaa642c3"},"childrenInfo":{},"schema":{"name":"vehicle","uuid":"2aa83a2b3cba40a1a83a2b3cba90a1de","version":"1.0","versionUuid":"f04dcf1a697545fc8dcf1a697505fcd8"},"container":false,"displayField":"name","displayName":"ğüü","fields":{"slug":"ğüiğü","niceUrl":null,"name":"ğüü","weight":null,"SKU":null,"price":null,"stocklevel":null,"description":null,"vehicleImage":null},"breadcrumb":[{"projectName":"demo","uuid":"4fbb0f45dcc5421cbb0f45dcc5a21cfb","schema":{"name":"folder","uuid":"0c5b8433b8f641659b8433b8f6f165c0"}},{"projectName":"demo","uuid":"ca6c7df3f45b48d4ac7df3f45ba8d42f","displayName":"Automobiles","schema":{"name":"category","uuid":"2ca2362b041247c4a2362b041227c4da"}},{"projectName":"demo","uuid":"7402161d8cbc4ad282161d8cbc6ad22b","displayName":"ğüü","schema":{"name":"vehicle","uuid":"2aa83a2b3cba40a1a83a2b3cba90a1de"}}],"version":"0.2","permissions":{"create":true,"read":true,"update":true,"delete":true,"publish":true,"readPublished":true}}

Jotschi · 2019-06-20T14:00:29Z

@kaplanke Thanks for your feedback. I reopened the issue.

kaplanke · 2019-06-20T19:33:04Z

As a workaround you may provide

-Dfile.encoding=UTF-8

option while starting mesh. E.g.

java -Dfile.encoding=UTF-8 -jar mesh-demo-0.36.1.jar

Jotschi · 2019-06-27T11:20:33Z

The issue has been fixed. The fix will be included in the next release.

Jotschi · 2019-07-03T11:13:15Z

Fix has been released with 0.36.2

patrickhoefer added the bug label Sep 13, 2018

Jotschi added the f/elasticsearch label Apr 24, 2019

Jotschi mentioned this issue Jun 12, 2019

Display problem with Chinese String #610

Closed

Jotschi added the major Major Issue label Jun 12, 2019

Jotschi closed this as completed Jun 18, 2019

Jotschi reopened this Jun 20, 2019

Jotschi self-assigned this Jun 26, 2019

Jotschi closed this as completed Jul 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elastic Search Error - Invalid UTF-8 byte (demo data) #498

Elastic Search Error - Invalid UTF-8 byte (demo data) #498

patrickhoefer commented Sep 13, 2018 •

edited

Loading

Jotschi commented Jun 12, 2019

Jotschi commented Jun 18, 2019

Jotschi commented Jun 18, 2019

kaplanke commented Jun 20, 2019 •

edited by Jotschi

Loading

Jotschi commented Jun 20, 2019

kaplanke commented Jun 20, 2019

Jotschi commented Jun 27, 2019

Jotschi commented Jul 3, 2019

Elastic Search Error - Invalid UTF-8 byte (demo data) #498

Elastic Search Error - Invalid UTF-8 byte (demo data) #498

Comments

patrickhoefer commented Sep 13, 2018 • edited Loading

Gentics Mesh Version, operating system, or hardware.

Operating System

JVM

Problem

Reproducer

Jotschi commented Jun 12, 2019

Jotschi commented Jun 18, 2019

Jotschi commented Jun 18, 2019

kaplanke commented Jun 20, 2019 • edited by Jotschi Loading

Jotschi commented Jun 20, 2019

kaplanke commented Jun 20, 2019

Jotschi commented Jun 27, 2019

Jotschi commented Jul 3, 2019

patrickhoefer commented Sep 13, 2018 •

edited

Loading

kaplanke commented Jun 20, 2019 •

edited by Jotschi

Loading