Nebula nodejs SDK implementation #6

wujjpp · 2021-12-02T07:31:31Z

Nebula Nodejs SDK

This repository provides Nebula client API in Nodejs.

Features

Muti-Server Support
Auto-reconnection support

Client will try to reconnect forever, until the server is available again.
Connection pool support
Disconnection detection

A heartbeat mechanism is implemented, client will send ping to server each pingInterval ms for detect connective
Thrift enhancement

fix auto reconnect issue#2407

fix performance issue in huge data scene#2483

API

Connection Options

parameter	type	description
servers	string[]	nebula servers
userName	string	username for login
password	string	password for login
poolSize	number	Pool size for each server(Optional, default：5)
bufferSize	number	Command cache in offline or before established connect (Optional, defaul: 2000)
executeTimeout	number	Command executing timeout in ms (Optional, default：10000)
pingInterval	number	for keepalive, ping duration in ms, (Optional, default：60000）

How To

Install

For compiling C++ native module, node-gyp is required, you can install node-gyp by npm install -g node-gyp

npm install nebula-nodejs --save --unsafe-perm

Simple and convenient API

// ESM
import { createClient } from 'nebula-nodejs'

// CommonJS
// const { createClient } = require('nebula-nodejs')

// Connection Options
const options = {
  servers: ['ip-1:port','ip-2:port'],
  userName: 'xxx',
  password: 'xxx',
  database: 'database name',
  poolSize: 5,
  bufferSize: 2000,
  executeTimeout: 15000,
  pingInterval: 60000
}

// Create client
const client = createClient(options)

// Execute command
// 1. return parsed data (recommend)
const response = await client.execute('GET SUBGRAPH 3 STEPS FROM -7897618527020261406')
// 2. return nebula original data
const responseOriginal = await client.execute('GET SUBGRAPH 3 STEPS FROM -7897618527020261406', true)

Events

parameter	description
sender	the individual connection in connection pool
error	Nebula Error
retryInfo	Retry information
retryInfo.delay	delay time
retryInfo.attempt	total attempts

const client = createClient(options)

// connection is ready for executing command
client.on('ready', ({sender}) => {

})

// error occurs
client.on('error', ({ sender, error }) => {

})

// connected event
client.on('connected', ({ sender }) => {

})

// authorized successfully
client.on('authorized', ({ sender }) => {

})

// reconnecting
client.on('reconnecting', ({ sender, retryInfo }) => {

})

// closed
client.on('close', { sender }) => {

}

About hash64 function

nebula-nodejs exports hash64 function for converting string to string[], it's based on MurmurHash3.

// ESM
import { hash64 } from 'nebula-nodejs'

// CommonJS
// const { hash64 } = require('nebula-nodejs')

const results = hash64('f10011b64aa4e7503cd45a7fdc24387b')

console.log(results)

// Output:
// ['2852836996923339651', '-6853534673140605817']

About Int64

nodejs cannot repreent Int64, so we convert Int64 bytes to string

// ESM
import { bytesToLongLongString } from 'nebula-nodejs'

// CommonJS
// const { bytesToLongLongString } = require('nebula-nodejs')

const s = '-7897618527020261406'

const buffer = [146, 102, 5, 203, 5, 105, 223, 226]
const result = bytesToLongLongString(buffer)

// result equals s

Development

Build

git clone https://github.com/vesoft-inc/nebula-node.git
cd nebula-node
npm install --unsafe-perm
npm run build

Unit Test

npm run build
npm run test

Unit Test Coverage

npm run coverage

Publish

npm run build
cd dist
npm publish

TODO

Not implemented data type for auto parser

Data Type	property name in nebula response
NMap	mVal
NSet	uVal
DataSet	gVal

CLAassistant · 2021-12-02T07:31:35Z

All committers have signed the CLA.

Reid00 · 2021-12-02T07:39:19Z

@wey-gu here is the pr, pls take a look.

wey-gu · 2021-12-06T01:55:27Z

@wey-gu here is the pr, pls take a look.

Thank you @Reid00 @wujjpp! Great Job! You even fixed two issues on upstream apache thrift 👍🏻. We will look into it.

Best Regards, Wey

README.md

nianiaJR · 2021-12-06T12:29:00Z

package.json

+    "sinon-chai": "^3.7.0",
+    "source-map-support": "^0.5.19",
+    "typescript": "4.1.5",
+    "uuid": "^8.3.2",


should these lib be in the package.json dependencies instead of devDependencies ?

browser-or-node, isomorphic-ws, node-int64, q, ws dependent by thrift, because we have copy & modified thrift in local package, so we should add thoese libs as dependencies

bindings, lodash dependent by nebula-node

all other libs treat as devDependencies

👌，I just find all the libs in the devDependencies.

nianiaJR · 2021-12-06T12:30:14Z

package.json

+    "inquirer": "^8.1.1",
+    "mkdirp": "^0.5.1",
+    "mocha": "^8.4.0",
+    "moment": "^2.29.1",


Just for advice: Is it better to use day.js instead of monent.js for reducing the package size? https://medium.datadriveninvestor.com/https-medium-com-sabesan96-why-you-should-choose-day-js-instead-of-moment-js-9cf7bb274bbd

moment is useless, I will remove it

nianiaJR · 2021-12-06T17:25:44Z

src/nebula/Client.ts

+
+try {
+  // eslint-disable-next-line @typescript-eslint/no-var-requires
+  const asyncHooks = require('async_hooks')


I'm curious about why not just import import async_hooks directly, is it possible that the module can't be imported?

BTW, I find you use import other place: https://github.com/vesoft-inc/nebula-node/blob/5d71bfd743ae512d2dc490b47897022cce5d6d4b/src/nebula/types.ts

async_hooks module added in nodejs v8.1.0 and in fact nebula-node can be run on nodejs below v8.1.0, although I have specifed v10.0.0 in package.json. we use require for detecting the target node compatibility. If it not support, our module can also working fine.

Directory Layout:
.
├── /dist/ # Build results for publishing (this folder generated after npm run build, its for publish)
├── /specs/ # Misc knowledge and docs
├── /src/ # Source Dir
├── /tests/ # Unit test case, if you have nebula server, Put your test case in this folder for testing
├── /tools/ # Tiny build system
└── .... # Supporting files

Build process:

Build & Publish:
In short: typescript -> ESM -> CommonJS -> Copy & Generate necessary assets -> publish

wujjpp · 2021-12-16T16:20:49Z

@wey-gu 这个PR还合吗？

nianiaJR · 2021-12-20T02:06:10Z

We'll deal with it in this week, thanks for your following🤝

wey-gu · 2021-12-20T02:14:06Z

@wey-gu 这个PR还合吗？

Sorry for the late response, sure we will, hopefully it could be done in short time as aligned with @nianiaJR

nianiaJR

@wujjpp Is there any plan to make up the unit test/example in future?

I find it only like a todo demo about test?

nianiaJR · 2021-12-20T03:29:12Z

src/nebula/types.ts

+  // 密码
+  password: string;
+  // 数据库名称
+  database: string;


It's for space? maybe it's more clear to rename it to space ?

@nianiaJR rename database to space done

nianiaJR

I've simply tried this lib, work fine. Because of it's too big changes, it's important to make up the enough unit test/examples for long-term maintaining. Thanks for your contributions. I'll approved it and make it merged soon🤝

wujjpp · 2021-12-21T10:09:16Z

@wujjpp Is there any plan to make up the unit test/example in future?

I find it only like a todo demo about test?

I have a full test case located at tests/nebula.test-disabled, but I didn't have nebula enviroment for testing in github env.
So, I rename nebula.test.ts to nebula.test-disabled, you can revert its name and add unit tests for testing.
BTW, those test cases are based on our nebula production server.

/* eslint-disable max-len */
/**
 * Created by Wu Jian Ping on - 2021/06/09.
 */

import { expect } from 'chai'
import createClient, { ClientOption } from '../dist'

const relationServer: ClientOption = {
  servers: ['10.0.1.100:9669', '10.0.1.101:9669', '10.0.1.102:9669'],
  userName: 'xxxxx',
  space: 'xxxxx',
  database: 'partion1',
  pingInterval: 5000,
  poolSize: 2
}

const riskInfoServer: ClientOption = {
  servers: ['10.0.1.100:9669', '10.0.1.101:9669', '10.0.1.102:9669'],
  userName: 'xxxxx',
  space: 'xxxxx',
  database: 'partion2',
  pingInterval: 5000,
  poolSize: 2
}

const commands = {
  cmd1: 'GET SUBGRAPH 2 STEPS FROM -7897618527020261406',
  cmd2: 'fetch prop on Company -7897618527020261406',
  cmd3: 'go from 815677140545765099 over Relation yield Relation._src as src, Relation._dst as dst, Relation.relation_name as relation_name, Relation.prop1 as prop1, Relation.prop2 as prop2, Relation.prop3 as prop3, Relation.prop4 as prop4, $^.Person.name as src_p_name, $^.Company.name as src_c_name,  $$.Person.name as dst_p_name, $$.Company.name as dst_c_name | limit 1000',
  cmd4: 'find noloop path from -4075961126534726064 to 815677140545765099 over Relation bidirect upto 2 steps',
  cmd5: 'fetch prop on Relation -4075961126534726064->815677140545765099@-5101473608195470769'
}

describe('nebula', () => {
  // eslint-disable-next-line prefer-arrow/prefer-arrow-functions
  it('test-case-1', async () => {
    const client = createClient(relationServer)

    const response1 = await client.execute(commands.cmd1)

    expect(response1.data?._vertices?.length).greaterThan(0)
    expect(response1.data?._edges?.length).greaterThan(0)

    await client.close()
  })

  it('test-case-2', async () => {
    const client = createClient(relationServer)

    const response1 = await client.execute(commands.cmd2)

    expect(response1.data?.vertices_?.length).greaterThan(0)

    await client.close()
  })

  it('test-case-3', async () => {
    const client = createClient(riskInfoServer)

    const response1 = await client.execute(commands.cmd3)
    expect(response1.data?.src?.length).greaterThan(0)

    await client.close()
  })

  it('test-case-4', async () => {
    const client = createClient(riskInfoServer)

    const response1 = await client.execute(commands.cmd4)

    expect(response1.data?.path?.length).greaterThan(0)

    await client.close()
  })

  it('test-case-5', async () => {
    const client = createClient(riskInfoServer)

    const response1 = await client.execute(commands.cmd5)

    expect(response1.data?.edges_?.length).greaterThan(0)

    await client.close()
  })

  it('test-promise-all', async () => {
    const client = createClient(relationServer)
    const [resp1, resp2, resp3] = await Promise.all([
      client.execute('GET SUBGRAPH 1 STEPS FROM -7897618527020261406', false),
      client.execute('GET SUBGRAPH 2 STEPS FROM -7897618527020261406', false),
      client.execute('GET SUBGRAPH 3 STEPS FROM -7897618527020261406', false)
    ])

    console.log(`resp11:${resp1.data._vertices.length}, ${resp1.data._edges.length}`) // eslint-disable-line
    console.log(`resp21:${resp2.data._vertices.length}, ${resp2.data._edges.length}`) // eslint-disable-line
    console.log(`resp31:${resp3.data._vertices.length}, ${resp3.data._edges.length}`) // eslint-disable-line

    await client.close()
  })

  after(async () => {
    process.exit()
  })
})

nianiaJR

LGTM

wujjpp added 6 commits December 1, 2021 14:26

empty origin repository

051a489

initial

2b0a3f6

update package name

e37e5f6

add npm ignore file

9c4b313

enhance

acd92b3

enahnce readme

a7ca049

wujjpp changed the title ~~Dev v0.1.0~~ Nebula nodejs SDK implementation Dec 2, 2021

wujjpp and others added 21 commits December 2, 2021 16:27

remove internal dependences

b7586b5

update readme.md

66f1398

update readme

adf24ed

update readme

d280fee

update readme

fb87924

update readme

1cc1053

update readme

1651597

update readme

e1fc3f1

update readme

5e51e5f

update readme

45e3367

update readme

7501e95

.

4c48955

remove github workflow

d435bff

Create test.yml

92cd6b5

Update test.yml

8e61938

Update test.yml

0ba56d9

.

71b7da4

Merge branch 'master' of https://github.com/wujjpp/nebula-node

8e315ed

.

c37cebb

remove github workflow

c94b822

update build.js

3f38a8c

wujjpp added 4 commits December 2, 2021 20:43

add keywords in package.json

b779141

update readme

f17fc8d

update readme

8f833a3

convert stmt from string to buffer

85479e2

wey-gu reviewed Dec 6, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

wujjpp added 3 commits December 6, 2021 11:44

update readme, for fix colon in config option example

bd946c7

update readme

81d2761

add CommonJS import

40ab88c

nianiaJR reviewed Dec 6, 2021

View reviewed changes

remove useless package

5d71bfd

nianiaJR reviewed Dec 6, 2021

View reviewed changes

fix utf8 length issue

a468a7b

nianiaJR reviewed Dec 20, 2021

View reviewed changes

rename database to space

9c46e8a

update readme and test case

07f6a50

nianiaJR approved these changes Dec 21, 2021

View reviewed changes

wey-gu merged commit e4ae63a into nebula-contrib:master Dec 22, 2021

wey-gu mentioned this pull request Dec 22, 2021

contribute node.js client #5

Closed

jamieliu1023 mentioned this pull request Dec 25, 2021

Weekly Report 2021-12-24 vesoft-inc/nebula-community#80

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nebula nodejs SDK implementation #6

Nebula nodejs SDK implementation #6

wujjpp commented Dec 2, 2021 •

edited

Loading

CLAassistant commented Dec 2, 2021 •

edited

Loading

Reid00 commented Dec 2, 2021

wey-gu commented Dec 6, 2021

nianiaJR Dec 6, 2021

wujjpp Dec 6, 2021 •

edited

Loading

nianiaJR Dec 6, 2021

nianiaJR Dec 6, 2021 •

edited

Loading

wujjpp Dec 6, 2021

nianiaJR Dec 6, 2021

nianiaJR Dec 6, 2021

wujjpp Dec 7, 2021 •

edited

Loading

wujjpp Dec 7, 2021 •

edited

Loading

wujjpp commented Dec 16, 2021

nianiaJR commented Dec 20, 2021

wey-gu commented Dec 20, 2021

nianiaJR left a comment

nianiaJR Dec 20, 2021

wujjpp Dec 21, 2021

nianiaJR left a comment

wujjpp commented Dec 21, 2021 •

edited

Loading

nianiaJR left a comment

Nebula nodejs SDK implementation #6

Nebula nodejs SDK implementation #6

Conversation

wujjpp commented Dec 2, 2021 • edited Loading

Nebula Nodejs SDK

Features

API

Connection Options

How To

Install

Simple and convenient API

Events

About hash64 function

About Int64

Development

Build

Unit Test

Unit Test Coverage

Publish

TODO

CLAassistant commented Dec 2, 2021 • edited Loading

Reid00 commented Dec 2, 2021

wey-gu commented Dec 6, 2021

nianiaJR Dec 6, 2021

Choose a reason for hiding this comment

wujjpp Dec 6, 2021 • edited Loading

Choose a reason for hiding this comment

nianiaJR Dec 6, 2021

Choose a reason for hiding this comment

nianiaJR Dec 6, 2021 • edited Loading

Choose a reason for hiding this comment

wujjpp Dec 6, 2021

Choose a reason for hiding this comment

nianiaJR Dec 6, 2021

Choose a reason for hiding this comment

nianiaJR Dec 6, 2021

Choose a reason for hiding this comment

wujjpp Dec 7, 2021 • edited Loading

Choose a reason for hiding this comment

wujjpp Dec 7, 2021 • edited Loading

Choose a reason for hiding this comment

wujjpp commented Dec 16, 2021

nianiaJR commented Dec 20, 2021

wey-gu commented Dec 20, 2021

nianiaJR left a comment

Choose a reason for hiding this comment

nianiaJR Dec 20, 2021

Choose a reason for hiding this comment

wujjpp Dec 21, 2021

Choose a reason for hiding this comment

nianiaJR left a comment

Choose a reason for hiding this comment

wujjpp commented Dec 21, 2021 • edited Loading

nianiaJR left a comment

Choose a reason for hiding this comment

wujjpp commented Dec 2, 2021 •

edited

Loading

CLAassistant commented Dec 2, 2021 •

edited

Loading

wujjpp Dec 6, 2021 •

edited

Loading

nianiaJR Dec 6, 2021 •

edited

Loading

wujjpp Dec 7, 2021 •

edited

Loading

wujjpp Dec 7, 2021 •

edited

Loading

wujjpp commented Dec 21, 2021 •

edited

Loading