Sugr Server Development From Scratch

This article is about the way we choosing frameworks during Sugr Server development.

We change the frameworks many times until we found the best one, which helps us on coding, deploying, debugging, and saving the cost of VPS.

SQL, NoSQL, QueryBuilder, ORM

At first, We tried Postgres and NoSQL.

Use NoSQL Database saves a lot of time wasted in writing SQL. There is no SQL statements anymore.

Table will be created at the first time we use it. Without any SQL like CREATE TABLE ... before using it.

db.inventory.find( {} )
db.inventory.find( { type: "snacks" } )

For example. In mongodb, table inventory will be created after line 1.

You can save and load any fields in any type as you want.

db.inventory.insert({
 item: "ABC1",
 details: {
    model: "14Q3",
    manufacturer: "XYZ Company"
 },
 stock: [ { size: "S", qty: 25 }, { size: "M", qty: 50 } ],
 category: "clothing"
})

db.inventory.insert({
 item: "ABC2",
 details: "description of ABC@",
})

In the above 2 records. detail have different types. But it also works fine.

However, NoSQL performance worse than traditional SQL.

Mongodb always wastes more than 2G memory and produces huge amount I/O operations.

We don't want to pay more for VPS, So we must make a compromise.

QueryBuilder can build SQL Query quickly.

knex.avg('sum_column1').from(function() {
  this.sum('column1 as sum_column1')
  .from('t1').groupBy('column1').as('t1')
}).as('ignored_alias')

>>>>

select avg(`sum_column1`) from (
    select sum(`column1`) as `sum_column1` from `t1` group by `column1`) as `t1`

knex.js make traditional SQL can be written in tree-struct like JSON. much better than the original way. PHP's laravel framework did the same thing.

But it's still not the most agile way dealing with database. I still need to write knex.update / knex.select / knex.delete / knex.insert to perform CURD (Create / Update / Read / Delete).

Finally we found BackboneORM.

var Project = Backbone.Model.extend({
  urlRoot: 'sqlite://dbpath',

  schema: {
    created_at: 'DateTime',
    type: ['Integer', {nullable: false}],
    name: ['String', {unique: true, indexed: true}]
  }    
});

It can create table and tranform field types automatically like NoSQL, even if the database is using traditional SQL like PostgresSQL, SQLite, MySQL, etc.

This is a feature of RubyOnRails, but node.js is quite lightweight and easy to use.

BackboneHTTP in BackboneORM make front-end using backbone.js framework access database easily, which may help us writing front-end pages later.

LeanCloud

LeanCloud is a Cloud API Platform founded in China, focus on small start-up company like us developing Server APIs rapidly.

It gathers node.js, express.js, BackboneORM, Apple Push API, iOS APP Publish and Analysis API, etc. into one platform.

It also integrates Promise.js in BackboneORM and many other APIs, which is very important for node.js.

LeanCloud's Apple Push API is easily to use. The only thing you need to do is to upload the Apple Push Notify P12 Certificate and integrate AVOSCloud iOS SDK in Cocopods.

Here are some screenshots of LeanCloud's Console Panel.

iOS APP Analysis

Database View

Cloud code logger

HTTP Comet Server

Comet server is using to exchange message between hardware. Hardware always keeps a long polling HTTP connection with comet server.

At first we use nginx_http_push_module. Its problem is:

• has no API getting current online id list
• uses date string in Last-Modified header field, which can't deal with 2 requests in the same second

So we write our own comet server using node.js instead.

Supervisord & lighttpd

lighttpd can forward HTTP request with specific Host to specific localhost ports running node.js process. We don't need a big and powerfull server like apache or nginx, a lightweight server is better.

supervisord keeps track of node.js process, captures and rotates stdout and stderr log.

Docker

Docker is an open platform for distributed applications for developers and sysadmins.

In Sugr Server. Docker likes the binary version of git.

If lighttpd configuration, sqlite database, node.js modules, etc. is modified, Docker will consider it as a 'commit' like a commit in git.

So everything can be save and load like "Time Machine" in OS X.

Another great feature of docker is, it can "hide" ports in docker image. For example, port 2234,2235,2236,2237 in used in node.js process. port 80 is used for lighttpd and request may forward to 223x. I can pack a image and export port 80 only, then 223x ports is invisible in host, which avoids port duplicate problems.

Future

In future we will release our own comet server as a npm package.