In
the first post of this series I mentioned some popular modules in
the community, such as underscore, async, etc.. I also listed a module named “Wind (zh-CN)”, which is created by one of my friend, Jeff Zhao (zh-CN). Now I would like to use a separated post to introduce this module since I feel it brings a new async programming style in not only Node.js but JavaScript world. If you know or heard about
the new feature in C# 5.0 called “async and await”, or you learnt F#, you will find
the “Wind” brings
the similar async programming experience in JavaScript. By using “Wind”, we can write async code that looks like
the sync code.
The callbacks, async stats and exceptions will be handled by “Wind” automatically and transparently. What’s
the Problem: Dense “Callback” Phobia Let’s firstly back to my second post in this series. As I mentioned in that post, when we wanted to read some records from SQL Server we need to open
the database connection, and then execute
the query. In Node.js all IO operation are designed as async callback pattern which means when
the operation was done, it will invoke a function which was taken from
the last parameter. For example
the database connection opening code would be like this. 1: sql.open(connectionString, function(error, conn) {
2: if(error) {
3: // some error handling code
4: }
5: else {
6: // connection opened successfully
7: }
8: });
And then if we need to query
the database
the code would be like this. It nested in
the previous function.
1: sql.open(connectionString, function(error, conn) {
2: if(error) {
3: // some error handling code
4: }
5: else {
6: // connection opened successfully
7: conn.queryRaw(command, function(error, results) {
8: if(error) {
9: // failed to execute this command
10: }
11: else {
12: // records retrieved successfully
13: }
14: };
15: }
16: });
Assuming if we need to copy some data from this database to another then we need to open another connection and execute
the command within
the function under
the query function.
1: sql.open(connectionString, function(error, conn) {
2: if(error) {
3: // some error handling code
4: }
5: else {
6: // connection opened successfully
7: conn.queryRaw(command, function(error, results) {
8: if(error) {
9: // failed to execute this command
10: }
11: else {
12: // records retrieved successfully
13: target.open(targetConnectionString, function(error, t_conn) {
14: if(error) {
15: // connect failed
16: }
17: else {
18: t_conn.queryRaw(copy_command, function(error, results) {
19: if(error) {
20: // copy failed
21: }
22: else {
23: // and then, what do you want to do now...
24: }
25: };
26: }
27: };
28: }
29: };
30: }
31: });
This is just an example. In
the real project
the logic would be more complicated. This means our application might be messed up and
the business process will be fragged by many callback functions. I would like call this “Dense Callback Phobia”.
This might be a challenge how to make code straightforward and easy to read, something like below.
1: try
2: {
3: // open source connection
4: var s_conn = sqlConnect(s_connectionString);
5: // retrieve data
6: var results = sqlExecuteCommand(s_conn, s_command);
7:
8: // open target connection
9: var t_conn = sqlConnect(t_connectionString);
10: // prepare
the copy command
11: var t_command = getCopyCommand(results);
12: // execute
the copy command
13: sqlExecuteCommand(s_conn, t_command);
14: }
15: catch (ex)
16: {
17: // error handling
18: }
What’s
the Problem: Sync-styled Async Programming
Similar as
the previous problem,
the callback-styled async programming model makes
the upcoming operation as a part of
the current operation, and mixed with
the error handling code. So it’s very hard to understand what on earth this code will do.
And since Node.js utilizes non-blocking IO mode, we cannot invoke those operations one by one, as they will be executed concurrently.
For example, in this post when I tried to copy
the records from Windows Azure SQL Database (a.k.a. WASD) to Windows Azure Table Storage, if I just insert
the data into table storage one by one and then print
the “Finished” message, I will see
the message shown before
the data had been copied. This is because all operations were executed at
the same time.
In order to make
the copy operation and print operation executed synchronously I introduced a module named “async” and
the code was changed as below.
1: async.forEach(results.rows,
2: function (row, callback) {
3: var resource = {
4: "PartitionKey": row[1],
5: "RowKey": row[0],
6: "Value": row[2]
7: };
8: client.insertEntity(tableName, resource, function (error) {
9: if (error) {
10: callback(error);
11: }
12: else {
13: console.log("entity inserted.");
14: callback(null);
15: }
16: });
17: },
18: function (error) {
19: if (error) {
20: error["target"] = "insertEntity";
21: res.send(500, error);
22: }
23: else {
24: console.log("all done.");
25: res.send(200, "Done!");
26: }
27: });
It ensured that
the “Finished” message will be printed when all table entities had been inserted. But it cannot promise that
the records will be inserted in sequence. It might be another challenge to make
the code looks like in sync-style?
1: try
2: {
3: forEach(row in rows) {
4: var entity = { /* ... */ };
5: tableClient.insert(tableName, entity);
6: }
7:
8: console.log("Finished");
9: }
10: catch (ex) {
11: console.log(ex);
12: }
How “Wind” Helps
“Wind” is a JavaScript library which provides
the control flow with plain JavaScript for asynchronous programming (and more) without additional pre-compiling steps. It’s available in NPM so that we can install it through “npm install wind”.
Now let’s create a very simple Node.js application as
the example. This application will take some website URLs from
the command arguments and tried to retrieve
the body length and print them in console. Then at
the end print “Finish”. I’m going to use “request” module to make
the HTTP call simple so I also need to install by
the command “npm install request”.
The code would be like this.
1: var request = require("request");
2:
3: // get
the urls from arguments,
the first two arguments are `node.exe` and `fetch.js`
4: var args = process.argv.splice(2);
5:
6: // main function
7: var main = function() {
8: for(var i = 0; i < args.length; i++) {
9: // get
the url
10: var url = args[i];
11: // send
the http request and try to get
the response and body
12: request(url, function(error, response, body) {
13: if(!error && response.statusCode == 200) {
14: // log
the url and
the body length
15: console.log(
16: "%s: %d.",
17: response.request.uri.href,
18: body.length);
19: }
20: else {
21: // log error
22: console.log(error);
23: }
24: });
25: }
26:
27: // finished
28: console.log("Finished");
29: };
30:
31: // execute
the main function
32: main();
Let’s execute this application. (I made them in multi-lines for better reading.)
1: node fetch.js
2: "http://www.igt.com/us-en.aspx"
3: "http://www.igt.com/us-en/games.aspx"
4: "http://www.igt.com/us-en/cabinets.aspx"
5: "http://www.igt.com/us-en/systems.aspx"
6: "http://www.igt.com/us-en/interactive.aspx"
7: "http://www.igt.com/us-en/social-gaming.aspx"
8: "http://www.igt.com/support.aspx"
Below is
the output.
As you can see
the finish message was printed at
the beginning, and
the pages’ length retrieved in a different order than we specified. This is because in this code
the request command, console logging command are executed asynchronously and concurrently.
Now let’s introduce “Wind” to make them executed in order, which means it will request
the websites one by one, and print
the message at
the end.
First of all we need to import
the “Wind” package and make sure
the there’s only one global variant named “Wind”, and ensure it’s “Wind” instead of “wind”.
1: var Wind = require("wind");
Next, we need to tell “Wind” which code will be executed asynchronously so that “Wind” can control
the execution process. In this case
the “request” operation executed asynchronously so we will create a “Task” by using a build-in helps function in “Wind” named Wind.Async.Task.create.
1: var requestBodyLengthAsync = function(url) {
2: return Wind.Async.Task.create(function(t) {
3: request(url, function(error, response, body) {
4: if(error || response.statusCode != 200) {
5: t.complete("failure", error);
6: }
7: else {
8: var data =
9: {
10: uri: response.request.uri.href,
11: length: body.length
12: };
13: t.complete("success", data);
14: }
15: });
16: });
17: };
The code above created a “Task” from
the original request calling code. In “Wind” a “Task” means an operation will be finished in some time in
the future. A “Task” can be started by invoke its start() method, but no one knows when it actually will be finished.
The Wind.Async.Task.create helped us to create a task.
The only parameter is a function where we can put
the actual operation in, and then notify
the task object it’s finished successfully or failed by using
the complete() method.
In
the code above I invoked
the request method. If it retrieved
the response successfully I set
the status of this task as “success” with
the URL and body length. If it failed I set this task as “failure” and pass
the error out.
Next, we will change
the main() function. In “Wind” if we want a function can be controlled by Wind we need to mark it as “async”. This should be done by using
the code below.
1: var main = eval(Wind.compile("async", function() {
2: }));
When
the application is running, Wind will detect “eval(Wind.compile(“async”, function” and generate an anonymous code from
the body of this original function. Then
the application will run
the anonymous code instead of
the original one.
In our example
the main function will be like this.
1: var main = eval(Wind.compile("async", function() {
2: for(var i = 0; i < args.length; i++) {
3: try
4: {
5: var result = $await(requestBodyLengthAsync(args[i]));
6: console.log(
7: "%s: %d.",
8: result.uri,
9: result.length);
10: }
11: catch (ex) {
12: console.log(ex);
13: }
14: }
15:
16: console.log("Finished");
17: }));
As you can see, when I tried to request
the URL I use a new command named “$await”. It tells Wind,
the operation next to $await will be executed asynchronously, and
the main thread should be paused until it finished (or failed). So in this case, my application will be pause when
the first response was received, and then print its body length, then try
the next one. At
the end, print
the finish message.
Finally, execute
the main function.
The full code would be like this.
1: var request = require("request");
2: var Wind = require("wind");
3:
4: var args = process.argv.splice(2);
5:
6: var requestBodyLengthAsync = function(url) {
7: return Wind.Async.Task.create(function(t) {
8: request(url, function(error, response, body) {
9: if(error || response.statusCode != 200) {
10: t.complete("failure", error);
11: }
12: else {
13: var data =
14: {
15: uri: response.request.uri.href,
16: length: body.length
17: };
18: t.complete("success", data);
19: }
20: });
21: });
22: };
23:
24: var main = eval(Wind.compile("async", function() {
25: for(var i = 0; i < args.length; i++) {
26: try
27: {
28: var result = $await(requestBodyLengthAsync(args[i]));
29: console.log(
30: "%s: %d.",
31: result.uri,
32: result.length);
33: }
34: catch (ex) {
35: console.log(ex);
36: }
37: }
38:
39: console.log("Finished");
40: }));
41:
42: main().start();
Run our new application. At
the beginning we will see
the compiled and generated code by Wind. Then we can see
the pages were requested one by one, and at
the end
the finish message was printed.
Below is
the code Wind generated for us. As you can see
the original code,
the output code were shown.
1: // Original:
2: function () {
3: for(var i = 0; i < args.length; i++) {
4: try
5: {
6: var result = $await(requestBodyLengthAsync(args[i]));
7: console.log(
8: "%s: %d.",
9: result.uri,
10: result.length);
11: }
12: catch (ex) {
13: console.log(ex);
14: }
15: }
16:
17: console.log("Finished");
18: }
19:
20: // Compiled:
21: /* async << function () { */ (function () {
22: var _builder_$0 = Wind.builders["async"];
23: return _builder_$0.Start(this,
24: _builder_$0.Combine(
25: _builder_$0.Delay(function () {
26: /* var i = 0; */ var i = 0;
27: /* for ( */ return _builder_$0.For(function () {
28: /* ; i < args.length */ return i < args.length;
29: }, function () {
30: /* ; i ++) { */ i ++;
31: },
32: /* try { */ _builder_$0.Try(
33: _builder_$0.Delay(function () {
34: /* var result = $await(requestBodyLengthAsync(args[i])); */ return _builder_$0.Bind(requestBodyLengthAsync(args[i]), function (result) {
35: /* console.log("%s: %d.", result.uri, result.length); */ console.log("%s: %d.", result.uri, result.length);
36: return _builder_$0.Normal();
37: });
38: }),
39: /* } catch (ex) { */ function (ex) {
40: /* console.log(ex); */ console.log(ex);
41: return _builder_$0.Normal();
42: /* } */ },
43: null
44: )
45: /* } */ );
46: }),
47: _builder_$0.Delay(function () {
48: /* console.log("Finished"); */ console.log("Finished");
49: return _builder_$0.Normal();
50: })
51: )
52: );
53: /* } */ })
How Wind Works
Someone may raise a big concern when you find I utilized “eval” in my code. Someone may assume that Wind utilizes “eval” to execute some code dynamically while “eval” is very low performance. But I would say, Wind does NOT use “eval” to run
the code. It only use “eval” as a flag to know which code should be compiled at runtime.
When
the code was firstly been executed, Wind will check and find “eval(Wind.compile(“async”, function”. So that it knows this function should be compiled. Then it utilized parse-js to analyze
the inner JavaScript and generated
the anonymous code in memory. Then it rewrite
the original code so that when
the application was running it will use
the anonymous one instead of
the original one.
Since
the code generation was done at
the beginning of
the application was started, in
the future no matter how long our application runs and how many times
the async function was invoked, it will use
the generated code, no need to generate again. So there’s no significant performance hurt when using Wind.
Wind in My Previous Demo
Let’s adopt Wind into one of my previous demonstration and to see how it helps us to make our code simple, straightforward and easy to read and understand.
In this post when I implemented
the functionality that copied
the records from my WASD to table storage,
the logic would be like this.
1, Open database connection.
2, Execute a query to select all records from
the table.
3, Recreate
the table in Windows Azure table storage.
4, Create entities from each of
the records retrieved previously, and then insert them into table storage.
5, Finally, show message as
the HTTP response.
But as
the image below, since there are so many callbacks and async operations, it’s very hard to understand my logic from
the code.
Now let’s use Wind to rewrite our code. First of all, of course, we need
the Wind package.
Then we need to include
the package files into project and mark them as “Copy always”.
Add
the Wind package into
the source code. Pay attention to
the variant name, you must use “Wind” instead of “wind”.
1: var express = require("express");
2: var async = require("async");
3: var sql = require("node-sqlserver");
4: var azure = require("azure");
5: var Wind = require("wind");
Now we need to create some async functions by using Wind. All async functions should be wrapped so that it can be controlled by Wind which are open database, retrieve records, recreate table (delete and create) and insert entity in table.
Below are these new functions. All of them are created by using Wind.Async.Task.create.
1: sql.openAsync = function (connectionString) {
2: return Wind.Async.Task.create(function (t) {
3: sql.open(connectionString, function (error, conn) {
4: if (error) {
5: t.complete("failure", error);
6: }
7: else {
8: t.complete("success", conn);
9: }
10: });
11: });
12: };
13:
14: sql.queryAsync = function (conn, query) {
15: return Wind.Async.Task.create(function (t) {
16: conn.queryRaw(query, function (error, results) {
17: if (error) {
18: t.complete("failure", error);
19: }
20: else {
21: t.complete("success", results);
22: }
23: });
24: });
25: };
26:
27: azure.recreateTableAsync = function (tableName) {
28: return Wind.Async.Task.create(function (t) {
29: client.deleteTable(tableName, function (error, successful, response) {
30: console.log("delete table finished");
31: client.createTableIfNotExists(tableName, function (error, successful, response) {
32: console.log("create table finished");
33: if (error) {
34: t.complete("failure", error);
35: }
36: else {
37: t.complete("success", null);
38: }
39: });
40: });
41: });
42: };
43:
44: azure.insertEntityAsync = function (tableName, entity) {
45: return Wind.Async.Task.create(function (t) {
46: client.insertEntity(tableName, entity, function (error, entity, response) {
47: if (error) {
48: t.complete("failure", error);
49: }
50: else {
51: t.complete("success", null);
52: }
53: });
54: });
55: };
Then in order to use these functions we will create a new function which contains all steps for data copying.
1: var copyRecords = eval(Wind.compile("async", function (req, res) {
2: try {
3: }
4: catch (ex) {
5: console.log(ex);
6: res.send(500, "Internal error.");
7: }
8: }));
Let’s execute steps one by one with
the “$await” keyword introduced by Wind so that it will be invoked in sequence. First is to open
the database connection.
1: var copyRecords = eval(Wind.compile("async", function (req, res) {
2: try {
3: // connect to
the windows azure sql database
4: var conn = $await(sql.openAsync(connectionString));
5: console.log("connection opened");
6: }
7: catch (ex) {
8: console.log(ex);
9: res.send(500, "Internal error.");
10: }
11: }));
Then retrieve all records from
the database connection.
1: var copyRecords = eval(Wind.compile("async", function (req, res) {
2: try {
3: // connect to
the windows azure sql database
4: var conn = $await(sql.openAsync(connectionString));
5: console.log("connection opened");
6: // retrieve all records from database
7: var results = $await(sql.queryAsync(conn, "SELECT * FROM [Resource]"));
8: console.log("records selected. count = %d", results.rows.length);
9: }
10: catch (ex) {
11: console.log(ex);
12: res.send(500, "Internal error.");
13: }
14: }));
After recreated
the table, we need to create
the entities and insert them into table storage.
1: var copyRecords = eval(Wind.compile("async", function (req, res) {
2: try {
3: // connect to
the windows azure sql database
4: var conn = $await(sql.openAsync(connectionString));
5: console.log("connection opened");
6: // retrieve all records from database
7: var results = $await(sql.queryAsync(conn, "SELECT * FROM [Resource]"));
8: console.log("records selected. count = %d", results.rows.length);
9: if (results.rows.length > 0) {
10: // recreate
the table
11: $await(azure.recreateTableAsync(tableName));
12: console.log("table created");
13: // insert records in table storage one by one
14: for (var i = 0; i < results.rows.length; i++) {
15: var entity = {
16: "PartitionKey": results.rows[i][1],
17: "RowKey": results.rows[i][0],
18: "Value": results.rows[i][2]
19: };
20: $await(azure.insertEntityAsync(tableName, entity));
21: console.log("entity inserted");
22: }
23: }
24: }
25: catch (ex) {
26: console.log(ex);
27: res.send(500, "Internal error.");
28: }
29: }));
Finally, send response back to
the browser.
1: var copyRecords = eval(Wind.compile("async", function (req, res) {
2: try {
3: // connect to
the windows azure sql database
4: var conn = $await(sql.openAsync(connectionString));
5: console.log("connection opened");
6: // retrieve all records from database
7: var results = $await(sql.queryAsync(conn, "SELECT * FROM [Resource]"));
8: console.log("records selected. count = %d", results.rows.length);
9: if (results.rows.length > 0) {
10: // recreate
the table
11: $await(azure.recreateTableAsync(tableName));
12: console.log("table created");
13: // insert records in table storage one by one
14: for (var i = 0; i < results.rows.length; i++) {
15: var entity = {
16: "PartitionKey": results.rows[i][1],
17: "RowKey": results.rows[i][0],
18: "Value": results.rows[i][2]
19: };
20: $await(azure.insertEntityAsync(tableName, entity));
21: console.log("entity inserted");
22: }
23: // send response
24: console.log("all done");
25: res.send(200, "All done!");
26: }
27: }
28: catch (ex) {
29: console.log(ex);
30: res.send(500, "Internal error.");
31: }
32: }));
If we compared with
the previous code we will find now it became more readable and much easy to understand. It’s very easy to know what this function does even though without any comments.
When user go to URL “/was/copyRecords” we will execute
the function above.
The code would be like this.
1: app.get("/was/copyRecords", function (req, res) {
2: copyRecords(req, res).start();
3: });
And below is
the logs printed in local compute emulator console. As we can see
the functions executed one by one and then finally
the response back to me browser.
Scaffold Functions in Wind
Wind provides not only
the async flow control and compile functions, but many scaffold methods as well. We can build our async code more easily by using them. I’m going to introduce some basic scaffold functions here.
In
the code above I created some functions which wrapped from
the original async function such as open database, create table, etc.. All of them are very similar, created a task by using Wind.Async.Task.create, return error or result object through Task.complete function. In fact, Wind provides some functions for us to create task object from
the original async functions.
If
the original async function only has a callback parameter, we can use Wind.Async.Binding.fromCallback method to get
the task object directly. For example
the code below returned
the task object which wrapped
the file exist check function.
1: var Wind = require("wind");
2: var fs = require("fs");
3:
4: fs.existsAsync = Wind.Async.Binding.fromCallback(fs.exists);
In Node.js a very popular async function pattern is that,
the first parameter in
the callback function represent
the error object, and
the other parameters is
the return values. In this case we can use another build-in function in Wind named Wind.Async.Binding.fromStandard. For example,
the open database function can be created from
the code below.
1: sql.openAsync = Wind.Async.Binding.fromStandard(sql.open);
2:
3: /*
4: sql.openAsync = function (connectionString) {
5: return Wind.Async.Task.create(function (t) {
6: sql.open(connectionString, function (error, conn) {
7: if (error) {
8: t.complete("failure", error);
9: }
10: else {
11: t.complete("success", conn);
12: }
13: });
14: });
15: };
16: */
When I was testing
the scaffold functions under Wind.Async.Binding I found for some functions, such as
the Azure SDK insert entity function, cannot be processed correctly. So I personally suggest writing
the wrapped method manually.
Another scaffold method in Wind is
the parallel tasks coordination. In this example,
the steps of open database, retrieve records and recreated table should be invoked one by one, but it can be executed in parallel when copying data from database to table storage. In Wind there’s a scaffold function named Task.whenAll which can be used here.
Task.whenAll accepts a list of tasks and creates a new task. It will be returned only when all tasks had been completed, or any errors occurred. For example in
the code below I used
the Task.whenAll to make all copy operation executed at
the same time.
1: var copyRecordsInParallel = eval(Wind.compile("async", function (req, res) {
2: try {
3: // connect to
the windows azure sql database
4: var conn = $await(sql.openAsync(connectionString));
5: console.log("connection opened");
6: // retrieve all records from database
7: var results = $await(sql.queryAsync(conn, "SELECT * FROM [Resource]"));
8: console.log("records selected. count = %d", results.rows.length);
9: if (results.rows.length > 0) {
10: // recreate
the table
11: $await(azure.recreateTableAsync(tableName));
12: console.log("table created");
13: // insert records in table storage in parallal
14: var tasks = new Array(results.rows.length);
15: for (var i = 0; i < results.rows.length; i++) {
16: var entity = {
17: "PartitionKey": results.rows[i][1],
18: "RowKey": results.rows[i][0],
19: "Value": results.rows[i][2]
20: };
21: tasks[i] = azure.insertEntityAsync(tableName, entity);
22: }
23: $await(Wind.Async.Task.whenAll(tasks));
24: // send response
25: console.log("all done");
26: res.send(200, "All done!");
27: }
28: }
29: catch (ex) {
30: console.log(ex);
31: res.send(500, "Internal error.");
32: }
33: }));
34:
35: app.get("/was/copyRecordsInParallel", function (req, res) {
36: copyRecordsInParallel(req, res).start();
37: });
Besides
the task creation and coordination, Wind supports
the cancellation solution so that we can send
the cancellation signal to
the tasks. It also includes exception solution which means any exceptions will be reported to
the caller function.
Summary
In this post I introduced a Node.js module named Wind, which created by my friend Jeff Zhao. As you can see, different from other async library and framework, adopted
the idea from F# and C#, Wind utilizes runtime code generation technology to make it more easily to write async, callback-based functions in a sync-style way. By using Wind there will be almost no callback, and
the code will be very easy to understand.
Currently Wind is still under developed and improved. There might be some problems but
the author, Jeff, should be very happy and enthusiastic to learn your problems, feedback, suggestion and comments. You can contact Jeff by
- Email:
[email protected]
- Group: https://groups.google.com/d/forum/windjs
- GitHub: https://github.com/JeffreyZhao/wind/issues
Source code can be download here.
Hope this helps,
Shaun
All documents and related graphics, codes are provided "AS IS" without warranty of any kind.
Copyright © Shaun Ziyan Xu. This work is licensed under
the Creative Commons License.
