When designing a service oriented architecture (SOA) system, there will be a lot of services with many service contracts, endpoints
and behaviors. Besides the client calling the service, in a large distributed system a service may invoke other services. In this case, one service might need to know the endpoints it invokes. This might not be a problem in a small system. But when you have more than 10 services this might be a problem. For example in my current product, there are around 10 services, such as the user authentication service, UI integration service, location service, license service, device monitor service, event monitor service, schedule job service, accounting service, player management service, etc.. Benefit of Discovery Service Since almost all my services need to invoke at least one other service. This would be a difficult task to make sure all services endpoints are configured correctly in every service.
And furthermore, it would be a nightmare when a service changed its endpoint at runtime. Hence, we need a discovery service to remove the dependency (configuration dependency). A discovery service plays as a service dictionary which stores the relationship between the contracts
and the endpoints for every service. By using the discovery service, when service X wants to invoke service Y, it just need to ask the discovery service where is service Y, then the discovery service will return all proper endpoints of service Y, then service X can use the endpoint to send the request to service Y.
And when some services changed their endpoint address, all need to do is to update its records in the discovery service then all others will know its new endpoint. In WCF 4.0 Discovery it supports both managed proxy discovery mode
and ad-hoc discovery mode. In ad-hoc mode there is no standalone discovery service. When a client wanted to invoke a service, it will broadcast an message (normally in UDP protocol) to the entire network with the service match criteria. All services which enabled the discovery behavior will receive this message
and only those matched services will send their endpoint back to the client. The managed proxy discovery service works as I described above. In this post I will only cover the managed proxy mode, where there’s a discovery service. For more information about the ad-hoc mode please refer to the MSDN. Service Announcement
and Probe The main functionality of discovery service should be return the proper endpoint addresses back to the service who is looking for. In most cases the consume service (as a client) will send the contract which it wanted to request to the discovery service.
And then the discovery service will
find the endpoint
and respond. Sometimes the contract
and endpoint are not enough. It also contains versioning, extensions attributes. This post I will only cover the case includes contract
and endpoint. When a client (or sometimes a service who need to invoke another service) need to connect to a target service, it will firstly request the discovery service through the “Probe” method with the criteria. Basically the criteria contains the contract type name of the target service. Then the discovery service will search its endpoint repository by the criteria. The repository might be a database, a distributed cache or a flat XML file. If it matches, the discovery service will grab the endpoint information (it’s called discovery endpoint metadata in WCF)
and send back.
And this is called “Probe”. Finally the client received the discovery endpoint metadata
and will use the endpoint to connect to the target service. Besides the probe, discovery service should take the responsible to know there is a new service available when it goes online, as well as stopped when it goes offline. This feature is named “Announcement”. When a service started
and stopped, it will announce to the discovery service. So the basic functionality of a discovery service should includes: 1, An endpoint which receive the service online message,
and add the service endpoint information in the discovery repository. 2, An endpoint which receive the service offline message,
and remove the service endpoint information from the discovery repository. 3, An endpoint which receive the client probe message,
and return the matches service endpoints,
and return the discovery endpoint metadata. WCF 4.0 discovery service just covers all these features in it's infrastructure classes. Discovery Service in WCF 4.0 WCF 4.0 introduced a new assembly named System.ServiceModel.Discovery which has all necessary classes
and interfaces to build a WS-Discovery compliant discovery service. It supports ad-hoc
and managed proxy modes. For the case mentioned in this post, what we need to build is a standalone discovery service, which is the managed proxy discovery service mode. To build a managed discovery service in WCF 4.0 just create a new class inherits from the abstract class System.ServiceModel.Discovery.DiscoveryProxy. This class implemented
and abstracted the procedures of service announcement
and probe.
And it exposes 8 abstract methods where we can implement our own endpoint register, unregister
and find logic. These 8 methods are asynchronized, which means all invokes to the discovery service are asynchronously, for better service capability
and performance. 1, OnBeginOnlineAnnouncement, OnEndOnlineAnnouncement: Invoked when a service sent the online announcement message. We need to add the endpoint information to the repository in this method. 2, OnBeginOfflineAnnouncement, OnEndOfflineAnnouncement: Invoked when a service sent the offline announcement message. We need to remove the endpoint information from the repository in this method. 3, OnBeginFind, OnEndFind: Invoked when a client sent the probe message that want to
find the service endpoint information. We need to look for the proper endpoints by matching the client’s criteria through the repository in this method. 4, OnBeginResolve, OnEndResolve: Invoked then a client sent the resolve message. Different from the
find method, when using resolve method the discovery service will return the exactly one service endpoint metadata to the client. In our example we will NOT implement this method. Let’s create our own discovery service, inherit the base System.ServiceModel.Discovery.DiscoveryProxy. We also need to specify the service behavior in this class. Since the build-in discovery service host class only support the singleton mode, we must set its instance context mode to single. 1: using System;
2: using System.Collections.Generic;
3: using System.Linq;
4: using System.Text;
5: using System.ServiceModel.Discovery;
6: using System.ServiceModel;
7:
8: namespace Phare.Service
9: {
10: [ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple)]
11: public class ManagedProxyDiscoveryService : DiscoveryProxy
12: {
13: protected override IAsyncResult OnBeginFind(FindRequestContext findRequestContext, AsyncCallback callback, object state)
14: {
15: throw new NotImplementedException();
16: }
17:
18: protected override IAsyncResult OnBeginOfflineAnnouncement(DiscoveryMessageSequence messageSequence, EndpointDiscoveryMetadata endpointDiscoveryMetadata, AsyncCallback callback, object state)
19: {
20: throw new NotImplementedException();
21: }
22:
23: protected override IAsyncResult OnBeginOnlineAnnouncement(DiscoveryMessageSequence messageSequence, EndpointDiscoveryMetadata endpointDiscoveryMetadata, AsyncCallback callback, object state)
24: {
25: throw new NotImplementedException();
26: }
27:
28: protected override IAsyncResult OnBeginResolve(ResolveCriteria resolveCriteria, AsyncCallback callback, object state)
29: {
30: throw new NotImplementedException();
31: }
32:
33: protected override void OnEndFind(IAsyncResult result)
34: {
35: throw new NotImplementedException();
36: }
37:
38: protected override void OnEndOfflineAnnouncement(IAsyncResult result)
39: {
40: throw new NotImplementedException();
41: }
42:
43: protected override void OnEndOnlineAnnouncement(IAsyncResult result)
44: {
45: throw new NotImplementedException();
46: }
47:
48: protected override EndpointDiscoveryMetadata OnEndResolve(IAsyncResult result)
49: {
50: throw new NotImplementedException();
51: }
52: }
53: }
Then let’s implement the online, offline
and find methods one by one. WCF discovery service gives us full flexibility to implement the endpoint add, remove
and find logic. For the demo purpose we will use an internal dictionary to store the services’ endpoint metadata.
In the next post we will see how to serialize
and store these information in database.
Define a concurrent dictionary inside the service class since our it will be used in the multiple threads scenario.
1: [ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple)]
2: public class ManagedProxyDiscoveryService : DiscoveryProxy
3: {
4: private ConcurrentDictionary<EndpointAddress, EndpointDiscoveryMetadata> _services;
5:
6: public ManagedProxyDiscoveryService()
7: {
8: _services = new ConcurrentDictionary<EndpointAddress, EndpointDiscoveryMetadata>();
9: }
10: }
Then we can simply implement the logic of service online
and offline.
1: protected override IAsyncResult OnBeginOnlineAnnouncement(DiscoveryMessageSequence messageSequence, EndpointDiscoveryMetadata endpointDiscoveryMetadata, AsyncCallback callback, object state)
2: {
3: _services.AddOrUpdate(endpointDiscoveryMetadata.Address, endpointDiscoveryMetadata, (key, value) => endpointDiscoveryMetadata);
4: return new OnOnlineAnnouncementAsyncResult(callback, state);
5: }
6:
7: protected override void OnEndOnlineAnnouncement(IAsyncResult result)
8: {
9: OnOnlineAnnouncementAsyncResult.End(result);
10: }
11:
12: protected override IAsyncResult OnBeginOfflineAnnouncement(DiscoveryMessageSequence messageSequence, EndpointDiscoveryMetadata endpointDiscoveryMetadata, AsyncCallback callback, object state)
13: {
14: EndpointDiscoveryMetadata endpoint = null;
15: _services.TryRemove(endpointDiscoveryMetadata.Address, out endpoint);
16: return new OnOfflineAnnouncementAsyncResult(callback, state);
17: }
18:
19: protected override void OnEndOfflineAnnouncement(IAsyncResult result)
20: {
21: OnOfflineAnnouncementAsyncResult.End(result);
22: }
Regards the
find method, the parameter FindRequestContext.Criteria has a method named IsMatch, which can be use for us to evaluate which service metadata is satisfied with the criteria. So the implementation of
find method would be like this.
1: protected override IAsyncResult OnBeginFind(FindRequestContext findRequestContext, AsyncCallback callback, object state)
2: {
3: _services.Where(s => findRequestContext.Criteria.IsMatch(s.Value))
4: .Select(s => s.Value)
5: .All(meta =>
6: {
7: findRequestContext.AddMatchingEndpoint(meta);
8: return true;
9: });
10: return new OnFindAsyncResult(callback, state);
11: }
12:
13: protected override void OnEndFind(IAsyncResult result)
14: {
15: OnFindAsyncResult.End(result);
16: }
As you can see, we checked all endpoints metadata in repository by invoking the IsMatch method. Then add all proper endpoints metadata into the parameter.
Finally since all these methods are asynchronized we need some AsyncResult classes as well. Below are the base class
and the inherited classes used in previous methods.
1: using System;
2: using System.Collections.Generic;
3: using System.Linq;
4: using System.Text;
5: using System.Threading;
6:
7: namespace Phare.Service
8: {
9: abstract internal class AsyncResult : IAsyncResult
10: {
11: AsyncCallback callback;
12: bool completedSynchronously;
13: bool endCalled;
14: Exception exception;
15: bool isCompleted;
16: ManualResetEvent manualResetEvent;
17: object state;
18: object thisLock;
19:
20: protected AsyncResult(AsyncCallback callback, object state)
21: {
22: this.callback = callback;
23: this.state = state;
24: this.thisLock = new object();
25: }
26:
27: public object AsyncState
28: {
29: get
30: {
31: return state;
32: }
33: }
34:
35: public WaitHandle AsyncWaitHandle
36: {
37: get
38: {
39: if (manualResetEvent != null)
40: {
41: return manualResetEvent;
42: }
43: lock (ThisLock)
44: {
45: if (manualResetEvent == null)
46: {
47: manualResetEvent = new ManualResetEvent(isCompleted);
48: }
49: }
50: return manualResetEvent;
51: }
52: }
53:
54: public bool CompletedSynchronously
55: {
56: get
57: {
58: return completedSynchronously;
59: }
60: }
61:
62: public bool IsCompleted
63: {
64: get
65: {
66: return isCompleted;
67: }
68: }
69:
70: object ThisLock
71: {
72: get
73: {
74: return this.thisLock;
75: }
76: }
77:
78: protected static TAsyncResult End<TAsyncResult>(IAsyncResult result)
79: where TAsyncResult : AsyncResult
80: {
81: if (result == null)
82: {
83: throw new ArgumentNullException("result");
84: }
85:
86: TAsyncResult asyncResult = result as TAsyncResult;
87:
88: if (asyncResult == null)
89: {
90: throw new ArgumentException("Invalid async result.", "result");
91: }
92:
93: if (asyncResult.endCalled)
94: {
95: throw new InvalidOperationException("Async object already ended.");
96: }
97:
98: asyncResult.endCalled = true;
99:
100: if (!asyncResult.isCompleted)
101: {
102: asyncResult.AsyncWaitHandle.WaitOne();
103: }
104:
105: if (asyncResult.manualResetEvent != null)
106: {
107: asyncResult.manualResetEvent.Close();
108: }
109:
110: if (asyncResult.exception != null)
111: {
112: throw asyncResult.exception;
113: }
114:
115: return asyncResult;
116: }
117:
118: protected void Complete(bool completedSynchronously)
119: {
120: if (isCompleted)
121: {
122: throw new InvalidOperationException("This async result is already completed.");
123: }
124:
125: this.completedSynchronously = completedSynchronously;
126:
127: if (completedSynchronously)
128: {
129: this.isCompleted = true;
130: }
131: else
132: {
133: lock (ThisLock)
134: {
135: this.isCompleted = true;
136: if (this.manualResetEvent != null)
137: {
138: this.manualResetEvent.Set();
139: }
140: }
141: }
142:
143: if (callback != null)
144: {
145: callback(this);
146: }
147: }
148:
149: protected void Complete(bool completedSynchronously, Exception exception)
150: {
151: this.exception = exception;
152: Complete(completedSynchronously);
153: }
154: }
155: }
1: using System;
2: using System.Collections.Generic;
3: using System.Linq;
4: using System.Text;
5: using System.ServiceModel.Discovery;
6: using Phare.Service;
7:
8: namespace Phare.Service
9: {
10: internal sealed class OnOnlineAnnouncementAsyncResult : AsyncResult
11: {
12: public OnOnlineAnnouncementAsyncResult(AsyncCallback callback, object state)
13: : base(callback, state)
14: {
15: this.Complete(true);
16: }
17:
18: public static void End(IAsyncResult result)
19: {
20: AsyncResult.End<OnOnlineAnnouncementAsyncResult>(result);
21: }
22:
23: }
24:
25: sealed class OnOfflineAnnouncementAsyncResult : AsyncResult
26: {
27: public OnOfflineAnnouncementAsyncResult(AsyncCallback callback, object state)
28: : base(callback, state)
29: {
30: this.Complete(true);
31: }
32:
33: public static void End(IAsyncResult result)
34: {
35: AsyncResult.End<OnOfflineAnnouncementAsyncResult>(result);
36: }
37: }
38:
39: sealed class OnFindAsyncResult : AsyncResult
40: {
41: public OnFindAsyncResult(AsyncCallback callback, object state)
42: : base(callback, state)
43: {
44: this.Complete(true);
45: }
46:
47: public static void End(IAsyncResult result)
48: {
49: AsyncResult.End<OnFindAsyncResult>(result);
50: }
51: }
52:
53: sealed class OnResolveAsyncResult : AsyncResult
54: {
55: EndpointDiscoveryMetadata matchingEndpoint;
56:
57: public OnResolveAsyncResult(EndpointDiscoveryMetadata matchingEndpoint, AsyncCallback callback, object state)
58: : base(callback, state)
59: {
60: this.matchingEndpoint = matchingEndpoint;
61: this.Complete(true);
62: }
63:
64: public static EndpointDiscoveryMetadata End(IAsyncResult result)
65: {
66: OnResolveAsyncResult thisPtr = AsyncResult.End<OnResolveAsyncResult>(result);
67: return thisPtr.matchingEndpoint;
68: }
69: }
70: }
Now we have finished the discovery service. The next step is to host it. The discovery service is a standard WCF service. So we can use ServiceHost on a console application, windows service, or in IIS as usual. The following code is how to host the discovery service we had just created in a console application.
1: static void Main(string[] args)
2: {
3: using (var host = new ServiceHost(new ManagedProxyDiscoveryService()))
4: {
5: host.Opened += (sender, e) =>
6: {
7: host.Description.Endpoints.All((ep) =>
8: {
9: Console.WriteLine(ep.ListenUri);
10: return true;
11: });
12: };
13:
14: try
15: {
16: // retrieve the announcement, probe endpoint
and binding from configuration
17: var announcementEndpointAddress = new EndpointAddress(ConfigurationManager.AppSettings["announcementEndpointAddress"]);
18: var probeEndpointAddress = new EndpointAddress(ConfigurationManager.AppSettings["probeEndpointAddress"]);
19: var binding = Activator.CreateInstance(Type.GetType(ConfigurationManager.AppSettings["bindingType"], true, true)) as Binding;
20: var announcementEndpoint = new AnnouncementEndpoint(binding, announcementEndpointAddress);
21: var probeEndpoint = new DiscoveryEndpoint(binding, probeEndpointAddress);
22: probeEndpoint.IsSystemEndpoint = false;
23: // append the service endpoint for announcement
and probe
24: host.AddServiceEndpoint(announcementEndpoint);
25: host.AddServiceEndpoint(probeEndpoint);
26:
27: host.Open();
28:
29: Console.WriteLine("Press any key to exit.");
30: Console.ReadKey();
31: }
32: catch (Exception ex)
33: {
34: Console.WriteLine(ex.ToString());
35: }
36: }
37:
38: Console.WriteLine("Done.");
39: Console.ReadKey();
40: }
What we need to notice is that, the discovery service needs two endpoints for announcement
and probe. In this example I just retrieve them from the configuration file. I also specified the binding of these two endpoints in configuration file as well.
1: <?xml version="1.0"?>
2: <configuration>
3: <startup>
4: <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
5: </startup>
6: <appSettings>
7: <add key="announcementEndpointAddress" value="net.tcp://localhost:10010/announcement"/>
8: <add key="probeEndpointAddress" value="net.tcp://localhost:10011/probe"/>
9: <add key="bindingType" value="System.ServiceModel.NetTcpBinding, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"/>
10: </appSettings>
11: </configuration>
And this is the console screen when I ran my discovery service. As you can see there are two endpoints listening for announcement message
and probe message.
Discoverable Service
and Client
Next, let’s create a WCF service that is discoverable, which means it can be found by the discovery service. To do so, we need to let the service send the online announcement message to the discovery service, as well as offline message before it shutdown.
Just create a simple service which can make the incoming string to upper. The service contract
and implementation would be like this.
1: [ServiceContract]
2: public interface IStringService
3: {
4: [OperationContract]
5: string ToUpper(string content);
6: }
1: public class StringService : IStringService
2: {
3: public string ToUpper(string content)
4: {
5: return content.ToUpper();
6: }
7: }
Then host this service in the console application. In order to make the discovery service easy to be tested the service address will be changed each time it’s started.
1: static void Main(string[] args)
2: {
3: var baseAddress = new Uri(string.Format("net.tcp://localhost:11001/stringservice/{0}/", Guid.NewGuid().ToString()));
4:
5: using (var host = new ServiceHost(typeof(StringService), baseAddress))
6: {
7: host.Opened += (sender, e) =>
8: {
9: Console.WriteLine("Service opened at {0}", host.Description.Endpoints.First().ListenUri);
10: };
11:
12: host.AddServiceEndpoint(typeof(IStringService), new NetTcpBinding(), string.Empty);
13:
14: host.Open();
15:
16: Console.WriteLine("Press any key to exit.");
17: Console.ReadKey();
18: }
19: }
Currently this service is NOT discoverable. We need to add a special service behavior so that it could send the online
and offline message to the discovery service announcement endpoint when the host is opened
and closed. WCF 4.0 introduced a service behavior named ServiceDiscoveryBehavior. When we specified the announcement endpoint address
and appended it to the service behaviors this service will be discoverable.
1: var announcementAddress = new EndpointAddress(ConfigurationManager.AppSettings["announcementEndpointAddress"]);
2: var announcementBinding = Activator.CreateInstance(Type.GetType(ConfigurationManager.AppSettings["bindingType"], true, true)) as Binding;
3: var announcementEndpoint = new AnnouncementEndpoint(announcementBinding, announcementAddress);
4: var discoveryBehavior = new ServiceDiscoveryBehavior();
5: discoveryBehavior.AnnouncementEndpoints.Add(announcementEndpoint);
6: host.Description.Behaviors.Add(discoveryBehavior);
The ServiceDiscoveryBehavior utilizes the service extension
and channel dispatcher to implement the online
and offline announcement logic. In short, it injected the channel open
and close procedure
and send the online
and offline message to the announcement endpoint.
On client side, when we have the discovery service, a client can invoke a service without knowing its endpoint. WCF discovery assembly provides a class named DiscoveryClient, which can be used to
find the proper service endpoint by passing the criteria.
In the code below I initialized the DiscoveryClient, specified the discovery service probe endpoint address. Then I created the
find criteria by specifying the service contract I wanted to use
and invoke the
Find method. This will send the probe message to the discovery service
and it will
find the endpoints back to me.
The discovery service will return all endpoints that matches the
find criteria, which means in the result of the
find method there might be more than one endpoints. In this example I just returned the first matched one back. In the next post I will show how to extend our discovery service to make it work like a service load balancer.
1: static EndpointAddress FindServiceEndpoint()
2: {
3: var probeEndpointAddress = new EndpointAddress(ConfigurationManager.AppSettings["probeEndpointAddress"]);
4: var probeBinding = Activator.CreateInstance(Type.GetType(ConfigurationManager.AppSettings["bindingType"], true, true)) as Binding;
5: var discoveryEndpoint = new DiscoveryEndpoint(probeBinding, probeEndpointAddress);
6:
7: EndpointAddress address = null;
8: FindResponse result = null;
9: using (var discoveryClient = new DiscoveryClient(discoveryEndpoint))
10: {
11: result = discoveryClient.Find(new FindCriteria(typeof(IStringService)));
12: }
13:
14: if (result != null && result.Endpoints.Any())
15: {
16: var endpointMetadata = result.Endpoints.First();
17: address = endpointMetadata.Address;
18: }
19: return address;
20: }
Once we probed the discovery service we will receive the endpoint. So in the client code we can created the channel factory from the endpoint
and binding,
and invoke to the service.
When creating the client side channel factory we need to make sure that the client side binding should be the same as the service side. WCF discovery service can be used to
find the endpoint for a service contract, but the binding is NOT included. This is because the binding was not in the WS-Discovery specification.
In the next post I will demonstrate how to add the binding information into the discovery service. At that moment the client don’t need to create the binding by itself. Instead it will use the binding received from the discovery service.
1: static void Main(string[] args)
2: {
3: Console.WriteLine("Say something...");
4: var content = Console.ReadLine();
5: while (!string.IsNullOrWhiteSpace(content))
6: {
7: Console.WriteLine("Finding the service endpoint...");
8: var address = FindServiceEndpoint();
9: if (address == null)
10: {
11: Console.WriteLine("There is no endpoint matches the criteria.");
12: }
13: else
14: {
15: Console.WriteLine("Found the endpoint {0}", address.Uri);
16:
17: var factory = new ChannelFactory<IStringService>(new NetTcpBinding(), address);
18: factory.Opened += (sender, e) =>
19: {
20: Console.WriteLine("Connecting to {0}.", factory.Endpoint.ListenUri);
21: };
22: var proxy = factory.CreateChannel();
23: using (proxy as IDisposable)
24: {
25: Console.WriteLine("ToUpper: {0} => {1}", content, proxy.ToUpper(content));
26: }
27: }
28:
29: Console.WriteLine("Say something...");
30: content = Console.ReadLine();
31: }
32: }
Similarly, the discovery service probe endpoint
and binding were defined in the configuration file.
1: <?xml version="1.0"?>
2: <configuration>
3: <startup>
4: <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
5: </startup>
6: <appSettings>
7: <add key="announcementEndpointAddress" value="net.tcp://localhost:10010/announcement"/>
8: <add key="probeEndpointAddress" value="net.tcp://localhost:10011/probe"/>
9: <add key="bindingType" value="System.ServiceModel.NetTcpBinding, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"/>
10: </appSettings>
11: </configuration>
OK, now let’s have a test. Firstly start the discovery service,
and then start our discoverable service. When it started it will announced to the discovery service
and registered its endpoint into the repository, which is the local dictionary.
And then start the client
and type something. As you can see the client asked the discovery service for the endpoint
and then establish the connection to the discoverable service.
And more interesting, do NOT close the client console but terminate the discoverable service but press the enter key. This will make the service send the offline message to the discovery service. Then start the discoverable service again. Since we made it use a different address each time it started, currently it should be hosted on another address. If we enter something in the client we could see that it asked the discovery service
and retrieve the new endpoint,
and connect the the service.
Summary
In this post I discussed the benefit of using the discovery service
and the procedures of service announcement
and probe. I also demonstrated how to leverage the WCF Discovery feature in WCF 4.0 to build a simple managed discovery service.
For test purpose, in this example I used the in memory dictionary as the discovery endpoint metadata repository.
And when finding I also just return the first matched endpoint back. I also hard coded the bindings between the discoverable service
and the client.
In next post I will show you how to solve the problem mentioned above, as well as some additional feature for production usage.
You can download the code here.
Hope this helps,
Shaun
All documents
and related graphics, codes are provided "AS IS" without warranty of any kind.
Copyright © Shaun Ziyan Xu. This work is licensed under the Creative Commons License.
