When designing a service oriented architecture (SOA) system, there will be a lot of services with many service contracts, endpoints and behaviors. Besides the client calling the service, in a large distributed system a service may invoke other services. In this case, one service might need to know the endpoints it invokes. This might not be a problem in a small system. But when you have more than 10 services this might be a problem. For example in my current product, there are around 10 services, such as the user authentication service, UI integration service, location service, license service, device monitor service, event monitor service, schedule job service, accounting service, player management service, etc.. Benefit of
Discovery Service Since almost all my services need to invoke at least one other service. This would be a difficult task to make sure all services endpoints are configured correctly in every service. And furthermore, it would be a nightmare when a service changed its endpoint at runtime. Hence, we need a
discovery service to remove the dependency (configuration dependency). A
discovery service plays as a service dictionary which stores the relationship between the contracts and the endpoints for every service. By using the
discovery service, when service X wants to invoke service Y, it just need to ask the
discovery service where is service Y, then the
discovery service will return all proper endpoints of service Y, then service X can use the endpoint to send the request to service Y. And when some services changed their endpoint address, all need to do is to update its records in the
discovery service then all others will know its new endpoint. In WCF 4.0
Discovery it supports both managed proxy
discovery mode and ad-hoc
discovery mode. In ad-hoc mode there is no standalone
discovery service. When a client wanted to invoke a service, it will broadcast an message (normally in UDP protocol) to the entire network with the service match criteria. All services which enabled the
discovery behavior will receive this message and only those matched services will send their endpoint back to the client. The managed proxy
discovery service works as I described above. In this post I will only cover the managed proxy mode, where there’s a
discovery service. For more information about the ad-hoc mode please refer to the MSDN. Service Announcement and Probe The main functionality of
discovery service should be return the proper endpoint addresses back to the service who is looking for. In most cases the consume service (as a client) will send the contract which it wanted to request to the
discovery service. And then the
discovery service will find the endpoint and respond. Sometimes the contract and endpoint are not enough. It also contains versioning, extensions attributes. This post I will only cover the case includes contract and endpoint. When a client (or sometimes a service who need to invoke another service) need to connect to a target service, it will firstly request the
discovery service through the “Probe” method with the criteria. Basically the criteria contains the contract type name of the target service. Then the
discovery service will search its endpoint repository by the criteria. The repository might be a database, a distributed cache or a flat XML file. If it matches, the
discovery service will grab the endpoint information (it’s called
discovery endpoint metadata in WCF) and send back. And this is called “Probe”. Finally the client received the
discovery endpoint metadata and will use the endpoint to connect to the target service. Besides the probe,
discovery service should take the responsible to know there is a new service available when it goes online, as well as stopped when it goes offline. This feature is named “Announcement”. When a service started and stopped, it will announce to the
discovery service. So the basic functionality of a
discovery service should includes: 1, An endpoint which receive the service online message, and add the service endpoint information in the
discovery repository. 2, An endpoint which receive the service offline message, and remove the service endpoint information from the
discovery repository. 3, An endpoint which receive the client probe message, and return the matches service endpoints, and return the
discovery endpoint metadata. WCF 4.0
discovery service just covers all these features in it's infrastructure classes.
Discovery Service in WCF 4.0 WCF 4.0 introduced a new assembly named System.ServiceModel.
Discovery which has all necessary classes and interfaces to build a WS-Discovery compliant
discovery service. It supports ad-hoc and managed proxy modes. For the case mentioned in this post, what we need to build is a standalone
discovery service, which is the managed proxy
discovery service mode. To build a managed
discovery service in WCF 4.0 just create a new class inherits from the abstract class System.ServiceModel.
Discovery.DiscoveryProxy. This class implemented and abstracted the procedures of service announcement and probe. And it exposes 8 abstract methods where we can implement our own endpoint register, unregister and find logic. These 8 methods are asynchronized, which means all invokes to the
discovery service are asynchronously, for better service capability and performance. 1, OnBeginOnlineAnnouncement, OnEndOnlineAnnouncement: Invoked when a service sent the online announcement message. We need to add the endpoint information to the repository in this method. 2, OnBeginOfflineAnnouncement, OnEndOfflineAnnouncement: Invoked when a service sent the offline announcement message. We need to remove the endpoint information from the repository in this method. 3, OnBeginFind, OnEndFind: Invoked when a client sent the probe message that want to find the service endpoint information. We need to look for the proper endpoints by matching the client’s criteria through the repository in this method. 4, OnBeginResolve, OnEndResolve: Invoked then a client sent the resolve message. Different from the find method, when using resolve method the
discovery service will return the exactly one service endpoint metadata to the client. In our example we will NOT implement this method. Let’s create our own
discovery service, inherit the base System.ServiceModel.
Discovery.DiscoveryProxy. We also need to specify the service behavior in this class. Since the build-in
discovery service host class only support the singleton mode, we must set its instance context mode to single. 1: using System;
2: using System.Collections.Generic;
3: using System.Linq;
4: using System.Text;
5: using System.ServiceModel.Discovery;
6: using System.ServiceModel;
7:
8: namespace Phare.Service
9: {
10: [ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple)]
11: public class ManagedProxyDiscoveryService : DiscoveryProxy
12: {
13: protected override IAsyncResult OnBeginFind(FindRequestContext findRequestContext, AsyncCallback callback, object state)
14: {
15: throw new NotImplementedException();
16: }
17:
18: protected override IAsyncResult OnBeginOfflineAnnouncement(DiscoveryMessageSequence messageSequence, EndpointDiscoveryMetadata endpointDiscoveryMetadata, AsyncCallback callback, object state)
19: {
20: throw new NotImplementedException();
21: }
22:
23: protected override IAsyncResult OnBeginOnlineAnnouncement(DiscoveryMessageSequence messageSequence, EndpointDiscoveryMetadata endpointDiscoveryMetadata, AsyncCallback callback, object state)
24: {
25: throw new NotImplementedException();
26: }
27:
28: protected override IAsyncResult OnBeginResolve(ResolveCriteria resolveCriteria, AsyncCallback callback, object state)
29: {
30: throw new NotImplementedException();
31: }
32:
33: protected override void OnEndFind(IAsyncResult result)
34: {
35: throw new NotImplementedException();
36: }
37:
38: protected override void OnEndOfflineAnnouncement(IAsyncResult result)
39: {
40: throw new NotImplementedException();
41: }
42:
43: protected override void OnEndOnlineAnnouncement(IAsyncResult result)
44: {
45: throw new NotImplementedException();
46: }
47:
48: protected override EndpointDiscoveryMetadata OnEndResolve(IAsyncResult result)
49: {
50: throw new NotImplementedException();
51: }
52: }
53: }
Then let’s implement the online, offline and find methods one by one. WCF
discovery service gives us full flexibility to implement the endpoint add, remove and find logic. For the demo purpose we will use an internal dictionary to store the services’ endpoint metadata.
In the next post we will see how to serialize and store these information in database.
Define a concurrent dictionary inside the service class since our it will be used in the multiple threads scenario.
1: [ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple)]
2: public class ManagedProxyDiscoveryService : DiscoveryProxy
3: {
4: private ConcurrentDictionary<EndpointAddress, EndpointDiscoveryMetadata> _services;
5:
6: public ManagedProxyDiscoveryService()
7: {
8: _services = new ConcurrentDictionary<EndpointAddress, EndpointDiscoveryMetadata>();
9: }
10: }
Then we can simply implement the logic of service online and offline.
1: protected override IAsyncResult OnBeginOnlineAnnouncement(DiscoveryMessageSequence messageSequence, EndpointDiscoveryMetadata endpointDiscoveryMetadata, AsyncCallback callback, object state)
2: {
3: _services.AddOrUpdate(endpointDiscoveryMetadata.Address, endpointDiscoveryMetadata, (key, value) => endpointDiscoveryMetadata);
4: return new OnOnlineAnnouncementAsyncResult(callback, state);
5: }
6:
7: protected override void OnEndOnlineAnnouncement(IAsyncResult result)
8: {
9: OnOnlineAnnouncementAsyncResult.End(result);
10: }
11:
12: protected override IAsyncResult OnBeginOfflineAnnouncement(DiscoveryMessageSequence messageSequence, EndpointDiscoveryMetadata endpointDiscoveryMetadata, AsyncCallback callback, object state)
13: {
14: EndpointDiscoveryMetadata endpoint = null;
15: _services.TryRemove(endpointDiscoveryMetadata.Address, out endpoint);
16: return new OnOfflineAnnouncementAsyncResult(callback, state);
17: }
18:
19: protected override void OnEndOfflineAnnouncement(IAsyncResult result)
20: {
21: OnOfflineAnnouncementAsyncResult.End(result);
22: }
Regards the find method, the parameter FindRequestContext.Criteria has a method named IsMatch, which can be use for us to evaluate which service metadata is satisfied with the criteria. So the implementation of find method would be like this.
1: protected override IAsyncResult OnBeginFind(FindRequestContext findRequestContext, AsyncCallback callback, object state)
2: {
3: _services.Where(s => findRequestContext.Criteria.IsMatch(s.Value))
4: .Select(s => s.Value)
5: .All(meta =>
6: {
7: findRequestContext.AddMatchingEndpoint(meta);
8: return true;
9: });
10: return new OnFindAsyncResult(callback, state);
11: }
12:
13: protected override void OnEndFind(IAsyncResult result)
14: {
15: OnFindAsyncResult.End(result);
16: }
As you can see, we checked all endpoints metadata in repository by invoking the IsMatch method. Then add all proper endpoints metadata into the parameter.
Finally since all these methods are asynchronized we need some AsyncResult classes as well. Below are the base class and the inherited classes used in previous methods.
1: using System;
2: using System.Collections.Generic;
3: using System.Linq;
4: using System.Text;
5: using System.Threading;
6:
7: namespace Phare.Service
8: {
9: abstract internal class AsyncResult : IAsyncResult
10: {
11: AsyncCallback callback;
12: bool completedSynchronously;
13: bool endCalled;
14: Exception exception;
15: bool isCompleted;
16: ManualResetEvent manualResetEvent;
17: object state;
18: object thisLock;
19:
20: protected AsyncResult(AsyncCallback callback, object state)
21: {
22: this.callback = callback;
23: this.state = state;
24: this.thisLock = new object();
25: }
26:
27: public object AsyncState
28: {
29: get
30: {
31: return state;
32: }
33: }
34:
35: public WaitHandle AsyncWaitHandle
36: {
37: get
38: {
39: if (manualResetEvent != null)
40: {
41: return manualResetEvent;
42: }
43: lock (ThisLock)
44: {
45: if (manualResetEvent == null)
46: {
47: manualResetEvent = new ManualResetEvent(isCompleted);
48: }
49: }
50: return manualResetEvent;
51: }
52: }
53:
54: public bool CompletedSynchronously
55: {
56: get
57: {
58: return completedSynchronously;
59: }
60: }
61:
62: public bool IsCompleted
63: {
64: get
65: {
66: return isCompleted;
67: }
68: }
69:
70: object ThisLock
71: {
72: get
73: {
74: return this.thisLock;
75: }
76: }
77:
78: protected static TAsyncResult End<TAsyncResult>(IAsyncResult result)
79: where TAsyncResult : AsyncResult
80: {
81: if (result == null)
82: {
83: throw new ArgumentNullException("result");
84: }
85:
86: TAsyncResult asyncResult = result as TAsyncResult;
87:
88: if (asyncResult == null)
89: {
90: throw new ArgumentException("Invalid async result.", "result");
91: }
92:
93: if (asyncResult.endCalled)
94: {
95: throw new InvalidOperationException("Async object already ended.");
96: }
97:
98: asyncResult.endCalled = true;
99:
100: if (!asyncResult.isCompleted)
101: {
102: asyncResult.AsyncWaitHandle.WaitOne();
103: }
104:
105: if (asyncResult.manualResetEvent != null)
106: {
107: asyncResult.manualResetEvent.Close();
108: }
109:
110: if (asyncResult.exception != null)
111: {
112: throw asyncResult.exception;
113: }
114:
115: return asyncResult;
116: }
117:
118: protected void Complete(bool completedSynchronously)
119: {
120: if (isCompleted)
121: {
122: throw new InvalidOperationException("This async result is already completed.");
123: }
124:
125: this.completedSynchronously = completedSynchronously;
126:
127: if (completedSynchronously)
128: {
129: this.isCompleted = true;
130: }
131: else
132: {
133: lock (ThisLock)
134: {
135: this.isCompleted = true;
136: if (this.manualResetEvent != null)
137: {
138: this.manualResetEvent.Set();
139: }
140: }
141: }
142:
143: if (callback != null)
144: {
145: callback(this);
146: }
147: }
148:
149: protected void Complete(bool completedSynchronously, Exception exception)
150: {
151: this.exception = exception;
152: Complete(completedSynchronously);
153: }
154: }
155: }
1: using System;
2: using System.Collections.Generic;
3: using System.Linq;
4: using System.Text;
5: using System.ServiceModel.Discovery;
6: using Phare.Service;
7:
8: namespace Phare.Service
9: {
10: internal sealed class OnOnlineAnnouncementAsyncResult : AsyncResult
11: {
12: public OnOnlineAnnouncementAsyncResult(AsyncCallback callback, object state)
13: : base(callback, state)
14: {
15: this.Complete(true);
16: }
17:
18: public static void End(IAsyncResult result)
19: {
20: AsyncResult.End<OnOnlineAnnouncementAsyncResult>(result);
21: }
22:
23: }
24:
25: sealed class OnOfflineAnnouncementAsyncResult : AsyncResult
26: {
27: public OnOfflineAnnouncementAsyncResult(AsyncCallback callback, object state)
28: : base(callback, state)
29: {
30: this.Complete(true);
31: }
32:
33: public static void End(IAsyncResult result)
34: {
35: AsyncResult.End<OnOfflineAnnouncementAsyncResult>(result);
36: }
37: }
38:
39: sealed class OnFindAsyncResult : AsyncResult
40: {
41: public OnFindAsyncResult(AsyncCallback callback, object state)
42: : base(callback, state)
43: {
44: this.Complete(true);
45: }
46:
47: public static void End(IAsyncResult result)
48: {
49: AsyncResult.End<OnFindAsyncResult>(result);
50: }
51: }
52:
53: sealed class OnResolveAsyncResult : AsyncResult
54: {
55: EndpointDiscoveryMetadata matchingEndpoint;
56:
57: public OnResolveAsyncResult(EndpointDiscoveryMetadata matchingEndpoint, AsyncCallback callback, object state)
58: : base(callback, state)
59: {
60: this.matchingEndpoint = matchingEndpoint;
61: this.Complete(true);
62: }
63:
64: public static EndpointDiscoveryMetadata End(IAsyncResult result)
65: {
66: OnResolveAsyncResult thisPtr = AsyncResult.End<OnResolveAsyncResult>(result);
67: return thisPtr.matchingEndpoint;
68: }
69: }
70: }
Now we have finished the
discovery service. The next step is to host it. The
discovery service is a standard WCF service. So we can use ServiceHost on a console application, windows service, or in IIS as usual. The following code is how to host the
discovery service we had just created in a console application.
1: static void Main(string[] args)
2: {
3: using (var host = new ServiceHost(new ManagedProxyDiscoveryService()))
4: {
5: host.Opened += (sender, e) =>
6: {
7: host.Description.Endpoints.All((ep) =>
8: {
9: Console.WriteLine(ep.ListenUri);
10: return true;
11: });
12: };
13:
14: try
15: {
16: // retrieve the announcement, probe endpoint and binding from configuration
17: var announcementEndpointAddress = new EndpointAddress(ConfigurationManager.AppSettings["announcementEndpointAddress"]);
18: var probeEndpointAddress = new EndpointAddress(ConfigurationManager.AppSettings["probeEndpointAddress"]);
19: var binding = Activator.CreateInstance(Type.GetType(ConfigurationManager.AppSettings["bindingType"], true, true)) as Binding;
20: var announcementEndpoint = new AnnouncementEndpoint(binding, announcementEndpointAddress);
21: var probeEndpoint = new DiscoveryEndpoint(binding, probeEndpointAddress);
22: probeEndpoint.IsSystemEndpoint = false;
23: // append the service endpoint for announcement and probe
24: host.AddServiceEndpoint(announcementEndpoint);
25: host.AddServiceEndpoint(probeEndpoint);
26:
27: host.Open();
28:
29: Console.WriteLine("Press any key to exit.");
30: Console.ReadKey();
31: }
32: catch (Exception ex)
33: {
34: Console.WriteLine(ex.ToString());
35: }
36: }
37:
38: Console.WriteLine("Done.");
39: Console.ReadKey();
40: }
What we need to notice is that, the
discovery service needs two endpoints for announcement and probe. In this example I just retrieve them from the configuration file. I also specified the binding of these two endpoints in configuration file as well.
1: <?xml version="1.0"?>
2: <configuration>
3: <startup>
4: <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
5: </startup>
6: <appSettings>
7: <add key="announcementEndpointAddress" value="net.tcp://localhost:10010/announcement"/>
8: <add key="probeEndpointAddress" value="net.tcp://localhost:10011/probe"/>
9: <add key="bindingType" value="System.ServiceModel.NetTcpBinding, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"/>
10: </appSettings>
11: </configuration>
And this is the console screen when I ran my
discovery service. As you can see there are two endpoints listening for announcement message and probe message.
Discoverable Service and Client
Next, let’s create a WCF service that is discoverable, which means it can be found by the
discovery service. To do so, we need to let the service send the online announcement message to the
discovery service, as well as offline message before it shutdown.
Just create a simple service which can make the incoming string to upper. The service contract and implementation would be like this.
1: [ServiceContract]
2: public interface IStringService
3: {
4: [OperationContract]
5: string ToUpper(string content);
6: }
1: public class StringService : IStringService
2: {
3: public string ToUpper(string content)
4: {
5: return content.ToUpper();
6: }
7: }
Then host this service in the console application. In order to make the
discovery service easy to be tested the service address will be changed each time it’s started.
1: static void Main(string[] args)
2: {
3: var baseAddress = new Uri(string.Format("net.tcp://localhost:11001/stringservice/{0}/", Guid.NewGuid().ToString()));
4:
5: using (var host = new ServiceHost(typeof(StringService), baseAddress))
6: {
7: host.Opened += (sender, e) =>
8: {
9: Console.WriteLine("Service opened at {0}", host.Description.Endpoints.First().ListenUri);
10: };
11:
12: host.AddServiceEndpoint(typeof(IStringService), new NetTcpBinding(), string.Empty);
13:
14: host.Open();
15:
16: Console.WriteLine("Press any key to exit.");
17: Console.ReadKey();
18: }
19: }
Currently this service is NOT discoverable. We need to add a special service behavior so that it could send the online and offline message to the
discovery service announcement endpoint when the host is opened and closed. WCF 4.0 introduced a service behavior named ServiceDiscoveryBehavior. When we specified the announcement endpoint address and appended it to the service behaviors this service will be discoverable.
1: var announcementAddress = new EndpointAddress(ConfigurationManager.AppSettings["announcementEndpointAddress"]);
2: var announcementBinding = Activator.CreateInstance(Type.GetType(ConfigurationManager.AppSettings["bindingType"], true, true)) as Binding;
3: var announcementEndpoint = new AnnouncementEndpoint(announcementBinding, announcementAddress);
4: var discoveryBehavior = new ServiceDiscoveryBehavior();
5: discoveryBehavior.AnnouncementEndpoints.Add(announcementEndpoint);
6: host.Description.Behaviors.Add(discoveryBehavior);
The ServiceDiscoveryBehavior utilizes the service extension and channel dispatcher to implement the online and offline announcement logic. In short, it injected the channel open and close procedure and send the online and offline message to the announcement endpoint.
On client side, when we have the
discovery service, a client can invoke a service without knowing its endpoint. WCF
discovery assembly provides a class named DiscoveryClient, which can be used to find the proper service endpoint by passing the criteria.
In the code below I initialized the DiscoveryClient, specified the
discovery service probe endpoint address. Then I created the find criteria by specifying the service contract I wanted to use and invoke the Find method. This will send the probe message to the
discovery service and it will find the endpoints back to me.
The
discovery service will return all endpoints that matches the find criteria, which means in the result of the find method there might be more than one endpoints. In this example I just returned the first matched one back. In the next post I will show how to extend our
discovery service to make it work like a service load balancer.
1: static EndpointAddress FindServiceEndpoint()
2: {
3: var probeEndpointAddress = new EndpointAddress(ConfigurationManager.AppSettings["probeEndpointAddress"]);
4: var probeBinding = Activator.CreateInstance(Type.GetType(ConfigurationManager.AppSettings["bindingType"], true, true)) as Binding;
5: var discoveryEndpoint = new DiscoveryEndpoint(probeBinding, probeEndpointAddress);
6:
7: EndpointAddress address = null;
8: FindResponse result = null;
9: using (var discoveryClient = new DiscoveryClient(discoveryEndpoint))
10: {
11: result = discoveryClient.Find(new FindCriteria(typeof(IStringService)));
12: }
13:
14: if (result != null && result.Endpoints.Any())
15: {
16: var endpointMetadata = result.Endpoints.First();
17: address = endpointMetadata.Address;
18: }
19: return address;
20: }
Once we probed the
discovery service we will receive the endpoint. So in the client code we can created the channel factory from the endpoint and binding, and invoke to the service.
When creating the client side channel factory we need to make sure that the client side binding should be the same as the service side. WCF
discovery service can be used to find the endpoint for a service contract, but the binding is NOT included. This is because the binding was not in the WS-Discovery specification.
In the next post I will demonstrate how to add the binding information into the
discovery service. At that moment the client don’t need to create the binding by itself. Instead it will use the binding received from the
discovery service.
1: static void Main(string[] args)
2: {
3: Console.WriteLine("Say something...");
4: var content = Console.ReadLine();
5: while (!string.IsNullOrWhiteSpace(content))
6: {
7: Console.WriteLine("Finding the service endpoint...");
8: var address = FindServiceEndpoint();
9: if (address == null)
10: {
11: Console.WriteLine("There is no endpoint matches the criteria.");
12: }
13: else
14: {
15: Console.WriteLine("Found the endpoint {0}", address.Uri);
16:
17: var factory = new ChannelFactory<IStringService>(new NetTcpBinding(), address);
18: factory.Opened += (sender, e) =>
19: {
20: Console.WriteLine("Connecting to {0}.", factory.Endpoint.ListenUri);
21: };
22: var proxy = factory.CreateChannel();
23: using (proxy as IDisposable)
24: {
25: Console.WriteLine("ToUpper: {0} => {1}", content, proxy.ToUpper(content));
26: }
27: }
28:
29: Console.WriteLine("Say something...");
30: content = Console.ReadLine();
31: }
32: }
Similarly, the
discovery service probe endpoint and binding were defined in the configuration file.
1: <?xml version="1.0"?>
2: <configuration>
3: <startup>
4: <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
5: </startup>
6: <appSettings>
7: <add key="announcementEndpointAddress" value="net.tcp://localhost:10010/announcement"/>
8: <add key="probeEndpointAddress" value="net.tcp://localhost:10011/probe"/>
9: <add key="bindingType" value="System.ServiceModel.NetTcpBinding, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"/>
10: </appSettings>
11: </configuration>
OK, now let’s have a test. Firstly start the
discovery service, and then start our discoverable service. When it started it will announced to the
discovery service and registered its endpoint into the repository, which is the local dictionary. And then start the client and type something. As you can see the client asked the
discovery service for the endpoint and then establish the connection to the discoverable service.
And more interesting, do NOT close the client console but terminate the discoverable service but press the enter key. This will make the service send the offline message to the
discovery service. Then start the discoverable service again. Since we made it use a different address each time it started, currently it should be hosted on another address. If we enter something in the client we could see that it asked the
discovery service and retrieve the new endpoint, and connect the the service.
Summary
In this post I discussed the benefit of using the
discovery service and the procedures of service announcement and probe. I also demonstrated how to leverage the WCF
Discovery feature in WCF 4.0 to build a simple managed
discovery service.
For test purpose, in this example I used the in memory dictionary as the
discovery endpoint metadata repository. And when finding I also just return the first matched endpoint back. I also hard coded the bindings between the discoverable service and the client.
In next post I will show you how to solve the problem mentioned above, as well as some additional feature for production usage.
You can download the code here.
Hope this helps,
Shaun
All documents and related graphics, codes are provided "AS IS" without warranty of any kind.
Copyright © Shaun Ziyan Xu. This work is licensed under the Creative Commons License.
