Data Transformation Pipeline
Posted
by
davenewza
on Programmers
See other posts from Programmers
or by davenewza
Published on 2013-10-23T08:10:51Z
Indexed on
2013/10/23
10:15 UTC
Read the original article
Hit count: 285
I have create some kind of data pipeline to transform coordinate data into more useful information.
Here is the shell of pipeline:
public class PositionPipeline
{
protected List<IPipelineComponent> components;
public PositionPipeline()
{
components = new List<IPipelineComponent>();
}
public PositionPipelineEntity Process(Position position)
{
foreach (var component in components)
{
position = component.Execute(position);
}
return position;
}
public PositionPipeline RegisterComponent(IPipelineComponent component)
{
components.Add(component);
return this;
}
}
Every IPipelineComponent
accepts and returns the same type - a PositionPipelineEntity
. Code:
public interface IPipelineComponent
{
PositionPipelineEntity Execute(PositionPipelineEntity position);
}
The PositionPipelineEntity
needs to have many properties, many which are unused in certain components and required in others. Some properties will also have become redundant at the end of the pipeline.
For example, these components could be executed:
TransformCoordinatesComponent
: Parse the raw coordinate data into a Coordinate type.DetermineCountryComponent
: Determine and stores country code.DetermineOnRoadComponent
: Determine and store whether coordinate is on a road.
Code:
pipeline
.RegisterComponent(new TransformCoordinatesComponent())
.RegisterComponent(new DetermineCountryComponent())
.RegisterComponent(new DetermineOnRoadComponent());
pipeline.Process(positionPipelineEntity);
The PositionPipelineEntity
type:
public class PositionPipelineEntity
{
// Only relevant to the TransformCoordinatesComponent
public decimal RawCoordinateLatitude { get; set; }
// Only relevant to the TransformCoordinatesComponent
public decimal RawCoordinateLongitude { get; set; }
// Required by all components after TransformCoordinatesComponent
public Coordinate CoordinateLatitude { get; set; }
// Required by all components after TransformCoordinatesComponent
public Coordinate CoordinateLongitude { get; set; }
// Set in DetermineCountryComponent, not required anywhere.
// Requires CoordinateLatitude and CoordinateLongitude (TransformCoordinatesComponent)
public string CountryCode { get; set; }
// Set in DetermineOnRoadComponent, not required anywhere.
// Requires CoordinateLatitude and CoordinateLongitude (TransformCoordinatesComponent)
public bool OnRoad { get; set; }
}
Problems:
I'm very concerned about the dependency that a component has on properties. The way to solve this would be to create specific types for each component. The problem then is that I cannot chain them together like this.
The other problem is the order of components in the pipeline matters. There is some dependency. The current structure does not provide any static or runtime checking for such a thing.
Any feedback would be appreciated.
© Programmers or respective owner