I have a dataset from video streaming logs. Each video is identified by a FileGUID. The log entries record the FileGUID, the fragment of the video watched and the bandwidth it was watched at.
I would like to create a mapreduce outputting, for each video, a count for fragments both total and for each bandwidth. Ideally it would look like;
{"FileGUID":"50acb3a5796634df0e073285",
{
"1":{"total":76, "0832":34, "1028":42},
"2":{"total":42, "0832":28, "1028":14},
...
}
}
Is this possible with one mapreduce or is it a multi-step process, or should I use a different method?
Here is a sample of the data.
{
"_id": ObjectId("50acb3a5796634df0e073285"),
"IP": "46.7.1.88",
"DateTime": ISODate("2012-10-24T22:59:57.0Z"),
"FileGUID": "8cdde821fb934a6da7c125a012a26612",
"Bandwidth": NumberInt(1028),
"Segment": NumberInt(1),
"Fragment": NumberInt(237),
"Status": NumberInt(200),
"Size": NumberInt(576790),
"UserAgent": "Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko\/20100101 Firefox\/16.0"
}
{
"_id": ObjectId("50acb3a5796634df0e073284"),
"IP": "46.7.1.88",
"DateTime": ISODate("2012-10-24T22:59:52.0Z"),
"FileGUID": "8cdde821fb934a6da7c125a012a26612",
"Bandwidth": NumberInt(1028),
"Segment": NumberInt(1),
"Fragment": NumberInt(236),
"Status": NumberInt(200),
"Size": NumberInt(577100),
"UserAgent": "Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko\/20100101 Firefox\/16.0"
}
{
"_id": ObjectId("50acb3a5796634df0e073283"),
"IP": "46.7.1.88",
"DateTime": ISODate("2012-10-24T22:59:47.0Z"),
"FileGUID": "8cdde821fb934a6da7c125a012a26612",
"Bandwidth": NumberInt(0832),
"Segment": NumberInt(1),
"Fragment": NumberInt(234),
"Status": NumberInt(200),
"Size": NumberInt(576664),
"UserAgent": "Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko\/20100101 Firefox\/16.0"
}
{
"_id": ObjectId("50acb3a5796634df0e073282"),
"IP": "46.7.1.88",
"DateTime": ISODate("2012-10-24T22:59:42.0Z"),
"FileGUID": "8cdde821fb934a6da7c125a012a26612",
"Bandwidth": NumberInt(0832),
"Segment": NumberInt(1),
"Fragment": NumberInt(233),
"Status": NumberInt(200),
"Size": NumberInt(575692),
"UserAgent": "Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko\/20100101 Firefox\/16.0"
}
