MongoDB complex MapReduce of video logs

Posted by Justin Hourigan on Stack Overflow See other posts from Stack Overflow or by Justin Hourigan
Published on 2012-11-22T10:02:57Z Indexed on 2012/11/22 11:00 UTC
Read the original article Hit count: 243

Filed under:
|

I have a dataset from video streaming logs. Each video is identified by a FileGUID. The log entries record the FileGUID, the fragment of the video watched and the bandwidth it was watched at.

I would like to create a mapreduce outputting, for each video, a count for fragments both total and for each bandwidth. Ideally it would look like;

{"FileGUID":"50acb3a5796634df0e073285",
  {
    "1":{"total":76, "0832":34, "1028":42},
    "2":{"total":42, "0832":28, "1028":14},
    ...
  }
}

Is this possible with one mapreduce or is it a multi-step process, or should I use a different method?

Here is a sample of the data.

{
  "_id": ObjectId("50acb3a5796634df0e073285"),
  "IP": "46.7.1.88",
  "DateTime": ISODate("2012-10-24T22:59:57.0Z"),
  "FileGUID": "8cdde821fb934a6da7c125a012a26612",
  "Bandwidth": NumberInt(1028),
  "Segment": NumberInt(1),
  "Fragment": NumberInt(237),
  "Status": NumberInt(200),
  "Size": NumberInt(576790),
  "UserAgent": "Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko\/20100101 Firefox\/16.0"
}
{
  "_id": ObjectId("50acb3a5796634df0e073284"),
  "IP": "46.7.1.88",
  "DateTime": ISODate("2012-10-24T22:59:52.0Z"),
  "FileGUID": "8cdde821fb934a6da7c125a012a26612",
  "Bandwidth": NumberInt(1028),
  "Segment": NumberInt(1),
  "Fragment": NumberInt(236),
  "Status": NumberInt(200),
  "Size": NumberInt(577100),
  "UserAgent": "Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko\/20100101 Firefox\/16.0"
}
{
  "_id": ObjectId("50acb3a5796634df0e073283"),
  "IP": "46.7.1.88",
  "DateTime": ISODate("2012-10-24T22:59:47.0Z"),
  "FileGUID": "8cdde821fb934a6da7c125a012a26612",
  "Bandwidth": NumberInt(0832),
  "Segment": NumberInt(1),
  "Fragment": NumberInt(234),
  "Status": NumberInt(200),
  "Size": NumberInt(576664),
  "UserAgent": "Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko\/20100101 Firefox\/16.0"
}
{
  "_id": ObjectId("50acb3a5796634df0e073282"),
  "IP": "46.7.1.88",
  "DateTime": ISODate("2012-10-24T22:59:42.0Z"),
  "FileGUID": "8cdde821fb934a6da7c125a012a26612",
  "Bandwidth": NumberInt(0832),
  "Segment": NumberInt(1),
  "Fragment": NumberInt(233),
  "Status": NumberInt(200),
  "Size": NumberInt(575692),
  "UserAgent": "Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko\/20100101 Firefox\/16.0"
}

© Stack Overflow or respective owner

Related posts about mongodb

Related posts about mapreduce