Storing data to SequenceFile from Apache Pig

Posted by asquithea on Stack Overflow See other posts from Stack Overflow or by asquithea
Published on 2010-03-11T09:52:51Z Indexed on 2010/03/12 12:27 UTC
Read the original article Hit count: 559

Filed under:

piglatin

|

pig

|

hadoop

Apache Pig can load data from Hadoop sequence files using the PiggyBank SequenceFileLoader:

REGISTER /home/hadoop/pig/contrib/piggybank/java/piggybank.jar;

DEFINE SequenceFileLoader org.apache.pig.piggybank.storage.SequenceFileLoader();

log = LOAD '/data/logs' USING SequenceFileLoader AS (...)

Is there also a library out there that would allow writing to Hadoop sequence files from Pig?

© Stack Overflow or respective owner

Related posts about piglatin

How can I load a file into a DataBag from within a Yahoo PigLatin UDF?

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a Pig program where I am trying to compute the minimum center between two bags. In order for it to work, I found I need to COGROUP the bags into a single dataset. The entire operation takes a long time. I want to either open one of the bags from disk within the UDF, or to be able to pass… >>> More
Storing data to SequenceFile from Apache Pig

as seen on Stack Overflow - Search for 'Stack Overflow'
Apache Pig can load data from Hadoop sequence files using the PiggyBank SequenceFileLoader: REGISTER /home/hadoop/pig/contrib/piggybank/java/piggybank.jar; DEFINE SequenceFileLoader org.apache.pig.piggybank.storage.SequenceFileLoader(); log = LOAD '/data/logs' USING SequenceFileLoader AS (...) Is… >>> More
Does throwing an exception in an EvalFunc pig UDF skip just that line, or stop completely?

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a User Defined Function (UDF) written in Java to parse lines in a log file and return information back to pig, so it can do all the processing. It looks something like this: public abstract class Foo extends EvalFunc<Tuple> { public Foo() { super(); } public Tuple… >>> More
ORDER BY job failed in the Pig script while running EmbeddedPig using Java

as seen on Stack Overflow - Search for 'Stack Overflow'
I have this following pig script, which works perfectly using grunt shell (stored the results to HDFS without any issues); however, the last job (ORDER BY) failed if I ran the same script using Java EmbeddedPig. If I replace the ORDER BY job by others, such as GROUP or FOREACH GENERATE, the whole… >>> More
Splitting input into substrings in PIG (Hadoop)

as seen on Stack Overflow - Search for 'Stack Overflow'
Assume I have the following input in Pig: some And I would like to convert that into: s so som some I've not (yet) found a way to iterate over a chararray in pig latin. I have found the TOKENIZE function but that splits on word boundries. So can "pig latin" do this or is this something that… >>> More

Related posts about pig

Storing data to SequenceFile from Apache Pig

as seen on Stack Overflow - Search for 'Stack Overflow'
Apache Pig can load data from Hadoop sequence files using the PiggyBank SequenceFileLoader: REGISTER /home/hadoop/pig/contrib/piggybank/java/piggybank.jar; DEFINE SequenceFileLoader org.apache.pig.piggybank.storage.SequenceFileLoader(); log = LOAD '/data/logs' USING SequenceFileLoader AS (...) Is… >>> More
Pig_Cassandra integration caused - ERROR 1070: Could not resolve CassandraStorage using imports:

as seen on Server Fault - Search for 'Server Fault'
I'm following basic Pig, Cassandra, Hadoop installation. Everything works just fine as a stand alone. No error. However when I tried to run the example file provided by Pig_cassandra example, I got this error. [root@localhost pig]# /opt/cassandra/apache-cassandra-1.1.6/examples/pig/bin/pig_cassandra -x… >>> More
Big Data – Interacting with Hadoop – What is PIG? – What is PIG Latin? – Day 16 of 21

as seen on SQL Authority - Search for 'SQL Authority'
In yesterday’s blog post we learned the importance of the HIVE in Big Data Story. In this article we will understand what is PIG and PIG Latin in Big Data Story. Yahoo started working on Pig for their application deployment on Hadoop. The goal of Yahoo to manage their unstructured data. What… >>> More
Hadoop/Pig Cross-join

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi I am using Pig to cross join two data sets, both with format. This will result as . In my case, if I have both tuples and , it is a duplication. Can I filter those duplications (or not joining them at all)? thanks, Sagie >>> More
ORDER BY job failed in the Pig script while running EmbeddedPig using Java

as seen on Stack Overflow - Search for 'Stack Overflow'
I have this following pig script, which works perfectly using grunt shell (stored the results to HDFS without any issues); however, the last job (ORDER BY) failed if I ran the same script using Java EmbeddedPig. If I replace the ORDER BY job by others, such as GROUP or FOREACH GENERATE, the whole… >>> More