A question on webpage representation in Java

Posted by Gemma on Stack Overflow See other posts from Stack Overflow or by Gemma
Published on 2010-04-11T05:19:09Z Indexed on 2010/04/11 5:23 UTC
Read the original article Hit count: 204

Filed under:
|

Hello there. I've followed a tutorial and came up with the following method to read the webpage content into a CharSequence

 public static CharSequence getURLContent(URL url) throws IOException {
       URLConnection conn = url.openConnection();
       String encoding = conn.getContentEncoding();
       if (encoding == null) {
         encoding = "ISO-8859-1";
       }
       BufferedReader br = new BufferedReader(new
           InputStreamReader(conn.getInputStream(),encoding));
       StringBuilder sb = new StringBuilder(16384);
       try {
         String line;
         while ((line = br.readLine()) != null) {
           sb.append(line);
           sb.append('\n');
         }
       } finally {
         br.close();
       }
       return sb;
     }

It will return a representation of the webpage specified by the url. However,this representation is hugely different from what I use "view page source" in my Firefox,and since I need to scrape data from the original webpage(some data segement in the original "view page source" file),it will always fail to find required text on this Java representation. Did I go wrong somewhere?I need your advice guys,thanks a lot for helping!

© Stack Overflow or respective owner

Related posts about java

Related posts about html