A question on webpage representation in Java
Posted
by Gemma
on Stack Overflow
See other posts from Stack Overflow
or by Gemma
Published on 2010-04-11T05:19:09Z
Indexed on
2010/04/11
5:23 UTC
Read the original article
Hit count: 204
Hello there. I've followed a tutorial and came up with the following method to read the webpage content into a CharSequence
public static CharSequence getURLContent(URL url) throws IOException {
URLConnection conn = url.openConnection();
String encoding = conn.getContentEncoding();
if (encoding == null) {
encoding = "ISO-8859-1";
}
BufferedReader br = new BufferedReader(new
InputStreamReader(conn.getInputStream(),encoding));
StringBuilder sb = new StringBuilder(16384);
try {
String line;
while ((line = br.readLine()) != null) {
sb.append(line);
sb.append('\n');
}
} finally {
br.close();
}
return sb;
}
It will return a representation of the webpage specified by the url. However,this representation is hugely different from what I use "view page source" in my Firefox,and since I need to scrape data from the original webpage(some data segement in the original "view page source" file),it will always fail to find required text on this Java representation. Did I go wrong somewhere?I need your advice guys,thanks a lot for helping!
© Stack Overflow or respective owner