Wednesday, 29 May 2013

Scraping Pavement: Mortgage Rates Hit New Lows

The ups and (mostly) downs of the housing market, combined with tax incentives have left many wondering whether or not now might be a good time to finally buy that dream home. Average interest rates on thirty year mortgage loans have plummeted once more, falling a full percentage point below the lowest rate at any time last year. According to Yahoo! Finance, the average thirty year mortgage rate is now a pavement-scraping 4.78%. That’s down from 4.83% just last week, and it matches the record low set in April of this year. Putting the numbers into further perspective, Freddie Mac reports the average 30 year rate at this time last year was 5.97%.

Thirty year mortgage rates were not the only ones to drop this week. Average rates on fifteen year fixed mortgages also sank, from 4.32% last week to 4.29% this week. That’s the lowest 15 year fixed mortgage rates have been since statistics were kept on them beginning in 1991. Additionally, five year, adjustable-rate mortgages saw their average rates fall to 4.18% this week from 4.25% last week. One year, adjustable rate mortgages held steady at 4.35% for the second week straight.

2711097891_952dd3025b

(Daryl Davis)

The rapid fall can be traced back to last November, when the Fed began its gigantic $1.5 trillion shopping spree of toxic, mortgage-backed securities to lower lending rates. By and large these efforts have succeeded, as rates have hung around the 5% neighborhood since April and triggered a flurry of mortgage refinancing. According to Freddie Mac chief economist Frank Nothaft, a homeowner who refinanced today would stand to save about $100 per month on a $200,000 fixed-rate mortgage.

On the surface, the falling rates discussed above seem to bode well for the struggling housing market. But according to the Wall Street Journal, refinancing activity may be slowing to a crawl. In the week ending November 20, refinancings reportedly fell 9.5% in spite of falling rates. The Journal also states that the “overall pace of mortgage applications also dropped in the week ended Nov. 13, down 2.5%.” It appears that the housing market’s problems may run too deep for low interest rates to correct, at least presently.


Source: http://www.mint.com/blog/goals/scraping-pavement-mortgage-rates-hit-new-lows/

Sunday, 26 May 2013

Limitations and Challenges in Effective Web Data Mining

Web data mining and data collection is critical process for many business and market research firms today. Conventional Web data mining techniques involve search engines like Google, Yahoo, AOL, etc and keyword, directory and topic-based searches. Since the Web's existing structure cannot provide high-quality, definite and intelligent information, systematic web data mining may help you get desired business intelligence and relevant data.

Factors that affect the effectiveness of keyword-based searches include:
• Use of general or broad keywords on search engines result in millions of web pages, many of which are totally irrelevant.
• Similar or multi-variant keyword semantics my return ambiguous results. For an instant word panther could be an animal, sports accessory or movie name.
• It is quite possible that you may miss many highly relevant web pages that do not directly include the searched keyword.

The most important factor that prohibits deep web access is the effectiveness of search engine crawlers. Modern search engine crawlers or bot can not access the entire web due to bandwidth limitations. There are thousands of internet databases that can offer high-quality, editor scanned and well-maintained information, but are not accessed by the crawlers.

Almost all search engines have limited options for keyword query combination. For example Google and Yahoo provide option like phrase match or exact match to limit search results. It demands for more efforts and time to get most relevant information. Since human behavior and choices change over time, a web page needs to be updated more frequently to reflect these trends. Also, there is limited space for multi-dimensional web data mining since existing information search rely heavily on keyword-based indices, not the real data.

Above mentioned limitations and challenges have resulted in a quest for efficiently and effectively discover and use Web resources. Send us any of your queries regarding Web Data mining processes to explore the topic in more detail.

Source: http://ezinearticles.com/?Limitations-and-Challenges-in-Effective-Web-Data-Mining&id=5012994

Saturday, 18 May 2013

Searching and retrieving yahoo answers in PHP

I am working on an WordPress autobloggin plugin where I had to search yahoo answers site for any keyword and post the answers in WordPress. After checking their site, I see it is very easy now as they provide API for the questions and answers. I do not know when they introduced API for Answers but I do remember during my last check (quite long time ago), they did not have any API and scrapping was the only way.

Their API for searching and getting answers is very simple and easiest! However, for the better handling of my project, I have mad a class to find questions by any given keyword and then retrieve the answers of any specific question by question ID.

How to use?

First of all you need to download the class file. Click here to download it. Then unzip the file.

Then include the class in your script like (assuming your script and this class file exists in same directory):
?

   
require('yahooanswer.class.php');
$params = array(
    'query'     =>      'keyword',   //enter your keyword here. this will be searched on yahoo answer
    'results'   =>      10,         //number of questions it should return
    'type'      =>      'resolved',  //only resolved questiosn will be returned. other values can be all, open, undecided
    'output'    =>      'php',      //result will be PHP array. Other values can be xml, json, rss
);
$query  = 'yoga'; //enter your keyword here to search for
$yn = new yahooAnswer($appid);
//search questions
try{
    $questions = $yn->search_questions($params);
} catch (Exception $e){
    echo ($e->getMessage());
}

There are more options to be set in $params. Please check this URL to know all options. You just need to set any option as key=>value pair in the $params array. It will be conveyed correctly for you :) .

So, if everything correct, we will get results like the following:
?
   
[0] => Array
        (
            [id] => 20080610153832AAStQwZ
            [type] => Answered
            [subject] => Yoga??????
            [content] => I usually do yoga and relaxation things and stuff like that. and i have a couple questions
1. How long do you think it would take me to lose about 5 pounds while doing yoga?
2. What are the best yoga tips and exersises?
3. What are some good ways to relax?
4. Any advise about yoga?
Those are my questions..hope you can answer them <img src="http://thehungrycoder.com/wp-includes/images/smilies/icon_smile.gif" alt=":)" class="wp-smiley">  thanks!
            [date] => 2008-06-10 15:38:32
            [timestamp] => 1213137512
            [link] => http://answers.yahoo.com/question/?qid=20080610153832AAStQwZ
            [category] => Diet & Fitness
            [userid] => yqaYuvZxaa
            [usernick] => Sjharr S
            [userphotourl] =>
            [numanswers] => 5
            [numcomments] => 2
            [chosenanswer] => I would add walking or aerobic exercise plus meditation and yoga.
            [chosenanswerid] => soZI0hNlaa
            [chosenanswernick] => carpal-tunnel-provider
            [chosenanswertimestamp] => 1213411114
            [chosenanswerawardtimestamp] => 1213681330
        )

Let’s say we need to fetch the answers of all questions. Simply call the get_question() method inside a loop.
?

foreach ($questions as $question) {
    //now get the answers for the question_id;
    try{
        $answers = $yn->get_question(array('question_id'=>$question_id));
        echo '<pre>';
        print_r($answers);
        echo '<pre>';
    } catch (Exception $e){
        echo($e->getMessage());
    }
}

So, you got answers of all the questions. That’s it. To know the detail option for getting answers you can check this page.  There are other APIs to use with Yahoo Answers however, in my class I implemented only the methods I needed for my project. If you needed any other methods, you can extend this class and implement those. Please ping this blog if you do same so that we all know about your work.

Source: http://thehungrycoder.com/scripts/php-scripts-2/searching-and-retrieving-yahoo-answers-in-php.html

Wednesday, 15 May 2013

Yahoo data scraper

Yahoo data scraper is a data scraping software that enables you to scrape the search result listing from yahoo query results pages. It is very fast, accurate and convenient to use. The results obtained from this data scraping tool can be used to solve various business problems and queries and present the solution in an organized manner.

Google Search Result scraper Features:

- Browses through all the yahoo search result listing of all the yahoo query results pages for the specified searching keywords, and extract the result listing.
- Search result by keyword, question, news topics, blog category or other parameters.
- Extracts data fields like : Yahoo Data URL, Yahoo Images, Yahoo Answers, Yahoo Blog, Yahoo News, Yahoo Finance, Yahoo Video and many more...
- Extract data can be saved in various forms such as CSV, MySQL, MS-Access, XML, MSSQL, HTML Files
- Download images, video from yahoo website
- Avoid IP Blocks with multiple proxy feature. Scrap anonymously, and without getting blocked.
- Set custom delay between web requests.
- Easy to use tool | Quick Learning curve and right to the point.
- Requires minimal user inputs.
- Compatible with Microsoft XP/Vista/Windows 7/8


Manually extracting and pasting the data can be very time consuming. To save on the time and effort of the user, various data scraping tools are available online. One of the most powerful tool is the our Yahoo Data Scraper.