Using External Input Data in Off-the-shelf Web Scrapers

There is a question I’ve wanted to shed some light upon for a long time already: “What if I need to scrape several URL’s based on data in some external database?“.

For example, recently one of our visitors asked a very good question (thanks, Ed):

    “I have a large list of amazon.com asin. I would like to scrape 10 or so fields for each asin. Is there any web scraping software available that can read each asin from a database and form the destination url to be scraped like http://www.amazon.com/gp/product/{asin} and scrape the data?”

This question impelled me to investigate this matter. I contacted several web scraper developers, and they kindly provided me with detailed answers that allowed me to bring the following summary to your attention:
Visual Web Ripper

An input data source can be used to provide a list of input values to a data extraction project. A data extraction project will be run once for each row of input values. You can find the additional information here.
Web Content Extractor

You can use the -at”filename” command line option to add new URLs from TXT or CSV file:

    WCExtractor.exe projectfile -at”filename” -s

projectfile: the file name of the project (*.wcepr) to open.
filename – the file name of the CSV or TXT file that contains URLs separated by newlines.
-s – starts the extraction process

You can find some options and examples here.
Mozenda

Since Mozenda is cloud-based, the external data needs to be loaded up into the user’s Mozenda account. That data can then be easily used as part of the data extracting process. You can construct URLs, search for strings that match your inputs, or carry through several data fields from an input collection and add data to it as part of your output. The easiest way to get input data from an external source is to use the API to populate data into a Mozenda collection (in the user’s account). You can also input data in the Mozenda web console by importing a .csv file or importing one through our agent building tool.

Once the data is loaded into the cloud, you simply initiate building a Mozenda web agent and refer to that Data list. By using the Load page action and the variable from the inputs, you can construct a URL like http://www.amazon.com/gp/product/%asin%.
Helium Scraper

Here is a video showing how to do this with Helium Scraper:


The video shows how to use the input data as URLs and as search terms. There are many other ways you could use this data, way too many to fit in a video. Also, if you know SQL, you could run a query to get the data directly from an external MS Access database like
SELECT * FROM [MyTable] IN "C:\MyDatabase.mdb"

Note that the database needs to be a “.mdb” file.
WebSundew Data Extractor
Basically this allows using input data from external data sources. This may be CSV, Excel file or a Database (MySQL, MSSQL, etc). Here you can see how to do this in the case of an external file, but you can do it with a database in a similar way (you just need to write an SQL script that returns the necessary data).
In addition to passing URLs from the external sources you can pass other input parameters as well (input fields, for example).
Screen Scraper

Screen Scraper is really designed to be interoperable with all sorts of databases. We have composed a separate article where you can find a tutorial and a sample project about scraping Amazon products based on a list of their ASINs.


Source: http://extract-web-data.com/using-external-input-data-in-off-the-shelf-web-scrapers/

How to scrape Yellow Pages with ScreenScraper Chrome Extension

Recently I was asked to help with the job of scraping company information from the Yellow Pages website using the ScreenScraper Chrome Extension. After working with this simple scraper, I decided to create a tutorial on how to use this Google Chrome Extension for scraping pages similar to this one. Hopefully, it will be useful to many of you.
1. Install the Chrome Extension

You can get the extension here. After installation you should see a small monitor icon in the top right corner of your Chrome browser.
2. Open the source page

Let’s open the page from which you want to scrape the company information:



3. Determine the parent element (row)

The first thing you need to do for the scraping is to determine which HTML element will be the parent element. A parent element is the smallest HTML element that contains all the information items you need to scrape (in our case they are Company Name, Company Address and Contact Phone).  To some extent a parent element defines a data row in the resulting table.

To determine it, open Google Chrome Developer Tools (by pressing Ctrl+Shift+I), click the magnifying class (at the bottom of the window) and select the parent element on the page. I selected this one:


As soon as you have selected it, look into the developer tools window and you will see the HTML code related to this element:


As is seen from the highlighted HTML line, you can easily define a parent element by its class: listingInfoAndLogo.

5. Determine the information elements (columns)

After you have learned how to determine the parent element, it should be easy to specify the information elements that contain the information you want to scrape (they represent columns in the resultant table).

Just do this in the same way that you did it for the parent element -  by selecting it on the page:

As you can see, the company name is defined by businessName class.
6. Tune the ScreenScraper itself

After all the data elements you want to scrape are found, open the ScreenScraper by clicking the small monitor icon in the top-right corner of your browser. Then do the following:

    Enter the parent element class name (listingInfoAndLogo in our case) into the Selector field, preceding it with a dot (*see below for why)
    Click the Add Column button
    Enter a field’s name (any) into the Field text box
    Enter the information item class into the Selector text box, preceding it with a dot
    Repeat steps 2-4 for each information item element you want to be scraped


After you enter all these definitions you should see the preview of the scraped data at the bottom of the extension’s windowThat’s it! I hope the tutorial is clear enough. But if not, feel free to write your comments below and I’ll give additional explanations.


Source: http://extract-web-data.com/how-to-scrape-yellow-pages-with-screenscraper-chrome-extension/

Data Mining: The AdWords Problem Review

This post is a continuation of the previous post on Advertising on the Web and Data mining. Here we conclude by reviewing some basic algorithms for placing ads on the web. The adwords problem is to be solved with association graphs algorithm in Data Mining.
The AdWords Problem

With Adwords we consider Google’s example. Google’s policy for placing ads is not on the bid value but in the total amount expected to be received for the display of each ad. The value of an ad was taken to be the product of the bid and the click-through rate (the click-through rate is taken for each ad, based on the statistics of displays of an ad).
Match graphs

After we concluded that the ad placement challenge is to be done with on-line algorithms of greedy or generalized balance types (previous post), we need to consider also the match graphs for bid-query match.

This problem involves two sets of nodes and a set of edges between members of the two sets. The first set is the incoming search queries while the second set is the related ads from various sources. The goal is to find a maximal matching — as large a set of edges as possible that includes no node more than once. The On-Line Solution to the Matching Problem would be a greedy algorithm for finding a match in a bipartite graph. It is done by ordering the edges in certain way. The competitive ratio in this case should be no less than 1/2. That is the on-line greedy matching algorithm finds matches at least half as many nodes as the best off-line algorithm does.
Basic Adwords workflow

Of course, the decision regarding which ads to show must be made on-line. So, there should be only on-line algorithms considered for solving the adwords problem. The input data and algorithm results are as follows.
Input:
1. A set of bids by advertisers for search queries.
2. A click-through rate (CTR) for each advertiser-query pair
3. A budget for each advertiser. We shall assume budgets are for a
month, although any unit of time could be used.
4. A limit on the number of ads to be displayed with each search query.

• Output – Respond to each search query with a set of advertisers of those characteristics:
1. The size of the set is no larger than the limit on the number of ads per query.
2. Each advertiser has a bid on the search query.
3. Each advertiser has enough budget left to pay for the ad if it is
clicked upon.


Implementing the Adwords algorithm

Basically we now have an idea of how ads are selected to go with the answer to a search query. But we have not addressed the problem of finding the bids that have been made on a given query.
Finding bids for search queries

Which algorithm finds the bids that have been made on a given query? For a simple case we can represent a query by the list of its words, in sorted order. Bids are stored in a hash table or similar structure, with a hash key equal to the sorted list of words. A search query can then be matched against bids by a straightforward lookup in the table.

The simplest version of this implementation serves in situations where the bids are on exactly the set of words in the search query (in reality, search engines all offer to advertisers a ‘broad matching’ feature). It also does not consider the historical frequency of queries or statistics.

The task might not just be limited to the same set of words. For example, Google also matches adwords bids to emails. There, the match criterion is not based on the equality of sets but on the inclusiveness of the bid’s set in the related document (email). How could you compare the millions of words from bids with the millions of words in the stream of regular emails?
Matching Algorithm for Documents and Bids

The Matching Algorithm for Documents and Bids has been developed for this purpose. Briefly, it works by processing the words of the document, rarest-first. Word sets whose first word is the current word are copied to a temporary hash table, with the second word as the key. Sets already in the temporary hash table are examined to see if the word that is their key matches the current word, and, if so, they are rehashed using their next word. Sets whose last word is matched are copied to the output. For more details on this algorithm you might read more from the same book, paragraph 8.5.3.
Summary

The data mining algorithms for ad placement in response to a search query or larger documents (emails) are on-line greedy or generalized balance algorithms on the match graphs. The complex query/bid matching process is to be done with hashing sets-of-words tables with the consequent matching of rare to unrare word order in order to find the total match of the bid set in the document.



Source: http://extract-web-data.com/data-mining-the-adwords-problem-review/

How to Outsource Data Entry Work Effectively

In today's world it is a well known fact that many businesses now outsource data entry work. All businesses are concerned with the running costs of their business as well as keeping clients and staff happy. One of the ways to achieve all of these goals is to use outsourcing techniques, which are growing in strength each year.

Outsourcing is now a staple part of business life. Whether you are a large conglomerate or a small office based business, there are aspects of your business which are already outsourced. For example, you may likely have a contract with a cleaner to clean your office or gardener to tidy up that hedge.

It is true to say that many larger businesses have the time, resources and money to invest in employing their own in-house own data entry specialists. However, mid-sized and smaller companies need to be able to operate at the same level as the large companies, but with less money, time and resources. This is where they can benefit from outsourcing this kind of work.

If you want to outsource data entry work, you need to firstly analyze how much it is going to aid your business. Is it necessary for your data entry work to be outsourced? You need to have a solid idea of your future business plans and work out where the data entry outsourcing fits into the plan. You need to do a lot of research and communicate with prospective outsourcing companies or individuals. Do not be afraid to ask questions; it is your business at stake should anything go wrong.

By outsourcing your data compilation work, you are taking care of many business related issues. Many data entry specialists either work as independent freelancers or may be part of a company specializing in outsourced data entry. This results in lower costs for your business; you are likely to receive a quote from an outsourcing company that is very competitive. If the work is an ad-hoc project, you may find that a freelance data entry worker is the cheapest option.

As the years have shown, outsourcing has proved a viable and advantageous option for many businesses. Whether it is employing a call center supervisor or a data specialist, your lower core competences can be dealt with by outside help. This leaves you to concentrate on the core competences that are of higher importance to the business and allow you to use your valuable time wisely.

Outsourcing is also a lot cheaper than employing in-house staff. The companies that offer to outsource entry of data have skilled workers, who can increase productivity whilst keeping your costs to a minimum. There is also the advantage of focusing your in-house staff; if you outsource data entry work it will allow more interesting, less-time consuming and important projects to be enjoyed by your own staff.

New technology is also emerging each year in the business world. By employing companies to outsource data entry projects you can eliminate some of the risk, save some time and some money. Many outsourcing companies have the latest technology in order for them to keep producing world-class results for their clients.




Source: http://ezinearticles.com/?How-to-Outsource-Data-Entry-Work-Effectively&id=2449297

Data Entry in Outsourcing Businesses

The process in, which a business house engages another company to do a particular type of work instead of using its own employees to do the same work, is called outsourcing. This is basically practiced so that the company can concentrate more on the core function. The cheap cost of outsourcing work is also another reason.

Outsourcing companies are often referred as "business to business" companies. Their business is dependent on the service provided by them to other business houses. Nowadays, every company is engaged in outsourcing. When a sole proprietor gives responsibility to another to buy supplies for the office, then automatically this process becomes outsourcing. In a real sense, it is almost impossible to do everything by yourself. You have to become dependent on those who are skilled in certain fields.

Data entry is one of the oldest and well known as the most common outsourcing activities that have been widely accepted across the globe for a long period of time. Still today, the demand is sky rocketing and the scope of data entry companies are just expanding.

All companies value their data very much. In order to generate good business, you need to deal with your data efficiently. Thus, companies related to BTB activities take care of the data handling very seriously. The employees are trained and prepared for all sorts of detailed oriented work. The services vary from back office support for a banking institute, calculation of medical bills, maintaining payroll functions etc. Banks generally outsource the work of the business class customers. Lock box payment is one of such example.

There are plenty of companies in the market of outsourcing who are engaged in providing in different kinds of services to the clients all across the globe. Many companies, which are earlier engaged into hard core data entry operations, are now exploring the area of medical billing, research work, project work for various universities, marketing job, news agencies, trade and several types of insurance organizations.

You can help your company to grow and reach a tremendous height once you get accustomed to take the advantages from various available data entry work. The service providers take an extra step to make sure that the work those are being delivered are of high quality and fulfill all the requirements as asked by the clients. Accuracy and punctually are the keywords to survive in the outsourcing market. Companies prefer outsourcing as the cost is always lower than the company would require spending on salaries if the same work was done by their own employees. Outsourcing is a very lucrative option for many business houses as it gives you the freedom to concentrate on your core business process and even you end up saving a good sum of money by outsourcing data entry work.



Source: http://ezinearticles.com/?Data-Entry-in-Outsourcing-Businesses&id=2021508

Outsourcing Data Entry Services

Data or raw information is the backbone of any industry or business organization. However, raw data is seldom useful in its pure form. For it to be of any use, data has to be recorded properly and organized in a particular manner. Only then can data be processed. That is why it is important to ensure accurate data entry. But because of the unwieldy nature of data, feeding data is a repetitive and cumbersome job and it requires heavy investment, both in terms of time and energy from staff. At the same time, it does not require a high level of technical expertise. Due to these factors, data entry can safely be outsourced, enabling companies to devote their time and energy on tasks that enhance their core competence.

Many companies, big and small, are therefore enhancing their productivity by outsourcing the endless monotonous tasks that tend to cut down the organization's productivity. In times to come, outsourcing these services will become the norm and the volume of work that is outsourced will multiply. The main reason for these kinds of development is the Internet. Web based customer service and instant client support has made it possible for service providers to act as one stop business process outsourcing partners to parent companies that require support.

Data entry services are not all alike. Different clients have different demands. While some clients may require recording information coupled with document management and research, others may require additional services like form processing or litigation support. Data entry itself could be from various sources. For instances, sometimes information may need to be typed out from existing documents while at other times, data needs to be extracted from images or scanned documents. To rise up to these challenges, service providers who offer these services must have the expertise and the software to ensure rapid and accurate data entry. That is why it is important to choose your service provider with a lot of care.

Before hiring your outsourcing partner, you need to ask yourself the following questions.

* What kind of reputation does the company enjoy? Do they have sufficient years of experience? What kind of history and background does the company enjoy?

* Do they have a local management arm that you can liaise with on a regular basis?

* Do the service personnel understand your requirements and can they handle them effectively?

* What are the steps taken by the company to ensure that there is absolutely no compromise in confidentiality and security while dealing with vital confidential data?

* Is there a guarantee in place?

* What about client references?

The answers to these questions will help you identify the right partner for outsourcing your data entry service requirements.




Source: http://ezinearticles.com/?Outsourcing-Data-Entry-Services&id=3568373

Data Mining and Financial Data Analysis

Introduction:

Most marketers understand the value of collecting financial data, but also realize the challenges of leveraging this knowledge to create intelligent, proactive pathways back to the customer. Data mining - technologies and techniques for recognizing and tracking patterns within data - helps businesses sift through layers of seemingly unrelated data for meaningful relationships, where they can anticipate, rather than simply react to, customer needs as well as financial need. In this accessible introduction, we provides a business and technological overview of data mining and outlines how, along with sound business processes and complementary technologies, data mining can reinforce and redefine for financial analysis.

Objective:

1. The main objective of mining techniques is to discuss how customized data mining tools should be developed for financial data analysis.

2. Usage pattern, in terms of the purpose can be categories as per the need for financial analysis.

3. Develop a tool for financial analysis through data mining techniques.

Data mining:

Data mining is the procedure for extracting or mining knowledge for the large quantity of data or we can say data mining is "knowledge mining for data" or also we can say Knowledge Discovery in Database (KDD). Means data mining is : data collection , database creation, data management, data analysis and understanding.

There are some steps in the process of knowledge discovery in database, such as

1. Data cleaning. (To remove nose and inconsistent data)

2. Data integration. (Where multiple data source may be combined.)

3. Data selection. (Where data relevant to the analysis task are retrieved from the database.)

4. Data transformation. (Where data are transformed or consolidated into forms appropriate for mining by performing summary or aggregation operations, for instance)

5. Data mining. (An essential process where intelligent methods are applied in order to extract data patterns.)

6. Pattern evaluation. (To identify the truly interesting patterns representing knowledge based on some interesting measures.)

7. Knowledge presentation.(Where visualization and knowledge representation techniques are used to present the mined knowledge to the user.)

Data Warehouse:

A data warehouse is a repository of information collected from multiple sources, stored under a unified schema and which usually resides at a single site.

Text:

Most of the banks and financial institutions offer a wide verity of banking services such as checking, savings, business and individual customer transactions, credit and investment services like mutual funds etc. Some also offer insurance services and stock investment services.

There are different types of analysis available, but in this case we want to give one analysis known as "Evolution Analysis".

Data evolution analysis is used for the object whose behavior changes over time. Although this may include characterization, discrimination, association, classification, or clustering of time related data, means we can say this evolution analysis is done through the time series data analysis, sequence or periodicity pattern matching and similarity based data analysis.

Data collect from banking and financial sectors are often relatively complete, reliable and high quality, which gives the facility for analysis and data mining. Here we discuss few cases such as,

Eg, 1. Suppose we have stock market data of the last few years available. And we would like to invest in shares of best companies. A data mining study of stock exchange data may identify stock evolution regularities for overall stocks and for the stocks of particular companies. Such regularities may help predict future trends in stock market prices, contributing our decision making regarding stock investments.

Eg, 2. One may like to view the debt and revenue change by month, by region and by other factors along with minimum, maximum, total, average, and other statistical information. Data ware houses, give the facility for comparative analysis and outlier analysis all are play important roles in financial data analysis and mining.

Eg, 3. Loan payment prediction and customer credit analysis are critical to the business of the bank. There are many factors can strongly influence loan payment performance and customer credit rating. Data mining may help identify important factors and eliminate irrelevant one.

Factors related to the risk of loan payments like term of the loan, debt ratio, payment to income ratio, credit history and many more. The banks than decide whose profile shows relatively low risks according to the critical factor analysis.

We can perform the task faster and create a more sophisticated presentation with financial analysis software. These products condense complex data analyses into easy-to-understand graphic presentations. And there's a bonus: Such software can vault our practice to a more advanced business consulting level and help we attract new clients.

To help us find a program that best fits our needs-and our budget-we examined some of the leading packages that represent, by vendors' estimates, more than 90% of the market. Although all the packages are marketed as financial analysis software, they don't all perform every function needed for full-spectrum analyses. It should allow us to provide a unique service to clients.

The Products:

ACCPAC CFO (Comprehensive Financial Optimizer) is designed for small and medium-size enterprises and can help make business-planning decisions by modeling the impact of various options. This is accomplished by demonstrating the what-if outcomes of small changes. A roll forward feature prepares budgets or forecast reports in minutes. The program also generates a financial scorecard of key financial information and indicators.

Customized Financial Analysis by BizBench provides financial benchmarking to determine how a company compares to others in its industry by using the Risk Management Association (RMA) database. It also highlights key ratios that need improvement and year-to-year trend analysis. A unique function, Back Calculation, calculates the profit targets or the appropriate asset base to support existing sales and profitability. Its DuPont Model Analysis demonstrates how each ratio affects return on equity.

Financial Analysis CS reviews and compares a client's financial position with business peers or industry standards. It also can compare multiple locations of a single business to determine which are most profitable. Users who subscribe to the RMA option can integrate with Financial Analysis CS, which then lets them provide aggregated financial indicators of peers or industry standards, showing clients how their businesses compare.

iLumen regularly collects a client's financial information to provide ongoing analysis. It also provides benchmarking information, comparing the client's financial performance with industry peers. The system is Web-based and can monitor a client's performance on a monthly, quarterly and annual basis. The network can upload a trial balance file directly from any accounting software program and provide charts, graphs and ratios that demonstrate a company's performance for the period. Analysis tools are viewed through customized dashboards.

PlanGuru by New Horizon Technologies can generate client-ready integrated balance sheets, income statements and cash-flow statements. The program includes tools for analyzing data, making projections, forecasting and budgeting. It also supports multiple resulting scenarios. The system can calculate up to 21 financial ratios as well as the breakeven point. PlanGuru uses a spreadsheet-style interface and wizards that guide users through data entry. It can import from Excel, QuickBooks, Peachtree and plain text files. It comes in professional and consultant editions. An add-on, called the Business Analyzer, calculates benchmarks.

ProfitCents by Sageworks is Web-based, so it requires no software or updates. It integrates with QuickBooks, CCH, Caseware, Creative Solutions and Best Software applications. It also provides a wide variety of businesses analyses for nonprofits and sole proprietorships. The company offers free consulting, training and customer support. It's also available in Spanish.

ProfitSystem fx Profit Driver by CCH Tax and Accounting provides a wide range of financial diagnostics and analytics. It provides data in spreadsheet form and can calculate benchmarking against industry standards. The program can track up to 40 periods.



Source: http://ezinearticles.com/?Data-Mining-and-Financial-Data-Analysis&id=2752017