Web Data Extraction

Extract and analyze vital competitive data

Got Data?
What happens when you don’t have data? Or not enough data? Or the right data? Or clean data? Or you have great data but weak analytics? Simple. Businesses cannot compete, governments cannot run, militaries cannot win battles, households cannot function, executives cannot make good decisions, employees cannot do a good job, customers cannot make intelligent purchase decisions, teachers cannot teach, students cannot learn . . . need I go on? From a business perspective, Bill Gates described it quite succinctly: “the most meaningful way to differentiate your company from your competitors, the best way to put distance between you and the crowd is to do an outstanding job with information. How you gather, manage and use information will determine whether you win or lose.” And by ‘use information’, Bill was really talking about analytics. But back to the main ingredient of analytics, data!


Where is the biggest treasure trove of data to be found? The place where every business can perpetually decipher the best opportunities to pursue, the best decisions to make, the most profitable alternatives to follow, the customers with the highest lifetime value to target, the best products and services to offer, and of course, the hidden risks and costs to avoid? Answer: The world wide web. OK, you already knew this. And perhaps you already knew that web scraping is a technique that, although not ubiquitous with SMBs, is gaining a more significant role within Fortune 1000 companies every day. Although we call it “web data extraction,” what is web scraping AKA web harvesting, web data extraction, screen scraping, data scraping, web crawling, data mining all about? Web scraping is a data acquisition technique that involves collecting data from websites and then subsequently organizing the data in order to be analyzed and reported for a specific use case.


However, web scraping has (and still is) evolving into a mutant hydra – every time you conquer the data collection of one web site’s data, ten more possibilities appear, which typically offer your organization: A) New opportunities to collect more data and enhance your organization’s insight, or conversely; B) New roadblocks, threats, and risks. Web Scraping is akin to what Robert Ballard does. As an underwater archaeologist (i.e. ocean explorer), Robert perpetually searches and sifts the ocean for hidden treasures. He discovered the RMS Titanic. But metaphorically speaking, “ocean of data” grossly underestimates the magnitude of data on the world wide web, which can be more contextually described as a perpetual series of exponentially expanding oceans that add trillions of data points a day. And just as using a scuba mask and snorkel to search the ocean floor for treasure is ridiculously primitive when compared to Robert Ballard’s Exploration Vessel Nautilus – which is equipped with the powerful computers, sonars, and sensing equipment as well as the remotely operated vehicles Hercules, Argus, Diana, and Echo that can search the ocean to a depth of 6,000 meters – it is just as ridiculously primitive to have your organization deploy staff to gather data via telephones, manually copying and pasting data from the web into Excel, and/or using generic web scraping apps or services. Is it time your organization deploy your own “exploration vessel” and initiate a value added web scraping initiative?
Got Scrapers?
Although web scraping has been around for 10+ years, it’s more en vogue today than ever. But first, let me blow-up two inaccurate perspectives that appear to be gaining traction. One: There is no data scraping software program available on earth that just magically grabs the data you need and automatically wrangles this data into perfectly clean and usable databases. Two: Businesses and consultants that sell web scraping services will promise you this aforementioned magic, yet will often fail to capture and deliver the data you really need. Why? Because data scraping is both an art and science requiring innovative methodologies, deft programming skills, scientific ingenuity, and keen mathematical expertise. Moreover, “value-added data scraping” involves what we call creative analytics [1]. So, just buying a retail-grade web scraping application, then, dumping your data into Excel, then, generating scatterplots and line charts will seldom deliver your organization much insight in order to base important decisions upon.


An effective web scraping project and/or data extraction initiative requires multiple disciplines in order to dig-out the near-invisible gemstones of useful and meaningful information hidden within the nearly 5 billion web pages [2] and approximately 5 zettabytes of data on the web [3]. Web scraping, without the proper experience and expertise is like a five year old kindergartner trying to hit a Nolan Ryan fastball or Bert Blyleven curveball . . . blindfolded. In order to hit the metaphorical home run with web scraping, you need to: know what to look for, see what you’re looking for, connect with what you need, then run the right type of advanced analytics. In essence, value added web scraping, done correctly, will empower your organization to transform your data into actionable insight so you can make home run decisions. If done incorrectly, it will inevitably cost you money, cost you time, and yield strikeouts.
Competitive Analytics developed a copious and proven process of delivering Value-Added Web Scraping (“VAWS”), which is in stark contrast to generic web scraping apps and services. Perfecting our craft of VAWS is a perpetual pursuit built upon our 22 foundational disciplines:


01. Understand Strategic Objectives
02. Understand Data Objectives
03. Diagnose URLs and APIs
04. Prescribe Optimized Process
05. Set-Up Scraping Tools and Protocols
06. Run and Monitor Scrapers
07. Load Data into ETL Tool
08. Organize Data into Usable Databases
09. Union and Join Data
10. Aggregate Necessary Data Fields
11. Clean Data
12. Interpolate & Extrapolate Data
13. Develop Analytical Workflows
14. Conduct Analytics
15. Decipher Relationships
16. Develop Data Visualizations
17. Design & Develop Interactive Dashboards
18. Customize Dashboards, Reports, Alerts
19. Program Auto-Updates and Alerts
20. Provide Strategic and Tactical Insight
21. Audit, Review, and Re-Scrape Data
22. Benchmark Decisions
To learn more about our value-added process, download From URL to BI: 22 Steps to Developing Value-Added Web Scraping, the comprehensive guide to Competitive Analytics’ value-added web scraping process.
More about why I need Web Data Extraction (data scraping)
Web Data Extraction is probably the most valuable competitive data process your company can deploy. By acquiring competitive, market, industry, and economic data on an intraday, daily, weekly, or monthly basis, companies can gain more insight into almost anything they wish, including A) All sectors of the economy: Basic Materials, Conglomerates, Consumer Goods, Financial, Healthcare, Industrial Goods, Services, Technology, Utilities, and Government. B) Financial data such as stock prices, bond prices, options, futures, etc. C) Entertainment such as event dates, concert tickets prices, advertisement data; D) Travel data such as airports, airlines, flights, trucking, automobiles, and motorcycles, E) Contact information such as phone numbers, white pages, yellow pages, and emails; F) Real Estate data such as apartment rents, home prices, mortgage data, foreclosures; G) Crime data; H) Education such as data on colleges, universities, admissions, curriculums; I) Employment such job growth, unemployment, job candidates, resumes; J) Environmental data; K) Journalism; L) SEO Data; M) Competitive Industry Data; N) Healthcare, drug and pharmaceutical Data; O) Government data; P) Food, beverage and restaurant data; Q) Social media data from LinkedIn, Facebook, Twitter, Pinterest, Xing, Google+, Snapchat, Tumblr, YouTube, and Instagram; R) Search Engines such as Google, Yahoo, Bing; S) Retail Sites such as Amazon, Best Buy, Target, Walmart, Staples, eBay, Apple, Craigslist, Airbnb, GitHub . . . and much much more!
Why choose Competitive Analytics for Web Data Extraction
First of all, there are some people and companies that deploy web scraping illegally by copying and reselling data without changing the nature and use of the data itself. At Competitive Analytics, we are extraordinarily mindful of all the legal, moral, and ethical guidelines of what data we extract. Secondly, we deploy the most powerful and robust scraping tools available, with plans to invest nearly a million dollars in 2016 to develop new analytics for a variety of industries and organizations. And last but not least, we take the data we extract and transform it into extremely valuable new information and insight for our clients.


Similar to the myth of Sisyphus (where the King of Corinth punished Sisyphus for an eternity by enslaving him to roll a heavy stone up a hill, only to have the stone escape his grip and roll to the bottom, thus, forcing Sisyphus to start again) . . . web scraping is an endless push-and pull battle of managing constant data influx from new and changing websites, while producing advanced business intelligence, only to be tasked with managing additional data created. Competitive Analytics ensures this process is effective and efficient by following our comprehensive 22 step process . . . ultimately transforming URLs to BI.
For more information or a telephone consultation, please call us at 714-545-2555 or email us at info@competitiveanalytics.com


[1] Click here to read our white paper on creative analytics

[2] http://www.worldwidewebsize.com

[3] https://en.wikipedia.org/wiki/Zettabyte

  • Thank you for another informative forecast session. We find it provides an excellent perspective. Many thanks for the invaluable insights. Given the outstanding feedback from clients and caliber of your lecture, I would like to extend you through the next 6 months. Please let me know if you can accommodate.

    Director, The Landing For Men of Newport Beach
  • One of the high points of my job is from time to time having the good fortune to experience phenomenal business structures and intellectually driven individuals. You definitely fit both of these in spades.

    General Manager, South Coast Plaza/C.J. Segerstrom
  • Thank you again for presenting at our council meeting. We were all very impressed by you and your company’s service.

    Executive Director, IREM Orange County
  • Simply stated, CA exceeded our expectations on all projects. With CA’s passion for accuracy, obsession with using the latest cutting-edge technology, and devotion to client service, I strongly recommend without hesitation the selection of CA as the top choice among market consultants in business today.

    VP Sales and Marketing, SunCal Companies
  • A few weeks ago I sent our friends at Competitive Analytics our monthly report to import into a new iPad tool they created. WOW! It is amazing how they turned an almost impossible collection of data into an intuitive one with built-in analyzers for your iPad!

    Director of Sales & Technology, NES Rentals
  • Competitive Analytics’ approach and methodology is well beyond any firm we experienced. They utilize advanced mathematics and proprietary software in order to assist us with specifically identifying revenue maximization and product optimization. Needless to say, they were worth their fee.

    Chief Financial Officer, Boeing Realty Corp.
  • We use a company called Competitive Analytics to provide projections, trends and metrics based on the information from our own actual results as well as other companies. We have found that this firm is significantly more accurate than any of the other “Economic Forecast” that are done in Southern California.

    Vice President, Business Development, Western National Group
  • When I need comprehensive economic analysis, precision market research, and strategic advice with measurable results, I call David Savlowitz at Competitive Analytics. Competitive Analytics has completed an array of demand studies, pricing studies, forecasting, and qualitative research for our company; and on each and every occasion CA provided phenomenal deliverables that added much insight and value to our business.

    Division President, KB Home
  • Since early 2000, Western National Property Management has engaged Competitive Analytics to assist in developing quantitative and qualitative forecasts specific to the Southern California apartment market, and to assist in forecasting strategic and tactical plans for our clients’ properties. The information garnered from this annual analysis is an integral component of our annual budgeting and business plan process. Due to the proven track record of Competitive Analytics with regard to our portfolio, we have come to rely on their insights and observations about the economic climate that directly impacts how we do business. Competitive Analytics is an intelligence firm that conducts comprehensive market research and precision analytics.  They employ a myriad of innovative methodologies and technology to produce objective, evidence-based market knowledge to assist companies to reach higher levels of performance and profitability.

    President, Western National Group