Multi-Parameter Website Scraping With Power Query

Callum Green shows how to build up a URL based off of multiple parameters, scraping data from a page for each permutation of parameters:

The sections highlighted in red are the parameters and sit in between some of the hard-coded URL text

Code Breakdown:

–          Text = http://www.boxofficemojo.com/monthly/?page=

–          Parameter = [Page]

–          Text = &view=calendargross&yr=

–          Parameter = [Year]

–          Text = &month=

–          Parameter = [Month]

–          Text = &p=.htm

This is a rather clever solution, and if your parameters are functionally dependent (unlike this example, where it was a simple cross join of the three domains), you can still use the solution the same way; you just need to populate your parameter combination table differently.

Related Posts

Diagramming Databases With Power BI

Philip Seamark shows how to visualize the relationships between tables using Power BI: The network navigator was another good visual, and if you have an R instance installed on your local machine, you can play with some of the custom R visuals. The catalog views could be used in a similar way to generate power […]

Read More

Web Scraping With Power BI

Imke Feldmann shows how to use Power BI to scrape multiple tables from a webpage: I will present 2 methods here: Append-method: This is the obvious one and is fast for just a few tables. Add-Column-method: A bit more complicated but will be faster for a large number of tables and is also suitable for […]

Read More

Categories

October 2017
MTWTFSS
« Sep Nov »
 1
2345678
9101112131415
16171819202122
23242526272829
3031