Module ScraperWiki
In: lib/scraperwiki.rb
lib/scraperwiki/version.rb

Methods

close_sqlite   config=   convert_data   default_table_name   get_var   save   save_sqlite   save_var   scrape   select   sqlite_magic_connection   sqliteexecute  

Constants

VERSION = '3.0.2'

Public Instance methods

get_var(name, default=nil, _verbose=2)

Allows the user to retrieve a previously saved variable

Parameters

  • name = The variable name to fetch
  • default = The value to use if the variable name is not found
  • verbose = A verbosity level (not currently implemented, and there just to avoid breaking existing code)

Example

ScraperWiki.get_var(‘current’, 0)

save(*args)

legacy alias for save_sqlite method, so works with older scrapers

save_sqlite(unique_keys, data, table_name=nil,_verbose=0)

Saves the provided data into a local database for this scraper. Data is upserted into this table (inserted if it does not exist, updated if the unique keys say it does).

Parameters

  • unique_keys = A list of column names, that used together should be unique
  • data = A hash of the data where the Key is the column name, the Value the row 
             value. If sending lots of data this can be a array of hashes.
  • table_name = The name that the newly created table should use (default is ‘swdata’).
  • verbose = A verbosity level (not currently implemented, and there just to avoid breaking existing code)

Example

ScraperWiki::save([‘id’], {‘id’=>1})

save_var(name, value, _verbose=2)

Allows the user to save a single variable (at a time) to carry state across runs of the scraper.

Parameters

  • name = The variable name
  • value = The value of the variable
  • verbose = A verbosity level (not currently implemented, and there just to avoid breaking existing code)

Example

ScraperWiki.save_var(‘current’, 100)

scrape(url, params = nil, agent = nil)

The scrape method fetches the content from a webserver.

Parameters

  • url = The URL to fetch
  • params = The parameters to send with a POST request
  • _agent = A manually supplied useragent string

NB This method hasn‘t been refactored or tested, but could prob do with both

Example

ScraperWiki::scrape(‘scraperwiki.com’)

select(sqlquery, data=nil, _verbose=1) {|row_hash| ...}

Allows for a simplified select statement

Parameters

  • sqlquery = A valid select statement, without the select keyword
  • data = Bind variables provided for ? replacements in the query. See Sqlite3#execute for details
  • verbose = A verbosity level (not currently implemented, and there just to avoid breaking existing code)
  • [optionally] a block can be also be passed and the result rows will be passed

one-by-one to the black rather than loading and returning the whole result set

Returns

An array of hashes containing the returned data

Example

ScraperWiki.select(’* from swdata’)

sqlite_magic_connection()

Establish an SQLiteMagic::Connection (and remember it)

[Validate]