Tag Archives: data visualizations

Package rinat use case: map of iNaturalist project

11 Mar

iNaturalist projects are collection of records posted on iNatualist. Now that we have a R package rinat from rOpenSci I thought of playing around with the data. Here is a function I wrote, to quickly map all the records of a project using ggmap package.

library(ggmap)
library(rinat)

inatmap <- function(grpid){
  data1=get_inat_obs_project(grpid, type = "observations")
  data1=data1[which(!is.na(data1$Latitude)),]
  map <-get_map(location =c(min(data1$Longitude),
                            min(data1$Latitude),
                            max(data1$Longitude),
                            max(data1$Latitude)),
                messaging = FALSE)
  p <-ggplot()
  p= ggmap(map)+geom_point(data=data1,
                           aes(x=as.numeric(Longitude),
                               y=as.numeric(Latitude)))
  p
}

We can used get_inat_obs_project function from rinat package to get all the observation from the specified project. get_map function form ggmap package to download google maps base layer and ggplot function form ggplot2 package to actually plot the map with points.

Now call to the function with a group name will produce a map with all the records in the project.

inatmap("birdindia")

inatmap_birdindia

We can use other ggplot options to add title, legend etc. to the map. This is just a simple example.

Temporal visualization of records of IndianMoths project using bdvis

13 Aug

I was looking for some data set which has some bias in terms of temporal data. I thought of checking out the data from iNaturalist project IndianMoths. This project is aimed at documenting moths from India. This project was initiated in July 2012 but really caught steam in January 2013, with members contributing regularly, minimum of 100 records per month. The reason that this project has not yet completed one year, I thought it might have some bias form the missed out months. Another reason for bias could be the fact that moths are not seen in the same numbers through out the year.

IndianMoths project on iNaturalist

IndianMoths project on iNaturalist


To explore this data, I first downloaded the data in a .csv file and loaded into R.

The data summary looked like this:

Total no of records = 2958
Bounding box of records Inf , Inf - -Inf , -Inf
Taxonomic summary...
No of Families : 0
No of Genus : 0
No of Species : 0

This tells us that the data is read by the package, but it has not understood the format well and we might have to do some transformations to get this going with our package. So let us use the function fixstr

to get the data into (somewhat) required format.

imoth=fixstr(imoth,Latitude="latitude",
                 Longitude="longitude",
                 DateCollected="observed_on")

Now let us check the summary again


 Total no of records = 2958
 Date range of the records from  0208-07-26  to  2013-08-07
 Bounding box of records  6.660428 , 72.8776559  -  32.5648529099 , 96.2124788761
 Taxonomic summary...
 No of Families :  0
 No of Genus :  0
 No of Species :  0

Now we have date and Latitude-Longitudes in a form that our package can understand. A quick glance at this data summary shows us that there is some problem with dates. In our data set we have one record form year 208 (which must be typo for year 2008). And the data is all form in and around India looking at the bounding box values of records.
We still need to get the taxonomy in place, but we will leave that for later time, and start working with this data. Let us create temporal plots of this data for different timescales of Daily, Weekly and Monthly.

tempolar(imoth,title="Daily Records")
tempolar(imoth,title="Weekly Records",timescale="w")
tempolar(imoth,title="Monthly Records",timescale="m")

would produce following three plots.

Indian Moths Daily Records

Indian Moths Daily Records

These are records per calender day and we see that 2-3 days in April have very high number of records compared to other dates. This could be due to some targeted survey during that time. This also shows us that we do not have much data records from September till April.

Indian Moths Weekly Records

Indian Moths Weekly Records

The weekly aggregation of same records highlights the fact that April month does have some spike in numbers, and otherwise the number of records seem to fairly uniform.

Indian Moths Monthly Records

Indian Moths Monthly Records

Monthly plot shows that April has recorded more than 800 records, where as no other month have more than 500 records in a month.

This could be due to several reasons, but mainly because of the activity of this particular project.

bdvis development version available for early feedback

31 Jul

Google Summer of Code 2013 is half way through. Mid term evaluations are underway. I thought this is a good logical point for us to share what we have been doing for Biodiversity Data Visualizations in R project and open up the package for testing and some early feedback. We have named the package bdvis. The package is on github, and I would appreciate if you could install and test it. Feedback may be given in the comments here, using issues on github  by twitter or email.

Getting data

The data was obtained from the Data portal of Global Biodiversity Information Facility. (http://data.gbif.org). The data set we are looking for is iNaturalist research grade records. We accessed the datasets page at http://data.gbif.org/datasets/ and selected the iNaturalist.org page from the alphabetic list which is at http://data.gbif.org/datasets/provider/407. Once on this page use link Explore: Occurrences and then from the next page click Download: Spreadsheet of results. On this page make sure  Comma separated values is selected and then press Download Now button. Website may take a few minutes to make your download ready. Once it is ready, the download link will be provided. Typically the name of the file will be occurrence-search-12345.zip The number of digits would be as many as 40.  Use the link to download the .zip file and then extract the data file occurrence-search-12345.csv in the working directory of R. Since this file has a long name, let us rename it to inat.csv for convenience.

Now we are ready to load our data.

inat = read.csv("inat.csv")
dim(inat)

If it shows something like

[1] 66581    47

we are on right track. Our data is loaded into R. For the time being, this package handles only GBIF provided data format, but getting user generated biodiversity data in this format using some built in functions is being worked out.

Package installation

Now let us install bdvis package. First we need to get devtools package which will let us install packages from github (rather than CRAN).

install.packages("devtools")
require(devtools)

install_github("bdvis", "vijaybarve")
require(bdvis)

if this produces something like

Loading required package: bdvis

Attaching package: ‘bdvis’

The following object(s) are masked from ‘package:base’:

summary

we are on right track. Our packages is installed and loaded into R.

Package functions

1. summery

Let us start playing with the functions now. We have the data loaded in inat data frame.

bdvis::summary(inat)

Should produce something like:

Total no of records = 66581
Date range of the records from  1710-02-26  to  2012-12-31
Bounding box of records  -77.89309 , -177.37895  -  78.53431 , 179.2615
Taxonomic summary...
No of Families :  1394
No of Genus :  5089
No of Species :  11299

What does this tell us about our data ?

  • We have 66581 records in the data set
  • The date range is from 1710 to 2012. (Really we have record form 1710? Looks we have a problem there.)
  • The bounding box is almost the whole world. Yes, this is global data set.
  • We have so many Families, Genus and Species represented in this data set.

I have two questions here:

  1. What more would you like to get in the summary?
  2. Should I rename the function summary to something else, so it does not clash with usual data frame summery function name?

2. mapgrid

Now let us generate a Heat map of the records in this data set. This map will show us the density of records in different parts of the world. To generate this map

mapgrid(inat,ptype="species")
mapgrid output for iNaturalist data

mapgrid output for iNaturalist data

ptype could be records if we need the map with raw records rather than aggregated to species. Again the questions:

  • What more options would you like to see here?
  • Ability to zoom in certain region?
  • Control over color pallet ?

3. tempolar

Now coming to Temporal visualizations, the function tempolar would make polar plots of temporal data into daily, weekly and monthly plots. The code and samples are as follows:

tempolar(inat,color="green",title="iNaturalist daily"
          ,plottype="r",timescale="d")
tempolar(inat,color="blue",title="iNaturalist weekly"
          ,plottype="p",timescale="w")
tempolar(inat,color="red",title="iNaturalist monthly"
          ,plottype="r",timescale="m")
Dailyly plot of Temporal data. Each line is records on each day of the year.

Dailyly plot of Temporal data. Each line is records on each day of the year.

Weekly plot of Temporal data. Plottype polygon is used here.

Weekly plot of Temporal data. Plottype polygon is used here.

Monthly plot of Temporal data. Each line is representing records in that month.

Monthly plot of Temporal data. Each line is representing records in that month.

Here options to control color, title, plottype and of course timescale are provided.

We are less than half way through our original proposal, and will continue to actively build this package. As I build more functionality, I will post more information on the blog. Till that time keep the feedback flowing telling us what more you would like to see in this package.

GSoC Proposal 2013: Biodiversity Visualizations using R

29 Apr

I am applying for Google Summer of Code 2013 with this “Biodiversity Visualizations using R” proposal. I am posting this idea to get feedback and suggestions from Biodiversity Informatics community.

[During next few days I will keep updating this to accommodate suggestions. The example visualizations here are crude examples of the ideas, and need lot of work to convert them into reusable functions.]

Backgrouond

R is increasingly being used in Biodiversity information analysis. There are several R packages like rgbif and rvertnet in rOpenSci suite to query, download and to some extent analyse the data within R workflow. We also have packages like dismo and SDMTools for modelling the data. It will be useful to have a package to quickly visualize biodiversity data. These visualizations would be helpful to understand extent of geographical, taxonomic and temporal coverage, gaps and biases in data.

The proposal is to work on a R package to provide functionality to quickly generate the visualizations of the data set user has gathered or generated.

The functions provided would be for following tasks:

  • Data preparation – The data needs to be converted into suitable format for visualizations and analysis i.e. date format, taxonomic classification and geographical co-ordinates should be in uniform and usable formats.
  • Data summary: Function(s) to quickly summarize the data set telling user number of records, number of records with Lat Long values, Bounding box of Lat Long Values, Date range and so on.
  • Geographic coverage – functions to visualize the data points on maps, density maps at different scales like Country level, Degree grid and so on.
Density of the records worldwide

Density of the records worldwide. Darker color indicates higher density of records.

Temporal coverage of the records

Temporal coverage of the records. Each line represents number of records on that particular day.

  • Taxonomic coverage – functions to visualize the taxonomic coverage of data in Tree Map formats by Number of records per species and number of species covered.
Familywise records

Family wise records present in the data set. (White block indicates records with unassigned family)

  • Completeness analysis – functions to assess and visualize completeness of biodiversity inventory of the region or in other words a measure of how exhaustive is the sampling in the study area [Ref:http://dx.doi.org/10.1111/j.0906-7590.2007.04627.x ]

Mentor(s): Javier Otegui

Data set: The data set used for the sample visualizations here is records published by iNaturalist.org on GBIF data portal. This data set contains Research Grade records (~46K) for all the organisms posted. The details of the data set are available here. The description on GBIF dat postal says “iNaturalist.org is a website where anyone can record their observations from nature. Members record observations for numerous reasons, including participation in citizen science projects, class projects, and personal fulfillment.”

References:

  • Chamberlain, S., & Barve, V. (2012). rvertnet: Search VertNet database from R. Retrieved from http://cran.r-project.org/package=rvertnet
  • Chamberlain, S., Boettiger, C., Ram, K., & Barve, V. (2013). rgbif: Interface to the Global Biodiversity Information Facility API methods. Retrieved from http://cran.r-project.org/package=rgbif
  • Hijmans, R. J., Phillips, S., Leathwick, J., & Elith, J. (2012). dismo: Species distribution modeling. Retrieved from http://cran.r-project.org/package=dismo
  • Otegui, J., & Ariño, A. H. (2012). BIDDSAT: visualizing the content of biodiversity data publishers in the Global Biodiversity Information Facility network. Bioinformatics (Oxford, England), 28(16), 2207–8. doi:10.1093/bioinformatics/bts359
  • Soberón, J., Jiménez, R., Golubov, J., & Koleff, P. (2007). Assessing completeness of biodiversity databases at different spatial scales. Ecography, 30(1), 152–160. doi:10.1111/j.2006.0906-7590.04627.x
  • VanDerWal, J., Falconi, L., Januchowski, S., Shoo, L., & Storlie, C. (2012). SDMTools: Species Distribution Modelling Tools: Tools for processing data associated with species distribution modelling exercises. Retrieved from http://cran.r-project.org/package=SDMTools