PDQ ATS research system
- Users new to R
- Mostly TAQ level II data
- Running in Amazon EC2
- Research focused on transaction costs
Support diverse exploratory research programs…
…involving all tickers over days to months or so…
…lightweight software footprint, easy to use
You know how this goes
- Some data, maybe a lot of data
- Maybe a database
- Complicated modeling and research; need R or Python to be productive
- Then, maybe throw even more analysis tools into the mix
- You find yourself adding more computing horsepower
- Before you know it, you end up with…
Eventually, some of those arrows connect to R
Moving data around data centers not as big of a deal as you might think
http://www.eecs.berkeley.edu/~ganesha/disk-irrelevant_hotos2011.pdf
Cache/fast storage reuse is usually more important.
Marshaling those data in and out of various formats is probably a bigger deal than you think!
Yuck!
Feather: a common data frame serialization format for R and Python and …
SciDB streaming API
(experimental)
Databases a great at managing mutable data
Append-only data are much simpler to work with
Feathercache: a fast object store interface for R (experimental)
Simple GET/PUT/DELETE-like interface
Elastic computing with R and Redis on Amazon EC2
doRedis update due to CRAN soon, for now use GitHub
Vignettes
Amazon EC2 Recipe
- Instantiate a Linux image
- Install R and desired packages, including doRedis
- Run the doRedis service installer script
- Make an AMI from this instance
- Re-launch from AMI as needed, supplying config as EC2 user data