Posts tagged High
[Case Study] Lessons in High Performance Computing with Open Source
Feb 2nd
Providing adequate software and tools for researchers has always been of great importance to organizations, but has often come at a great cost. In an era of constantly evolving technology and rapidly dwindling budgets, my IT team has had to work with a large pool of researchers to provide cost-effective solutions that meet the ever-growing demand for innovation and computing power.
I am an Information Technologist for the Department of Statistics and Probability at Michigan State University. The Department is home to award-winning faculty with a wide variety of expertise in fundamental and interdisciplinary research, and over 100 graduate students from all over the world. Keeping the faculty and students ahead of their research is a constantly evolving challenge for my team and I.
Sponsor
Evolution of Statistical Software
Erik Segur is an Information Technologist for the Department of Statistics and Probability at Michigan State University.
For many years, most statistical analysis in our department was done in Matlab, S-Plus, SPSS or SAS. Even with a Higher Educational discount, most of the software required yearly renewal fees that quickly devoured our IT budget. Things started to change when the R language, which was first developed in 1993, began to gain traction in statistics communities in the early 2000s. R is an open source programming language and software environment that is used for statistical computing and data analysis. Several years ago, we began the transition at Michigan State to R; today, it is used for the majority of the research in the department–as well as being a central focus of our statistics curriculum. By switching to the free, open source version of R, our department has been able to cut thousands of dollars each year in software costs and have focused more on fueling and expanding research.
Lesson #1: The Shortcomings of Open Source
As more people began to use R and the analysis became increasingly complex, researchers began to face a large problem: time. Research was taking several months to complete in terms of processing jobs. Often, there is a need to run the calculations several times to ensure accuracy; waiting three months for one to complete was simply not feasible. It was taking R this long to process the jobs because the iterations were computed in serial, one right after another, using only one processor core at a time.
Bo Cowgill from Google once said “The best thing about R is that it was developed by statisticians. The worst thing about R …is that it was developed by statisticians.”
Until the spring of 2010, R was a 32-bit application and could only access a limited amount of memory. The maximum amount of memory that could be accessed by R was only 3GB. When dealing with large datasets researchers were quickly running out of memory as well as discovering they needed a solution to deal with large data efficiently.
Bo Cowgill from Google once said “The best thing about R is that it was developed by statisticians. The worst thing about R …is that it was developed by statisticians.” Even though R was–and still is–constantly evolving, the department needed a solution that could keep up with hardware technology and compute calculations in an efficient, scalable manner.
Lesson #2: Find Commercial Enhancements for Open Source
Our search for a more effective version of R ultimately brought us to a product called Revolution R Enterprise by Revolution Analytics, which provides commercial support and software for open source R. It takes advantage of multiple processor cores by using optimized assembly code and efficient multi-threaded algorithms that use all of the processor cores simultaneously. Although this addressed a lot of the issues of open source R, professors were only using Revolution R on their desktops. The next question was, how we could combine the power of our servers to dramatically decrease our computation times?
Lesson #3: Expanding to Infinity and Beyond
Open Source R is a memory-bound language. This means that all of the data, matrices, lists etc. need to be stored in memory. Issues quickly arose when data sets became several gigabytes large and were too big to fit into memory. This required implementing parallel external memory algorithms and data structures to handle the data. These challenges were tackled by Revolution Analytics as they developed the R language for a High Performance Computing (HPC) environment.
There are often great pieces of software created through open source, but they generally lack key features needed for an enterprise environment. Combined with commercial backing and expertise, these projects can be further developed and expanded to meet the needs of large-scale enterprise environments.
In 2010, Revolution Analytics offered Revolution R Enterprise free for academic users and shifted the focus of their enterprise software to big data, large scale multiprocessor computing and multi-core functionality. Revolution Analytics was going to tackle everything the department needed. The evolution was complete: open source R went from an inefficient single core program to a HPC environment.
Once the department could schedule R jobs in an HPC environment, the demand began to drastically increase. The HPC cluster is now scheduling more than four times the amount of jobs that were scheduled in previous semesters, from 200 jobs over a year ago to over 800 jobs this past semester. Jobs that were taking over three months to complete on open source R were completed in less than a few days with Revolution R. Computational jobs are now run multiple times with significantly higher levels of accuracy than ever before.
Conclusion
There are often great pieces of software created through open source, but they generally lack key features needed for an enterprise environment. Combined with commercial backing and expertise, these projects can be further developed and expanded to meet the needs of large-scale enterprise environments. IT departments can provide enhanced solutions to their users that adapt to the expanding world of cloud and High Performance Computing environments–all while minimizing the impact on a shrinking budget.
Photo courtesy of Shutterstock.
View full post on ReadWriteWeb
“Dream High 2″ Releases Last Official Poster of its Six Leads – Soompi
Jan 15th
|
"Dream High 2" Releases Last Official Poster of its Six Leads
Soompi This last poster also features the six leads of the drama – Park Seo Joon, JB, T-ara's Park Ji Yeon, Kang So Ra, 2AM's Jung Jin Woon , and SISTAR's Hyorin – displaying their character's personalities through their expressions. Park Seo Joon, JB, … |
View full post on SEO – Google News
SEO & On-Page Optimization: Aim Wide, Not High, Study Says – EContent (press release)
Jan 5th
|
SEO & On-Page Optimization: Aim Wide, Not High, Study Says
EContent (press release) Conductor, the SEO technology provider, has come along with some solid new research demonstrating that you can get as much bang for your SEO buck without nabbing top billing on the highest-volume search terms. A study by Conductor's Senior Research … |
View full post on SEO – Google News
The High Price of Add-on Online Fees
Dec 27th
New rules will go into effect in a month for US airline advertising that take the emphasis off the asterisk and adds transparency to their add-on ticketing fees. Ironically, some of the low-fare airlines such as Spirit are fighting the changes, claiming freedom of speech infringement by the government. I guess the right to deceive their customers should be part of the Constitution, or at least left to free enterprise to sort this all out.
The rules were supposed to go in effect earlier this fall, but were extended to January 24th to give the airlines time to legally outmaneuver them. I mean, to comply.
Sponsor
It used to be that one went online to avoid add-on fees by travel agents, but as the airlines have made visits to bankruptcy courts more frequent (American is the most recent), they have come up with a variety of ways to squeeze more money out of the public without being so, well, obvious and clear. There are baggage fees, fees for using particular forms of payment, and fees for sitting in those extra-spacious exit row seats. Buying a ticket online is now akin to buying a computer from Dell: there are so many screens to click on to avoid various up-sells (rental cars? hotels? a new printer cartridge, maybe?), that the process is a user experience nightmare.
But the fee asterisk and its associated add-ons are onerous. Spirit, for example, charges all sorts of fees depending on the ticket. They claim the online booking fees are “optional” since you can avoid them by going to the airport and buying your tickets there. Right. As if we don’t spend enough time waiting in airport lines. You can read their fine print here and it will take you a while. Here is a screenshot showing you that a $164 round-trip ticket would cost you an additional $62 in fees.
Granted, they aren’t alone in charging high fees for online purchases. Many of these are local and other taxes that the hospitality industry has to bear. But the objection is how the consumer is informed of them during the online purchase process. Spirit doesn’t divulge all the fees until you have selected your flights and are getting ready to pay them, which is understandable given the many nuances of its complex fee structure. But that is still reprehensible, from an airline that advertises $9 fares that turn into something quite a bit more.
As another example, two tickets for Book of the Mormon, the popular Broadway show, can have almost as much in add-ons as Spirit does. The site Broadway.com is at least a bit more upfront about what they charge you, as you can see here at right.
Easyjet, the European low-fare airline, has different ticketing prices depending on the way you pay. “All bookings will incur a £8.00 booking fee except for bookings made by Visa electron (a debit card that isn’t available in the US and Canada) which is free.” Then you are presented at checkout with this somewhat confusing screen at left.
So the biggest number on the page is the fee that you would pay with a regular credit card, that includes the surcharge. Got it? This is a user experience nightmare.
Concert tickets for years have had high add-on fees, but at least the online concert ticket booking sites are more up-front about them. Still, two tickets for an upcoming show of George Strait for example that started out costing $80 a piece add $13 for various fees at Ticketmaster, more if you want them mailed quickly to you.
That is still less of a proportion than what the airlines add on.
Some thoughts for the future to improve the experience:
- Limit the upsells. I shouldn’t have to click through more screens to decline offers at checkout than it took to buy the original item.
- Disclose the fees at the beginning rather than at the end of the purchase process.
- Simplify, as Thoreau once said. If you need to explain your fees with an asterisk, it is time to get rid of both.
- Keep them below 10% of the total price. Otherwise it isn’t really a fee, it is just generating more revenue.
- Make it easy to find the fees. At least Spirit posts them on their site in plain sight.
View full post on ReadWriteWeb
Search In Pics: Google Keg, Panda Bread & Lego High Heels
Dec 16th
In this week’s Search In Pictures, here are the latest images culled from the web, showing what people eat at the search engine companies, how they play, who they meet, where they speak, what toys they have, and more. Panda Bread: Android Balloon Art: Free Chromebook Carts: Google-like Lego…
Please visit Search Engine Land for the full article.
View full post on Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing
How Brands Can Fly High on the New Twitter
Dec 15th
Twitter’s much-anticipated redesign launched last week, causing quite a bit of industry buzz. From new brand pages to embeddable tweets, the recent changes to the social networking site are major.
The new version enhances the way we use Tw…
View full post on Search Engine Watch – Latest
Facebook 101: New High School and College Social Media Classes Help Train Next … – San Francisco Chronicle (press release)
Dec 15th
|
Facebook 101: New High School and College Social Media Classes Help Train Next …
San Francisco Chronicle (press release) SEO.com is hiring some of these graduates as growth rates in social media marketing are expected to increase about 33 percent each year, from $400 million in 2010 to $2.3 billion in 2015, according to a forecast by BIA/Kelsey. … |
View full post on SEO – Google News
Search Retargeting: Delivering Bargains For Search Marketers In High CPC Verticals
Dec 14th
Coupons, Rebates, Daily Deals…it seems everyone is looking for a bargain. But for search marketers, bargains in paid search can be hard to find. CPC prices continue to rise, making it harder and harder for them to expand search campaigns and achieve their goals. However, search retargeting can…
Please visit Search Engine Land for the full article.
View full post on Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing
New SEO Tool by Optimum7 Makes It Easy to Manage High Volume Content – DigitalJournal.com (press release)
Nov 18th
|
New SEO Tool by Optimum7 Makes It Easy to Manage High Volume Content
DigitalJournal.com (press release) SmarPost is fully integrated as it monitors the content process step by step, preventing the SEO Specialist from skipping a crucial step. For example, the article will not be allowed to syndicate until it has been published on the clients' website, … |
View full post on SEO – Google News
How To Use Rich Snippets, Structured Markup For High Powered SEO
Nov 7th
I recently received a question from a reader asking about the way search engines were using microformats and other forms of structured markup. Today, I am going to address that topic from the perspective of its impact on SEO. What is particularly interesting about this topic is that structured…
Please visit Search Engine Land for the full article.
View full post on Search Engine Land: News & Info About SEO, PPC, SEM, Search Engines & Search Marketing
