Posts tagged Open
[Case Study] Lessons in High Performance Computing with Open Source
Feb 2nd
Providing adequate software and tools for researchers has always been of great importance to organizations, but has often come at a great cost. In an era of constantly evolving technology and rapidly dwindling budgets, my IT team has had to work with a large pool of researchers to provide cost-effective solutions that meet the ever-growing demand for innovation and computing power.
I am an Information Technologist for the Department of Statistics and Probability at Michigan State University. The Department is home to award-winning faculty with a wide variety of expertise in fundamental and interdisciplinary research, and over 100 graduate students from all over the world. Keeping the faculty and students ahead of their research is a constantly evolving challenge for my team and I.
Sponsor
Evolution of Statistical Software
Erik Segur is an Information Technologist for the Department of Statistics and Probability at Michigan State University.
For many years, most statistical analysis in our department was done in Matlab, S-Plus, SPSS or SAS. Even with a Higher Educational discount, most of the software required yearly renewal fees that quickly devoured our IT budget. Things started to change when the R language, which was first developed in 1993, began to gain traction in statistics communities in the early 2000s. R is an open source programming language and software environment that is used for statistical computing and data analysis. Several years ago, we began the transition at Michigan State to R; today, it is used for the majority of the research in the department–as well as being a central focus of our statistics curriculum. By switching to the free, open source version of R, our department has been able to cut thousands of dollars each year in software costs and have focused more on fueling and expanding research.
Lesson #1: The Shortcomings of Open Source
As more people began to use R and the analysis became increasingly complex, researchers began to face a large problem: time. Research was taking several months to complete in terms of processing jobs. Often, there is a need to run the calculations several times to ensure accuracy; waiting three months for one to complete was simply not feasible. It was taking R this long to process the jobs because the iterations were computed in serial, one right after another, using only one processor core at a time.
Bo Cowgill from Google once said “The best thing about R is that it was developed by statisticians. The worst thing about R …is that it was developed by statisticians.”
Until the spring of 2010, R was a 32-bit application and could only access a limited amount of memory. The maximum amount of memory that could be accessed by R was only 3GB. When dealing with large datasets researchers were quickly running out of memory as well as discovering they needed a solution to deal with large data efficiently.
Bo Cowgill from Google once said “The best thing about R is that it was developed by statisticians. The worst thing about R …is that it was developed by statisticians.” Even though R was–and still is–constantly evolving, the department needed a solution that could keep up with hardware technology and compute calculations in an efficient, scalable manner.
Lesson #2: Find Commercial Enhancements for Open Source
Our search for a more effective version of R ultimately brought us to a product called Revolution R Enterprise by Revolution Analytics, which provides commercial support and software for open source R. It takes advantage of multiple processor cores by using optimized assembly code and efficient multi-threaded algorithms that use all of the processor cores simultaneously. Although this addressed a lot of the issues of open source R, professors were only using Revolution R on their desktops. The next question was, how we could combine the power of our servers to dramatically decrease our computation times?
Lesson #3: Expanding to Infinity and Beyond
Open Source R is a memory-bound language. This means that all of the data, matrices, lists etc. need to be stored in memory. Issues quickly arose when data sets became several gigabytes large and were too big to fit into memory. This required implementing parallel external memory algorithms and data structures to handle the data. These challenges were tackled by Revolution Analytics as they developed the R language for a High Performance Computing (HPC) environment.
There are often great pieces of software created through open source, but they generally lack key features needed for an enterprise environment. Combined with commercial backing and expertise, these projects can be further developed and expanded to meet the needs of large-scale enterprise environments.
In 2010, Revolution Analytics offered Revolution R Enterprise free for academic users and shifted the focus of their enterprise software to big data, large scale multiprocessor computing and multi-core functionality. Revolution Analytics was going to tackle everything the department needed. The evolution was complete: open source R went from an inefficient single core program to a HPC environment.
Once the department could schedule R jobs in an HPC environment, the demand began to drastically increase. The HPC cluster is now scheduling more than four times the amount of jobs that were scheduled in previous semesters, from 200 jobs over a year ago to over 800 jobs this past semester. Jobs that were taking over three months to complete on open source R were completed in less than a few days with Revolution R. Computational jobs are now run multiple times with significantly higher levels of accuracy than ever before.
Conclusion
There are often great pieces of software created through open source, but they generally lack key features needed for an enterprise environment. Combined with commercial backing and expertise, these projects can be further developed and expanded to meet the needs of large-scale enterprise environments. IT departments can provide enhanced solutions to their users that adapt to the expanding world of cloud and High Performance Computing environments–all while minimizing the impact on a shrinking budget.
Photo courtesy of Shutterstock.
View full post on ReadWriteWeb
[Poll] Does An Open Source webOS Have A Legitimate Future?
Jan 27th
This week, Hewlett-Packard announced the open source roadmap for webOS along with the next edition of its application framework, Enyo 2.0. As we wrote yesterday, the time for webOS to shine may lie ahead. What it comes down to is how well the open source community responds to webOS and whether or not the original equipment manufacturers will ever decide to build webOS devices.
The favorable response of the community and OEMs is not guaranteed. Many think webOS is as dead an operating system as Aramaic is a language. That may include former Palm CEO Jon Rubinstein who is leaving HP after his commitment to the company elapsed. Is there still potential for webOS and Enyo or have we seen the last of the once-promising mobile operating system? That is the topic of this week’s ReadWriteMobile poll.
Sponsor
There may or may not be a future for webOS. The timeline stretches to September this year and is licensed under the Apache 2.0 open source license. HP has said that developers are free to suggest new aspects of the project and bounce them off the experts in the in the Enyo Forum. The company believes it is more likely that proposals concerning the outer branches of webOS will be undertaken than anything touching the core of the source code and kernel.
The biggest gain that open sourcing webOS may garner could have less to do with webOS itself than with Enyo. The application framework is fundamentally Web-based. In mobile terms that means it will rely heavily on HTML5 and CSS and work through WebKit and Direct Canvas. While there are other HTML5 frameworks developers can use to create mobile Web apps, such as those provided by appMobi and Sencha Touch, one of the biggest desires of mobile HTML5 developers has been a consistent, easy-to-use framework. Enyo might be the option that developers have been looking for.
For the OEMs, there may be an advantage in contributing to the webOS open source project. These are turbulent days for many OEMs. HTC was one of the companies that helped make Android popular, but it has seen its growth stall with the dominance of Samsung in the ecosystem. Motorola, which reported a loss for the 2011, is stuck to Android through its potential acquisition by Google. Samsung has shown a willingness to adopt any mobile platform that it thinks it can create future growth. Secondary OEMs such as LG and Huawei could hedge bets against a reliance on Android with webOS.
Will anybody adopt it? Or are the dissembled parts of webOS, like the standard Linux kernel or the application ecosystem that could be created through Enyo, more valuable? Take the poll below and let us know your thoughts in the comments.
View full post on ReadWriteWeb
On Magic & SEO: Open Letter to Forbes, Bill Barol
Jan 25th
SEW’s reply to a blog post on Forbes.com entitled, “Sex, Free iPad, Naked Pictures of Cat Deeley, 8 Ways to Get Rich Right This Second and Lose Weight While You Sleep” which urged demystification in SEO but instead reproduced endemic superstitions.
View full post on Search Engine Watch – Latest
New Open Group Cloud Standard Introduces “XaaS” – Something as a Service
Jan 19th
As prominent as cloud computing has already become in today’s enterprises, it’s amazing to realize that the world’s reference standards are only now catching up with the concept. On Tuesday, the consortium of industry stakeholders known as The Open Group updated its reference standards for Service-Oriented Architecture. You remember SOA, don’t you?
Well, if you’ve been following along with the SOA story, you know that cloud computing platforms have catapulted the service concept onto a huge and growing platform. Now, the consortium – led by software giants IBM, Oracle, and SAP, along with HP, and business consultancy CapGemini – has produced a formal interpretation of the role services play in the cloud, by offering a new term for the concept. Say it with me (if you can): XaaS.
Sponsor
If a component delivers a service over a network using a service-oriented infrastructure, the Open Group now explains, in whatever form that takes, the concept will be referred to as XaaS. Literally, the X stands for… anything.
“This is the essence of cloud computing,” reads the Open Group’s new Service-Oriented Cloud Computing Infrastructure (SOCCI) framework. “It refers to an increasing number of services that are delivered over a network. Anything as a service requires an understanding of the service objectives and the accounting of service use and quality. The objectives, use, and quality can be determined from the underlying reference model for SOI: Broad network access (cloud) + resource pooling (cloud) + business-driven infrastructure on-demand (SOI) + service-orientation (SOI) = XaaS.”
SOCCI is a necessary adaptation to the OG’s existing SOI concept, mainly because certain aspects of cloud services had become incongruous with the formal framework for SOI even up until last week. The expectation for SOI was built around software contained within the fixed space of server hardware (note: no virtualization) in an enterprise data center, or perhaps (begrudgingly) through cohosting services. Resources were provisioned directly and manually by administrators, and financing was often expected to be handled as capital expenditures.
The new SOCCI framework embraces the modern understanding of cloud services as spelled out by the National Institute of Standards and Technology. There are three principal divisions – SaaS, IaaS, and PaaS – and usually anything else that vendors may attach to an “-aaS” is arbitrary and often self-serving. Open Group leaves the door open for something else to fit there later, but makes clear that these three pillars are the only ones that need to hold up the cloud for now.
Previously, the SOI framework helped organizations to understand how to design, using architecture, the hardware foundations for their services. With cloud computing, much of that architectural process is rendered moot. You provision the basic characteristics of the virtual systems you need to deliver services. And if they don’t work well or properly, you change those characteristics. SOCCI has adopted this concept now, and is advancing it up until the time it needs to be completely redefined all over again.
Quoting from the newly revised framework:
Cloud computing puts new demands on the IT infrastructure and management thereof. It requires an abstract approach to the operational environment. A cloud computing provider cannot any longer tailor its environment for each subscriber. It means that instead of a physical device, cloud computing offers an abstraction of a server, file system, storage, network, database, etc. Moreover, increasing providers’ profitability and maximizing the utilization of resources requires multi-tenancy, dynamic allocation of resources, and metering with charge-back.
At the same time, subscribers expect to see implementation of a utility model since they want to allocate resources on-demand and pay exactly for their usage while being able to sustain their operations, much like the electric bill. Hence, new infrastructure should be agile and elastic and create an illusion of infinite computing resources available on-demand. While SOI did not offer the whole spectrum of the characteristics desired, it became an enabler for what came to be known as Service-Oriented Cloud Computing Infrastructure (SOCCI). SOCCI can be defined as service-oriented, utility-based, manageable, scalable on-demand infrastructure that supports essential cloud characteristics, service, and deployment models. In other words, SOCCI describes the essentials for implementing and managing an Infrastructure as a Service (IaaS) environment.
View full post on ReadWriteWeb
Troubling Google Contractor Allegedly Caught Vandalizing Open Street Map
Jan 16th
The official blog of Open Street Map reports tonight that someone at a range of Google IP addresses in India has been editing the collaboratively made map of the world in some very unhelpful ways, like moving and deleting information and reversing the direction of one-way streets on the map.
The IP addresses match the same ones that were caught last week running a long-term scam wherein telephone directory listings were scraped from a crowd-sourced phone directory in Kenya called Mocality. A Google contractor then systematically called those phone numbers claiming to have a paid placement deal jointly offered by the Kenyan company and Google! A Google spokesperson told BoingBoing on Friday that the company was “mortified” by the discovery – but now it appears the same Google contractor may be behind mayhem rippling throughout one of the world’s biggest maps. Google says it’s investigating these latest allegations.
Sponsor
Open Street Map said tonight that two user accounts have been found vandalizing streets in New York, London and elsewhere since at least last Thursday. Full investigation of the actions may take time, OSM said, because at least 17 user accounts have accessed OSM from those Google IP adresses more than 100,000 times over the past year.
Is this a Google contractor with something against crowdsourced projects? That’s one thing both targets have in common. Neither offense seems short-lived or trivial though, either.
Open Street Map is like Wikipedia for world maps. It’s a fabulous and inspiring project, I think, but not everyone agrees.
In August 2010, Open Street Map co-founder Steve Coast wrote a long blog post titled Enough is Enough: Disinfecting OSM from Poisonous People. That post has been read by almost 175,000 people.
Coast said that divisive conversations “have spilled over now from poisonous people merely making life difficult on the mailing list, to paralyzing the project and even systematically corrupting the data we serve out using bots…Many (if not most or all) of the key people in OSM are feeling drained, distracted and upset. Some are talking of hiatus or resign. These are the key people who write code, build things, maintain things and run our working groups.”
Three months after writing that post, Coast left the company that supports Open Street Map and became the Principal Architect at Bing Maps.
Coast was one of three signers of tonight’s blog post that concludes as follows:
“These actions are somewhat baffling given our past good relationship with Google which has included donations and Summer of Code work. As a community we take the quality of our data extremely seriously and look forward to an explanation from Google and an undertaking to not allow this kind of thing to happen in the future.”
In response to our request for comment, a Google spokesperson said tonight, “We’re aware of OpenStreetMap’s claims that vandalism of OSM is occurring from accounts originating at a Google IP address. We are investigating the matter and will have more information as soon as possible.”
It will be interesting to see how the company responds to this, the second serious allegation of wrongdoing by one of its contracting companies inside of a week.
It would be nice if Open Street Map could continue to flourish and grow. Bad actors may be an inevitable issue for an open site building collective knowledge at scale, but it would be good if people sitting in Google offices around the world were all helping instead of hurting such efforts. Reversing the direction on one-way streets is a particularly nasty thing to do.
View full post on ReadWriteWeb
SEO Imperial Beach: SEO Services in Imperial Beach from Head4 Marketing – The Open Press (press release)
Jan 10th
|
SEO Imperial Beach: SEO Services in Imperial Beach from Head4 Marketing
The Open Press (press release) Imperial Beach, CA (OPENPRESS) January 10, 2012 – SEO Imperial Beach: Looking for the best results driven SEO Services in Imperial Beach, CA? The top Search Engine Optimization (SEO) Marketing Services in Imperial Beach are provided by Head4 Marketing, … |
View full post on SEO – Google News
Want To Revive The Economy? Open Public Buildings To Remote Workers
Jan 5th
The City of Palo Alto, Calif. and mobile workspace-finding app LiquidSpace have teamed up for an exciting step in public co-working. The Palo Alto City Library will make rooms available on LiquidSpace in a 3-month pilot. This is the first instance I can find in the U.S. of a public facility using a location-aware mobile app to reduce its unused capacity.
Co-working is the new normal, and city governments could drive lots of high-tech productivity if they make their latent space available to flexible, remote workers. Palo Alto is an obvious place to start, but every city in the world should start thinking like this.
Sponsor
The partnership began this week. Two of the library’s study rooms, with room for around 10 people, are available on LiquidSpace, a free iPhone app that lists available workspaces in business centers, hotels, offices or co-working spaces. During the pilot, the library will assess whether it will benefit the public, as well as the library itself. If so, it will work with other city departments to expand the program to other facilities.
We like the looks of this. Coffee & Power, another remote working story we’re watching, has figured out how to make a win-win out of helping private spaces open up to remote workers. LiquidSpace is doing a civic service by pushing municipal governments to open their doors to co-workers, too.
If you want to learn more about this topic, Phil Shapiro wrote a cool post about co-working in public libraries in PCWorld.
Do you work remotely? Share your experiences in the comments.
View full post on ReadWriteWeb
IBM Promises to Keep Green Hat’s Platform Support Open, Broad-based
Jan 5th
For several years, a company called Green Hat (not associated with Red Hat) has been in the business of creating sophisticated software testing equipment for developers, particularly for service-oriented applications that use messaging queues. The problem with distributed application testing is that it’s getting more and more complicated, especially as a multitude of new and independently evolving frameworks introduce dependencies that can’t always be accurately simulated in a test environment.
So yesterday’s acquisition of Green Hat by IBM brought up an interesting question: Will a company whose test environments were developed to support Oracle, Java Message Service, SAP, Software AG, and TIBCO as well as WebSphere MQ continue to do so after being acquired by the maker of WebSphere MQ? Today, we have the answer.
Sponsor
In a statement to ReadWriteWeb, the director of product management and strategy for IBM’s Rational division – whose umbrella will now cover Green Hat’s tools as well – said IBM plans to keep Green Hat’s support broadly-based.
“We have every intention of maintaining and extending the existing Green Hat support for multiple applications, protocols, formats, and the like,” Charles Chu tells RWW. “A primary value of Green Hat is its ability to help manage the challenge of application testing in a complex world. In this sense, expanding the support for ever [greater] levels of complexity is part of our core mission.”
The way organizations often build distributed applications is by delegating responsibility for the individual components to separate teams. While ideally it would be nice for all of those teams to cooperate on scheduling, in practice components end up with at varying stages of completion for any one point in time. For this reason, when one development team needs to test the viability of a component intended to communicate with other components that may not even exist yet, the team utilizes a testing environment that can create stubs – substitute components that can interact with the ones being tested in a realistic manner.
The crown jewel of Green Hat’s test automation suite has been GH Tester. Historically, it’s been used to rapidly generate stubs that can respond to Web services using messaging protocols like SOAP or JMS. Because Web services must behave in a more protocol-agnostic fashion, testing suites like GH Tester must be more open to multiple platforms – which is why the support question for Green Hat and platforms like .NET is so important.
More recently, though, Green Hat has been working toward weaning GH Tester from the use of stubs, and toward a more sophisticated system of virtualized applications it calls GH Virtual Integration Environment (VIE). Rather than a stub, VIE generates a component that truly is an application, responding more logically and believably to a component’s request using logic that’s adaptable to a variety of test scenarios.
As Green Hat explains, “GH VIE is part of the GH Tester suite, not another ‘product,’ so when GH Tester starts a test, it can automatically start the necessary virtualized applications to go with it, ensuring that unresolved system dependencies can be satisfied. The user is in control of which virtualized applications are used, allowing a wide variety of different situations to be modeled, depending on the testing scenario.”
IBM is now calling the VIE scenario “testing in the cloud.” At the moment, Green Hat may not technically be “as-a-service,” though it’s conceivable that as its portfolio is integrated into IBM Rational, VIE functionality could be offered at some point in the future through a cloud portal.
View full post on ReadWriteWeb
ISO 9001 Certification Gives SEO Copywriting Firm, Fountain Partnership, A … – The Open Press (press release)
Jan 4th
|
ISO 9001 Certification Gives SEO Copywriting Firm, Fountain Partnership, A …
The Open Press (press release) Norwich (OPENPRESS) January 4, 2012 – Fountain Partnership, a SEO Copywriting and digital agency, has become the first online marketing agency to become ISO 9001 certified in the East of England. By achieving the ISO 9001 standard, they are committing … |
View full post on SEO – Google News
SEOPartner Goes To Texas – SEO Dallas Now Open For Business – DigitalJournal.com (press release)
Jan 3rd
|
SEOPartner Goes To Texas – SEO Dallas Now Open For Business
DigitalJournal.com (press release) SEO Partner chief James Schramko further extends his company's area of operation, targeting major US cities with geo-specific search engine optimization. The latest expansion takes the Sydney-based SEO provider to Dallas, Texas. The SEO Dallas service … |
View full post on SEO – Google News
