Posted:
Cross-posted on the Google for Work Blog.

At Google Cloud Platform, we strive to deliver innovative cloud computing technology to developers at low costs with massive scale. Currently, our cloud offerings are powered by best-in-class data centers, virtualization software and container-based applications, but as we consider the future of cloud computing, we realize there’s a major opportunity ahead that would allow us to deliver an unprecedented improvement in the power, efficiency and reliability of our infrastructure. We can elevate cloud computing like never before.

We’re answering the question that’s been in front of us the whole time: why isn’t cloud computing built in actual clouds? Well, as of today, it is. We’re excited to announce Google Actual Cloud Platform: all of our incredible services and products running in actual clouds in the troposphere.

Starting on April 1, Google Actual Cloud Platform brings with it a number of exciting new features:

  • New compute zone: We’ve added a new compute zone, troposphere-1a, to make it easier than ever to provide your app with non-earth-bound availability. Now, you no longer have to make a choice between high availability and high altitude.
  • New machine types: Alongside our current machine types, we’ve added a new category of “physical” devices: actual-cloud machine types. Choose from cumulus-16gb, cirrus-32gb and stratocumuliform-64g (created specifically for data intensive workloads).
  • Stormboost: Drawing on charges from electrical fields during thunderstorms, we’re able to supercharge read/write performance on all persistent disks and offer 50% higher IOPS.
  • CloudDrops: A new, game-changing content distribution system. CloudDrops can provide blazingly fast content delivery to all of your users using—you guessed it—rain drops.
  • Weather Dashboards in the Developer’s Console: With new weather-dependent performance features, you need a way to monitor the atmospheric conditions of your servers. Now, you can monitor humidity/request, watch your app’s altitude and see a 7-day forecast right next to the rest of your stats.
  • Bare-metal container support: Applications deployed on Actual Cloud Platform can run in containers too. The lightweight shared kernel model of containers makes them ideal for non-terrestrial deployments.

From all of us on the Google Cloud Platform team, here’s to clear skies ahead.

-Posted by Greg Demichillie, Director of Product Management

Posted:
A few weeks ago we announced Perfkit to make it easy for you to benchmark popular workloads on the cloud. As we mentioned, it’s a living benchmark, and we are evolving it to include a new tool to measure the impact on latency when you grow the number of servers that power your application.

We call the new performance benchmark Online Data Intensive Simulator, or OLDISIM, written in collaboration with the Multiscale Architecture and Systems Team (MAST) at Stanford. It models the distributed, fan-out nature of many modern applications with tight tail latency requirements, such as Google Search and some NoSQL database applications.

We use OLDSIM internally to measure the impact of both hardware and software improvements on our scale out workloads and analyze their scaling efficiency. Scale out efficiency allows us to meet new user demand by adding the fewest number of servers possible while maintaining great user experience. The fewer servers we add, the more energy efficient we are, and the cheaper the solution is. Predicting how a service will scale out is usually very hard under laboratory conditions, but experiments show that OLDISIM results strongly correlate with our current Google Search performance in scaling efficiency, as the chart below demonstrates.
Our needs within Google are similar in many ways to other scale out Internet workloads, and we're making a version of OLDISIM available to the open source community through PerfKit Benchmarker. We shared it using the Apache V2 license. With OLDISIM, you can more easily model and simulate most applications with a fan-out/synthesis model, including Hadoop and several NoSQL products. You can specify which workload you plug in to each leaf node, and measure the scaling efficiency and tail latency of your applications.
You can run OLDISIM by itself by following the instructions on GitHub, or use PerfKit Benchmarker to run it on many of the most popular cloud providers. The command line is as simple as “pkb.py --benchmarks=oldisim”.

Both OLDISIM and PerfKit Benchmarker teams get your feedback through GitHub. We’d love to hear what you think, so please send us your suggestions and issue reports.

Happy Benchmarking!

Posted by Ivan Santa Maria Filho on behalf of the Cloud and
Platforms Performance Teams

Posted:
Imagine being away from your desk and receiving automatic alerts when an issue occurs in your Google App Engine app. Or waiting at the airport and stopping your test VMs before leaving for vacation. With the beta launch of Cloud Console for Android, managing Google Cloud Platform from your phone or tablet is possible (and yes, an iOS version is in the works).

With just a few taps you can quickly glance at the status of your solution, set up alerts, manage your Cloud Platform resources and access Google Cloud Monitoring performance and health graphs.

Get it now from the Google Play Store.

Quickly view app status

Want to determine the status of your solution with just a quick glance? Customize the home page with your personal selection of monitoring graphs, a billing estimate or Cloud Platform service status information.

Get alerts and manage incidents

Want to be told when something goes wrong, for example when your Google Compute Engine instances rise above their expected load of 50% CPU for one hour? Cloud Console for Android integrates with Cloud Monitoring, enabling automated incident tracking when system metrics deviate. You can configure alerts to display directly in the Android notification drawer, and you can comment so that your team knows you’re working on the issue.

View App Engine and Compute Engine properties and make quick changes

When investigating an issue, you often need to check the health and properties of your resources, such as running state, zone or IP. The app supports viewing details and monitoring graphs for App Engine and Compute Engine instances. You can also invoke a number of core operations, such as changing the App Engine version or starting/stopping a Compute Engine instance.

Get started now

The app is available in the Google Play Store, just search for Cloud Console. For tips on how to configure the app, take a look at this quick guide.

Feedback wanted

Cloud Console for Android is currently in beta, and an iOS app is expected to launch later this year. Over the coming months, we’ll continue to add new features and resolve issues. To influence our future work, please send your feedback, ideas and suggestions to android-cloud-console@google.com!

­- Posted by Michael Thomsen, Product Manager

Posted:
Google App Engine is a great place to run your applications, but for some workloads you may want more fine-grained control of the environment your app runs in. You may need to fine-tune how or when scaling occurs, customize the load balancer or code in a language that App Engine doesn’t support.

Today we’re excited to introduce a solution paper and tutorial for Scalable and Resilient Web Applications to help you – you guessed it – build a scalable and resilient web application on Google Cloud Platform. The solution includes a technical paper that discusses the application architecture and key design decisions as well as a functional, open source application and tutorial hosted on GitHub that you can deploy or even use as a starting point for your own applications.

You may have read our previous post about Google Compute Engine Load Balancer easily handling 1,000,000 requests per second or watched the live demo where Compute Engine Autoscaler added enough instances to handle over 1,500,000 requests per second and wondered, how exactly did they do that?

The sample implementation uses Cloud Deployment Manager to provision a load balancer as well as multi-zone auto-scaled Compute Engine instances to serve the Redmine project management web app. The architecture uses Google Cloud SQL and Google Cloud Storage to reliably and scalably store the app’s data. Here’s an overview of the complete architecture:


You’ll also learn how to use Chef and Compute Engine startup-scripts to configure and install software on instances at boot time. There’s a lot of technical content we think you’ll find useful – check out the article, then head over to the GitHub project page (where you’ll also find the tutorial and can ask questions or make suggestions in the issues section) and start building more scalable and resilient apps.

-Posted by Evan Brown, Solutions Architect

Posted:
One of the most compelling benefits of building and deploying solutions on public cloud platforms is the speed at which you can move from idea to running applications. We offer you a continuum of compute options – from high performance VMs and container-based services to managed PaaS – so you can choose the most suitable option.

For those of you who need a VM-based solution, deploying an application requires that all underlying runtime components and packages be in place and configured correctly. This often becomes a labor-intensive, time-consuming task. Developers should spend most of their time on design and writing code. Time spent finding and deploying libraries, fixing dependencies, resolving versioning issues and configuring tooling is time away from that work.

Today, we're introducing Google Cloud Launcher, where you can launch more than 120 popular open source packages that have been configured by Bitnami or Google Click to Deploy. Deployment is incredibly straightforward: users simply select a package from the library, specify a few parameters and the package is up and running in a few clicks. Cloud Launcher is designed to make developers more efficient, removing operational deployment and configuration tasks so developers can focus on what matters – their application and their users.

Cloud Launcher includes developer tools and stacks such as Apache Solr, Django, Gitlab, Jenkins, LAMP, Node.js, Ruby on Rails, and Tomcat. It also includes popular databases like MongoDB, MySQL, PostgreSQL and popular applications like Wordpress, Drupal, JasperReports, Joomla and SugarCRM. Many of these packages have been specifically built and performance-tuned for Google Cloud Platform, and we’re actively working to ensure these packages are well integrated with Google Cloud Monitoring so you can review health and performance metrics, create custom dashboards and set alerts for your cloud infrastructure and software packages in one place. This will roll out to all supported packages on Cloud Launcher this spring.

When you visit Cloud Launcher, you can search for your desired package, or filter and browse categories such as Database, CRM or CMS.


“We are excited to partner with Google to simplify the deployment and configuration of servers and applications and look forward to continue to expand our integration with Google Compute Engine. Delivering an exceptional user experience is important to us, and Compute Engine gives Bitnami users another great way to deploy their favorite app in just few clicks,” said Erica Brescia, COO at Bitnami Inc.

You can get started with cloud launcher today to launch your favorite software package on Google Cloud Platform in a matter of minutes. And do remember to give us feedback via the links in Cloud Launcher or join our mailing list for updates and discussions. Enjoy building!

-Posted by Varun Talwar, Product Manager, Google Cloud Platform

Posted:
Currently, the way that doctors and clinicians approach medical treatment is to look at a patient’s symptoms, determine a prognosis, and assign the appropriate treatment. While sensible, this reactive approach leaves a lot open for interpretation and may not hone in on critical clues such as predisposition to genetic mutation or length of time an illness lingered before symptoms appeared. With added insights about genetic makeup, environment, socioeconomic factors and family medical history, doctors and clinicians gain the ability to better tailor and individualize medical treatment.

Doctors need new technologies in order to provide this individualized care. Researchers devoted to personalized medicine can now use big data tools to analyze clinical records, genomic sequences, and laboratory data. All of this valuable data may reveal how differences in an individual’s genetics, lifestyle, and environment influence reactions to disease. And ultimately, it may show us that customized treatments can improve outcomes. To get there, we first need to overcome the challenge of data inundation. Vast health datasets create significant impediments to storage, computation, analysis, and data visualization. The raw information for a single human genome is over 100 GB spanning over 20,000 genes, and the doctors’ handwritten notes are hard for computers (and people) to make sense of. There just aren’t enough tools and data scientists available to leverage large scale health data.

At Northrop Grumman, we’ve prototyped a personalized health analytics platform, using Google Cloud Platform and Google Genomics, to improve knowledge extraction from health data and facilitate personalized medicine research. With our personalized health analytics platform, a genomics researcher would be able to evaluate diseases across a set of patients with genomic and health information. In the past, a simple question about what genetics are linked to a medical condition might take hours, or even days, to execute. By leveraging Google Cloud Platform, in combination with our own algorithms, the analysis of 1,000 patients’ genomic data, across 218 diseases, generates near real-time results.

Northrop Grumman’s analytics platform would provide multiple benefits to researchers. With Google Genomics and Google BigQuery, terabytes of genomics information can be analyzed in only a few seconds, so researchers would see faster research results. This increase in the speed of discovery deepens our understanding of how genetic variations contribute to health and disease. In addition, the scalable storage and analysis tools provided by Google Cloud Platform and Google Genomics reduce costs and increase security when compared against in-house IT systems. And lastly, our platform aims to improve patient health by expanding the knowledge base for personalized medicine with discovery of complex hidden patterns across long time periods and among large study populations.

The Architecture

To make personalized medicine research easier, we architected our health analytics platform in layers. Here they are starting from the base layer, progressing upward:

  1. Massive Data Storage: A storage layer leverages Google Genomics to efficiently store and access genomic data on the petabyte scale and Northrop Grumman knowledge engines and framework to efficiently process and store electronic health records (EHR) data.
  2. Annotation Layer: The annotation layer provides tools to extract clinical knowledge from structured and unstructured EHR data sources. It also includes a database containing aggregated phenotypic and disease associations from public sources. These enable improved functional annotation of the genomic data.
  3. Analytics Layer: The analytics layer is built on top of Google BigQuery and Google Compute Engine to provide high-performance modeling and analytics tools. With these, we can demonstrate genomic risk modeling with analysis time scales of only several seconds.
  4. Visualization & Collaboration Layer: The visualization and collaboration layer provides a framework for high-level analytics, visualization, and collaboration tools.
The system architecture for Northrop Grumman’s personalized health analytics platform. A layered approach is designed to provide an integrated research environment with greater access to storage infrastructure, improved information extraction and annotation tools, more powerful computational platforms and improved collaboration and visualization tools. 

New Breakthroughs in Personalized Medicine

Today our personalized health analytics platform is a prototype, but the results are promising. Our health analytics platform may improve a researchers’ speed of discovery, lower the costs of storing massive amounts of health data, offer better security than in-house IT systems and ultimately lead to breakthroughs in personalized medicine and treatment. If you're interested in learning more, please email Northrop Grumman at PHC@ngc.com.

- Posted by Leon Li, Future Technical Leader and Systems Engineer at Northrop Grumman Corporation

Posted:
More and more organizations have learned, through experimentation, how much latent value exists in large scale data and how it can be unearthed via parallelized data processing. Bringing these practices into production requires faster, easier and more reliable data processing pipelines.

Google Cloud Dataflow is designed to meet these requirements. It’s a fully managed, highly scalable, strongly consistent processing service for both batch and stream processing. It merges batch and stream into a unified programming model which offers programming simplicity, powerful semantics and operational robustness. The first two of these benefits are properties of the Dataflow programming model itself, which Google released in open source via a SDK, and is not tied to running on Google Cloud Platform.

Today, we’re announcing another deployment option for your Dataflow processing pipelines. The team behind the fast-growing Apache Flink project has released a Cloud Dataflow runner for Flink, allowing any Dataflow program to execute on a Flink cluster. Apache Flink is a new Apache Top-Level project that offers APIs and a distributed processing engine for batch and stream data processing.

By running on Flink, Dataflow pipelines benefit not only from the power of the Dataflow programming model, but also from the portability, performance and flexibility of the Flink runtime. It provides a robust execution engine with custom memory management and a cost-based optimizer. And best of all, you have the assurance that your Dataflow pipelines are portable beyond Google Cloud Dataflow: via the Flink runner, your pipelines can execute both on-premise (virtualized or bare-metal) or in the cloud (on VMs).

This brings the number of production-ready deployment runtimes for your Dataflow pipelines to three and gives you the flexibility to choose the right platform and the right runtime for your jobs, and keep your options open as the big data landscape continues to evolve. Available Dataflow runners include:




For more information, see the blog post by data Artisans, who created the Google Cloud Dataflow runner for Flink.
We’re thrilled by the growth of deployment options for the portable Dataflow programming model. No matter where you deploy your Dataflow jobs, join us using the “google-cloud-dataflow” tag on StackOverflow and let us know if you have any questions.

-Posted by William Vambenepe, Product Manager