In second day of the QCON, we have following tracks
Software Architecture Improvements
Applied Data Science and Machine Learning
Java Innovations - The Latest Trends in Java Technology
The Evolving Cloud
You can view the details of the tracks download slides at https://qconnewyork.com/schedule-2014. I was mainly focus on Machine Learning and The evolving Cloud sections.
Engineering Velocity: Shifting the Curve at Netflix by Dianne Marsh , Netflix
try http://www.infoq.com/presentations/netflix-continuous-delivery for last year's presentation.
Build a Blameless culture
Freedom & Responsibility Culture
A story of brining python into Netflix. Employee are free to bring new techniques and new tools into the company, To be good Netflix, when a engineer introduce Python to the company, he needed to build the same support of building support for the tools as well.
Manager's Role in Netflix:
Developers deploy their own code, Micro-service
The developer who make the change is the right person who make call to when to deploy and how to fix the issue
Shifting the curve with culture at Netflix:
Switch the culture of Netflix
Technique used in Netflix
Netflix build language
Jenkins Job DSL:
Create AMI from base AMI
Image contains server and everything needed to run it
Builds unit of deployment for test and prod
Abstracts Cloud Details
Deploys Netflix to the Cloud
Developed to address deploys in rollback
Hot topic this afternoon
Continuous Delivery Workload
Balance between Regional Isolation and Multi-Region Consistency
Limit Impact of Human Error
Netflix Chaos Monkey
Kills Running instances
Structure, Personalization and Scales: A deep dive into Linked Search
Daniel and Asif from Linked-in talked about linked-in 's new search platform. I think it make sense for us as well since we also need a search platform for our apps, documents, messages.
The overall of linked search
What is unique in Linked-In Search
What are our system challenges?
Search quality pipeline
Spelling check => query tagging => vertical intent => query expansion
Regular full index builds using Hadoop
Partial live updates supports
Indexing and searching cross graph entities/attribute
Single engine, single stack
Search as service
Improved relevance support
Ranking is highly personalized
Exploring the economic graph
Machine Learning at Netflix Scale
try last year's talk at http://www.infoq.com/presentations/machine-learning-netflix
Aish Fenton from Research Engineering talk about machine learning at Netflix. He explained Netflix's data mining pipeline with three time requirement, he give an simple tutorial on SVC algorithm and describe many good areas in machine learning. I suggest you watch it online
Everything is a recommendation in Netflix
Personalize homepage for you,
75% of plays come from homepage
Focus on rates more than recommendation (people only click the top rated items)
Netflix Prize: $1 Million
The people who build High quality recommendations which can beat running system with 10%
Accuracy in predicted rating
Similarity based recommendation
Based on what you learned
SVC and DBM are commonly used algorithm
Netflix has a data mining framework mixing with different data pipelines. They have three pipelines/machine learning phases : real-time, near real-time and offline
Make data flow using different tools, such as storm, kafka, S3, Hadoop, etc.
Nvida card make computation easier. Single machine is doing SVC computation
Computational Patterns of the Cloud by Ines Sombra
Lead Data Engineer at Engine Yard
This talk is very interesting, Ines talked a lot of lesson learned from Engine Yard, a very popular cloud provider. I suggest you can watch in details if you are in service areas. The slides can be found at https://speakerdeck.com/randommood/computational-patterns-of-the-cloud-qcon-nyc-2014
Your apps as collection of services, connected via apis
Pets v.s Cattle is an very interesting blog
Cloud resource are reused
Provision=> consume -> maintain => release
Our experiences + a few stories
Excel at Process:
Anticipate failure and plan for it
Everything is a recipe
Infrastructure is maintained as code
Resource are used to increase the av and red of applications
Importance of monitoring &benchmarking
What does healthy means?
Resource familiar match user cases
Think about resources in fluid terms: compute & release. Harvesting services
App design for the cloud
Operational experiences can become siloed
The team who build billing only know how money flow into the company
Service dependency and failure planning
Importance of API design, maintaince
Distributed Systems as a new norm. Distributed system and the end of the API
Availability & coordination
What does it means to be "up"
Are they fast? Can we trust them?
What type of tests?
What coverage do we have?
Are databases …
Setting up replication?
Testing is critical and frameworks should help by streaming chose
Automation helps with the process &certification
Stop thinking a tool will fix your agile problem
Canary Analyze All The Things: How we learned to Keep Calm and Release Often
This is related to our testing in production principle. I suggest you watch it online to get more details
You need better Testing?
"I am going to push to production, though I'd pretty sure it is going to kill the system"
Trade-off of Rate of changes vs. availability
Change to you need better deployment !
Canary analysis is a deployment process, but Is not
replacement for any sort of software testing
Are we there are yet?
How you start:
Focus on the goal:
The road to get there
Automated canary analysis
700 canary runs per day.
Avoid deploy during weekend.
Customer heavy usage during weekend
Developer don't work during weekend
Our next trick for configuration system:
Canary analysis make your change
Most people can start doing it now.
Weathering the Data Storm
This talk is very interesting for people in machine learning and ads area. She explained in details how her team can generate ~5000 models automatically to do Ads Campaigns and her cluster algorithm improve 5 to 10 time more than random selection.