Quora’s Technology Examined

2024-03-17 10:32
文章标签 technology quora examined

本文主要是介绍Quora’s Technology Examined,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!

转载自:http://www.philwhln.com/quoras-technology-examined/comment-page-1#comment-1691

 

Quora has taken the tech and entrepreneurial world by storm, providing a system that works so fluidly that it is sometimes hard to see what the big fuss is all about. This slick tool is powered, not only by an intelligent crowd of askers and answerers, but by a well-crafted backend created by co-founders who honed their skills at Facebook.

It is not surprising that, with all the smart people using this smart tool, there are many pondering on how it works so well. The NoSQL boffins scratch their heads and ponder such questions as, “Why does Quora use MySQL as the data store rather than NoSQLs such as Cassandra, MongoDB, CouchDB, etc? “.

In this blog post I will delve into the snippets of information available on Quora and look at Quora from a technical perspective. What technical decisions have they made? What does their architecture look like? What languages and frameworks do they use? How do they make that search bar respond so quickly?

  • Components Of Quora
  • What’s Cooking Under That Hood?
    • The Search-Box
    • Webnode2 And LiveNode
    • Amazon Web Services
      • Ubuntu Linux
      • Static Content
    • HAProxy Load-Balancing
    • Nginx
    • Pylons And Paste
    • Python
    • Thrift
    • Tornado
    • Long Polling (Comet)
    • MySQL
    • Memcached
    • Git
  • Charlie Cheever Follows “14 Rules for Faster-Loading Web Sites”
  • Conclusion
  • Recommended Reading
  • Resources

Components Of Quora

The general components that make up Quora are…

  • You can ask questions
  • You can answer questions (anonymously if you desire)
  • You can comment on answered questions
  • You can vote -up or vote-down answers to questions
  • Questions can be assigned to topics
  • You can write a post (a informative statement, rather like a orphaned answer or blog post)
  • You can follow questions, topics or other users
  • A super-fast auto-complete search -box at the top, which doubles as the method for entering new questions

The last point, the super-fast auto-complete search-box, is one of the defining features of Quora. You can see immediately, as you begin to enter a question, whether somebody else has already asked that question or if there is a topic or post on the subject. Let’s start there…

What’s Cooking Under That Hood?

Only the questions, topic labels, user names or post titles are indexed and served up to the search-box. There is no full-text search, so searching the content of questions and answers will not work. The text that is indexed is tokenized so that words in a different order will still be matched. Prefix matching enables best matches to be shown before the entire word is entered. For instance, typing “mi” might immediately show “Microsoft” in the results.

There is some simple stemming of words, since “nears” matches “near”, but “pony” does not match “ponies”. “Topic-aliases” allow for similar matches on topic names, such as “startup” and “start-up”. These topic-aliases have been manually entered by users. Otherwise these would not match.

If a duplicate question is redirected to another question (a feature of Quora), then that original question will still appear in the search results, since it increases the chances of a match. There is no n-gram indexing, so slight mis-spellings will not match. For instance, “gooogle” (with an extra “o”) finds nothing.

Previously, they did use an open source search server, called Sphinx . It supports the features they are using above, but they have since moved from this due to real-time constraints . Their new solution is built in-house and allows them better prefix indexing and control over the matching algorithms. They built this in Python.

What libraries does Quora use for search?
Adam D’Angelo, Quora Founder (Nov 13, 2010)
Our search is custom-written. It doesn’t use any libraries aside from Thrift, and Python’s unicode library, which we use for unicode normalization.

Speedy Queries

Did I mention that the search-box is fast? I did some tests and found the responses to be impressive. Queries are sent over AJAX as a GET request. Responses come back as JSON with the rendered HTML embedded inside the JSON. Rendering of the results on the server-side, as opposed to rendering them in JavaScript, seems to be due to the need to highlight matching words in the text. This is sometimes too complex for JavaScript. For instance, typing “categories” might highlight the world “category” in the result text.

I was seeing responses of roughly 50 milliseconds per query from my Linode machine. Quora does not short-change you when sending requests. From within the browser, I found typing “Microsoft” (9 characters) would result in nine requests to the Quora search server, no matter how fast you type. As you will see later , the server is in control, so if it did become over-loaded, then it could update the results less frequently without changing the JavaScript.

Quora uses persistent connections. A HTTP connection is established with the server when you start typing the search query. This connection is kept open and further requests are made on this same open connection. The connection will terminate (times-out) if not used for 60 seconds. If a connection times-out then a new connection is established when typing begins.

To simulate the typing of a word into the search-box, I sent the following requests, character-by-character, across a persistent connection. For instance “butler” is six requests (“b”, “bu”, “but” … “butler”).

"butler" (6 chars) duration: 0.393 secs 0.065 secs per query
"butler monkeys" (14 chars) duration: 0.672 secs 0.048 secs per query
"fasdisajfosdffsa" (16 chars) duration: 0.746 secs 0.046 secs per query

That last query was used to test if there was a slow-down for a word that would obviously not be in a caching layer. I saw no slow-down. This means that they are not caching, caching is only used to take the load off the backend search engine or they are doing something smarter (e.g. if there is no match for “fasd” then there will be no match for “fasdi”, so abort).

Is Quora going to implement full-text search?
Adam D’Angelo, I made a lot of the early Quora … (Sep 1, 2010)
Yes, eventually. We haven’t implemented this yet because we’ve prioritized other things, but we will definitely do it in the future.

Webnode2 And LiveNode

Webnode2 and LiveNode are some of Quora’s internal systems, which were built for managing the content. Webnode2 generates HTML, CSS and JavaScript and is tightly coupled with LiveNode, which is responsible for managing the display of the content on the webpage. Charlie Cheever says that he were to start a similar project without LiveNode, then the first thing he would do is rebuild it .

They seem very pleased with the technology they have built and struggled to find its weaknesses . One weakness is that it is tricky for LiveNode to keep track of what is happening within the browser as it pushes changes from the server. If users A and B are viewing the same question then ones interactions will affect the other. For instance, if user A up-votes an answer then that answer will be promoted and will visibly move up the page. This display change will be pushed over AJAX to user B’s browser. Any prior browser-side change that user B made, such as expanding a comments section, might be lost.

LiveNode is written in Python, C++, and JavaScript. jQuery and Cython is also used.

While they would like to open-source LiveNode and have tried to keep code separation, doing so right now would be too much work and would take time away from their main goal, which is making Quora better.

Charlie Cheever points out that webnode2 is unrelated to the “free and easy website builder” called Webnode at webnode.com .

Amazon Web Services

Amazon EC2 and S3 is used for their hosting. While this is not as cost-effective in the long-term as running your own servers, it is perfectly designed for fast growing companies like Quora.

Ubuntu Linux

Quora uses Ubuntu Linux as its OS of choice. No major surprises there. It is easy to deploy and manage on Amazon EC2. Adam D’Angelo points out that he used Debian Linux at high school and college and stuck with it because “it works and there hasn’t been a compelling reason to switch” .

Static Content

You only need to look at the source HTML of any Quora webpage to see that they are using Amazon’s distributed content delivery network, Cloudfront . URLs are in the form…

http://d2o7bfz2il9cb7.cloudfront
.net/main-thumb-670336-25-7kmigSSkkdusoE6gHRkdQsXfjuTCaxQs.jpeg

CloudFront is used for all static images, CSS and JavaScript (except for Google’s Analytics JavaScript, which is hosted by Google). Images are uploaded to the EC2 machine, then resized and uploaded to S3 . This is managed using the Python S3 API .

HAProxy Load-Balancing

Quora uses HAProxy at the front-line, which load-balances onto the distributed Nginx servers behind them.

Nginx

Behind the load-balancer, Nginx is used as a reverse-proxy server onto the web-servers.

To understand more about this setup I recommend reading “Using Nginx As Reverse-Proxy Server On High-Loaded Sites “.

Pylons And Paste

Pylons , a lightweight web framework, is used as their main web-server behind Nginx. They use the default Pylon + Paste stack .

Pylons was selected much like you would select a pumpkin for Haloween. They scooped out the insides, such as templates and the ORM, and replaced it with their own technology, written in Python. This is where LiveNode and webnode2 reside .

MochiMedia was also one of the inspirations for using Pylons, since they were using it themselves.

Python

Coming from Facebook, it was a good bet that Charlie and Adam would choose PHP for their development language. As Adam points out, “Facebook is stuck on that for legacy reasons, not because it is the best choice right now “. From this experience they knew that choosing technologies, especially programming languages, for the long-run was very important. They also looked at C#, Java, and Scala. Discounting Mono , C# would be a choice of more than just the language. It would require them to build on-top of a Microsoft stack. Python won over Java because it is more expressive and quicker to write code than Java. Scala was too new. Adam mentions speed and the lack of type-checking as drawbacks with Python, but they both already knew the language reasonably well. Where Python lacks speed for performance critical backend components, they opt to write them in C++. They saw Ruby as a close match to Python, but their experience with Python and lack of experience in Ruby, made Python the winner. Python 2.6 , to be precise.

Additional benefits for using Python are the fact that data-structures that map well to JSON, code readability, there is a large collection of libraries and the availability of good debuggers and reloaders. Browser-server communication using JSON is major component of what Quora does, so this was an important factor.

No IDEs are used for development as most use the Emacs text editor . Obviously this is personal choice, and would change as the team grows.

PyPy , a project that aims to produce a flexible and fast Python implementation, was also mentioned as something that might give them a speed-boost.

Thrift

Thrift is used for communications between backend systems. The Thrift service is written in C++.

Why would you write a Thrift service in C++?
Adam D’Angelo, I’ve written a lot of Python, in… (Sep 4, 2010)
Mainly if you want to keep data in memory between requests, and want to keep your Python code stateless. Writing a Python wrapper around a C library involves some memory management with reference counting that requires some understanding of the Python internals, but writing a thrift interface is simple. You also isolate failures this way – if the service goes down it won’t take the Python code down with it.

Tornado

The Tornado web framework is used for live updating. This is their Comet server, which handles the large volumes of open connections used for long-polling and pushes updates to the browsers.

Long Polling (Comet)

Quora does not display just static web pages. Each page will update will new content as questions, answers and comments are submitted by other others. As Adam D’Angelo points out, one of the best ways to do this currently is with “long polling”. This is different to “polling”. With polling the browser will repeatedly send requests to the server saying “Any updates?” and the server will respond with “No”. A few seconds later it will ask again, “How about now?”. “No”. “How about now?”, “No, already!”. This puts the client (web-browser) in the driver’s seat. This is backwards because the client does not how long to wait before asking again. If the client asks the server too frequently then it will unduly overload the server. If it pings the server too infrequently, then server will be sitting on updates while it waits for the client to request them and the end-user will not see updates immediately.

Long polling, also known as Comet , puts the server in control, by making the client wait for responses. The conversation between the client and server is the same, but instead of the client waiting before making another request, the server waits before it makes the response. The server can keep the connection open for a long period of time (e.g. 60 seconds) while it waits to see if any updates come in. When updates do come in it can respond immediately to the client. On receiving the update, the client then immediately sends a new request for more updates. The server, once again, delays responding until it knows something worth telling the client or enough time has past that it would be rude not to respond.

The benefit to long-polling is that there is less back-and-forth between the client and server. The server is in control of the timing, so updates to the browser can be made within milliseconds. This makes it ideal for chat applications or applications that want really snappy updates for their users.

The down-side is that you are going have lots of open connections between the clients and your servers. If you have a million users (Quora will soon) and, if only 10% of them are online on your site, you will need an architecture that can hold open at least 100,000 concurrent connections. This assumes they only have one tab open to your site. Right now I have 7 tabs open for quora.com in my browser. Each tab usually has multiple connections open to quora.com. In short, Quora must maintain a lot of open connections.

The good news is that there are technologies specifically designed for this. It costs very little to hold open connections in memory if you free up all the resources used for that connection. For instance, Nginx (Quora uses this for proxying requests) is a single-threaded event-based application and uses very little memory for each connection. Each Nginx process is actively dealing with only one connection at a time. This means it can scale to tens of thousands of concurrent connections.

How do you push messages back to a web-browser client through AJAX? Is there any way to do this without having the client constantly polling the server for updates?
Adam D’Angelo, Quora (Sep 29, 2010)
There is no reliable way to do this without having the client polling the server. However, you can make the server stall its responses (50 seconds is a safe bet) and then complete them when a message is ready for the client. This is called “long polling” and it’s how Quora, Gmail, Meebo, etc all handle the problem.

If you have a specialized server that uses epoll or kqueue, you should be able to hold on the order of 100k users per server (depending on how many messages are going). This is called the “c10k” problem. http://www.kegel.com/c10k.html

MySQL

Just like Facebook, where co-founder Adam D’Angelo previously worked, Quora heavily uses MySQL. In answer to the Quora question “When Adam D’Angelo says “partition your data at the application level”, what exactly does he mean? “, D’Angelo goes into the details of how to use MySQL (or relational-databases generally) as a distributed data-store.

The basic advice is to only partition data if necessary, keep data on one machine if possible and use a hash of the primary key to partition larger datasets across multiple databases. Joins must be avoided. He sites FriendFeed’s architecture as a good example of this. FriendFeed’s architecture is described by Bret Taylor in his post “How FriendFeed uses MySQL to store schema-less data “. D’Angelo also states that you should not use a NoSQL database for a social site until you have millions of users.

It is not only Quora and FriendFeed who are heavily using MySQL. Ever heard of “Google”? It is hard to imagine, since everything Google does has to scale so well, but in the words of Google , “Google uses MySQL [...] in some of the applications that we build that are not search related”. Google has released patches for MySQL related to replication, syncing, monitoring and faster master promotion.

How does one evaluate if a database is efficient enough to not crash as it’s put under increasing load?
Adam D’Angelo, Quora (Oct 10, 2010)
One option is to simulate some load. Write a script that mimics the kinds of queries your application will be doing, and make sure it can handle the amount of load you want it to be ready for (especially as the size of the dataset changes).

Memcached

Memcached is used as a caching layer in front of MySQL.

Git

Git is used for version control .

JavaScript Placement

If you look at Quora’s source you will see that the JavaScript comes at the end of the page. Charlie Cheever suggests that this give the feeling of a quicker loading page, since the browser has content to display before the JavaScript has be seen.

Charlie Cheever Follows “14 Rules for Faster-Loading Web Sites”

Steve Souders, author of High Performance Web Sites and Even Faster Web Sites, lists the following rules for making websites faster . This list is mentioned by Charlie Cheever, the co-founder of Quora, as one of the reasons for Quora’s speed.

“One resource we used as a guide is Steve Souders’ list of rules for high performance websites: http://stevesouders.com/hpws/rules.php ”
– Charlie Cheever, Quora

Steve Souders’ 14 rules are…

 

  • Make Fewer HTTP Requests
  • Use a Content Delivery Network
  • Add an Expires Header
  • Gzip Components
  • Put Stylesheets at the Top
  • Put Scripts at the Bottom
  • Avoid CSS Expressions
  • Make JavaScript and CSS External
  • Reduce DNS Lookups
  • Minify JavaScript
  • Avoid Redirects
  • Remove Duplicate Scripts
  • Configure ETags
  • Make AJAX Cacheable

 

 







===========================================

Conclusion

Quora is a great example of a modern tech start-up. They are very small team who understand the technologies they are using very well. They have made considered choices in the technology they have selected and have a good vision of which components would be better written from scratch. They seem keen to share these in-house technologies with the open-source community and I look forward to when they have the time to make this a reality.

I intend to keep following Quora and writing about them more in future blog posts .

If you found this post useful then please leave a comment or follow this blog .

Resources

  • Quora.com
  • How FriendFeed uses MySQL to store schema-less data
  • Steve Souders’ 14 Rules for Faster-Loading Web Sites

这篇关于Quora’s Technology Examined的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!



http://www.chinasem.cn/article/818691

相关文章

【ZOJ】2071 Technology Trader 最大权闭合子图

传送门:【ZOJ】2071 Technology Trader 题目分析:最大权闭合子图问题。源点向订单建边,容量为利益,汇点向组件建边,容量为成本,原图所有边变成容量无穷大的边,最后跑一遍最小割,订单利益和减去最小割容量就是最大净利润。 输出方案就从源点跑一遍dfs,能从源点到达的所有点都标记上。然后看从源点出发的边的弧尾是否被标记,被标记表示被使用,然后再看从汇点出发的点,如果被标记

IPKISS 服务器部署环境如何找到 get_technology() 目录

IPKISS 服务器部署环境如何找到 get_technology 目录 引言正文 引言 随着对 IPKISS 了解的加深,作者本人逐渐开始探索 IPKISS 的 TECH 目录,IPKISS 版图绘制中所有重要的参数几乎都被收集在了这个目录下。本文,我们将介绍如何进入到这个目录中查看对应的参数。 正文 首先我们需要明确,get_technology() 方法实际上进入的是

The International Journal of Advanced Manufacturing Technology投稿相关记录

特刊链接 https://link.springer.com/collections/ffdjajgahc 下载Latex模板 https://www.iotword.com/17785.html 改为双栏 分区 图片转换为eps格式 https://convertio.co/zh/pdf-eps/ 编译超时 注释75,82,86可以了 编译再次超时 改用TeXworks

VMware错误:CPU does not support long mode Intel Virtualization Technology

win7 系统vmware平台下安装64位centos提示如下错误 解决办法: 1.开机时按下Del键进入Bios 2.进入Security选项,按Tab键切换到Virtualization Technology,回车 3.将2个选项默认为Disabled 切换成Enabled 4.按F10 保存重启

数字取证技术(Digital Forensics Technology)实验课II

@数字取证技术(Digital Forensics Technology)实验课II 本文是我本学期的教学课题目,不包含任何博客知识分享,无关的读者可忽略; 实验练习题 (♞思考):请对工作邮件进行签名;“problem3_1.txt"里存储的是由John Doe撰写的真实的邮件,而"problem3_2.txt"里存储的是其他人恶意伪造的邮件(关键时间作了篡改);内容分别如下,请

论文《Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review》详细解析

论文《Sensor and Sensor Fusion Technology in Autonomous Vehicles: A Review》详细解析 摘要 该论文对自动驾驶汽车中的传感器和传感器融合技术进行了全面回顾。它评估了各种传感器(如相机、LiDAR、雷达)的能力和技术性能,并讨论了多传感器校准的重要性和现有的开源校准包。论文还总结了主要的传感器融合方法和用于自动驾驶环境中的障碍物检

ST-SLAS Technology 实验室自动化与筛查学会技术

文章目录 一、期刊简介二、征稿信息三、期刊表现四、投稿须知五、出版支持 一、期刊简介 SLAS Technology ——SLAS技术强调促进和改进生命科学研发的科学和技术进步;药物递送;诊断;生物医学和分子成像;以及个性化和精准医疗。这包括高通量和其他实验室自动化技术;微/纳米技术;分析、分离和定量技术;合成化学和生物学;信息学(数据分析、统计学、生物、基因组和化学信息学)和

Quora 首席执行官亚当·德安杰洛 (Adam D’Angelo) 谈论了 AI、聊天机器人平台 Poe,以及 OpenAI 为什么不是竞争对手

每周跟踪AI热点新闻动向和震撼发展 想要探索生成式人工智能的前沿进展吗?订阅我们的简报,深入解析最新的技术突破、实际应用案例和未来的趋势。与全球数同行一同,从行业内部的深度分析和实用指南中受益。不要错过这个机会,成为AI领域的领跑者。点击订阅,与未来同行! 订阅:https://rengongzhineng.io/ 去年 11 月,亚当·德安杰洛 (Adam D'Angelo) 置身于科

Semtech ClearEdge™ technology的理解

EML(External Cavity Laser)外腔激光 DML(Distributed Feedback Laser)分布式反馈激光 EML激光器,即光电调制激光器,其工作原理基于光电效应。通过在半导体材料上施加电压来调制激光的振幅和相位,从而实现高速光调制。EML激光器具有高速、高效率、低噪声等优点,在光纤通信、光学成像、光学传感等领域有着广泛的应用。 **DML激光器,即直接调制激光器,